 Good afternoon. Good morning for people in the other side of the world. Good afternoon for people here in Europe or in other continents. Okay, we are ready to start our European Distance Learning Week. This is I think the fourth webinar of this week and it's a very interesting topic and a very actual topic which is artificial intelligence in higher education. Let me introduce also Diana Andone, which is Vice President of Communication of Eden. Diana and myself will be the moderators. And we have today four presenters, well-known presenters about this topic, artificial intelligence in higher education. Professor Tony Bates, Professor Olaf Savakir-Richter from University of Oldenburg, and Alexandra Kristea from Bergen University and Christo Balcobo from Oxford Internet Institute. Just a few words to start remembering that this is an event organized by Eden. I'm sure all of you know very well which is Eden, European Distance Learning Network, which aims, of course, to share knowledge and improve understanding among professionals about the topics of the network. And this network is organized with different kind of participations. We have close to 200 institutional members and also more than 200 individuals and 1,000 people in our labs, network, academics and professionals. At the end, we will share with you some future events of our network. And then you also remind that Eden is supported by the European Commission. And now I give the floor to Diana and Doné in order to introduce a little bit this webinar and also to start with our first speaker. Diana, please put your mic on. Diana, I cannot hear you. Please, Diana, your microphone. Okay, I think Diana is having some problems with her microphone. Okay, I think it's better if we can continue in order to... Let me introduce the first speaker today. I think it's well done for all of us. It's Professor Tony Bates, which is president of Tony Bates Associates. It's a private company in consulting and training. And he is also a distinguished visiting professor and well known for all of us. Tony, please, if you can put Tony's presentation here on the screen and Tony, the floor is yours. Well, good morning, everybody. Or good afternoon. Good evening, wherever you are. It's a great pleasure for me to be presenting to you on this topic. I don't have a great deal of expertise in artificial intelligence. And I actually want to discuss the issue for many educators around this issue of knowing what's going on in artificial intelligence. But I have been following developments in the literature fairly closely, both in learning analytics and in artificial intelligence. And there are a number of key issues that come out of this work that I want to raise today. First of all, I think there are some definitional problems that I just want to mention very briefly. And I want to make a distinction between three areas. One is statistical analysis. That's been around for a very long time. This would include things like multiple variant analysis, multiple regression analysis, analysis of variants. And this is commonly used in doing research in education and it's well-established. And then there's learning analytics, which is basically trawling data, trawling educational data in a search for patents. And it doesn't actually make decisions itself. It may suggest decisions. But in the end, somebody has to look at that patent and make the decisions. And it's usually educators or administrators that make those decisions. And then there's what I would call for blown artificial intelligence, which uses algorithms to identify and interpret patterns of student behavior. In other words, it actually can make decisions and does make decisions. Although again, it may be used in partnership with an educator. In other words, it may recommend a decision, but an educator may overrule that decision. But in other words, artificial intelligence goes further than the other two. And sometimes I see papers that have the title artificial intelligence such as really learning analytics or just statistical or advanced statistical analysis. And so I've drawn this little diagram to show the overlap because you can have statistical analysis within artificial intelligence and you can have learning analytics can be driven by artificial intelligence or it can stand alone. But the key thing in artificial intelligence to me seems to be the algorithms which look for patterns and may actually come up with some decisions as well. So I just wanted to put that out there so that when we hear these terms, we don't get them confused. What I actually wanted to talk about is artificial intelligence here. And there are three core features of any application of artificial intelligence. Access to massive amounts of data. The more data, the better. The more powerful the tools are, the more data there are. Powerful algorithms for helping to find, search and identify data, analyzing that data and then recognizing patterns within that data and also there may be some algorithms in there for decision making as well. And for that to work, you need very powerful computing and usually that means going into the cloud to use very powerful computing unless you've got a very big computer like Watson, for instance. So those are the core features. What are the applications of artificial intelligence in higher education? Well, there's three levels of application. One is at an institutional level. For instance, to help institutions make decisions about which students to admit, where to market students, how to go out and use data out there for marketing and making decisions about marketing and also to some extent curricular decisions. I think we'll see more and more where artificial intelligence is used to go out and draw data that's already out there like open educational resources and analyze what kind of content goes well together and how to put that content together into a curriculum. And that might well be linked, say, to what employers are demanding and the way of skills, for instance. And I'll come back to that towards the end because I think this is one critical area for artificial intelligence that has not been very much explored at the moment. The second area is student support. Intelligent tutoring, such as chatbots, these are software that runs around. Let's say that you've got a massive online course of MOOC and there's lots of comments going on. The chatbots will go through all the comments that students are making and where they see students may be misinterpreting information or needing help. The chatbot will give automated responses. And again, this may be used in conjunction with real-life people. In other words, the chatbots might identify areas where students are having difficulty and flag that for an instructor to intervene. It's also used for providing automated feedback so that students can know whether they've learned something correctly or not. And also prediction, in other words, providing students with early warnings that they're struggling or they may have to do more work and so on. But again, it's not actually teaching them. It's just providing support to their learning. And the third one is actual teaching, instructional, where such things as adaptive learning, where students are tested and if they get it wrong, they're redirected to another to do more work and assessment such as automated grading. And again, what we're seeing in the literature is there's quite a bit of literature on institutional applications and student support and less on what I would call advanced artificial intelligence in the instructional area. It's still mainly old artificial intelligence such as adaptive learning and assessment, which has been around for some time. What's been the main impact to date on higher education? Well, I have to say from my search of the literature and there's a big issue here and I'll come back to that in a minute, but I see very little major impact to date on higher education. As I said, it's mainly at the institutional level screening. It's mainly old artificial intelligence that doesn't have powerful tools such as deep learning algorithms and so on being applied. Some attempt to essay marking, which is using linguistic analysis and so on, but again, that's not proved to be very successful for higher level learnings. Some learning analytics and where there has been learning analytics is often no significant difference. In other words, the learning analytics haven't really helped to identify strategies that instructors should be following, for instance. There have been some experiments with MOOCs applying because there's massive amounts of data. Again, the results are not impressive. And most importantly, where it has been used, it's been applied to what I would call low levels of learning although they're quite important levels such as comprehension and understanding. What the National Research Council of the United States calls declarative knowledge. Name of plants, testing people on the name of plants. It does do a little bit of process. You can do process teaching, teaching processes like problem solving in mathematics, but it tends to be what Bloom would call the lower levels of learning objectives. It's not proved successful to date at tackling the more difficult things such as concept development, cognitive learning, high level problem solving and so on. So it's been disappointing to date. Why? Well, most of the papers written on artificial intelligence are written by computer scientists who tend to have a very limited understanding of learning. They have a particularly behaviourist approach to teaching and learning. Although they don't, I'm not sure they understand it's a behaviourist approach, but it's just the approach they take. You test, redirect, test again, redirect, test again and redirect and so on. And it's data driven rather than theory driven. In other words, it's hoped that the patterns will provide some kind of meaning, but often they don't. If anybody's actually done multiple regression analysis in statistics, you've got a lot of variables and you've got some weights to the different variables, but knowing how they interact, it's really hard to interpret. And probably the most worrying thing for me is often the applications of artificial intelligence are contrary to what I think are some of the core values of education, such as equitable access and the agency of the individual. I don't want to go into detail there because I'm sure others will be talking about that, but it seems to me an attempt to make life easier for administrators but not necessarily further the interests of students. And I think another reason is that in education, and I'll come back to that in a minute, the data sets are too small. If you look at the higher education system, it's very fragmented. I guess you might have a big university of 20,000 students, but what you really want to look at is the 20 million students that might be in the system, not just the 20,000 in the university to get some really good results. But my last message is don't be complacent. I think that artificial intelligence could have a major impact on higher education because once some of the more advanced levels of artificial intelligence are intelligently applied to teaching and learning. And this change is going to be driven by the global internet platforms. The higher education market globally is very, very large. There's a lot of money to be made in this area. And let's be clear what the artificial intelligence goal is here is not to improve the higher education system as it exists at the moment. It's to meet employer expectations and if possible bypass the institutions by reducing costs. And I've done a little diagram here about one potential model that might be used. Each of those arrows reflect areas where artificial intelligence is used. So for instance, give you an indication LinkedIn is now possible to go into LinkedIn and identify all the skills that employers are requiring over the next 15 to 20 years. An internet platform could go in and trawl that data and produce a skills inventory. That's number two. Commercial educational providers can go into that skills inventory and using artificial intelligence could develop automated programs that learners can then enroll in. And also employers and internet platforms could create a qualifications warranty or skills bank using for instance blockchain. So one could have a system that completely bypasses institutions and I think that's the danger. So my questions are, is artificial intelligence an opportunity or threat? Is it all smoke and mirrors? Will it disrupt the higher education system? And what are the implications for learners, teachers and institutions? Thank you. Thank you very much, Tony. I think maybe we can change the order of our presenters. Maybe because our lab is having some travels with the communication. Maybe Diana, if you want to introduce Alexandra. Diana? Diana, can you hear me? Okay, yeah. Okay. I can introduce myself. I'm sorry about that. We are losing our time in these things, but okay. Professor Alexandra Cristia, if you want to start your presentation, remember that you have 15 minutes. Professor Cristia is a member of the Dohan University. And as you can see here on this slide here, you have a special presentation. Okay, Alexander, the floor is yours. Thank you. Can you just, I said those are my slides. Yes, thank you very much. So thank you very much for inviting me to this panel and hopefully you can hear me, yes. So my name is Alexandra Cristia and I work in Durham University. And I've been working in the area of AI for quite some time now before it was so very popular. I've also been working on distance learning for quite a while and AI applied to education. And I think we're living in quite exciting times in terms of AI. So I'm less scared perhaps to some extent than Tony in this area because I think it's a great opportunity for us that AI is moving so fast. And hopefully unlike in other areas, I mean unlike in other advances of science, we can catch up in higher education and in education in general and e-learning in general with these strengths and with these developments and make good use of them. I do share his concerns in terms of actual deeper understanding and deeper concern regarding the educational process but I think there is a great opportunity as well. So yes, I'm also chairing the Innovative Computing Group so these are some slides from there and we're also hiring people in AI and bias in AI and all sorts of other wonderful areas around this. And we have 15 staff members there. I'm going to talk a little bit about learning analytics because this is something that I'm doing a lot of. And first of all, there's several definitions of learning analytics and my favorite one is one by a very good friend that unfortunately passed away. Maybe many of you know him, Eric Dival, very famous professor from Leuven. He was saying that learning analytics is about collecting traces that learners leave behind and using those traces to improve learning. So I highlighted on these slides the two bits that I think are very important. One is the collecting traces part. So it doesn't actually say specifically what to collect as long as it's something left behind by the learners. And importantly, and I think slightly conflictually to what Tony was saying to some extent, it's about improving learning. So learning analytics shouldn't be enough if it doesn't at least attempt to use this data to some extent towards improving learning. Now, having said that, it's not all... Let me just try and go to the next slide. It's not all that simple because basically there's lots of aspects to learning analytics and this is what you see here on my slide. You have descriptive analytics and that would be kind of the type of view that I think Tony was presenting on learning analytics where we just collect all the data and show it and model it perhaps to some extent, perhaps visualize it. That also is a part of it so it could help in the decision process. But you also have other things. You have diagnostics where we're trying to respond to things like why did it happen? And that's actually quite old AI questions. It's not something new. Why and how? Explaining why something happened and usually this is a hard question to answer for neural networks again traditionally because they're black boxes and we only know the inputs and the outputs. And then we have things like predictive analytics where you're trying to figure out what's happened and make some predictions and here are some papers of mine on that. And then finally, prescriptive analytics is supposed to be the sort of more interesting part where you actually start telling people what to do as a result of the analytical data. So you already have done the prediction and then as a next step you're telling them what to do. Now, funnily enough, prescriptive is actually a bit easier to some extent than diagnostic because sometimes the diagnosis is harder. And I could talk about that more but let's move on to the stakeholders of learning analytics. So I think Tony has discussed some. So basically we have the whole pyramid as you can see it here. So all the way from higher level government, then institutions, teachers, students but you have also groups, classrooms, learning groups, academic. And all of these could be actually our stakeholders in learning analytics and could be consumers of the output or producing the data so they could be both on the input and on the output side. It's not always happening like that and of course people look at different aspects of this. So having said that, you know, you have all this spectrum that doesn't mean that everybody always uses the whole spectrum either in stakeholders or in learning analytics sites but I think this is basically what it all entails altogether. And of course not to forget the researchers as well in this area which are interested to find out new ways to generate this kind of metrics and measurement, et cetera. So I call them micro, metho, and macro levels. So different methodologies again. So yes, I do agree with the fact that, you know, we have at the bottom sub our statistics but that's not all. So we have also things like data mining. We have things like machine learning. We have things like network analysis which are really interesting as well in terms of lots of people interacting with each other especially when we talk about big learning just like in big data we have possibly something like big learning, right? With lots of stakeholders interacting. And then we have also visualization which is part of the sense making from the human side and usually what learning analytics has been traditionally used for. So to give some example from my own research on visualization here are some slides from a paper that we published long time ago now, 2016, from a system called Topolor. It was an e-learning system. And what I would point out here is that it also shows you the fact that, I mean the learning activities and the fact that you have gone through some of the pages that you are supposed to read, et cetera, but also shows you interactions or social interaction with others. And it has various ways of comparing yourself with previous work in different weeks and at the same time with the average in the course and with the top 20, et cetera. Another way of looking at things is to look at frequency. So for instance, once you are able to take all those traces that I was talking about from the definition of Eric Duval of learning analytics and group them into some semantically meaningful way. In this case, we opted towards the social navigation assessment, reading and auxiliary. You can start looking at how different students, for instance, at individual level perform. So for instance, the third student in a row has a big chunk of red. That means he's doing a lot of reading whereas the first student is doing a lot of social interaction and so on. And you can also look at these activities in a temporal way so you can see that the top student, the one with the blue, has a lot of social interaction. So a lot of both at the bottom level is a social interaction in this graph. But for instance, not so much reading which would be the third one from top to bottom or the red student has a lot of reading and has also social interactions as a bit of both. So you can start looking at this. So this would be very fine grade analysis at the level of a student or you can do it at various types of patterns kind of matching where in this case we were looking at probabilities of transition between states. So where the students all start and what would be the probability for them to transition to a different state from social interaction in this case they all started in this particular system with social interaction and then they go to navigation or to auxiliary actions and then for instance from navigation they go to reading, et cetera and we could look at patterns like that. And then from a different paper and completely more recent stuff we did actually work on MOOCs as well. This was a little bit of a crazy idea of mine. MOOCs usually, so traditionally the problem with MOOCs is that they have very low level of completion and there's lots of reasons why that is and there's lots of papers about the reasons. Some of them potentially not to do at all with the system themselves simply the people are looking for specific things and once they have learned about them they move on, that's fine. But still we're looking at various predictors or potential predictors in this case we were playing with the registration data as a potential predictor to see that may actually generate some kind of information about when if a student is completing or not and we found some statistically significant correlations there. And eventually because of that we got all the way to the prescriptive analytics part we were proposing some rules in pseudo code in terms of what to do next for a student. So for instance, well the simple part would be the top one that if they have registered long time ago we just tell them, you know, why don't you register a bit later because you might actually not do this course if you register so much in advance. But slightly more personalized so this is my background is also in personalized learning so the third one I believe from the top basically it says also the last sentence there said also please consider visiting these links for additional support. So that's where you would give them some recommendation of enrolling or de-enrolling or putting something in their schedule but also providing them with additional material which would help them in this case to catch up because they have those ones are people that have registered just a little bit after the course has started. So perhaps some kind of summary of what has happened so that they can catch up quickly. So there's various interventions you can have and actually there's a lot of literature on personalization and for me this is a very interesting time in terms of this big learning because basically we all of a sudden all this experiments we were doing a very small scale in personalized learning intelligent tutoring systems adaptive educational hypermedia in all these areas all of a sudden we can do them in big scale and that's very interesting and start having going beyond statistical significance basically where the big data lies. Other things in visualization so we were playing around with 3D visualization so we're looking at if the behavior in the first week and the last week are somehow correlated in terms of so this is actually three-dimensional but in four-dimension the fourth dimension being the color and the color red means that they're non-completers and blue means that they're completers and we're trying to see if students in this case answered questions correctly we're looking at incorrect answers and so forth so more the more traditional parameters to predict student behavior. But generally speaking you build a prediction models something like that you'll be cleaning feature engineering using various algorithms and reporting them and yes this is one of my favorite little slides on machine learning and then you can do some feature selection this is another work of ours where we're looking at features that are good predictors from the different week here we were looking for instance at number of accesses and time spent per axis in other places so we look at gamification and so forth this is not a MOOC study it's a different study on gamification but also big data study and we were looking at gender analysis I do a lot of gender analysis as well for various reasons and we're looking if females and males have in this case different preferences for the type of gamification elements that they would use in general my great believers that what we have done in the past is very much what I would call top down kind of analysis so we start from our educators, psychologists and teachers and they create some kind of models and then we try to implement these models and test them and very often small scale and this is what we've done in the past nowadays we're at the bottom up kind of approach we have all the student usage data and traces of the students like I said and from there we can build the system the other way around my actual belief is that the truth is somewhere nothing between and needs to combine these things but we're not there yet this is the direction I think we're moving towards and speaking of gender I wanted to mention very quickly that we're also analyzing other things that we're doing so take up is retraining in IT for women program that we've been working at we're training 100 women in IT from different backgrounds especially under privileged and we have actually a great percentage of BAME over 50% and quite a lot of social media presence and so some of the things we do what cloud but we do other kind of analytics this is just because to show that they were very happy with us same conclusion I think only analytic is a very interesting area at the moment this data analytics and data mining field in general they're both evolving really fast the new methods are creating all the time and gives us great opportunities for higher education for all parts in the process and for all the stakeholders so thank you very much I'll end here for now thank you very much Alexandra can you hear me? yes yes I can so if you can hear me that's really amazing thank you so much Alexandra it's really lovely and I have to introduce now Olaf Zavaki Richter who is our next Olaf Zavaki Richter is a professor of educational technology at the University of Oldenburg in Germany and his major work recently was by reviewing research on artificial intelligence in higher education and this is going to be published by the International Journal of Education Technology and Higher Education Olaf the floor is yours yeah thank you very much I hope you can hear me now sometimes we are struggling with technology and yeah I had big problems setting up my computer and the speakers and the microphone and had to get another computer but now it works so now I can talk about artificial intelligence in education okay the theoretical and mathematical foundations of artificial intelligence has been developed decades ago it was John McAfee who organized the first workshop in the USA and they made the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it and this is what he termed artificial intelligence and the algorithms for AI applications are extremely data hungry and need a lot of computing power and that's why we are seeing this huge wave of interest now because of the exponential growth of computing power and the abundance of digital big data that is now available via the internet and social networks or sensors that produce massive amounts of visual data never seen before and this development will have or already has an impact of our field of education and higher education particular according to various reports that predict massive growth rates of AI in education like the horizon report or contact north who conclude that there is little doubt that the technology is inexorably linked to the future of higher education and universities are making heavy investments like the technical university in Eindhoven they announced that they will launch a new AI research institute with 15 new professorships given the interdisciplinarity of the field I think it's necessary to clarify terminology first artificial intelligence does not describe a single technology it is an umbrella term to describe a range of technologies and methods such as machine learning, natural language processing, data mining or a single algorithm and machine learning and AI are often mentioned in the same breath machine learning is a method of AI for supervised and unsupervised classification and profiling like we have just seen and deep learning in turn is a method of machine learning based on artificial neural networks and this method was used by AlphaGo to defeat the best human Go player on Earth it remains a philosophical question whether machines will be able to think this would be called strong AI and it's still sanction all AI and machine learning applications today are based on weak AI or good old-fashioned AI machines are just simulating thinking and show hopefully rational behavior so computers act as if they were intelligent they are a one-trick course they can perform clearly defined tasks much better and more efficient than a human could do so AlphaGo defeated the best human Go player on Earth but the machine does not know how to play tic-tac-toe so what is artificial intelligence in education? some Spanish colleagues wrote this technology is already being introduced in the field of higher education although many teachers are unaware of its scope and above all of what it consists of there is a recent report on AI ad by Baker and Smith and they use the definition that I find very helpful they distinguish three different perspectives to categorize AI applications in education there are learner-facing applications like an adaptive LMS or an intelligent tutoring system teacher-facing applications like assessment tools or plagiarism detection tools and system-facing AI ad tools like monitoring tools on an institutional level we use this framework as well in our systematic review on AI in higher education that we wrote for this special issue in the International Journal of Educational Technology in Higher Education this paper is available now since the end of October and I'm glad to see that it has already been downloaded over 1,700 times so I think this also shows the great interest in that topic we do not have time to talk a lot about this systematic review method which comes from medicine and the health sciences the application in educational research or social science is a bit different so those who would like to learn more about it I refer to our new book that will be published in an open access format hopefully in December about the systematic review methodology so in our article we addressed review questions in three areas the first one was about mapping the research publications where are they published, where are the authors coming from and so on then we looked at the concepts and ethics how is AI in education conceptualized and what kind of ethical implications and challenges and risks are considered in the publications and finally what is the nature and scope of the AI applications in the context of higher education it is quite interesting to note that the vast majority of AI research is done in the context of higher education institutions we found only very few studies about AI in schools in continuing education, corporate training or vocational training the field is clearly dominated by colleagues coming from computer science and STEM departments over 60% were written by computer scientists, engineers and mathematicians only 13 papers less than 10% were authored by scholars from education departments we used the concept of the student life cycle as a framework to describe the various AI based services at the institutional and administrative level as well as at the academic support level for teaching and learning and then we described four broad areas of AI applications in our synthesis, profiling and prediction, adaptive systems and personalization intelligent tutoring systems and assessment and evaluation and they contain 18 sub areas and we cannot go into much detail here you can read this in our article I just would like to highlight a few things in the next slide Shannon Doe writes that the accurate prediction of students' academic performance is of importance for making admission decisions as well as providing better educational services and some colleagues from Turkey published a study in which they predicted admission decisions at the College of Physical Education and Sports with an accuracy of about 97% using different machine learning algorithms intelligent tutoring systems they can be used to simulate one-to-one personal tutoring and based on learner models, algorithms they can make decisions about the learning path of an individual student and the content to select and provide cognitive scaffolding and help and to encourage students in dialogue this example is Duolingo they use chatbots for language learning for example automated essay scoring articles that utilised automatic grading they came from a range of disciplines and they are used in biology, medicine, business studies, English as second language but they were mostly focused on its use in undergraduate courses and that's because these systems are practical for large courses due to the need to calibrate or train the systems with pre-scored assignments that's supervised machine learning and I found one very interesting example Geerle and colleagues used AES systems for large-scale assessment in Canada there's a medical council of Canada qualifying examination Canadian students have to take this test in 2013 over 5000 students took this test and 100 raters needed 4 days or up to 3200 hours to score the items of this test and Geerle and colleagues they used a machine learning classification system and with this approach they required the system required over about 3 hours of one rater to calibrate the system and then the system needed 10 seconds to score the 2013 examination I think this is quite interesting with an agreement of 97 or 98% and then there are of course also adaptive systems we heard about that Chantry is a company that claims to be the first offering an AI-based learning management system the system is intended for schools, not so much for higher education and the learning path is based on small nuggets of knowledge and well I think this is not what we understand by complex competence development at universities another question of our systematic review was how authors consider ethical implications challenges and risks implementing AI in education and as I already said AI applications are very data-hungry and require a lot of confidential data from students and faculty so this is where issues of privacy and data protection come in or faculty members and tutors might fear that an intelligent agent or an automated essay scoring system might take their jobs so I think generally we should not strive for what is technically possible but always ask ourselves what makes pedagogical sense in China systems are already being introduced to monitor student participation and expressions via faith recognition in classrooms and display them to the teacher and to the parents on a dashboard so-called intelligent classroom behavior management systems you can buy that in China but what does this do to children who believe that the teacher can now read their thoughts I would like to finish with a quote from Russell and Norvig who remind us in their leading textbook on artificial intelligence all AI researchers should be concerned with the ethical implications of their work and the stunning result of our systematic review is that only two papers out of the 146 studies included of our studies reflected on those ethical implications and I think this is something we must too on this fact in our discussion that's why we are asking where are the educators so here we are thank you very much for your attention thank you very much Olaf thank you very much for this presentation I am now it's time for the last presentation of this webinar it's a pleasure to me to introduce Cristobal Cobo Cristobal Cobo is now a senior education specialist at the World Bank he used to work with the table foundation in Uruguay I'm also doing research and before that I'm doing that also at the Oxford Internet Institute at the University of Oxford please Cristobal go ahead the floor is yours all right, can you hear me? yes awesome, well Joseph thank you very much for the invitation for the introduction thank you to the end for organizing this activity and to my fellow panelists for extremely insightful comments so I'll try to talk briefly connecting some of the ideas mentioned by my colleagues as Tony said my expertise is not on AI I've been working for 10 years or so in human-compete interaction and I'm especially interested in tackling some of the social implications of technology so the presentation of Olaf particularly the last part I think connect very smoothly with the ideas that I would like to offer to you and I guess the title in a way suggests in which direction I would like to go if you have a look at this Google trend which is an aggregated way of monitoring the searches of people you will see that between 2013 and 2019 the search queries on artificial intelligence have almost double which is a not scientific but very rough and illustrative vision of the growing interest in this as it has been presented now, I'm not going to go into detail AI in education had some pockets of expertise in approaches like automated early creating providing tools for automatic feedback crystallizes some of the contents that can be helpful for the learner plagiarism also is another strong player as well as spiritual teaching assistance now I think we are today in a good moment for exploring how to bring in this conversation the potentials of AI education as well as the arts and I trying to stay away from the extremisms either utopia or dystopian I think we have to navigate in between but I will try to emphasize in some of the tensions that I see in this field today so my impression so far is we have an important and growing gap between the expectations the expectations what is being discussed today and the reality and that gap I think in part is because we tend to search for quick solutions to complex problems digital technologies are extremely helpful to deliver contents but the learning experience is a complex challenge and the social experience tends to be highly embedded into the learning and probably the message about that was mentioned by Alexandra can be connected with that and I think there is a lack of critical analysis of that that's why I really value the ending of that presentation but as well as his paper where he highlighted over 2,600 papers the lack of critical analysis of some of the unintended consequences of the negative effects of digital technology is a matter of concern my take is today is the end of the digital honeymoon and it's important also to address the other side of the coin in digital education we know very well all of us who have been working in this field for years now that tend to be driven by solutionism using moral circumstances which is every social or educational problem has a quick fix through technology and when we have seen that option of technology into the learning experience we have to understand that the process is not growing exponentially as a law as the Moore's law it requires to address the system of change change of behaviour and overall adapting the culture of our organisations and you know better than me that higher ed tends to be an organisation that takes a lot of time to change its practices now I'm not going to deny some of the incredible things that we see today in the world of AI I mean the capacity of predict to adapt to search learn to identify patterns to reason in some ways to solve some problems even to recognise voices, images and languages is certainly incredible but the truth is when we think seriously in the concept of artificial intelligence there's a very very great oxymoron because intelligence is much more than the effective use of information we as humans when we learn or we build a network when we recognise these things when we adapt to uncertain environments we need to apply all these other capacities and intelligence which are critical social intelligence, emotional intelligence, self-efficacy, adaptability and I think artificial intelligence as has been described is really far from that so I don't think it's a concept that is particularly helpful that's why I appreciate it as being a scaffold in other much more specific concepts and the other element that tend to be ignored in many of the approaches with technology in general is there's a massive difference between knowing and understanding and many of these technologies are very effective for accumulating knowledge but understanding means a much more complex way of dealing with information because you have to understand which context is developed what implications may have, how it evolves, how it's influenced and this is because it's critical to understand the knowledge in the context it's not that easy to explore from one context into another that idea has to do with what I would call an asymmetry of information or sort of data field and I apologize for that after that but I think today higher education institutions are in a kind of very vulnerable position waiting for these adoptions and technology developments to be deployed and implemented so overall beyond AI the way that technology is being deployed and implemented in today's world is generated by a small number of developers, geeks, coders, whatever you want to call them the interfaces are implemented and designed and adapted by a small group of people and regulations are done by a small group of people using the digital field that is in Metaphora we could say are the scribes and we have a large, large sector of the society which are passive use of technology developed by somebody else and this we generate not only asymmetries but also generate a number of tensions that I think is fair to address them in today's world in which we have seen that many of the technological deployments have generated unintended consequences through this so-called extractive economy which organizations say that they would take very seriously the information of the user but they end up being in the third party hands without any clear notification or without all, with all the notifications but in a complete way that human beings don't understand so this data surplus that can generate improvements in learning or much more scalable technical solutions I think need to address a way of having the users in the conversation and not just as a passive one. Now my fourth element has to do with the idea of the bias. If you have a look at this search query that I did for CEO you would see that most of the ones who show up in the first result are male Western, many of them middle-aged and white and you may wonder why there's no women, why there's no young people why don't we have people from different backgrounds and I think that in a way is illustrative that the algorithms are not biased by itself but the way that they are fit with information make that there's a very clear imbalance and one of the concerns I have is there's a large number of the society that think that algorithms are neutral so there's an important challenge here to address how to develop higher levels of awareness that are not going to be solved only publishing AI ethic guidelines which is super trendy today as you may say, as you may see a large number of companies which shout very loudly that they are committed with transparency but we have heard this story before, at Google in 2018-1998 published this Don't Be Ideal and today we see that many of this data has been used in ways that were not clear for us and the context bias means that we don't see things as we are we see things as we are and that tends to affect the way that we interact with technology because that is replicating on the technology so we have a lot of work to do not only to improve the algorithms but also a lot of work to do to increase the level of fairness, accountability, transparency and explainability and as I said it's not enough to publish them in guidelines on ethical principles because this is directly connected with dealing with abilities, contradictions and complex environments when a larger sector of the society is adopting this and my last slide has to do with what I would call choosing not to choose so many of us use a large number of AI systems today not in the future with a number of apps and social networks that help us to think on our behalf so part of our cognitive capacity we download this capacity in the systems that help us to think on our behalf but there are some challenges behind that because we don't always understand how the systems work the black books are evident and as I said in this kind of passive position they don't understand what actions they need to take how to clarify that very easily when we design the systems I think it's fair enough to question what could be some of the ethical implications of that how the algorithm was built how representative is the data from minorities or communities that might not be necessarily represented as I showed you in the CEO search and this idea of the edit by design cannot be a feature at the end of the academic papers if you search on AI academic applications you will find that AI analytics is always in the last paragraph so I think we have to have an important task here in order to ensure mechanisms to open the black box to diversify the data that we use to train the algorithms to understand that these are leading us to new problems and complex problems will require complex solutions we don't solve these problems only showing a big banner with an announcement that your data is being tracked and I would say there are a few recommendations that I would like to conclude with the first one is we in the higher education but in education in general and probably in a larger sector as well we will have to update the society with this idea of AI literacy people need to understand at least the basics and need to understand how to react when they receive an orientation just given but not how to make a decision the second one has to do with we have a lot of work to do in terms of improving the quality of the data infrastructure we have today otherwise we will tend to replicate the problems of the past in the future decisions and that will be connected certainly with my self-recommendation which is to develop AI for education systems which are representative of a multidisciplinary perspective with different visions with people from computer science certainly from psychology, from the legal background from sociology and other disciplines because we need to bring all these voices into this technology that is very powerful and finally I think it's fair enough to go on this idea of certifying responsible uses of AI to ensure that we will reduce as much as possible the consequences and I want to finish with this quote from Guy Fouli who wrote this highly recommendable book, AI Super Power he said that AI is the new electricity and big data is the oil that power the generators and this is very nice but if we use the same analogy to seeing in the global warming these future fossil fuels that we are wasting to data in a responsible way I think are related with the poor use of data so perhaps we when they can change the paradigm moving from large volumes of data or algorithms are extremely data-hungry as Olaf said into probably a smarter use of this data in a more efficient way ensuring that it won't affect anyone thank you very much thank you very much Olaf can you all hear me now? excellent point Olaf so let's start now with some discussions and I will go back to our speakers and probably I will invite first Tony Bates which mentioned during his presentation that most worried thing for me this is Tony Bates is the application of AI and the ethics and what is AI doing in education in this case is it not AI more to make the life easier for administrator but not taking so much into context or into vision the students Tony if you want to say something more about this you want to say something okay if any other of the presenters want to say something about this Diana question if not there is other questions on the chat okay Tony is typing well Tony is typing I can't unmute my mic I know I had the same problem don't worry Tony okay I can ask you sorry okay thank you for unmuting the mic I think that at the moment the emphasis has been a little bit more in actual applications on providing information that is useful for administrators particularly in terms of admissions and marketing it doesn't have to be that way I think one of the issues is who is defining the applications at the moment I think it is a very good question to ask who is the client for this kind of application and who is pushing it if you like so I think there are some issues around that it seems to me that a lot of the applications are being driven by computer scientists doing research on the topic rather than actually an institution making a deliberate decision to use artificial intelligence the only example I can think of here in Canada is Athabasca University who is seriously working with IBM I believe it is to develop an automated tutoring system for its systems education programs so everything is piecemeal at the moment it doesn't seem to be developed system-wide which may be good in the sense that we get some idea of what is happening once it gets applied at the system-wide level I think it becomes much more difficult to stop it if it is not going in the right direction those are my comments Thank you very much for this Tony that is really a good input I have Olaf which is one of the comments Yes, I would like to agree with Tony I can't, speaking for Germany for German higher education institutions I also can't really see a wide application of these tools in the institutional practice like in Canada with Athabasca University it is also here in Germany the Fern University in Hagen they have a cooperation with the German Institute for Artificial Intelligence Research in Berlin and they are developing together tools they are developing chat boards and also doing some research on automated assay scoring but it is still all in a prototype project-based status and there is no institution-wide applications of these systems Thank you Alexandra, as you are one of the computer scientists here do you have a point as a computer scientist? Yes, I'm one of the unwanted computer scientists I'm also an educator I've also been in the education system forever and I've also been doing a lot of research these questions are in a way not new every time there is a new technology there is also great fear I was also following the chat and there was a discussion about bringing in AI replace people, replace jobs, etc this question if you look at the technological revolution every time you have a new technology there is a great fear and sometimes it is disruptive but more often than not it's disruptive for a while and after a while basically we find that our life is improved we use these technologies and then we find other better ways of employing our minds we don't all plow the fields these days and we could have done this in the past but now we have technology doing that for us so we said we're having these lovely discussions this whole discussion we're doing online over the internet and it's facilitated by technology and we couldn't do it over such great distances if we wouldn't have that etc, etc I mean we have the example of Duolingo before I believe Olaf put the slide up and I had been at the talk of the owner of Duolingo at AIID conference and one of the things that really to some extent shocked me was something he said about online tutors and human tutors because we always had this discussion in the past every time we talked about personalization and education we had a discussion about what about the teacher and will we be replacing the teacher and the usual standard reply to this was well no it's just enhancing the teacher, the teacher can still be there and we're just giving the teacher and the student more capabilities but what really shocked me was that the Duolingo owner said we want to replace the teacher we don't want any teachers anymore because teachers are basically expensive teachers are also not scalable they have millions and millions of users they want to do everything automatic and I was a little bit sad I stood back about this very upfront reaction I'm not sure it's the best way forward but it's a tiny application and it seems that it works for them so and we have a lot of online early we have universities that work online we have we also talked about bypassing learning bypassing traditional ways of learning like higher education and to some extent that is a potential there's lots of companies I mean the bigger companies that basically create their own training programs and sort of apprenticeship programs into the job so and this conversion course etc so I'm sure that there might be some changes and maybe higher education will have to change itself to follow these changes but at the same time I think there will be great opportunities so that's part of my answer I could talk more about it very much Alexandra yes I know I'm pretty sure yes this reminds me so for example back in the 90s when I done my master in artificial intelligence everybody was afraid of artificial intelligence because we were doing it for robots and then the robots will become much more clever than human beings and so on and it still hasn't happened and it's more than 20 years in them so I just want to remind you a study done for example at Georgia Tech University where they used a lot of chat booths and so on in Coursera a lot of the answers in the online platform for MOOCs Coursera are done and answered by chat booths and at the end they done a survey to see if the learners have realized if there are chat booths or not, if there are robots and in fact artificial intelligence answering to their questions and not human beings and more than 65% of the learners realize that there's a machine answering to their questions and they prefer mostly the questions and sorry the answers which were provided to the questions by the artificial intelligence by the machines than those by the educator so this is really worrying for me I will never forget when I read that paper and when I also listen that I call that presentation because that's really worrying but I would like to move now further down to something more which is also worrying me somehow. Can you really monitor learning? Can learning happen only online and you can monitor only with technology or is more an individual thing and is it really possible to see that if you learn something is more effective or not effective Alexandra spoke about behavior path patterns that also lead somehow to that area but maybe Alexandra, Olaf you want to comment if you really think that you can monitor learning as learning is more an intersect effect Is there any question of the presenters? Olaf, can you say something about this question from Diana? Can you also answer please? I think Olaf is right Olaf you have to put your mic on May I comment on that question? I am concerned about the assumption of what learning is in many artificial intelligence and learning analytics applications particularly on the testing side what is being tested it reflects a kind of learning that I don't have at all I see learning as developmental in a sense if you take a concept like heat how hot something is there are various levels of learning there you can feel heat from touching it you can learn that you can measure heat in terms of temperature you can look at the chemical processes that are happening so the concept of heat is not static at all for an individual it is very developmental a lot of the learning analytics particularly are trying to capture something at one specific point in time and then drawing a pattern from that when in fact I see this particularly in the predictive studies where they are trying to test somebody on where they are in the first week of the course and can they tell where they are going to be at the end of the course and it is not my view of how learning takes place it is an iterative process it is a reflective process and none of that is captured in the data now you could argue that is true in face-to-face teaching as well we don't know what the students are doing and in some ways we know more in an online environment about what they are doing and we do in a face-to-face environment but the danger is you measure the things that you can count and you don't measure the things that are hard to count or are less observable and I think that is a real problem not just for artificial intelligence for teaching in general but when artificial intelligence focuses almost entirely on the observable immediate state of learning of the individual then I think you've got problems Thank you very much Tony really a good insight I think yes we shall think about when we start having problems is Cristobal do you want to comment on that? Yes unfortunately I'm always I tend to agree with Tony in a regular base so this is not going to be the exception one of the concerns I have is we tend to overrate but we can measure and I think the saying don't value what you measure measure what you value is really applicable here because you can measure clicks but there are a few things that you cannot measure would you say for instance that education the learning experience is completely disassociated of trust or frustration I would say there's no learning without trust and there's no learning without some level of frustration with trust how do you measure frustration these are concepts that are completely difficult we can build some index we can build some proxies but it will be difficult to measure so I'm all about supporting learning analytics but we need to understand the massive limitations that this kind of approaches have Thank you Anna Cristina is writing something interesting Yes Alexandra yes you can you can pop in our ballet I'm not saying AI is the magic bullet but I'm just saying it's a great opportunity and we shouldn't miss it by saying oh we could answer all the questions because to be fair even human tutors can get things wrong and don't necessarily answer all the questions at the same time I'm a great believer that we have to very carefully look at these things and my second slide is about us hiring people and one of the areas we're hiring is bias in AI and I actually wrote that application and we're looking we're very interested to get people that work on all aspects of bias starting with ethics and ethical concerns all the way to algorithmics and algorithms producing the wrong results like the type that Cristobal was showing us where there are some plastic examples if you take a piece of text and translate it automatically with Google and say like she is the CEO he is the secretary if you translate it back and forth from a language you're going to get he is the CEO she is the secretary so this kind of stereotype so the AI has the wrong thing because of the many wrong examples that it has been given and so forth so this would be algorithmic kind of issues so we're interested in the whole scale and I think it's a very serious and interesting thing of these days so at the same time when we build this new system we need to take that into account absolutely so I agree with that Thank you Alexander it's a big debate about the ethics of artificial intelligence and I call like Ana Cristina comment about social development and learning I don't know if what she meant exactly about social development the thing which for example causes into my mind is that if our students know that they are monitored and their behavior is going to be improved for example we tested and we evaluated this with our students in my university in Timishwara and when they know that we watch them let's make it like that then they behave differently so is it not changing the behaviors of our students with the one of the impacts of artificial intelligence so if Olaf wants to comment on this or Cristobal or Tony any of you is free just pop in so basically my question is is the behavior of our students changing so much when they know that they are monitored and their analytics are considered towards either better results or to put them somewhere in separate groups or something like that anybody wants to comment on this Tony? Yes I agree that the ethics issues is important but one issue that hasn't been talked about as much is the value issues and what people value why we have public education for instance I mean my view of teaching and learning is that you need to help every student if possible and so when you get artificial intelligence for screening students out then I have a real problem with that it's not an ethical issue I mean from an institution's point of view they want the students who are most likely to succeed but from a societal point of view we really want all students to succeed so what do we do about the students who are screened out um so it's not so much the actual use of artificial intelligence it's it's um understanding the values that drive education and could we use artificial intelligence in a way that actually reflects those values rather than just runs right over it That's a rhetorical question Tony or one to which I think it's also very important to see all of these things from the learners perspective because usually we are talking here about from this institution also we were talking about predictions et cetera not bad it's possible control from the learning perspective all of this data that some other people are collecting and are predicting and are analyzing from from us as learners in that case Can I say something Diana because I a lot of discussion about personalization here and learners being different and stuff like that and I think especially now in so for instance if you go on mooks right and you participate you will notice that there's discussions and so forth and you have for each of those items in the discussion for each of the elements of the course for instance in future learn you can get for each little video or piece of text you can get hundreds if not thousands of comments of users and it's really hard to tell which are relevant to you in any possible way there's no personalization at the moment right so trying to filter that based on your your interest your needs et cetera is it could be very very helpful and you know guiding you through all that material and this is the kind of thing that adaptive personalized learning can do for you so it can actually cater for individuality even on a large scale whereas that's completely not scalable for one teacher so they do have human teachers and they do try to make some sort of summary at the end about what has been discussed there et cetera but they badly scratch the surface because they just can't reply to each and every student one of my PhD students is actually specifically thinking at where instruction intervention is needed on this massive online learning systems because it's a vital thing to know whom to answer to try to pick up those that in this case might actually be dropping out but there's all sorts of other levels that you know we could automatically look at which are just not feasible for one person basically as a teacher Cristobal if you want to comment on this please sure thank you yes so my take is it's difficult to navigate in the tensions because they're not black or white it's not with or without teachers teachers with AI replacing some activities now in the US there's a large a growing interest in automating all the assignment scores and this is gaining momentum and today we know that the system have bias but let's be honest I mean teachers and revising assignments will also have some level of bias and because of the speed of the scale it's very unlikely that this will be so I think we have to address that there's a new key to the block and we need to find systems to educate people not to be against technology to develop some level of algorithmic thinking to understand how the systems work the level of bias they may have the strengths and the weakness develop some level of skeptical smart skepticism which is understand that may fail incorporate this kind of ethical fluency which is understand who might be affected and when possible some level of self-regulation and with these elements we will have the people better prepared not to be against technology but to use technology in a wise way much for this crystal ball and everyone at the end because we are almost closing I would like each of you to choose one single word to describe the future of AI in education so you already are only allowed one single word and because we are we are presenters not presenters but we are moderators and to give you one minute of thought my word will be inclusive I think AI will make education more inclusive so that's my pick so please everyone who wants to start first I'll take the women first Alexandra I see potential if I am only allowed one word potential crystal ball hesitation Olaf scalability Tony Tony maybe Tony sorry no no Tony Joseph what your word that dangerous dangerous oh no I don't have a word Tony dangerous Joseph for you sorry Joseph thank you very much yes we have to finish Diana because it's 29 can you put my last slide please Dora or maybe I can do it I don't know oh yes I can do it ok just a few words just to thank you everyone thank you to all the presenters also thank you to the people the audience because it was a very interesting debate on the chat and just one minute to remind our next events of course tomorrow we will have another one another another webinar in European Distance Living Week and just remember that the next events next year the Eden Conference will be in Tunisio where in Diana in Diana University in Romania and also next year it's time for a research workshop this research workshop will be in October in Lisbon hosted by the University of Bertha of Portugal and in these two events as usual we are also preparing a PhD symposium for PhD students, PhD candidates in Timi Suara and also in Lisbon thank you very much Diana if you want to say something thank you very much it was really very very interesting and I'm expecting that we can continue the discussion about how we use AI in education and how is the impact of AI in education in Timi Suara so I'm welcoming you soon in Timi Suara thank you again to all of you thank you everybody and I need to thank specially to the comments and the questions which were raised we really like them a lot and it's food for thought everything what happened here today thank you bye