 Thanks to you, Tim, thanks to Indie for being with us today. I just introduced the webinar today as Chair of the NAP, Network of Academics and Professionals from Eden, European Distance and Learning Network. It is really a great pleasure for us to restart the webinar series that we offer as part of our engagement as a Network of Academics and Professionals Committee within the Eden Network. We had to stop for a little while after the emergency came up and we started the Eden series. In a minute we are going to show you some slides related to the activity of the NAP. The NAP, as you can see here, is at its members' service. We try to support, first of all, networking. We provide meeting and communication forums. We work in synchrony with a steering committee that tries to provide information for members and opportunities, build up a personal portfolio. This is what I always say NAP is for, is a professional development committee. It promotes communication, networking, as I was saying. We support also in finding partners for new projects and new research ideas. This is the steering committee as it is composed at the moment that our mandate is going to an end. That's the possibility also to step in and participate in the elections that are going to take place in a few months from now. Here you have a picture of the members' area where you can participate, take part, include your profile. Being at the members' service means that this is a large community and that's what we want to be with all the opportunities we offer. If you are an institutional member, you can delegate up to 30 individuals in the NAP. You can attend conferences at reduced fees, of course, when it will be possible to have face-to-face conferences again. But the most important thing is that we can work together. We can create new partnerships. We can discuss together as we are doing during these webinars for developing new ideas, new researches, new cooperation. Of course, there are also publications connected to the Eden where you can contribute too. We use social media, so we work on Twitter and on all the other social media. We offer different kinds of webinars and this is one of the series I was mentioning. We will have others until the end of the term and we try really to listen to our members' and colleagues' ideas and try to interact the most we can. Join the NAP, join us on our social networks and thank you. Get more information about our next events on our website. Follow us and thank you so much to ING and Timothy for being with us today and have a wonderful webinar today. Thank you to all the participants. I know there are so many today, so it's a very good opportunity for all of us. Thank you. Thank you very much, Antonella. Once again, welcome to this Eden NAP webinar today on student evaluation during and after COVID-19. I think it's a very challenging topic for you. We have high expectations and I think it's going to be very interesting because I don't think it's necessarily difficult for us to get our students online. Most of them are in fact actually on there. Trying to engage with them in a meaningful way is difficult. That's a harder problem, but the real killer problem in a way is how to actually evaluate them. And that's the question that's been coming up in some of our webinars and I think it's on a lot of people's lips. I'm not going to present you because I think most people already know you. Your bio data is on the Eden website. Just to say to people that ING is very kindly divided her presentation into three parts and after each one we'll have a brief pause. So she can have a chance to answer some of your questions and at the end we'll have a general question and answer session. Okay, thank you very much, ING. I'll pass it over to you. Okay, thank you. So I'll wait for the slides to come up, but in the meantime I will start with myself. I think as you said, Tim, it's a very important topic and it's also quite a challenging one. I truly believe that if we share each other's experiences and if we share each other's insights that we will find better solutions than before. So starting from that idea, I can share my own experience in terms of exams and I will make you all jealous. During my master and in order to get my master and in order to get my PhD, I only took part in two examinations. That's all. And one of those examinations was an online exam. So it was really rigorous exam and the other one was like a defense, a normal, regular PhD defense. So I think that from a personal experience, I'm really happy that I didn't have to go to more examinations because I'm really not that good at exams, which is why the topic interests me. Now on the other side, I have worked at the Institute of Tropical Medicine and as well as in the renewable or sustainable energy sector. These are two sectors that require quite a high standard of health and safety guidelines and standards. So I do have experience with really very tough and very high quality certified guidelines that need to be followed and that need to be assessed. So this is where I am positioned. I'm somewhere in between the belief that formal assessment, formal examinations are really important and useful. But on the other hand, that non-formal assessments or creative formal assessments in exams can benefit us all at the same time. Okay, this is how we get to this. In a way, I think we must all question ourselves before starting the full presentation. What is the purpose of an exam or any type of learning assessment? Because is it really necessary to only look at what we classically do in the classical examination since? I don't think so. So that's why this webinar is divided into three sections. We will have a look at proctoring tools, see what it is, what it can do. We will have a look at moving from a closed book to an open book exam and what the benefits are and of course always looking at the downsides as well. And then have a look at the team or group exams and where they can help us. So let's start with the proctoring tools. Now proctoring tools are basically kind of software that is added to your learning environment and which enables you to assess or evaluate students and learners at a distance. There are a number of proctoring tools out there and it's still increasing the amount of solutions that are out there, but basically they follow the same steps. And you can use them for students at a distance, but you can also use them for if you have big classrooms like huge auditoria where you need to provide examination options for huge amounts of students and allowing a different type of examination to take place. Now I don't know on the site, small site detail. The picture I show you there on the left is a picture of a WeConnect Barco lecture hall, which is already a fully fledged online lecturing hall that we are going to start using at InnoEnergy where I work as well, and which will allow easier online assessments. But so for the proctoring tools, I think we must wonder how far we need proctoring tools. Because in a way, if you think about one of the reoccurring ideas or remarks that I get when I mention proctoring tools to our own institutions and we work with a lot of European institutions, both universities as well as companies. One of the reoccurring ideas is, yeah, but how can we avoid that students cheat on their exams? That's the main issue. Well, my first answer to that is maybe it's not that bad. Of course, I can hear you and saying, yeah, look, we have our degrees, we need to have a certain amount of quality, which is true. But in a sense, if you look at cheating, it is problem solving. It is working with peers. It is being able to curate content really quickly to see what is right and what is wrong. So to me, it features 21st century skills in some way if you cheat. However, I totally agree that you don't want people to cheat and by cheating, they no longer know what they need to know. Then you have a problem cheating while still knowing everything and finding great innovative solutions. Then I say, okay, why not? So starting from there, how does a proctoring tool really work? Although there are a lot of proctoring tools out there, the steps they use are basically the same. You have a software solution which can be embedded or linked to your LMS or your LXP if you have a learning experience platform. And you can add it to it and use the proctoring tool as a digital examiner. And so what the proctoring tool basically asks you to do, a student, what it asks a student to do is to show and identify themselves as being that person. So by showing the identity card, then to take a picture of their face for later recognition and then activate the proctoring tool which will enable the screen of the computer to be fully recorded. So if you have tabs open as a student, if you are using some notes on your desktop, cheat notes, something like that, then it will be recorded. And it will also ask you to make like an environmental check, whether there's nobody in the back helping you or in front of you helping you out with some of the questions in sign language. Why not? Being innovative. And that is basically the first part of a proctoring tool to really look and whether the students don't have any additional tools working in the background. Now, this also means that each step that a student takes is actually logged into the system and can be reviewed later by proctors. These proctors can be live proctors, so people that are at a distance looking at the screens that the students use to see whether everything is working well and whether they don't cheat. But it can also be done by recording the exam as it moves forward and then sending that recording to a proctoring company and have it checked out by proctors once the recording has been sent to them. In that case, the feedback from those proctor comes back in a report 24 to 48 hours later. So from a student point of view, it's kind of, well, it's kind of safe in terms of cheating. Now, from a teacher point of view, of course, you need to invest some time in it. And you need to invest time for the admin because you need to check, can they use anything? Can't they use anything at all? Can they only access the questions that you have provided for the exam or something more, like in between open book, closed book exams, something like that? Because with each proctoring tool, as you can see here, you have the opportunity to add an additional source that a student can use. This can also be like a 3D design or sketch type of tool. It can be a different type of tool. It can also be a calculator, a dictionary when it's a language exam. So there are a couple of options open in a proctoring exam to teachers. But as I said before, it's an add-on. So as a teacher, you already have to have an exam set or like a question database set in your learning management system or in your learning experience system. Now, the nice thing is you don't need to walk around once your students or you don't need to be like a live proctor yourself once the proctoring tool is taken over. You can also really time the exams really up to the second almost of each student. And because it's linked to your LMS or LXP, it will be automated, the full process of grading, of possibly weighing some of the questions. But of course in this, you can see how with COVID-19 coming up, these proctoring tools become much more useful because it's a solution for many universities. And with our partners, there's, for instance, TU Delft, which is a university in the Netherlands that already is using proctoring tools for their students and who have rolled out exams up to 2,500 students for the same exam on the same day. Which is pretty, well, it's a huge amount of students. Nevertheless, there is a limitation to students being able to start the same exam. One of the, I compared a couple of proctoring tools and I will show you my little comparison later on in the presentation. But mostly you can have like 100 to 150 students starting the same exam within the same hour. So there are some drawbacks that you need to take into account when you set up an exam. But the good thing is students can't take print screens, send it, mail it to other students in the meantime, something like that is not, it's possible once you have a proctoring tool installed on the exam computers. So you have a lot of benefits. There's, of course, the obvious higher flexibility in terms of location. You have a higher flexibility in terms of timing because you can really set up the schedule. You can reduce some of the anxiety for some of the students because they can take the exam in their own natural studying environment, which to some students makes a difference as well. And in a way, you have quite a safe learning or examination environment because of the proctoring tools. In another university that we work with, KTH, which is the Royal Institute of Technology in Stockholm in Sweden. There they use a proctoring tool for all of the exams in the university. But the proctors, the life proctors are the teachers themselves who walk in between, well, who supervise a part of the exams. That's because using proctoring tools, even ad hoc in the university, is safer than if you don't use them at all. So they're pretty cheap proof. Now, can also, if you think about post COVID, then it becomes of interest to use a proctoring tool for specific student groups like athletes. If they're on a mission or they prepare somewhere else, they're in training, then still they can take a remote exam. Or if you look at disabled students who cannot make it to the exam, then again, with a proctoring tool set on their computer, they can't take the exam or the assessment depending on what it is. And of course, it's the same for learners having some kind of long-term illness. So there are quite a bit of benefits. Now, on the downside, English language dominates all the proctoring tools. There is one tool who is now considering use, attracting life proctors who can speak the native language. In this case, it's Dutch, but it's pretty difficult because the standards of language and communication skills needs to be really high for proctors in order to understand what is happening when the student is taking the exam or the learner because there are also corporations using proctoring tools. And of course, you need, as a teacher, you need to become comfortable. You need to trust a proctoring tool as well. And there are some practical limitations as well, like you have sketches, drawings. Sometimes you need to draw it up in like an art class immediately. And you need to be able to confer with a teacher in order to say, is this well done? Is this Joker or something like that? So there's definitely a practical limitation in that area as well. And it needs a lot of preparatory work because, of course, IT needs to be involved, the teachers as well. You need to come to some kind of agreement. And you need to set up a special help desk for this for the students as well as for supporting the teachers. Now, in terms of other stakeholders, it's easy to come up with the fact that it needs to be chosen, which means that the university or the corporation in a whole must dedicate some time in selecting the best system. And every couple of years, you need to do the same exercise. You need to make sure that it is LTI and API compliant. And also, of course, if you have previous tools set up for this type of purpose, like an LMS and LXP or any other tools, it needs to be compliant from both sides. And you need to set up support. You need to consider whether to use your own proctors, your own teachers, or whether to rely on the external proctors. And of course, from a legal side, you need to know whether this policy or this solution is in fact consistent with your own internal policies. Now, in terms of pricing, I refer to a document someplace. It's not this one on the slide, but on the first slide with the title of the webinar, you can find a link to a full document where there's a full comparison of different proctoring tools. In my case, I only limited myself to looking at three different proctoring tools based on the proctoring tools that were already in use with our partners. And if you look at the pricing itself, if you look at the first column, it's clear that they all use a different type of description on how the pricing works, how it's calculated. So it's a really good practice to try and figure out some kind of even benchmark to see how much to be able to compare it. But it is within the same area, between for a thousand students taking 60-minute exams. The first one that's like the highest cost will vary between 14,000 and 18,000, let's say. So that's about it with the proctoring tools. And then afterwards, after some questions, I will move to the next open book exam. Thank you very much, Inge. That was a wonderful part of your presentation. There's been some questions coming up in the chat. For time restrictions, I'll just ask you two of them. There was a nice question come up from Colin about the practical realities of buying essays. Because even if we don't necessarily want our students to sit down and write essays, we know that there are lots of companies, well, companies in inverted commas that offer that. In fact, some of my colleagues have been contacted and offered money for producing model solutions. What do you think we can do about that? Well, the essay mills are, of course, for me, I would immediately say if you do an oral exam, it will soon be clear that a student cannot solve any question you have or solve it in a very low standard type of way. But of course, asking for essays is an example of a classic type of assessment, which means there are solutions that you can buy out there. But of course, if you make your question like, do you know the Kobayashi Maru type of... Well, everybody who loves Star Trek knows the Kobayashi Maru challenge. You ask a question that cannot be solved or you make a situation that cannot be won. And then you ask students to come up with a solution as close to the best option there is. And if you have such type of questions, it's much more difficult to buy an essay from one of those essay mills. So that's the only problem. I think that's a very good answer. And Jakob Katz also asked a question, which I think is very relevant at the moment. And that is, what about taking exams from home when students are actually in the middle of this, you know, COVID lockdown, we can't necessarily go to the university to take them? No, indeed. So with a proctoring tool, you can just do it at a distance. If you don't have access to or if the university doesn't have enough time to go through all the administration of buying a new tool and installing it and using it, because a proctoring tool is something you can send out to any computer wherever located in the world, if it does have a strong internet connection. Then you can fall back on best practices and best practices is actually what I have used in the past as well, where you only rely on an audio and a video connection to do the exam and you ask them to show the room before and you transform your questions into more of an oral examination type of questions. And then you can also do it from home in really short time and without the cost of a proctoring tool. Of course, this does demand extra work. You need to transform your questions into more like an open book type of question, which fits some content but not all of the content. But in the document that I referred to in the first slide, there is also... I think it's in that document where the best practices without the proctoring tool for taking or organizing online exams is also described. Wonderful. Thank you, Inge. I'll let you get on with the next part of your presentation. We'll come back to questions later. Thank you. Okay, so next part, next section of three sections. Section two, open book exams. As was mentioned before, not everybody has the means to purchase a proctoring tool and to get online exams going with this proctoring tool. Another option to do is to use classic audio video and allow students to have open book exams. In fact, it also relates back to distance education when it was like on paper-based. You would send out your questions and the answers would come back and it would take some time and you phrase the questions in that way that they were open questions because then you would get the best type of answers back. In a way, I like open book exams. So the best known exams are closed book exams, of course, where you have a fixed amount of knowledge and that knowledge will be assessed using multiple choice questions, essay-based questions. But the students or the learner isn't allowed to go and look at solutions on the web in the library or just simply ask their colleague. But you can do and so in a way, that's one who is best known. But if you go back into time, you can see that the Socratic method is much older and actually relies on question-answer-question-answer and there's a dialogue and you can do the dialogue yourself, one-on-one, I mean, or you can do the dialogue with multiple peers or multiple people and then dig into some of the knowledge and understanding and the ability to connect concepts using the Socratic method. So in a way, these type of more open book exams are much older if you look at time. Now, an open book exam stands right opposite to a closed book exam and there is, of course, a gray area in between. But with an open book, you ask questions, you have the exam prepared, but the student or learner can go out and use any type of resource that is out there or if you have partial open book exams, they can use some resources. So as I said, there are some benefits of moving towards open book exams. I think most of those choices are based either on pedagogical issues or on philosophical issues. It's again, like with the proctoring tool, what do you want them to do? Do you want to avoid cheating or do you want them to become ethical problem solvers using resources and, of course, using some ready-to-use knowledge? So an open book exam is useful if you want to organize some kind of online evaluation because it is easier to organize and it doesn't take too much money to provide an open exam where you ask to apply and analyze certain knowledge. Now, of course, where a closed book exam asks straightforward questions with an open book exam, it's about the process of the answer that comes back to you. You need to see if it's high quality. You need to see if all the key concepts that the students learned in their past period of time are actually mentioned in that process, that they use it in the right way and that they have an understanding of how it all fits together. So it's much more in-depth in a way. From a teacher's side, that means that you need to acquire the skill set to be able to relax and say, okay, with an open book exam you don't have fixed rights and wrongs. It's always somewhere in between. There's a gray area. The evaluation is somewhat, well, I say somewhat because of philosophical reasons, somewhat more subjective because you can say, look, the intention was right, but this concept was not completely used in the correct way. So as a teacher, you need to build a different type of rubric in order to evaluate the assessment. And if you look at open exams from a teacher point of view, you can say, look, the student can use a limited amount of resources the student can use within the time frame, all resources, or you can even step away from the time boundary and say, look, you have 24 hours to come up with the solution to this question, which is like what's it called a take home exam, which is really like very well known in distance education. So you have a couple of benefits. The benefits, I see that I have a typo here. You combine different concepts towards the solution and you request the students to relate that in the answer. You can show the bigger picture of a specific knowledge area. And a good one that I haven't mentioned is the qualitative and quantitative data interpretation that you can also ask. So then you provide the student with a set of data. They have studied the key concepts and you ask them to work with that data to come up with a new solution, which is a nice addition to open book exams, I think. And of course, it evaluates a broader array of student knowledge as well. Basically, from my perspective, it fits 21st century skills that people or students need to have. On the downside, in terms of grading, it takes a lot more time from a teacher. It can be. If it's purely like an oral exam, it goes quickly. But if the answers need to be read and interpreted, it will take you a lot more time. And of course, it also lacks in that fact response. If you look at the health examinations, you need to have, or for surgeons, you need to have immediate knowledge, prepared knowledge, and not just relating concepts together. So it doesn't fit all types of content. And it's, of course, less straightforward in terms of grading. But having set up some open book exams myself and looking at the brilliant open book exams some of my colleagues have come up with, I can tell you that it's much more difficult for students to really answer these exams and to answer them correctly in high quality in a specific limited amount of time. There's also this myth about you don't need prior knowledge, which is totally untrue. You need to have the basic knowledge. You need to start from that and build your concepts and understanding from there. And of course, people sometimes say, yeah, but if you open up all resources, they can just go there. And it's like the essay mill. They just copy a bit of content and put it in the answer. A good open book exam question is much broader than just give me a paragraph on this or this subject area. So it will not do to just simply go out to a resource and copy an answer from there. As an example, I refer to one of our energy data cases. We are now having two data sets for specific learner groups. We have data sets for academics, specifically those who can code and those who can't code. And the data sets for those who can't code are data visualizations where the students get a data set and need to use that data set to combine and distill and look at causality also to correlations to indicate how this data set could help them interpret a specific question or challenge in the energy sector, in this case, the energy sector. And so we have a similar energy data case for people working in corporations, so business people, again, divided into those people who know coding and those people who need to be able to use data visualization. And those data sets are really, they are in high demand and they really work well as an examination tool as well. So, yay! Time for questions. Thank you very much, Inge. That was a very interesting part of your talk. And we had some interesting comments in the chat as well. And I don't have a lot of time for questions, but I particularly like the comment by Drago Salomon who says that they actually like doing open exams and someone that can be done at home and smart student projects, et cetera. But I think Carol Kios makes a very interesting comment and question here that it really depends in a way on how you're actually going to formulate your learning objectives, you know, if you're using Bloom's voice technology, et cetera. And certain things do need to actually be assessed in a written way, especially if you're actually trying to, I guess, train the students in the sorts of skills they're going to need when they hit the real world. So, I don't want you to think about that. Yeah, definitely. Well, in terms, I totally agree that open book exams are only, well, the best type of exams in my view are a combination of closed book and open book exams because you need to have some, the basic knowledge, even if you have like high level knowledge, there's still this central bit of knowledge that you need to have immediate, that you need to recall immediately if necessary and use inside bigger picture, inside of concepts. So, in terms of that, I totally agree. Sometimes you need to go for the one or the other. But of course, in COVID times, at least if you can transform some of the closed book exams into open exams, it will alleviate some of the tension of having to be able to roll out all the exams online. In terms of coming up with intended learning outcomes or learning outcomes in general, then I agree that using Bloom's taxonomy won't be that easy to... Well, of course, you can, the higher levels, then it becomes easier again to build learning outcomes from that, from that. But I do feel that if you have an open exam, even if it's really open, it always comes back to understanding the concepts that they have seen or understanding concepts that they needed to find and analyze and internalize themselves. So, it can even be a learning outcome that is part of some self-directed learning initiative that you ask from them to take up prior to the examination and to transform that into a learning objective. So, although it's... you will most probably, but not for all, but in most cases you will move up in Bloom's taxonomy, you can, even with open book exams, you can describe the learning objectives and make them really strong and use a rubric to assess the answers that you get against those learning objectives. Yes, thank you. Thank you very much. I think an interesting comment I heard the other day as well is that sometimes you can use closed exams or with students, for example, even if they're at home, and then because you've been working with them over a period of time and you've established a relationship, then you know roughly how they think and how they're going to apply. So, if you suddenly find an answer that doesn't really fit in with your idea of that person, then you can even contact them for a normal follow-up. But I mean, that has to be done in a scale way. You can't obviously do it with all the students. Okay. That was wonderful. Let's roll on with the third part. Thank you. Third part? Where am I? Here. Okay. So, now we're going to have a look at group exams. I think the one posing the questions earlier, hopefully, and more of you, of course, will hopefully like it. These group or team exams, why I added this is because I work in a renewable energy organization, which means we work with master's students, PhD students, professional learners, all coming from an engineering, or mostly engineering background. Now, in engineering, as you know, you have the same pedagogical formats that are used, but some of them are used more frequently than others. For example, you have design thinking approaches used to come up with prototypes, to come up with solutions to challenge-based learning and all of that. Now, this is where I come from with thinking that this might be a good addition to the online exam options, because we needed them as well. Now, if you think about group exams, so you can think about projects, but the group exam in action, for me, can easily be visualized by thinking about hackathons. In a hackathon, you have a couple of challenges. These challenges are normally not yet answered, so it's a real open challenge. You ask those who want to take part in the hackathon to choose one of these challenges, and they can choose it, and then they form a team, because they know this is a good hacker, this person is good in marketing. Here's the one we need for communication and structuring things or managing. You build teams, and these teams within 24 hours or whichever timing that particular hackathon has needs to come up with a solution within that period of time, and they have access to a lot of options and they can consider different choices and build from there. In a way, I feel that a good group exam has these elements in them. It's a team effort, but at the same time, you rely on the expertise of each individual being part of the team, and everyone's input lifts a normal challenge to a higher level and making the end product or the solution, or in this case of exams, the answer makes it better. I added Digi Edo hack because this is one of the hackathons I'm involved in as a supporter, and this specific hackathon looks for teams who are willing to come up with solutions for education, so this is really useful in these times. Now, a team exam, normally you would plan a team or a group exam if you already have the prior learning in some way team-related, like if people work on a project, people work on a specific challenge, then that's the moment to consider online group exams. Now, the group, so let's see, here we go. In a group exam, many of you will have had some experience with evaluating teams. It's the same here. You have external and internal experts who will look at the specific content that needs to be evaluated. You have a team presentation provided by the designated team speaker, or you can have the team presentation consisting of smaller bits of the presentation each time taken online, so one after the other, taking part of the slides and explaining that part of a project or that part of a solution which is evaluated. Then you get feedback from the examiners to the full team as well as to the individual team members in order to see if you understood everything correctly if everything is transparent to the examiners. After that, part of the group exam, you have an option to add questions and answers on an individual level. Why is it important to add these questions and answers on an individual level in an online group exam as well? That's because there's always, as I said with the closed and open book exams, there's always this basic knowledge that needs to be there, the theories, the concepts, the descriptions or the processes that they need to know, the formulas that they need to know. You need to be sure that each member of that team understands the basic knowledge needed or required for that evaluation. That's why you would put in questions and answers as well. Of course, it's also important to have an open discussion. Again, I referred to the Socratic Method to really dig in deeper and to see the extent of knowledge and comprehension that each of the individual of that team has. Then once it comes to grading, you can either use a rubric that you set up in advance or you can use a voting system talking to one another and seeing how much you would or which grade you would give to which member of the group. That's generally the overview of a group exam, which you can do online if of course you have access and if each of the team members has access to good audio-video connection because that's the basic need or tool that you need. Now, in terms of benefits, it's clear that it benefits the multidisciplinary team evaluations. It's also good for some real-life project evaluation, which we do and I will show you in a minute. It also fits innovation, but that's, of course, because in innovation, you never have straightforward answers. There's always the process of coming closer to a solution and teamwork is really, well, something that works well in these conditions. And you have a near-immediate grading which saves time, especially if you compare it to open exams that you need to read through. Now, the downside is it takes multiple people, multiple examiners, so it takes much more time, people, time from the teachers. And the grading is, of course, an interpretation. Again, like with the open book exams, it's less straightforward. Yet, if you have a good instrument, you can deliver the same rigor of grading. And normally, if we look at great teams, we also consider how they came across, because of course everybody, because we are a European organization, we take into consideration 21st-century skills and it makes a difference in how they deliver each part of the presentation. And although it's an online team exam, still you can see that certain persons are more open to delivering what they know and others would be more hesitant to know it. So you need to take into account different personalities as well. In terms of preparation, from a teacher's side, of course, you need to prepare the project questions that you will ask. In an ideal world, you will set up a rubric as an instrument for grading. You need to have the team members all gathered at the same time, including supervisors. And from an ICT or an IT side, you need to have a durable, consistent and online meeting tool, like Adobe Connect here. You can also use Zoom, which is rolled out quite massively during these COVID times. And of course a recording, because you know that each exam can be questioned by the students or learners taking the exam. So it's better to have a recording for possible litigations or something. And from a student's side, you have to have a clear understanding of what is expected of you, of course. You need to understand the concepts, you need to be really well prepared as a student, and you need to have social skills, because it is a team effort. This means that everybody has a responsibility towards the whole team as well. But this is, I think, the last slide of this section. Well, last but one. The example I want to share very briefly, looking at the time, is the InnoEnergy Smart Cities Innovation Journey Example. So this is normally something that would be done in real life. It's teams entering into a contest to find the best solutions for smart cities. And each time you have these very specific topics. In this case, it was Andorra Smart Country Edition. So finding solutions for Andorra, looking at what they have, what the best solutions would be from an engineering energy perspective. And normally it's something that, so it's a physical event. And all of a sudden in just a couple of, I think it was 12 days time, or something they needed to transform it completely into an online event. Now, Mar-Martinez and Xavier Crusad, two of my colleagues, which are like brilliant teachers, managed to transform it completely into an online event. And with all of the planned sections taking place, and all of the students completely interacting and collaborating with each other to come up with real products in such a brief period of time. So here we tested out group exams, which were immediately put on the spot. And it came out as a real strong contender for student evaluations, which made us all even more in favor. So again, before closing or opening up the last questions, I think we must really question the idea of exams before we make any kind of selection in our exams or assessment methodologies and wonder what is the purpose of an exam. And also think what are the other questions besides the classic close book examination option or the using proctoring tools in the limited version, so not opening up resources and only thinking about trying to limit the amount of cheating that is being done. I think it's much more important to think, look, if we consider different types of student evaluations that we look at, does it enhance self-directed learning? Does it enable self-evaluation on where they stand, what they need to do to increase their knowledge. And in the end, I think it would be brilliant for me to give rise to ethical problem solvers using all the tools available. So that is it. I think this is the last part of the presentation. So I'm ready. That was a wonderful part of the presentation. We have been having a fantastic conversation in the chat. We should be finishing now, but what the heck? We will take a few minutes to enjoy this opportunity of asking you some of the possibilities that might actually take place. I think it was a question asked by one of our members of the public. I can't quite find the exact question here. Do you think that we will make the most of this opportunity that the unfortunate circumstances has given us with the lockdown virus? Do you think we will come out fighting and stronger and have improved and learn from this experience to make learning and evaluation better? Now you are asking. I can only answer from getting a little bit older. And I have the feeling that although you can have a slight tweak of a movement, so let's say I would say there will be a slight tweak because people think I assume it's nice and I can record some of the slides in use of flipped classroom more frequently let's say something like that. Or maybe let's try an open book exam for this bit of my syllabus. In the end a person is prone to stay close to their own personality. My personality is normally positive, normally optimistic or it's terribly pessimistic and I think even though with COVID we could test a couple of tools, some of those tools will stick to most COVID as well, but I don't think there will be like a massive shift towards distance education because our institutions aren't yet built for it and we don't really have the administrative structure to most of our institutions, some of them do to do it. So I think although there is I hope there will at least be a small shift but not that it will be massively taken up now with brilliant solutions. I think that's a very sensible, very practical and pragmatic answer and knowing you're quite appropriate. The question I meant that I asked before was from Philip Furran, I found it now in my list of questions. Okay, from another perspective I'd like to know what you think about structuring assessment with the overall core structure because if you can imagine that you've got a series of training which is like one course after another after another and you've got like an onion structure of learning building, so therefore in a way the students if they are teaching, if they aren't really learning and being assessed appropriately at the lower levels then as they get further out of the onion if you like further up the hierarchy of learning what they've learned, I mean surely they're going to have problems anyway. So in a way it's the self-fulfilling prophecy, this question of failure. I think so too, but the difficulty here with the problem is if we each of us would have known COVID was coming we would have built our courses in a different way already at the beginning. Now we are suddenly faced with having delivered content in a certain way taking into account the assessments that we would normally use in the end and all of a sudden everything is twisted, so now we are like piddling with in a very unstable canoe. We have all of this knowledge that we have but indeed in the end even if we decide to use more in evaluations or not once the COVID is passed indeed it's this onion layer type of approach that will sift the students that aren't really that knowledgeable or that just don't comprehend the basics that will well fall out of the rest of the full four years of master or any type of learning journey that they have in front of them. So I do feel but I think in a way an open exam can make it easier to filter them out but of course do you need to filter or do you take students along as as long as you can because they will learn something and it's good for everybody to learn or do you say look if you can't take it this is your cut of time and move. There's a philosophical part. Indeed there is I think this fits in quite nicely with a comment that our colleague ever made in the chat but in a way that when you're thinking about assessment and its purpose it fits into a more macro level it's a question of institutional and authority requisites and priorities so I think it's very interesting I'd like to ask you a question that came up also for my colleague of ours Alastair Krillerman who said he actually liked what you were saying about group assessment process and he specifically wondered which parts of this process would be synchronous and which parts would be asynchronous. Specifically for the group exams right? For the group assessment yes well normally for the online group assessments it's completely synchronous. So I must honestly tell you that I never thought about defining it into asynchronous is synchronous but it's a really good idea because then you take then you limit the time of the synchronous evaluation and you already get some idea like the questions and answers well thinking out loud but the questions and answers moments within synchronous team exams could be taken out and just sent out to them and then of course the synchronous part is reduced and you already get a feel of who will be the stronger one or the less stronger one. So that's... Okay I think that makes a lot of sense. To finish on a very practical question from Natalie Ross that the poor thing asked twice in the chat and I didn't get round to asking you earlier on the proctoring hall you showed at the very beginning on one of your first slides can you tell us where that is please or which one it is? The proctoring room? Just to be sure that I have the... almost there one back maybe Great. Yes, I think it's that one. Is it this one? This one is actually from Reconnect from a Barco system so Barco systems are the ones who are making these huge IMAX and 4D cinema screens but now they have developed an online lecturing hall and I tested it myself to me this is the closest you can get to a classical lecture hall as possible because you have a whiteboard so you see this man here behind him he has a whiteboard he can use polls he can even use the learning analytics coming back from the system and transform them into a graph to see who is talking more than someone else and so make complete learning analytics from the logs as well so personally I was really I'm still enthusiastic about it and I know that in Barcelona they have one in a business school which is called ESE I think and they build it together with Barco so this system they provided the academic and the lecturing feedback and then Barco went back to the design and everything and so you can ask them to visit one of their they have these rooms which is a completely virtual one and then they have hybrid rooms so you have basically present and then those screens that you see in the back are on the wall of the classroom and that's for the students at a distance but I don't have any benefit from telling this just to be sure I'll let you scream down to the end whether you can leave your final slide I'm very grateful for your presence here today thank you very much and we're also grateful for everyone who's more than welcome those are the people who've attended we have our next Eden Together in time of crisis webinar coming up next Monday at 5pm Central European Standard Time and our Eden conference which is now fully online please connect to our website for that, thank you very much to everybody and keep safe, bye Thank you, thank you for being here, bye