 So I'd like to invite our panellists now and introduce them to you. We have with us Professor Alison Little-John from the University College Knowledge Lab and Alison's work concentrates on professional training and learning. And that's usually within a global context so that would be a very interesting perspective for us. Matt from Australia, he's an actual assessment specialist so we're looking forward to hearing what Matt has to say and we've got Marco Kaltz with us from Heidelberg and Marco among his many interests is formative assessment and assessment for learning and we've also got Tarja with us. Tarja from Uvascular, her work is very related to inclusive higher education, a perspective we really need in this discussion and she's also a specialist in e-assessment looking at proctoring systems that she took part in a big European project. So thank you Tarja. So I'm going to ask the participants a number of questions and I'm sure they will build upon each other's answers and let's make a start now and I'm going to start with Alison. And for all of you, what was your university's approach to summative assessment and now that we've been through some of this, what were the pros and cons, what have we learned in our areas? So over to you Alison, welcome. Thank you Denise and thank you so much for the invitation to this panel, it's on such a relevant and topical, timely topic. So looking forward to discussing these issues. So I'm from University College London in the UK and in March 2020 like many universities around the world in line with the government guidance, our campus went into a lockdown and that's one we really had to change the assessment. Now to give you an idea about UCL and its scope and scale, there's about 44,000 students and many of those students are international and we advise them to go to wherever their place of residence, main residence was. There's around 8,000 academics spread over 11 faculty so a lot of different topics, disciplines and ways of examining. So I would describe how we changed as being an evolving approach where we did some test and learn, although it wasn't always well structured and it was a little bit ad hoc because it had to happen so quickly. But in 1920 academic year, so around about March 2020, all exams and assessments were changed to online capstone. So in first year for example, there was a high stakes exam which was going to be online for first years and usually those were over 24 hours and could be book exams. Now we have a variety of different approaches to assessment but in general a lot of the exams would have been a three-hour exam in a lecture hall where we had invigilators watching over so that was quite different. We implemented a no detriment policy to make sure that there was no discrimination depending on the circumstances of the students. Actually we didn't have to use that policy and it had some consequences in later years. The students could self certify if they had extenuating circumstances that could affect their exams and I think that was in line with many universities across the UK. So as I said an evolving approach, so in the next academic year, 2021, the academic year that's just ended, what we discovered was that we benefited from purchasing a platform which was Uniwise. This is a Danish-based company that we've been working with and that meant that the online exams could be managed centrally and most of the exams were 24-hour but some went back to being timed exams. We discovered that because we had put in so many clauses in the previous year that there had been some grade inflation and it was very difficult to then go back and say well we don't want these clauses in place because obviously the students themselves felt that these different checks should still be in place. So this year we've learned from all of that and we now have a new team of digital assessment advisors across all faculties and this is to encourage more online assessment as we go forward and as we move out of the pandemic. So we're encouraging people to use the Uniwise platform so that there is some central management and also support for colleagues and most centrally managed exams will be online this year with timed windows. So as I said going back to what were the problems we overcompensated in the first year. We wanted to make sure that everything was as fair as it could be but that did lead to some grade inflation but what were the benefits? Well it was like a massive leap forward in terms of thinking differently about how we could use technologies so that's what happened at UCL. Thank you Alison. I'm now going to hand over to Marco. Let's hear what was happening in Heidelberg. Thank you Denise and also thank you Eden for organizing this webinar on a very timely topic. So I think in the panel here I'm representing a very small high education institution the Heidelberg University of Education with approximately 5 000 students and 300 academics and the development was of course the same like Alison reported with an ad hoc need to completely redesign our services and also examination and assessment approach and what I remember from it is that the first thing we did was we changed and relaxed the examination policies in the first place to get stress out of the system more or less to be sure that neither lecturers nor students in any case frightened to do exams in a different way and we we built in a lot of flexibility and repetition options for different kinds of exams. I think this was one of the first activities that we did. At the same time of course we needed to do that to change also the practices for the examination to adhere to the new reality and to allow for example online examinations which was not foreseen in many study programs. So that was more of the policy-based approach. Like I said we are a small and high education institution. Our whole infrastructure is running on open source software without any external service providers. So we set up also in a very short amount of time a pilot study to find out what kind of assessments we can realize in which systems. In the end we implemented the most of the assessment in line survey in Moodle and in a social surveys also a service system and what we saw in practice what has been implemented was a mix of open book exams together with time-based testing so mostly multiple choice but in a in a very short amount of time and from what we've learned I think in general this approach has worked quite well. We also had to work on usability issues for students with disabilities because not all of these systems were by default following rules that also students with different types of disabilities could be using them. And one problem that has not really solved is everything related to authentication of course because that is still for me a very complex issue if you have only remote exams. Yeah how can you be sure that the person that is doing the exams is the person and so on and so on. The open book exams were mostly focused on more complex problems and procedural knowledge which I think was also interesting because some exams were transferred to more knowledge testing to more complex problems which was difficult in a short amount of time to do but on the other hand a very interesting move. The time-based testing were mostly really multiple choice questions with knowledge testing and some of the assessment practices just used so many items that you didn't more or less even have time to think about cheating in such a such a stress framework. We also of course had an evolving cyclic approach to development. We also had to see if our infrastructure can take so many uses at the same time because our institution is a presence institution and we're using normally these services just accompanying to our teaching presence in the institution but that has also worked quite well. We looked into proctoring solutions at some moment in time but nobody was really interested to to go into that solution also because we had the feeling that besides the authentication problem most of the assessment practices could be done in the systems we've set up. Making a jump into now we are now tomorrow I know today actually we started again some some presence teaching but we're still using a lot of online systems and I would expect that some of the assessment practices are also still existing when we're going fully back to the campus. Thank you. Thank you Marco. Allison there are a couple of questions in the chat perhaps you could answer them. One of them was what do you mean by great inflation and the other one was what company were you using? Unimwise isn't it but could you give some details? Yes in reverse order the company's called Unimwise you can find out about them online just search for them. By great inflation what I mean is that well the whole system changed overnight and so colleagues were left in terms of thinking about how they assess so I think that colleagues basically would err on the on the side of being as fair to students as possible and that maybe meant that how they assess the final grade if it wasn't a number if it was you know an A or a B or a pass or fail would generally be fairer and more open than it might have been in previous years. Of course we look at the data it was very different exam conditions so we have to be careful in terms of how we interpret that but these were for first year so we're going into second year and I think colleagues wanted to make sure that all of these students had a fair chance. Thank you Alison and we'll now move to Tarjan and let's find out what was happening in Uvascular. Good afternoon everybody and maybe I started with a couple of words about our university so I come from the University of Uvascular and it's quite a traditional university meaning that students mainly come to the campus. As a part of the traditional university we have an open university which offers studies mainly online and those studies are open for all no matter what their educational background is. At the open university meaning online we do not offer or produce or give a decrease but parts of the decrease. But with open university studies our students can complete their degree in faculties it's a kind of special system in the world I guess. And in Finland higher education institutions and teachers have a quite a big autonomy in terms of assessment especially when compared to many other European countries and there are not too many regulations at the national level and only few in institutional level. Of course we have curricula to follow. During the pandemic and lockdown our strategy was to offer some options for staff or and teachers and and it was strongly the choices were strongly based on choices of individual staff members or teams working and and teaching together. So I guess the most common choice was to add the flexibility of studies by using formative assessment instead of summative assessment. So very quickly teaching staff had to change assessment mode and and the pedagogy of the courses. And then secondly we have an exam system for students to take exam almost whenever they like. It's organized by consortium of 27 higher education institutions and institutions have so-called exam rooms in campus. So in I am not sure but I think there are those kind of rooms in all the higher education institutions who participate in this consortium. So it means that students for example in our from our institution can go to the University of Helsinki to have exam and things like that. So it's something but during the total lockdown those rooms were closed. Nowadays a student can use them with some restrictions. So then the third option was to have a scheduled kind of open book exams in Moodle. So it's something same you already heard about from other institutions. So it's in open universities the should I say basic way to study and have formative assessments. So it was something not so new but it was we didn't have any proctoring system integrated to our Moodle. We tried to use also we tried Zoom exams but we quite suddenly we got feedback or not actually our university but other universities in Finland that there are some challenges with the GDPR when using Zoom as for the exams. So and then we have exams in Moodle with the proctoring but that wasn't very often used option partly because we have or had a very limited number of licenses. The system was integrated to the Moodle and as far as I know the costs were remarkable and it didn't work very well or they had a lot of challenges with it. So yeah but nowadays like Denny's already mentioned I participated in a European project called Tesla project. I think some of you are familiar with that system and now we have acquired Tesla system to our university and we are looking forward to see what kind of possibilities we can offer with it especially when traditional face-to-face exams are not an option and summative assessment is needed. Yeah we are lucky to have many solutions and options available and we are lucky to have a flexibility in our institution so I have a feeling that we did it quite well so far. Thank you. Thank you Taja there's a question for you. Okay. A floor. When you said students can take exams whenever they want. Yeah it means that we have a kind of room so it's a place where students can take exams written at the computer at time suitable for them. So the process requires that the examiner has created the exam and questions in the electronic exam system. The exam situations are automatically recorded for exam supervising and exam rooms adds the flexibility of studies not whenever so they have to travel to campus but now we are working and developing the so-called I don't know how to translate it but kind of students can use their own laptops and they can take an exam with their own laptop at home or in our exam rooms in campus. So the flexibility comes from that they can just schedule the individual exam time so that's how it goes. Thank you and last but not least here's Matt to tell us what's been going on down under. Hi everybody I'm Matthew I'm from Sydney Australia I'm currently working at Macquarie University which is a mid-sized university of about 40,000 students in the northwestern suburbs of Sydney. I don't live in the northwest but that's another thing. The pandemic I guess the shutdown the lockdown hit our session in was our first session number one in 2020. We were halfway through so there wasn't really any time to plan anything. So essentially the solution was we stuck everything in our VLE our Moodle system. All the exams were digitized in the sense they were either quizzes or there were word documents and in some cases the students would hand write their math problems and then take a photo of it and then upload it to the Moodle. They were a mix of they were mostly 100% online but it was all no invigilation, no supervision, no proctoring. You know it was an emergency situation at that time. There was probably around 500 different examinations that happened over their exam period in the first session and then after the first session happened that was in when was that June-ish in Australia at that time and then in about July the coronavirus situation in Australia really decreased a lot. So in the second half of 2020 and the first half of 2021 we started coming back to have some on-campus teaching again but a lot of the exams were still done online so in those two in the second half of 2020 and the early part of 2021 there was a mix of on-campus exams and a mix of online exams. However we're also looking forward to thinking about okay how come we do online invigilation as well online proctoring as they say we piloted some online proctoring service providers. We found that were not very satisfactory but we also piloted using Zoom as an invigilation technique with staffing people hired and trained by the university by our central exams office. So they provided some supervision. They observed the groups of students through Zoom. That was a really small-scale pilot in the second part of 2020. Then in 2021 we increased it a little bit more. In this case mostly we were catering to the students who could not attend the on-campus exam. So if the unit coordinator for that subject has said okay my examination is going to be on campus therefore it is a supervised examination but of course we have students who cannot come because there was travel restrictions and other things that were happening. In those cases we allocated those students to do a Zoom exam and then in this current semester we're in now we're in now in the second part of our academic year. So for the session two in 2021 coming up in early November we are now scheduling about 23,000 sittings for online invigilated or online proctored Zoom exams that we will staff ourselves. We also have exams that are non not invigilated or not supervised online exams and there's a small number of on-campus exams because we're crossing our fingers and hoping we can still do those. But there's also been a strategy by the university to try and diversify away from examinations so the unseen timed exam to try and get teachers to do other types of assessments such as mini projects or time limited assignments I would call them mini projects. There's also a very small number of online oral assessments like Viva exams or Viva assessments or interactive assessments are again done through Zoom but they're a small number at this time. So I think what we're hoping for moving forward is more diversity of final assessments so more alternative assessments but also improving our internal capacity to do the Zoom based examinations as well. So that's I think where we're at at my university at this time. Thanks Matt and we'll start with you with a second question which really follows on about online proctoring. Now what do we know about online proctoring? A lot of this is outsourced to companies. What's your experience and opinion about this sort of way of working going forward? Yeah so as I did mention we did pilot two different methods of online proctoring let's just call it proctoring because I know that's a familiar word to many people. In Australia we tend to call it exam supervision or exam invigilation but I'll use the word proctoring. So we piloted one company who's well known in the industry which will remain nameless that provided live humans and another one that provided an automated approach where there wasn't a live human but it was just face tracking and screen recording. Our pilots found that they were unsatisfactory I might say. Let's just say they were quite unsatisfactory. We found that things like technical issues were difficult to solve when things went wrong it tended to result in the exam session being abandoned or failed for that student and it was really quite difficult to figure out what went wrong so that we could then fix the problem. So that was an issue. The service providers would allocate a person on the fly so we couldn't train those people. We couldn't talk to those people to discuss what kind of examinations we want. We had to write the specification down and we found that was a very imprecise and messy process. They weren't plugged into our problem solving IT support, student support, counselling services etc that we make available to students during the exam periods and none of that was connected or joined up. I would say that those kinds of online remote proctoring companies, three types of torture for students is what I referred to them as in my report. I said three types of torture. Let's not do that again. However, what we did find though is a sort of a kind of method of doing it with Zoom because we employed the people to do it ourselves. They were trained by our central exams office who were very good at this process. They were good at running examinations and they then upskilled themselves in the technology and I think that was a good solution for us at that time. I'm not saying that proctoring services can't work in certain circumstances. There are universities that do use them and seem to be happy with them but it certainly did not work for us and it didn't work for the flexibility that we wanted. The proctoring company we tried, they wanted everything to be a Moodle quiz and nothing else whereas we wanted different kinds of examinations to happen. So a pen on paper exam, a word document exam, a quiz exam and a mix of those things. Sometimes we wanted the students to use a spreadsheet application for example to solve a problem. We wanted students to be able to draw diagrams and then upload the diagrams. They wouldn't allow that but within our in-house method when we staffed it we set the rules and it seemed to work quite flexibly for that point of view. So I guess my sort of telltale thing is if you want it done properly do it yourself. Thanks Matt. Over to you Tarja. Anything to add to that? You're muted Tarja. Sorry okay when we are talking about online proctoring I guess we can talk about our experiences with Tesla and some of you are probably familiar with that but if I shortly try to describe we had a European project funded by Horizon 2020 and we developed or actually we for example in our university we tested the system that included it was it was developed for the e-authentication and it included the students face and voice recognition keystroke dynamics and included prology orism checker and forensic analysis was also made about students exams and assessments and we have of course it was the project where the system was developed but it's easy to say that system must be very stable and reliable which that system wasn't in that time when we we piloted with it and so we also made some research about the the experiences of students and teachers and and my team me and my team was we our team was we had a focus on usability and accessibility of the system so it's very very easy to say that the system system we are using or developing have to sorry sorry I have to be accessible because as we know we have about 10 percent of our students have some kind of disabilities or some kind of situational situational factors infecting they they are affecting they learning and and performance so what we learned during that tesla project I would like to underline that we we need the systems that diversity of students can use and we tested the system like with the with the students using some assistive technology for example screen readers and and as somebody already mentioned they found some accessibility issues with the the online proctoring so yeah I think that's that's all so far thank you Tarja and over to you Marco yes as reported since we decided against the implementation of proctoring system I have much less practical experience compared to mad attire nonetheless from from first tests of such a system I was found it pretty appealing that you could integrate a proctoring system with existing learning management system so the ease of of use of the system we looked at seemed to be quite high and nonetheless I was very surprised by the yeah by the discourse about proctoring as a whole during the pandemic because I think it was a very black and white discussion it seemed very much focused on like lecturers wouldn't trust their students and I don't know and wouldn't care for students I don't know I I thought that's a very biased impression that you got from following the discourse on proctoring because I see many people taking care for students and needs and special situations and I had the feeling that the proctoring topic was mostly based on this lawsuit happening right now and not so much on the potentials of proctoring and furthermore I was like I said already I was surprised that open universities are keeping study centers just for authentication purposes and proctoring could offer solutions for that without all the other services but in fact many higher education institutions didn't use any authentication methods in this specific time so I'm I don't know I'm I found proctoring an interesting idea especially for the purpose for authentication not so much for screening the room looking for cheating and so on and I was surprised that this topic has been in some my education institutions nearly untouched thank you thanks Marco so Allison yes very interesting points that have been raised by colleagues and so one thing I'd like to focus on is this change of relationship that that proctoring brings about it's not that there there hasn't been proctoring in the past but that proctoring has generally in traditional exams been invigilators in the same room checking the identity of students making sure that they're doing the exam as as as they usually are and so what we've moved to is um identification of that process so the change in relationship between universities and students is profound from that perspective at UCL we didn't really use proctoring because in first year we used Moodle as the platform where we were managing the online exams but you know going forward working with Uniwise they're looking at different proctoring systems I would say that I'm heartened by the open response that they're taking they recognize the difficulties that that there are with some forms of proctoring I would say that proctoring is not online proctoring is not completely new phenomenon I mean in the gig economy for example there are platforms that check every 10 minutes or so whether or not the person that they're paying to do a job is actually doing the job so they use the camera monitoring every 10 minutes they check the keystrokes and so on so there are other industries where proctoring is is being undertaken but I guess that it does put both academics and students in I would say in a difficult position where the relationships between them have changed thank you Alison I'm going to come back to you now with another set of questions because we've discussed how we had to move online quickly with our teaching and then of course we've got exams students need to graduate how do we check all these exams but also before the lockdown we were experienced in companies which we often call essay mills that would students could pay to the company for them to write their papers maybe an open book exam even and what with the change in circumstances students not coming to the university how do we think contract cheating is developing and controversially I've said is it going to be the next normal I think that's a really important question Denise so thanks for asking I I like to think that the majority of students who come to university are still there to learn they're there so so that they can go through a life-changing experience however obviously with different technologies being developed and the number of students being scaled up there are more opportunities and and potentially more possibility for some students to to you know pay for essay mills to to do their assessments for them so we do have to to think about ever more sophisticated ways of finding out whether the students are are going through assessments as we expect or not I mean I actually think it calls for a fundamental change to how we think about exams if it's possible to pay someone else to complete an assessment for you then I think we really do have to think about how we assess our students and unfortunately finding ways where we can assess at scale but not allow essay mills companies to come and you know complete an assessment for you is very difficult so in short I'm very optimistic about the future I think students are coming to university to learn so I don't think it's going to be the norm but I'd like to flag that we need to find more sophisticated ways of finding out whether or not there is cheating I was actually in a viva for PhD last year where the student had paid someone to to do the work and for me as an academic that was quite shocking and it was very hard work by the registry team to to take up that complaint to go through an investigation to prove it and and therefore to fail the student that that was was very shocking and it's not fair for the students who have actually done their own exams so I'll let someone else answer this question it's very important thanks Alison Marco I have to admit I have very little insight and experience with contract cheating I have never actually faced such a case in my career so far at least not knowingly on the other hand I would concur with Alison that it's also the result of a system that is thinking of allowing only students to succeed if we have tested everything that we have thought so before so I studied before the Bologna reform in which not I mean I could just go to a lecture and I mean I didn't need to be tested for that to get some credit points so I think in fact the more stress we put on exams for the whole system the more likely it would be that students would need to use these kind of solutions to succeed because I mean the complexity of students lives is also has also very much increased in terms of funding their situation finding housing in big cities which are very expensive finding jobs and so on so I think if we talk about contract cheating we need to look at that more on a systemic level and think about if we don't put students into looking for that solutions by stressing examination for every single module that we are teaching so sometimes I wish I would could go back with my students to the time before Bologna where I just could offer them something and they could make use of that so a little bit more freedom and a little bit more general approach to knowledge acquisition and learning and not testing everything that we taught them thank you Mark and Tarja thank you I don't have too much to add for that but but I'm also like Alison said very optimistic optimistic with the future so he sometimes said that we will finish Finland is a trust-based society and we might sound quite naive sometimes because we trust our students so much but there is also one point that because we are quite a small how you say language group or or there isn't so it gives a kind of protection for for us because everything is not available for our students they have to use Finnish language so the markets are so much smaller there thanks thanks Tarja and Matt now because Matt there's been a lot of work going on in Australia with fields group about contract cheating and SNLs so perhaps you'd like to let us know about that interest and work too yeah so Professor Phil Dawson at Deakin University a former colleague of mine from Webloom we're both at Monash also has has done a lot of research around in digital cheating he's actually got a book out on each cheating but I also refer to the work of the now former professor Tracy Breitag who done a lot of research around contract cheating and the factors that you know the column hygiene factors if you will that that maybe perhaps lead students down that path the kinds of assessment designs that that can be more or less cheatable etc in Australia I think it is it is a concern for our system it's been in the media the government has introduced laws to outlaw commercial cheating service providers in fact the first one that the law is relatively new the first website was recently just the legislation had been enacted and the the the national regulation body for the tertiary education sector has just banned one website so far with a list of about 3000 more websites to be similarly banned on a national basis but I don't think they've heard of VPNs so we'll see how that goes I yeah look I think contract cheating has always been there it's not a new phenomenon however I guess you know like e-commerce like anything has turned a local problem into a global service providing problem you know so that particularly because English is the medium instruction for us it's probably the one of the most common languages in higher education there are many service providers all around the world wanting to sell their services to students and we have a diversity of students in our system so I think there are vulnerabilities there higher education institutions are aware of them you know to tackle this problem it's a wicked complex problem and you need a wicked complex solution to address it you can't there is not there's no one solution to this problem a law is not going to stop it online proctoring is not going to stop it you know assessment design changes alone are not going to stop it you need a holistic approach to addressing this issue so you know student education campaigns you know making sure your students have got the skills to succeed providing you know health and financial support to support students so they don't make silly decisions as well as of course you know detection and punishment and all of those sorts of things you need a holistic approach and I think the university sector in Australia is certainly realizing that and they're doing better and better at implementing academic integrity frameworks and policies and support mechanisms to support the students to succeed so that they don't have to make those bad decisions and when they do to try and have an educated approach I think that's that's the sort of future I wouldn't say that contract cheating per se is the future it's like everything we have to adapt and our university systems and our educators and our administrators and support staff have to adapt to support our students and I think we also try to have the best view with the say that most students don't cheat it's a minority problem but we can't be naive and blind to it either so that would be my my take on it thank you Matt and I've one last question before we open the floor and I'd like you to just answer in a couple of sentences very quickly um what do you think is the future of the exam now so we'll start with Matt we'll go reversely okay what do I think is the future of exam versus what do I hope is the future of the exam okay let me start with what hope is not the future of exams I don't want to see massive online multiple choice quizzing systems I think that would be a mistake um you know there's that's there's that triad of authenticity integrity and scalability what I've observed is a lot of the commercial service providers have given us scalability tried to give us integrity but have forgotten about that authentic assessment side of the equation what I would like to be able to see is approaches that facilitate authentic genuine interesting assessments that the students are more engaged with that doesn't necessarily mean an exam but it could be an exam like experience um so that's what I hope the future of online exams are much more authentic richer diverse forms of assessment that we can have students do that would be my take so far thanks so Tarzan oh let's see how it goes in the future um because I'm interested in diversity of students and equal possibilities to study I hope we could more often look at the assessment from universal design for learning perspective so so meaning that there are many ways to show what you have learned and there are of course differences between the discipline discipline is how to how to show it so I think that in the future the quite a traditional exam will be one of the many options so we'll get more solutions instruments and tools to implement exams online the exam itself is not good or bad how the exam is implemented makes the difference I think so thanks thank you Marco so I have very very um yeah low wishes for the future I have to say because one of the futures of the exam is hope hopefully without paper one of the side effects that I have seen at our institution is that actually we have exams now in a digital format and that already brings freeze up some time for lecturers in the administration of the exams and I hope that that this will also stay regarding exam innovation I would wish that we would see much more formative assessment opportunities for students throughout their studies and then a type of more complex authentic problem solving where it doesn't even make sense to cheat or I mean and where you can use all means that you want to use so that would be my ideal future but similar like Matt I think this is will be heavily influenced by economic and scalability limits to this kind of approaches especially for individual lecturers they will not buy in if it will take more time for them um yeah timing and resources always a problem thanks Marco and Allison you've got the last word yes well well I would agree with what what colleagues have been saying um but I think it's particularly important to um to focus on authenticity because as Matt said that tends to be the the the part of the the three things that he emphasized that gets overlooked and I'll tell you why I think authenticity is really important years ago I did a lot of work with Shell who measure timed competence for new graduates so that's the time their new graduate can be in a job and can work competently without a lot of supervision and that time is getting bigger and bigger partly because the jobs are becoming more complex but also because something was happening in university education where students were coming out with even better grades but were less able to move into a job now I'm not arguing that education is only about um getting a job and about where it is much more than that's about our whole lives but the more authentic the assessment then the more we're focusing on assessment as learning and enabling students to be able to learn through the assessments that they have thank you Allison we have some questions coming through and I'd like to bring those to the attention of the panel and then there might be some more questions from the floor but Matt the question was about what training did you give your people when they were going online with exams online exams yeah so I humbly say that I wrote the initial versions of the training materials for our staff and our students I developed some quite a lot of guides but I based it on the good work of other institutions such at University of Melbourne they'd also started off doing zoom exams as well I then passed that to our excellent people in the central exams team who are the real experts I should say at doing the exams so they took my guidance notes if you will for how I think the e-exams or the online exams in zoom can go ahead and then they conducted training sessions with the staff that they had had a hired or allocated to do the online invigilation tasks and at the same time in the in the two weeks prior to the exam period we also run zero stakes practice runs for the students so the students can get familiar with how it all just works they can come along to a practice session and experience a mock exam in zoom so that they know they can test their equipment they can test their internet they can test the procedures they can ask questions and it also acts as training for our our new invigilators as well so they can get some practice in remembering it was really early days for us so a lot of the people we're having doing this job are doing it for the first time so the training sessions for the students are also the training sessions for the staff but you know it's a bit of a train the trainer thing happening in the exams office so that's how we sort of went forward with that I'm happy to share the the guides and things that we've developed for staff and students around this as well if people are interested I did stick my email address into the chat oh thank you Matt because the people could you share that also Taja and people are struggling to get through to the Tesla project and would you be able to put a link into the chat and maybe perhaps you could explain about the tools some of the tools because um Alison mentioned you know keystroke dynamics in the gig economy and that's one of the tools it might be helpful for people to understand it's not just a single tool for a proctoring system sorry I'm not sure if I if I understood the whole question so you wanted me to explain a little bit about Tesla did I did I get it right yeah yeah so I'm sorry I I guess the link doesn't work anymore so so uh yeah Tesla is a kind of system that how how I say it the those uh recognize face recognition voice recognition keystroke dynamics and plagiarism checker uh it's possible to integrate those systems together and it's it's works for example you can integrate the Tesla system for example to the Moodle so it just goes behind the the how you say Moodle and and there are many ways to to use it so I'm very looking forward how the finally it will go go with us because because uh I'm not sure how how the instruments works right now so but but it's it means that teacher for example can choose between different uh instrument or tool so for I think it's very important because for example when we tested it with our students using sign language they didn't feel very comfortable to use the voice for example for the recognition and then they could choose the the face recognition or keystroke dynamics it's something something all those those tools are built how you say inside but you can you can also add for example uh Urkund plagiarism tool that instrument or we tested it with the Urkund and maybe you can turn it in also goes together I I guess what else to say and it it's not not how you say recording all the the session or all the the exams it just tested the the occasionally what is going on for example with the video and and it's it's something that the the how we say students were very worried about the personal data who can see it or who cannot and so on so so it's a system that that uh if everything is okay if the you have a sorry this is very very uh difficult to explain so you have to first of all you have a kind of enrollment you give a sample of your voice and your face and your keystroke dynamics and when it comes to the the assignment or exam it compares the the original data enrollment data to the the one data the system is collecting during the the exam does it make any sense so so uh and and if if there is nothing wrong or nothing to worry about nobody sees your face or or sees your what you are doing but in case of some issues teacher can see the face and so on so and it's it's uh the the personal data is locally how you say locally shared or how you say so it's it's not going to to out of your institution that's the way I suppose it will work and and you can use it as a as a part of the summative assessment or formative assessment or so on thank you Taja so using the Tesla system you've got more than one tool that you can use in different circumstances also we had a group of lawyers as well from the more working on this project to make sure that where the data was stored that the users were protected with their data and also they were um implemental in the um usage that you sign that you want to use the system and that everyone would be protected in this way so a lot of work went into that now before we close because we're very near to closing it's come it's upon me before I thank everyone to to sort of give a summary of of where I think we are from our learned speakers and their opinions so what can we say we're all forced to move online and to do examinations online and um you know edtech is really in the forefront it's come together been forced on everyone which is really interesting and how people responded to that of course has to meet the resources that were available at the time in the university and also to we haven't actually mentioned this explicitly but in the answers you could hear today how the student is first student is at the center we're not trying to do anything to um upset any sort of examination system where you know what they do and how they can tell us about what they know is actually done in a fair way so what about online proctoring well as Matt said do it yourself don't go outside but there's a lot to learn about online proctoring how we can do this and the best way of doing it and the test to learn is probably a good way forward before you plunge into you know whole scale proctoring but um is contract cheating the next normal well we need to authenticate don't we and we need to have this trust and most students don't cheat we know that but there is a question there that we've hinted at in an answer and that is about the type of assessments we give our students and Allison hits upon you know are our students ready for work what is authentic assessment and we didn't talk about assessment literacy that is in the literature now you know and so what we're saying there is do the students really understand what knowledge they should be demonstrating for the questions we're asking them do they really understand the questions and and what's required of them and I think at the heart of all our discussion which is so encouraging is you know as assessment specialists we're here to help the student demonstrate what they have learned and what they do know and that is our asset research agenda going forward I think I would like to thank all our speakers Tarja, Matt, Allison and Marco for a very interesting discussion and how they built upon each other's answers and had their own take on these very important questions so thank you to our speakers and perhaps we can I'm not sure we can take our microphones off but I think we can give a clap thank you very much but thank you to you in the audience for your interesting questions and I'm sure we're going to be talking about this for a long time thank you to you all and I wish you all a very good evening and goodbye