 So again, a huge welcome to everybody here today and for this our second webinar on what is the future for the assessment of learning through final examinations. As you know, at our first webinar, we shared perspectives on the final examination as an assessment method. Today we're looking at alternatives and broadening out the conversation in a little bit. I'm delighted that we have some wonderful speakers here with you, which our chair will be introducing you to shortly. But before I hand over to my chair, I'd just like to say that we're going to the format for today's we're going to look at some of the key issues that emerge of the insights we got from our fresh webinar. And we're going to particularly explore alternatives to the final examination using case studies from what's happening across the sector. And proctoring came up in the first webinar where so we're going to have a little focus on that. And in the second half of this particular webinar, and the talks will be lightning talks. So the guest speakers know that they have five minutes and Terry's Turkey will emerge after five minutes. If the if anybody talks over the the time that they've been allocated, and we're hoping to have a discussion both with the panel and with yourselves at the end. It's a packed schedule. So it gives me great pleasure to quickly hand over to our chair for the day, Dr. Nile Siri, who is the director of the Technology University Project for the ITLIT consortium. And Nile is also a national forum board member. Nile is over to you. Sorry, I needed to unmute. Terry, thanks very much for the introduction and welcome everybody to what promises to be a very informative and interesting insight into the assessment discussion, particularly thanks to Terry and her team for organizing this and bringing to the fore this discussion, which is both timely and important. I suppose I'd like to start with some brief housekeeping in terms of the webinar. So the webinar will be recorded again. The purpose of that is to support the National Forum Insights Publications and Assessment. We want this to be as interactive as possible, so we welcome comments in the chat box throughout the entire seminar. We pick those up as we go. And in addition to that, in the chat box, we will post a link to a Google Docs, which will allow us to expand and elaborate on any points of the chat that we want to discuss over the course of the seminar and beyond, in fact. We also invite you to participate in a number of polls that we run over the course of the event today. So I suppose without delaying anything, we might start with the first poll, which is really raises a couple of questions. One question that we want is a really quick response to, which is we'd like to get a sense of your discipline area. And that's important again, just to get a sense of the breadth and perspectives on activity. People could fill that out really quickly, that would be great. The second thing is, on the last seminar, we had several speakers, but our final speaker, Darryl Ryder, from the Association of Higher Education Access and Disability really challenged us with a question regarding the disability of our assessment system. And I just, if we can look at the second question, which is what is your sense or what do you think about that question in relation to the disability of our assessment system? And if we can get a quick sense of that, that will be fantastic too. And I will hand over to Claire for the next session, which is the speakers' perspectives on webinar one. Thank you, Claire. Thank you very much, Nyle. I'm screen sharing here, so I'm just going to move the poll slightly. Hopefully it's not obscuring anything on the slide here. And I'll leave it there. People are still responding to that. We thought it might be useful just to revisit briefly some of the perspectives shared in the first webinar. Many of you may have been with us for that session and the recordings also available, if people would like to go back to that from the teachingandlearning.ie website. Speaker perspectives, first of all, really were grouped into three themes when we listened back and we reviewed this again. Partnership, balance and the notion of ladders and walls or barriers and how we might take down some of those barriers. So, first of all, in terms of partnership, a view that the review and redesign of assessment would be something that we'd need to undertake in a team context, working across programmes rather than in isolation or just with particular individuals and their modules. That dialogue would be essential in this process. And that needed to be not only with colleagues, but with students and also that small changes could have significant effects. Whereas if we try to put large scale change in front of us, we can find that very overwhelming and our students find that overwhelming as well. So there was a consensus amongst the speakers that those small changes might be more effective. But all that's based on an assumption of change and really the question on any haypost for us at the beginning of the webinar was whether we do need to consider a change. Are we going to have a system reset back to how things were before the pandemic? Or are we going to build what she called a better normal? What kind of assessment blend can we construct? Maybe we're assessing too much. And again, speakers referred to that. We are over assessing, most likely. And if we are over assessing, can we construct something different? And can we choose where to include exams rather than assuming an exam always has to be there? We probably need then discussion with professional bodies and other accreditation agencies and other stakeholders who, again, have been very accustomed to using exams as part of the admission to those professional areas of practice and expertise. The question was asked in what other settings in the 21st century are we asked to sit and write with pen and paper for a long time under very pressurized conditions and also whether this is really integral and as part of assessment integrity, because we don't tend to include phrases like write with pen and paper for a long time as part of any of our learning outcomes. So that raises a real challenge to us in terms of our curriculum and assessment design and aligned to that. And whether we're really designing universally for all our students and giving choice or whether there are barriers there to how they're being assessed. It's said that changing assessment is going to take time and effort and resources. But one of the points that our rider made was the accommodations we have to put in place around exams are also taking substantial time, effort and resources, so perhaps that should be channeled towards something else. So those were some of our speaker perspectives and I'm going to hand to Catherine to hear a little bit more about participants' perspectives from webinar one. Many thanks, Claire. Is my microphone OK now? Had a little issue earlier. In addition to, you know, the diversity of speaker perspectives, we invited everyone who participated in the last webinar to share their thoughts in the Google Doc and also in the chat. And this is these are some of the themes that arose from the participants in that webinar. And the first was what's listed here, things that were mentioned by, you know, more than one participant, things that kind of rose as key themes. And in the area of exams, there was a feeling that final exams will still have a place, but not a priority place, not as priority places perhaps they have now. Close book final exams, as mentioned by the speakers, are were seen as an artificial unrealistic way of assessing. Remote proctoring came up quite a bit and the notion of the fact that we're outsourcing something so integral to higher education to third parties, you know, outside technology companies raises issues around data and authority. No matter what kind of proctoring we might implement, these things can be subverted and attempt to counter subversion lead to privacy infringements. And then a number of people pointed out additional hurdles for students with disabilities and other marginalized students. In the area of continuous informative assessment, there was a wealth of contributions, links, all kinds of things in the doc, but really these focused on the fact that we can assess we can assess the depth of learning, critical thinking, the application of knowledge and skills and traces of learning over time, particularly through digital assessment and continuous informative assessment enable inclusive assessment and inclusive feedback. So around this whole umbrella of reassessing assessment. We're talking about, you know, how might we move to more continuous informative assessments, recognizing that this has significant resource issues, recognizing that we have to consider the total weight of continuous assessment and avoid over assessment. There is a perceived challenge of getting external examiners and professional bodies and regulators on board. Some of that work is in progress, some is yet to be done. And obviously all of this work requires program level consideration and design. I will point you once again that we invite your sharing just as this rich feedback shows, we invite your sharing today as well. There are a couple of questions in the Google doc. Add a new heading if you'd like to add information about something else. The link is in the chat and we invite you to contribute throughout the webinar. And I'd like to hand back to the chair now. Great, Claire and Catherine, thank you very much for those insights from the previous activity, previous seminar. It gives me great pleasure now to introduce our lightning speakers for this part of the session. And they are Dr. Brendan O'Connell, assistant professor in the School of English and Trinity College Dublin, you're very welcome. Irene Hayden, Lecturer Department of Building and Civil Engineering, Colway Mayo Institute of Technology. Again, you're very welcome. Dr. Fiona O'Reardon, academic developer teaching enhancement unit at Dublin City University and Rob Lowney, learning technologist teaching enhancement unit at Dublin City University. You are all very welcome. And thank you for sharing with us your expertise in this session. So I'll hand it over initially to Dr. Brendan O'Connell. Thank you very much. Thank you and thank you all for inviting me. I'm delighted to be with you today. Just to let you know a little bit about me at the moment. I teach in the School of English at Trinity. I am currently the director of undergraduate studies in undergraduate teaching and learning in the school. And I've also recently taken on the role, very recently taken on the role of the school champion for Trinity's inclusive design project, inclusive curriculum design project, which is ongoing at the moment. So I'm delighted to just share a few thoughts with you from, I suppose, my perspective and drawing on some of the things that we've thought about it over the last year or so. The first thing I suppose has already been mentioned and which came up in the last kind of webinar. Is this question that I suppose in many disciplines within the humanities, including, I suppose, languages, teaching, the teaching of languages, there's going to be a continued need for examination in various contexts. But obviously we're all here to think about how we can assess differently. I suppose during the first lockdown and subsequent lockdowns, our own assessment strategy was really simply to convert our planned closed book to our exams, to open book assignments. And I suppose the first thing I would comment on is really the relative ease with which that happened in, I'm not saying that happened in every discipline with relative ease, but in our discipline, it was not actually tremendously complicated to do that. And in fact, I think that raised a lot of questions about why exactly we were using exams in the first place and whether we should continue to do that in the future. Many of the exam questions, when we looked at them, they simply are, in our discipline, essay-style questions that lent themselves readily to an open book context. And certainly we were not overly concerned about the students having the books in front of them as they wrote. In fact, many colleagues felt it was perfectly welcome to allow students to have this opportunity to reflect. I suppose I want to comment briefly on one of the issues that came up in the last webinar and that I know a number of colleagues are concerned about, which is questions about academic integrity. But I suppose in our schooling and in our experience, this was really no more of a problem when we converted to open book examinations and assessments than really it would be in, for example, those modules where the module is assessed by essay or other forms of continuous assessment. And we also found overall that the quality of responses was actually broadly in line with what we expected in a normal year and that our results at the end of the year were really around the same or maybe slightly higher than they had been in previous years. I guess one thing I do want to highlight from our experience in the humanities is that there was a very broad range of approaches in terms of how long should be scheduled for an open book exam. And initially, many schools in our faculty offered, say, 24-hour open book exams or 48-hour exams. In some cases, it was actually more than that. In some cases, it was less than that or like 12 hours. And I suppose there are different ways of thinking about that. But one of the unexpected advantages of the longer time frames was in relation to students who were registered with the disability services. One of the things I found from engaging with a lot of these students was that in many cases, even when extra time was allowed and allocated for such students, many of them didn't actually feel the need to avail of it. And I think that's one of the things I found that, well, this 24, 48 hours or whatever it was, was sufficient for them to do what they wanted to do and they were happy to just do that. And I think, again, that raises the questions about the point that was raised about what kinds of disabilities are present in our assessments. But I do want to point out as well, however, that there are other problems presented by open book assessments, and I think that's one of the reasons that I think we have to think about that. And I think that's one of the reasons that I think we have to think about the long durations in that sometimes students get burnt out and overwhelmed when you give them a long time to do something that previously they would have expected to do in a couple of hours. And we certainly had reports of students spending all their time for those few days just working on assignments and getting quite worked up about it. And I think that's one of the reasons that I think we have to think about is the importance of literacy and the importance of embedding within our disciplines, a kind of culture in which students understand why they're being assessed, that they don't feel that assessment is something that is being done to them and recognize that it's part of the learning process. Certainly in our school and some other schools, colleagues have been experimenting with different ways of doing this, including under frameworks of assessment or part of the assessment for the modules themselves. And while we don't have firm quantitative data on this yet, anecdotal evidence does suggest that students do find this quite a fair and reasonable way to approach it. Now, I'm running out of time, so I'll just say in conclusion there are obviously many other ways of dealing with this and people have looked at things like reflective journals, blog posts, presentations, podcasts, group work, etc. Just to echo that I suppose as was said at the start there, that this process does not happen quickly. It's important to have a kind of conversation with all the stakeholders, including colleagues and students, but also the teaching and learning structures within our units and our administrative staff. So these changes are not going to happen quickly. They won't all be in place by the start of the new academic year, but now is the perfect time to get information going. Thank you. Brendan, thank you very much for that. I'll hand straight over to Irene. Thanks very much. I'm just going to share a PowerPoint if you could tell me if you're able to see it. I should share now. Now you're able to see that. Yep, good sharing. Perfect. I'm going to present your mode instead of this. Okay, so I'll get started then. Thank you very much for the invitation to come and talk and I will keep it brief to the five minutes as best I can. I wanted to just present a small little case study if you like of my experience through the COVID pandemic. We were innovated of an accreditation process as it started with professional bodies. So we had to revisit quite quickly and think on our future in terms of how we assessed throughout the entire program. So just to give you an idea to background in terms of timelines to begin that process before everything went hardly wrong, nationally internationally, we had actually received accreditation principle from a professional body coach, the Chartered Institute of Architecture and Technologists to have been awarded that in September 2019. So at that point our program has been reviewed quite thoroughly and it is a four-year program. It doesn't work currently and there was a mixed method of assessments. Some modules had final exams some were mixed modules with exam CA or exam project work. Some of the core modules were already 100% continuous assessment and a few modules had practicers or labs because of the nature of the discipline and final exams combined with them. As we all know the first lockdown hit in March 2020 and we all had to redress things on our feet straight away. So overnight assessment was reviewed by every individual lecturer and by the program board as well. And in terms of kind of default back to certain things that we were familiar with as well as exams such as we do assessment, project-based learning. Looking at purpose for use of technology and health learning, for some staff it has to be the first time that they have had to look at MCQs for what they want to move the platform can do and kind of think on a patient terms of assessment. We weren't facility with practising for a lot of different reasons by May 2020. So most lecturers actually published their own exams, we timed them and they were done on the time schedule and students were taught how to use office lens, job load, hand-driven things. Now because we got more sophisticated I suppose but now I would say the lecturers who needed to do an exam continued to do an exam but roughly maybe 36% of the previous exams had been done as an alternative at that point, so by April and May 2020 across the program some changes had happened. When I look at my own modules, the ones that I was leading before, I reviewed them myself at that point, so early March, mid-March and March we of course addressed the program learning outcomes to double check that we were still okay but I reviewed my own assessments that I checked, what did I feel that they were authentic were they good for purpose and I thought that they were. Do they need the module learning outcomes that are very important to check when you are based in that predicament and what other assessments do I need to add in to work with a missing mother to the learning outcomes and then I did an exercise as everyone did where I mapped continuous assessment of project work against the module learning outcome and I have to say I put the hand up, I did feel that definitely in some modules may have been over assessing uniformly and you know it was quite eye-opening really that process to do that and then to report back and we did check with external examiners to check with management to make sure that our changes were agreeable and we run the system and it seemed to be a quite common theme and I got a lot of people talking across the entire institute and then to go back to my timeline we ended up obviously kind of blended delivery up until December 2020 for that semester back online so there was this different shift away from some final exams from that point on and the decisions were made early on actually to do that we had a virtual accreditation visit in February 2021 the same accreditation panel, the board from the CIET reviewed everything, they see the students work through the full year from including the pandemic and then that semester from December to September to December 2020 and they were actually on board with us, they had no issues with any of the assessment changes that module leaders put forward and presented, it wasn't an issue for them at all and that's a testament really to the hard-working efforts of every single person involved in that particular program so it does put the question out there moving forward, we're going through a question with you now currently actually so we're going to all have to look at this and address it again checking again are assessment choices authentic are they fit for what they're supposed to do with the particular modules do they make the module learning outcomes of course very important are they student-centered we have time to take stock of that and reflect on it and I know that a few people have mentioned issues like that and it came up in in the last webinar and are the module learning outcomes over assessed absolutely not, I sincerely hope not so just to conclude I would suggest that perhaps you know we've got to look at and address the learning outcomes to make sure that they're measured, look at multiple assessments versus deeper assessment techniques respect the module leader regarding their assessment choices and we have to remember to use tools and paper so you know sorry for a while reflect your practice evaluation tools, look at peer review exam review, professor body review and then look at small pilots and reevaluate thank you for your time Thank you very much and again I think you've touched on the issue of you know bringing it to a fore and allowing us to really think deeply about our assessment but also identifying the resilience within the system to actually be able to deal with this so thank you very much for that and again we all have evidence of trying to manage technology too so thank you very much for managing that well so if I can invite Fiona and Rob now maybe is that I think you're presenting together so I'll introduce you collectively so maybe over to you first Fiona I'm actually going to pass to Rob first but thank you Niall Thank you Fiona and thank you Niall and thanks everybody I've just popped a little link in the chat if you guys would like to click on it and I'm going to share my screen briefly and just take you quickly through a resource we've developed here in DCU to I suppose demystify some approaches to student partnership in assessment as we know from previous work from the forum and from NSTEP and so on that student partnership is very good for student learning and student engagement and we in particular were interested in looking at well what does student partnership look like in assessment and what are some of the practical approaches that we can get students involved in assessment get them engaged and ultimately have a better experience so we undertook this project and with the help of Ruth Naguolan from Hibernia College we conducted a literature review to kind of see what was out there around practical approaches to student partnership and then we developed this resource for our academics and this resource which you can look at in your own time and further detail maps out student partnership in assessment along a continuum from low level partnership to high level partnership and low level partnership can include very low risk things to involve students in their assessment and give them a sense of agency things like negotiating a submission date giving them choice in the topics for their assessment or the methods or the formats for them to complete their assessments or giving them some sort of input into negotiating elements of the assessment brief and as we move up along the continuum we see students become more engaged engaging in things like self assessment, peer assessment or a combination thereof getting involved in more detailed approaches like co-designing assessments or co-designing marking criteria and so on and some of our academics at DCU took this resource and implemented some approaches along this continuum and we've evaluated it and going back to something Brendan said earlier about believing that student partnership in assessment is a good thing we and certainly our experience at DCU as we know it is a good thing we'll have some case studies out in the next few weeks and I encourage you to take a look at them but generally speaking in the research that we conducted all the students felt more engaged by one or more of these partnership approaches they felt they learned more as a result and both they and the lecturers kept doing this kind of thing into the future I'm going to stop there and hand over to Fiona who's going to just tell you a little bit about one example of how our academics have used student partnership in the past year Thanks very much Rob and it's great to see Ruth here today as well yeah so a lot of the work that we did in the SACRA project led to lots of other work we have a key interest as everyone I think at this point has in academic integrity we had it before and obviously we are now getting great traction people like to hang out with us now which is really cool as a result of the pivot and the evidence and the research you're showing us there's no doubt about it that involving students in any type of assessment is going to promote academic integrity so really core to any alternative assessment and we have one example of community of practice so we're implementing interactive walls as an alternative assessment to exams and it's working out really well we've got two semesters of experience now and lots of data telling us that students believe it's very authentic it's promoting academic integrity so essentially what the interactive oral is it's a two-way conversation based around a role-play scenario so preparing them for professional life or employability and it's a good way of checking in with students it's best used if it's in a scaffolded assessment environment where you've kind of received a report or some piece of work already it's a proper conversation some examples of the authentic conversation one would be a French lecturer her authentic scenario was where she was interviewing her students as part of a radio show and they were to discuss a book they'd read in French how we partnered students in this they co-designed the rubrics in all cases or informed them they used an exemplar that was produced on a sample video recording that was produced by the lecturer and me usually in advance they used the rubric to grade their lecturer and me and say which of us were performing well and not so well and I never performed well because it was never my discipline area but they loved calling their lecturer out and all of this contributed to them having greater sense of assessment shared assessment literacy and agency in the learning and the other thing was where possible we allowed them to negotiate or have some choice around the artifact or the scenario that they wanted to play with their lecturer but it would have to be it would have to reflect some part of the work that they've done already in the module so I think that's it so essentially any involvement from the student has a positive impact in the case our preliminary findings are showing us it's enhancing academic integrity it's helped develop that shared literacy it's engaging students by using authentic assessment that speaks to issues that Brendan and Arlene have already shared thanks for your time excellent thanks very much Fiona and Rob and thank you for sharing as well that schematic of that journey through that partnership with the students it's very very useful we will hopefully have time at the end for a further discussion on all of these agendas and issues that are emerging I suppose I'd like to maybe just acknowledge the fact that you know the initial poll that we had said that 77% of people on the call think that assessment has disabilities so maybe just to acknowledge that and that's the agenda for unpacking these discussions as we go forward I'd like to hand over to Terry now who's going to introduce and focus on the second poll Terry okay thank you very much at the last at the first webinar the issue of proctoring came up so what we thought before we start having a conversation about proctoring we'd just like to get a sense of your own experience of proctoring so could you use the poll to tell us if you've administered a remotely proctored exam if you've actually taken a remotely proctored exam or if you've actually done both and I'll give you a few seconds just to get the answers in as many people as possible to participate we have 104 participants so it would be good to have we have a few more to have their say yes okay so we can see our poll is flawed so anybody that says neither just put it into the chat as you have done so thank you okay so I can tell you now that the majority are probably have not they have not administered nor taken a proctored exam we have some people that have administered it and less people that have actually taken actually done a proctored exam itself and a very very small number that have done both so we can just stop the poll just now yes Terry thank you very much for that and I just picked up a comment in the chat box which captured quite eloquently some of the previous speakers views which is any involvement which students has a positive impact I think that's a really powerful message and something that from my from the very first time I engaged with the national forum the centrality of the student has always been central to every discussion that we've had so it gives me great pleasure to introduce the next session of this seminar and I'd like to introduce John Begley who is the USI class rep for the 4th year school of medicine UCC Diana Jung BA on her psychology class rep year two and peer mentor of the arts program in the business in DBS SU and vice president for education in Dublin business school Dr Adrian Fleming department of science school of science and computing in TU Dublin and Dr Lisa Padden who is project lead university access and lifelong learning in UCD and all of our speakers today are going to capture their experience as a student and on this whole idea of proctored exams and actually doing proctored exams and sharing with us some of their perspectives from having that experience so we will start with John who's going to join us by video link Terry and I think you're going to maybe introduce that so John and Diana will be the first speakers on this so thank you very much John at last unfortunately was not able to come in person but he did submit a video he had gone out to his class who have just completed a proctored examination to get their feedback on the process and he's captured their feedback in a very short video which I'll just share sorry just one moment today I hope you're enjoying this event my name is John Begley and I'm a tertiary medical student who's been asked to share the experiences of his class with the Examiners Online Proctoring System firstly I'd just like to apologise for not making the event live itself due to work commitment this is why I took the time to actually record this video and try and give you all a bit of an insight into mine and my class's general experiences with Examiners I hope you enjoy and if you're into any specific feedback I'd just like to give a brief summary of the experience that my class had with Examiners Online Proctoring System so basically we had a trial exam back in the late March with the Examiners Online Proctoring System unfortunately the system was non-functional for most of our class as a result of this a lot of the feedback that was given by my classmate was negative but we have to consider the fact that a lot of the negative feedback was generated towards the actual system itself as opposed to the concept of Proctored Exams which is something that's important to keep in mind when watching this video so basically we decided that the best way to get feedback from a class was to ask them for rest of these simple questions about their Examiners experience so the first question we asked them was how did their general experience with Examiners go for them and mixed range of feedback on this most of which was generally quite negative but we weren't surprised because a lot of the issues did come back to entrance and exiting issues in terms of access to the exam and the site itself a lot of our students also felt that the levels of support that were provided by the site itself just weren't up to standard even though we did receive adequate support from our medical school a lot of our students also felt that the site itself was too invasive and it collected too much personal information the cameras, microphones and screen sharing put a lot of people off and although some people did say that the site did seem relatively well put together the issues arose because the site was relatively non-functional another large area of concern that came out in all of the different domains of feedback were the issue of faults being accused of cheating and there's around that due to actions such as head movement eye movements that kind of thing so a lot of our students felt the site was generally just a bit too invasive and we're worried that it would alter some of the settings on their computers so then we decided to ask our class about the more general topic of proctored exams and how they'd feel about them going forward in future and unlike the feedback about examity itself the feedback on this topic was a lot more mixed so I concluded from the feedback that a lot of our students felt actually willing to do proctored exams and understanding of proctored exam systems given the current climates if there were adequate systems in place which means that potentially in future proctored exam systems could be introduced as an option for students however some concerns did remain about issues like recording, false accusations of cheating, a software and systems being used some of our students suggested open book exams and other students would have preferred a system where there was an individual invigilator that they knew maybe a lecturer who was actively watching them or a group of students that they were assigned to so then we decided to ask our class if they felt that proctored exam systems affected their performance in any way but as I said earlier it's important to bear in mind that our class unfortunately did not have the best experience with proctored exam systems so the feedback was generally quite negative some of the specific issues that were pointed out were connectivity issues and a lot of issues around false accusations of cheating head movements, that kind of thing a lot of our students also felt that the site was a bit too invasive as I previously mentioned and did not always feel comfortable being recorded while doing an exam they were also wondering about the artificial intelligence algorithms that could call them out for cheating falsely finally we decided to ask our class about how they felt about proctored exams in comparison to their more traditional exam experiences unsurprisingly a lot of our class wanted to go back into an exam hall and felt it was preferable to the proctored exam experience however given my own personal opinion and a lot of our students they did feel somewhat more comfortable seeing their exams at home so it was a place and up and running that students could avail of we have to also remember the fact that for exam hall exams there is a lot of time we see involved that simply isn't there with proctored exams for example having to go in having to get to the venue having to get to your seat and all that level of organisation and stress involved on the actual day of an exam so to conclude personally the best system to be in place will be a combination of proctored exams and in real life exams and our students could have a choice if there was an up and running adequate proctored exam system that they could avail of thanks very much and I hope you enjoyed my feedback I appreciate John's work commitments and he couldn't join us today but I also appreciate the time and effort he has put into sharing with us that he is very active on proctored exams within his class group and the amount of work that he has done to prepare that so thank you very much John if I can hand over to Dr Adrien Fleming Adrien Thanks Nad I'm going to share a very short presentation of just a couple of slides and into transitions already and so thank you for the opportunity to speak to you today about the experience of taking remote exams using proctoring in 2020 and 2021 I completed two different courses where the end of course assessment involved a proctored exam online so I suppose it's been a long time since I was a student in years gone by I never had issues preparing or studying and then sitting exams but it was always in a really formal face-to-face environment but I did feel approaching these exams but there was extra pressure associated with it I'm a lecturer so I set exams I supervised students taking face-to-face assessments and in the last year pivoting not just to teaching online but also to online assessment some of which did include timed exams online I did understand the pressures that existed in terms of taking exams so having to put on the hat of a student it added that new dimension for me I'm going to say it scared me just a little and listening to some of your other speakers talking about assessments and indeed the previous speaker in terms of how it went for them I really feel some of that so actually there was an awful lot of learning for me it wasn't just for me sitting the exams as a student but also to understand the experience of the students so the first exam that I sat was using zoom for the online exam so in advance a setup where there were two cameras one from my laptop facing the main screen where I was going to take the exam and then another one on my phone looking at me so I had to log in twice there were three other students taking the exam alongside with me and the exam was recorded using zoom to observe the activity that was going on and in advance the proctor asked me to sweep around my area and show him that there was nothing on my desk or under my desk and I suppose initially I thought this was I did have a feeling that it was quite invasive it was a two hour exam and I found that I was under pressure to ensure I finished it I think I certainly write faster than I type I don't know other people maybe have that same experience but it came from a background where I always did hand write my exams the proctor though the one thing they did a time check so like you're sitting in a formal exam hall they told you how long you left at different stages which was certainly useful the second exam however and that was indeed my second exam that I took it was proctored using artificial intelligence so this was a three hour online exam and it was invigilated by means of you know a camera and screen sharing on my device the one thing I'd say is that what really prepared me for that was I'd run a mock exam the previous week so I went in and I locked into the system to make sure that I understood the technology and how it worked and that gave me some comfort that I was going to be well placed to use the software and the platform and understand how the exam paper would look understand how the calculator would look so for both exams very much preparation I felt was key to me being comfortable in that environment you know but the second exam it was showing your identification so getting the passport out and certainly I was a little bit cautious on that so the provider worked through it with me in terms of what that meant but I can certainly see concerns that people have associated with showing identification I didn't have to show my credit card or anything so certainly I think that would have been maybe a step that I would have had to explore further if that happened the organisations that I took the exams with I have to say the one thing that they did was make sure that actually how well it ran very smoothly because they very much prepared us all of us students in advance so the key really was that preparation for me making sure that the environment in which I was taking the exam was really safe and quiet environment and indeed my family understood there was no interruptions to happen on those Tuesday evenings and nobody was allowed to use broadband for fear my technology went down and so they were well warned on it how did I feel about the exams in this way I was conscious that someone was looking at me or someone was observing and indeed at one stage I pressed control on my laptop and this red flashing screen appeared and I'd love to look back on the video to see the expression on my face because I can imagine if anybody was to examine it say this girl is definitely not trying to cheat because I got such a fright but it said if this happens five times more we're reporting you so I felt a little bit secure then you know but immediately following each exam I tell you that the one thing I remember feeling was I struggled to remember all aspects of the exam paper because I didn't have a tangible exam paper to take away or to download and if somebody asked me what questions came up I certainly will tell you I don't really remember all of them because of the feeling of you know the pressure showing the exam so unlike other exams I suppose I was unable nearly to do a post mortem which may be a good thing as well so based on my experience proctoring you know I can certainly see the discomfort and the challenges that present for students but I can also see for some examinations if possibly a critical one to ensure the integrity of the exam so I think it has its place but I think if it's to be considered I think preparation is key and technical but also from the emotional side I think it's important that we address that but I also think it's important that we look to see is it the right assessment in order to understand how we meet those learning outcomes so to finish off today I suppose well after completing the exams and coupled with assignments I did well in them so that was certainly a bonus for me but my learning was not just for me taking the exams because I wanted to complete certain modules it was also for me to understand to be a student again in this really technology driven world Thanks Naan Thank you very much for that it's quite useful to have those perspectives particularly around that experience and that emotional experience as well as the pragmatics of just being able to manage a proctored exam online so thank you very much I was very very useful I'll hand over to Dr Lisa Padden now if you would like to take the virtual stage Sure, thanks very much I'm going to share my screen but I have very few very few slides so I'm going to take you back at 18 months to our previous world where this was our exam hall in the RDS I'm sure it's very familiar to lots of people who are on this call where we would have had two and a half thousand students in every sitting four times a day, six days a week and we had also an alternative exam location on campus in Bellefield in UCD so that's where students with reasonable accommodations would sit their exams but in recent years we saw a dramatic increase in the number of students who were applying for temporary exam supports on the basis of exam anxiety so that they could sit their exams on campus rather than going to this large exam hall in the RDS and we had gotten to the point where for some programs students there were more students in the program on campus in the alternative location than there were in the RDS so there were obviously issues in terms of this particular exam format before COVID but then we switched online and we thought great you know that's going to eliminate this particular barrier but unfortunately some of those barriers came with us in the online format so in terms of online exams I suppose it's one of the most common things that happened so for some students they went they had assignments as Brendan described switching from the timed exam traditional exam to an assignment but for others it switched directly to an online exam instead and I suppose our online exams inclusive well the first thing we have to do is ask our students whether or not they're inclusive and it's brilliant to hear the student perspective today as well and we spoke to our students the issues that they reported were first of all a lack of clarity in terms of what is an online exam is it a timed online exam it was called lots of different things it's a live exam it's a synchronous exam it's a take home exam it's an open book exam and the expectations of what that meant for the students really weren't clear so Brendan talked about this issue actually so students felt if they were doing it at home then what was expected of them was to produce an assignment in the same way they would have produced for continuous assessments throughout the semester when they knew if they went into an exam they had two hours and it's a very different piece of material that they would produce so there was a real lack of clarity for some students in terms of what was expected of them proctoring obviously was an issue so particularly in the first exam session that we saw online we did have some colleagues who proctored the exams themselves over Zoom so lots of students felt uncomfortable in terms of privacy and being uncomfortable and even the idea that their own lecturer was the invigilator in the exam was making students feel uncomfortable and we know with proctoring software there are lots of different issues which have been mentioned already in terms of how accessible it is in terms of how it reacts to students with certain disabilities and behaviors so there are definitely issues with proctoring and connectivity was a real source of stress for students where they had to do a live online exam and didn't have a reliable broadband connection and the environment that they were in because we know that the idyllic image that I showed you of the person with the perfectly clean desk and the coffee cup is not the environment that a lot of our students are in they're in shared space where they didn't have privacy and they really didn't have a comfortable environment to do an exam in so how do we make assessment inclusive or make the alternatives work well the universal design for learning says we should be supporting diversity of assessment so we have a diverse group of students and that means the assessment that we offer students should be diverse we can offer choice and you know Dr. Jordan O'Neill has done a huge amount of work around choice of assessment and how we can make that work as well and authenticity and I was delighted to see one of the previous speakers talk about authenticity and how important that is and I loved her checklist for that type of assessment as well so how do we plan for success so when we run the universal design for learning digital badge we've had hundreds of people who've made changes to their assessment and sometimes people introduce a diverse or a new method of assessment moving away from traditional assessment and they're a bit disappointed in terms of maybe with choice students are still sticking with the traditional assessment method but really what can work is having just one more choice for students so we're not overwhelming students with lots of new methods of assessment which has been the case for the past 18 months when we've all had to change but really just offering one additional choice scaffolding the skills so where you're asking a student to do something different that you actually support them to develop the skills to do that so if it's creating a video that you're giving the students the skills they need to create the video and even John spoke about having the opportunity to practice with the proctoring software which didn't work for lots of the students so giving the students a chance to practice a chance to develop the skills that they need and it's really the students assessment literacy as well my final point really is around students as partners which has been spoken about lots which is fantastic but bringing students into the discussion around how they're going to be assessed and why they're being assessed in a particular way can have a really positive impact particularly where you're introducing diversity or choice of assessment so that you're really making the students central in terms of what you're doing with assessment and I know John talked about some students wanting to go back to the exam environment but my big post-COVID wish is that we never ever go back to the ODS and never ever have two and a half thousand students sitting together doing those exams again and that we look at this diversity and choice and we take this as a real opportunity now to look at our alternatives and how we assess students so thanks very much Lisa thanks very much for that and I think that photo certainly sparked some emotional responses from people on it that had experienced that venue and I think your summation of all of the issues around validity, reliability of assessment constructs was well summated but also concluded with a very strong message again which is not only the preparation it's also the consultation with students and the students being a real central partner in valid and reliable and authentic assessment constructs so I really welcome that and thank you very much for that with that in mind I would like to maybe open the floor initially if there are any questions but I would like to bring in Diana Young Diana was part of our earlier panel representing the students with John but Diana you haven't had a chance to maybe get so I'd love to hear your views on this idea of proctoring and maybe some of the considerations that need to be taken into account from a student's perspective thank you very much Nile well I'll just introduce myself my name's Diana I was the 2020-2021 VP for Education in my students union in DBS so I actually got the opportunity to work alongside the examinations office of my college to kind of set up and to help get running up of the proctoring system for students during the exams they originally our school did a kind of a trial run in January for post grad students and that kind of went relatively smoothly only thing or only feedback that we got was just that students needed to be more familiar with the entire online system of the proctoring system because we all know exams are quite stressful already so adding another component of it being online is extra stressful then moving forward we started doing the proctoring exams setting up for the end of semester exams the end of term exams for some of the students and one thing that came up was the word proctoring in itself the word proctoring scared a lot of students just number one being it's not a word that we're all familiar with at all and it's something new so we kind of ended up using the word invigilating to kind of ease the students more into the concept of proctoring and the whole experience so I think one thing to note is just perhaps the way we introduce it to students if we are doing proctoring for exams another thing I would mention is that just from the experience of getting to work in the background of how to start it up for students and stuff it was really really important to have the ability to call for additional resources and outsourcing other systems like John mentioned having adequate systems in place to support students not just prior to the exam to prepare them but also during the exam if something bad happens say if it's connection or anything of that matter not being able to submit exam papers online also that has become an issue also and then as well as not just before and during but also after to kind of ensure that students are kind of what I'm not sure the exact words kind of help students round up afterwards and see how they thought the exam was after the whole experience of going through proctoring so that was really important and that required I would say a lot of reassurance not just for the students but for the lectures too so that all parties were familiar and understood how the experience was going to plan out and I also actually personally I had the opportunity to actually undertake a external proctoring exam and I would say it was quite different in that for every student that was taking an exam there was a specific proctoring person that would be looking so throughout your whole three hour exam there was one person on the other end of the screen that would be just looking at you while you're doing the exam so it wasn't necessarily a online system that was trying to track your eye movements it was an actual person who was keeping eye on you throughout the whole exam also with that at the beginning we had to kind of bring I would have to bring my computer and show the top of my table the bottom of my table the front of my table and I would also have to go up the mirror and show my computer in the mirror just to show that there wasn't anything on my computer that was perhaps helping me cheat I'd also have to show the different walls the four walls and the ceiling and the floor and my back would have to kind of face the wall just like how it is now so I think that having to include that I think it is a very good practice to ensure that you have a good environment for having exams but also brings about how some students may not have the opportunity or have the privilege to be in a space that is exam conducive for them and also it brings about some students not being having or not having technology devices that have cameras sometimes they're on a desktop that may not use a camera and that would call for students to have to go out and buy a camera for the exam and then be familiar so it also I think just to reiterate and to kind of finish off what I'm about to say is just that proctoring I don't think is necessarily a bad thing and it's great that we have different options for examination purposes but it definitely brings about an added on stress and pressure and for us to kind of eliminate that it's to ensure that we have resources and supports for students and lecturers for both parties so that we can ensure the experiences as smooth as possible Excellent Diana thank you very much for that and Terry I almost kept everything to time so my apologies to everybody for running a little bit over if I can hand over to Terry maybe just to maybe just look at the next steps but before I do I'd like to just thank a lot of our speakers and our participants online this was a really informative session and a pleasure to be involved with so thank you very much Terry Thank you Emilia Nyland and thanks to everybody and there's lots of things happening in the chat I know that this conversation could go on and on and we will be producing a forum inside so please do continue to contribute to the Google Docs and have your perspectives and share your perspectives there and if we if you just go we have another just to let you know we have another webinar coming up on the 10th of June Catherine do you quickly want to do 10 seconds and what we're going to be talking about Yeah five seconds we already have a guide on developing enabling policies for digital and open teaching and learning we're redeveloping it it's an opportunity for you to join that conversation we'd appreciate it Thank you Emilia Can I just thank Nyla's chair can I thank all of her speakers and very much and you as participants for joining us and I look forward to seeing you on the 10th of June for our next webinar Thank you everyone Thanks everybody