 Okay, so good afternoon everyone and thank you all for coming to this yet another fantastic Tech Thursday sessions. Now we're getting in the flow of it. This is the third in a row. We did one in end of February, one first week of the March and then this is the third one. As you probably are aware that we are running a working party where we've launched a working party on inclusive assessment and sort of driving change to make assessments more just inclusion inclusive and so on. So if you would like to join the group in that capacity, we would share a link later on. So please do feel free to join us in making that happen in whichever institutions you're representing in whichever your practices. Okay, so we make a start. I'm here today with my colleague from Portsmouth University in the School of Energy and Electronic Engineering, Dr. Jibril Kohli Nizad and him and I are going to talk to you about directed study chat GPT led independent study and an exam revision and and our findings from some recent as well as some very long standing interventions. So we'll highlight the problem statement, show some case studies on directed studies including examopedia, which is an exam revision service and three exam period style documents in classrooms being used to run directed study events. Also talk about some during and post pandemic disruptors like the open book exam to assessment and chat GPT also a big disruptor at the minute and later on we'll have some time for Q&A. Okay, and I will highlight a potential opportunity for us to to be taking part in a collaboration if anybody is interested in that front. Okay, and what I'll do is I'll go through the first few slides and then I'll pass over the control to Jeb for sharing his example and then Jeb will do the same back to me for for the rest of the presentation and feel free to use the chat to put any questions we may be able to look at those and and come back to you. Right, so for for this particular project we are looking at exams and and really end of module exams that we're talking about here they often you know used as assessment of the learning rather than assessment for learning and there's some research to suggest that and that's because they tend to take place after all the instruction has has happened and basically because of which they become separated from from the educational process like when you're teaching and and they have a opportunity to ask you questions and so exams kind of stuck at the end of a module that's problematic what we are saying. Almost exclusive purpose of such assessments tend to be to test what the student already knows you know their sufficient knowledge and not kind of engage them in in in more learning if if it was possible so that's the the predicament of of an end of module exam that's the difficulty and we often find students saying what is important for the exam sir or teacher whatever they refer you as what will come in the exam and sort of showing their anxiousness their anxiety about this format of assessment a closed book exam maybe even you know so that's what used to happen before in most cases and before the pandemic hit us and the feedback they get is yes it's useful for reflecting on it but maybe not as useful for other modules and you know or a future exam so it's a thing that happens in a you know on a retrospective basis and rather than you know helping somebody to become better for for that particular module that time so yeah not assessment for learning it's assessment of learning that kind of scenario a systematic review done recently on not so recent but still relevant on exam as an assessment suggests that traditional exams are linked to anxiety which we have kind of printed before but also anxiety is linked to lowering exam performance and another point that this paper highlighted was that exam anxiety you know if you were if you worried about who are going to be my worried students you know if you if you were to put a finger on on who which of the students have the lowest self-esteem in the class they were the they will probably be the ones and this tends to be the middle of the class maybe towards those who are really you know somewhere in the middle from 30 to to 60 65 sort of that touch for the failure they'll be worried about the exam the most the ones who are going to excel in exams they're going to they're going to excel and they're not worried the ones are going to fail and that they know that they're going to fail perhaps also not so much worried but yes they will be worried to an extent but not as much as the ones in the middle tends to be more more worried that's the research paper was suggesting and then self efficacy is another sort of related term which is also positively linked with with performance so if you're more feeling confident about a subject and you're likely to do well right so and another systematic review looked at the comparison between open and closed book exam and it said that the latter which is the closed book exam may promote superficial learning with things like requiring a lot of memorization and stuff it is also therefore more stressful and a less preferred mode of assessment by many students and it's no doubt less inclusive because if you're relying on memory then of course some people have learning difficulties which prevent them to be very good with you know lots of memorization an open book on the other hand as this research suggests also that they can be more engaging and they require a different set of questions a higher order thinking skills are being tested in in those type of exam so that's the problem statement that there is there is this issue of their divide between assessment for learning and off learning here are some other sort of quotations from different papers about what is assessment for learning then the benefits are that you know tutor is able to point out problems in a student's work and they can then benefit and have follow-up discussions if necessary as in the first case the second one suggesting that feedback is information about a gap between an actual level and a reference level so if you're if an engineer you have a feedback control system if you provide some feedback from the output to the input you could probably tune the system to to reduce that gap somehow so this is again a real time sort of system where the feedback is and if you apply that knowledge to to exams obviously not possible because you can't have feedback coming in in the process of examination so there's that difficulty for doing AFL in exam seeking and interpreting evidence for use by learners and their teachers to decide where the where the learners are in the learning journey and where they need to go and how best to get there again it's not possible inside examination situation but that's what assessment for learning is about likewise assessment for learning also has benefits for the teacher because you then get to know what the states of mind the student are and you can improve your learning and teaching process as you go along with them you're on the journey with them so that's the benefit of AFL but whereas the associate assessment of learning it's kind of the the word itself exam which can perpetuates anxiety and hinders the transition for students to become independent learners so for us we were interested in in answering these two questions how can directed study initiative help with this gap between AFL and AOL and how to then prepare students to become independent if these are the problems in this other background these are my questions we are interested in before the pandemic hit us we used to have mostly closed book exams in us in our courses and mostly used as assessment of learning as we've just been talking about revision tended to be quite an individual thing students I'm talking about here they would be sitting down on their own revising probably isolating and very few people will form groups and study in groups but majority of the class kind of doing it individually but we also noted that we had a diverse intake of students people coming who are into the first year with having no exam background especially those with p-tech qualifications and things like that so we felt that there is a need to do some kind of training in in in exams for students and make it collaborative to overcome these isolating sort of stressful experiences that the student have been telling us so we created this I created this service called examopedia which is basically we will show you how it works in a minute and Jeff and I have some some case studies to to share with you today but it's essentially a collaborative exam revisions service but which gives the opportunity to do this assessment for learning business that that we can't achieve with the exam otherwise and it has been used since 2007 in my school and has benefited many students as you will see some some testimonies and some data later on and it reaches each time it reaches more or at least the same number of people who are in the class in the in the let's say the revision session that tends to happen towards the end of the the block again and moreover in the last few years it has triggered new innovations based on the idea being used by many people and as it happens when people take an idea and they obviously innovate around it as well further so that's been happening in my school and it is it is after you've used examopedia for two to three years then you would have learned enough from your from your students and the students obviously would have learned enough from you as well but you you would end up doing things slightly differently in in your class hence there's a potential to improve teaching and learning in using examopedia and the students will tell you that they whatever you're doing right at the end perhaps we should now start doing it much earlier so we've done we've listened to them and we brought the examopedia activity into examopedia style documents throughout the the teaching weeks and that has helped in making the classrooms more active and even outside the classrooms more active through asynchronous sort of interaction between student and and and staff what is examopedia i'll just go through you know these points with you so basically it's a google document you can use a wiki for it and it has been structured in a specific way i will show you an example but what happens is the the the document has some questions on it and the questions have some spaces below which the students can fill with their own researched answers to to to whatever type of question you put in there and as students put their answers in obviously they could be diverse answers they could be similar answers and undoubtedly there will be somewhere where the answers are very different and there are some common mistakes in them and so on and you you then get a chance to to come back and give feedback to the students and make that visible to the whole class so it's shared and you never want to give the final answer until perhaps the the answers have been produced by the students and they're correct and you say yes finally did that that you come to that right place um you've understood it so really trying to make this as an opportunity for learning and uh rather than you know live with the fact that exams are assessment of learning and and there's no um yeah there's little to to do overcome that and if you see some questions that are unanswered obviously that's an opportunity for you to do some more teaching either this year if you want to do if you have time or for the next and you can choose what type of questions you want to put you want you can put past exam questions there or you could put very small short answer questions so too so the student can engage with little chunks of learning from time to time now at this point i'd like to pass the control back to my colleague jeb who is waiting i guess for me to do so yeah jeb if you can share your screen thank you manish i'm just trying to share my screen can you see my screen it's loading it says yes i can see it okay so let me this must present it by yourself um so hello everyone my name is jeb uh i am the team your lecturer at the school of energy electronic engineering um at university portsmouth i am also a course leader for two msc courses in the school um so i am here to provide some information some examples of my implementation of examopedia i am not the one who has given the original idea it is coming from manish but i have just tried to implement it on a regular basis in all my modules i have used it in since 2018 in four different modules and during this time out of 264 students in in the in the class 208 students has reached and used this document which is about nearly 80 percent of the students now in terms of the modules that i am teaching the class sizes are typically between 20 to 40 students except one module which has um which is larger cohort 89 t students so i have been implementing um this examopedia which is perhaps one of the good um methods of directed study but it's mainly for exam revision and as manish said we are trying to use it um even before getting to that point where the exams take place so during the teaching uh but the examples that i'm showing today are mainly for exam revision a few weeks before the exam um again manager already explained how the exam period is set up but just to repeat the instructor provides prepares some questions which is typical questions similar to exam exam questions and put it in a tabulated form there will be different rows under each question which is used for the students to contribute and write their answers then this document is shared with the students the students are invited to contribute we also put a link in our virtual learning environment so they can click on it but you know with the students it is always good to repeat um this invitation to you know to to um again remind them during your lectures sending emails etc so it's um so so all the students are aware of it and they can come and contribute but now at the top of the document we add some notes um and the most important point is that we should emphasize to the students that they don't rely on this um piece of resource only so they for the exam they should use all the material that we have told during the term um and this one should not be the only source of revising the exam the revision for the exam um in fact i i i have a kind of disclaimer which are used even in my lecture revision lectures where i say to the students that these questions that i'm solving today um don't think that the exam questions are going to be exactly the same or even similar to these questions there they may or may not be similar and they will have to eventually um read all the document all the all the material that has been provided available to the students um so that's one point that needs that needs to be emphasized as you can see in red we have put some dates and times which are check points and that's when we go and check the student contributions we'll be able because this is these questions are set up in the google doc which is online so it has a good feature that you can comment and give feedback to the students you can give your comments you can provide feedback now having so many dates and times doesn't mean that you will have more work to do because you know that students usually leave the revision let's say majority of the students not all of them leave the revision for the last few days so for the first few occasions the dates and times that you go and check and you see no one has contributed and perhaps you use that as a prompt to remind the students again to to come and contribute to the document but the document become let's say you become busy and document become busy near the exam perhaps one or two weeks before the exam um i usually put the exam date on this document as well i think it's a good practice to leave it there because some students um forget to check their timetable for the exam and get confused by the different dates so it is always good that they see it here they know when exactly their exam is going to take place um it is also important to have a tough date because um if you keep doing this until the last minute before the exam first of all some students will not be able to check it before the exam and also they will focus too much on this one piece of resource for the revision so one or two days before the exam you will give a tough date after which you will not be checking the document or at least you will not be providing any feedback to the students answers so in terms of the examples of modules that I have used I have used examopedia so I have only inserted module quote here m21983 is the title is actually numerical and computational methods um as engineers you know we have lots of modules with design questions computations calculations um that's a matter of fact I'm also a little bit nerd of mathematics so I like calculation based questions so you would see from my snapshots also that most of majority of our questions are calculation based but it is possible to use um um of course theoretical questions in the same way um or text-based uh questions and one thing that we have been recently practicing is to try and integrate chat gpt with examopedia because chat gpt is good for perhaps um for text-based text-based interventions rather than calculations because I have seen calculations it is uh a little bit um having problems and errors but um but yes um so in this module um in penetration that I have had about 63% of the students used examopedia the last one was 2020 because after that this module the course was closed and um the module um did not run but I I have been teaching this module for few years before it was closed um so 63% of the students used the examopedia I compared these numbers with the revision lectures I would normally have one revision lecture before the exam um those revision lectures usually the contribution is high the the attendance is high it is not only because of the exam students are thinking about the exam and they want to focus on and exams but also I normally tell them that I will provide some I would provide some information about the exam so that encourages them to come to the session where we we solve some questions similar to political questions about from the material but I mean comparing it we can just say that they are comparable sometimes in some instances it is higher than these revisions 10 000 these revision lectures but I think that's good enough to have comparable numbers using those documents um another module is M30615 this is a module which is sold to all msc students and the title of the module is actually engineering management economics and risk analysis um I have been been using the examopedia in this module for the last two years nearly 70% last year and more than 80% this year contributed to the examopedia now this contribution by the way doesn't mean that every each and every students come and write their answers some students after you give the comments you provide the feedback some students come and check and learn from those feedback all those comments they may not need to to contribute especially if you say the answer is correct and they don't need to write the answer again but these this is in fact we are talking about numbers of the students it is not repetition of you know it will be different students in terms of the numbers so this percentage is for different students with unique student numbers so yes we will have some snapshots of again this this module the examopedia for this module as well but this is the chart simply shows the contribution from the students in two occasions in 2002 and 2023 so in revision sessions we have 57% in 2022 in the revision lecture I found the data from our our attendance monitoring system and 80% this year now I always try to find reasons why we have low or high attendance so that 57% is is probably because last year it was an online revision session and those online revisions sessions are recorded so they would have had access to the recorded session after after that date so perhaps some of them just prefer to to watch the videos so they didn't turn up for the revision session but I'm talking about revision lecture in terms of examopedia it was 70% last year I had model module m24608 the class size was not too big about 19 students and 84% of the students again tried to use the examopedia and for the revision session it was a 74% those who attended the revision session before the exam so some examples perhaps some lessons that I have learned during this time from these examopedia the implementation of the examopedia and some snapshots from the examopedia in my comments first of all it helps the students understand how your marking style is and how the marking scheme works for each module for each exam because you in your comments for example you will mention that the answer that you have given will not get high mark because you are missing this bit of information or that type of information so they will know what is important in that particular question what what should should they cover when they're answering that that question so for example in this case I have said that if they don't write their workout in the calculation it is important for us that they write their workout they don't just give the final answer so if they give final answer if the question has 10 marks it will only be one or two marks that they can get so they have to show they're working and it is important to emphasize this beforehand so they will know that in the exam they should write their they're working or in this case I have asked them to write the formula well we don't ask them to memorize the formulas but when they are solving it whether they are getting from the material if it is open book or it is provided in the exam paper then they need to write it again write the formula again to solve the question so this helps them to to get an idea of how the marking will work you can direct students to to certain material you can refer them to certain material that you have made available to them whether it is your lecture slides or your tutorials or sometimes maybe a book you're into a certain book to read a certain page to understand the question and to realize what would be the best answer to that that question so for example in this case I have asked the student to refer to the tutorial one for the definitions or in this case I have asked them to look at slide six in the relevant lecture so that in fact prompts them to go and look at the material and perhaps look at the whole whole material they may take the time to look at you review the material the relevant material and that would be a revision or at least that part of the lecture it is also good that to good practice to identify students with different styles we have students who give I'm sure you all have had these kind of instances students who give very short answers sometimes just a matter of one word or if it's a definition just very short one one line definition it's not necessarily the weak students sometimes the good students also because they want to be concise they know the answer they think that if they they they sometimes have this perception that the instructor knows that they know the answer so they by default they think that if they write a short answer instructor will assume that they know the answer but in the exam we are marking it blindly so they should write what is necessary for the answer so even sometimes smart students also write a short answer and they are missing one piece of information or keyboard not getting the full mark for the question some students on the other hand write lengthy answers lots of text it is possible that in that long text you find the correct answer or you may just find it some nonsense without referring to any part of the answer but you will know when the students come and contribute you can identify these kind of students and perhaps these kind of answer you and you can perhaps emphasize on a few things to help them to improve their answer some of them prefer theoretical questions some prefer calculation based questions and also some of them you will always find students who have weak mathematical background so giving some hints to them helps for example those students usually write the mathematical equations incorrectly or in terms of fractions they don't do it correctly then the procedure of whether deriving or solving the equations or you can see they forget to write the units they forget to write the right decimal points so it is good to always remind them to give some comments and feedback about these things you can motivate these students to engage more with the examopedia if you just keep criticizing the answers and just saying that it is incorrect it is wrong then students may be discouraged to contribute more and so sometimes you have to motivate them by saying well done for doing this in fact sometimes students I will perhaps I was thinking to refer to it later on about you know improving the teaching but you find students who give kind of answer which is new answer not the same answer that you you are thinking but it is correct answer so in that way you also learn from it and in that case of course you can also so you know you can say well done for doing this and sometimes students come and create some communication with you sometimes they engage I personally wouldn't prefer that because I tell them that try not to use not to put comments in the examopedia document because that may confuse other students but you know sometimes it's you see that there's no harm in leaving those comments in the examopedia so sometimes they come and communicate with you and you try to get direct them to to do more and remember to always happy we remember to always have one version of this examopedia as a view only so in case because this is editable version students may just remove the questions and so on so you have a view only one as well and sometimes these actually discussions continue via email some students prefer for the various reasons they want to continue the discussion via email they contact you and ask about the same question or other questions or want to learn more so the good thing is that you can also learn from this you can try to eliminate the source of sources of confusion and misunderstanding and the following year when you are doing your teaching you have a better idea of what were what was the perhaps challenging part what was the confusing part so we can improve your teaching based on that you can identify the common mistakes that students make perhaps again in your teaching you refer to those common mistakes and try to give some ways of mitigating it and eventually identify the areas of strength and weakness for the module for the teaching and try to improve your teaching later on one point that I forgot to mention the students can also add images so they can write it by hand and then take a picture and add it because especially in our case it's calculation based typing might not be easy so they can write some paper and scan it and put it up on this document I think yet that I just wanted to give some examples of implementing this examopedia but as I said there are ways of integrating chat gpt with this with this tool and also using similar documents for during your teaching rather than just leaving for the revision so at this point I will just give the floor back to Manish thank you very much for listening okay great thank you so much for that Jeb that was fantastic let me continue from here I hope you can see my slides yeah I hope yeah this coming up yes excellent so as you could you could see from Jeb's presentation the assessment of learning you know you can't escape it pretty much from from the exam point of view but if you see if you think about this being used two or three weeks before the exam in that moment there is a lot of learning that is happening with the student and the staff and that's where the the goodness of this technique sits and that's what is kind of making yes there's an exam at the end of it that is by a fact an assessment of learning but there is this kind of collaborative revision happening before the exam which makes it a little bit more interactive and less isolating for students but also makes the focus more towards learning then sort of wrote memorizing information and then things which tend to happen when students are learning individually in preparing for an exam and there's no opportunity for interaction so I actually set out to ask these questions of students in the last 16 years in the first 10 years I would collect surveys every so often and you know for some of my modules sometimes all of my modules so I present to you now what I have been getting back from the students on that so 80 percent have agreed to the fact that this helps their exam revision of the 210 who completed the surveys from probably a larger number of sample who actually were in the class not everybody completed the survey top of my head maybe about 33 percent or 35 percent was probably the the completion rate but here we go 80 percent found that it is allowing you access to different views of fellow students which is by design basically those boxes below the questions are exactly for that reason sometimes it is the same view sometimes is different sometimes is both wrong sometimes is both correct sometimes is different anyway 68 percent were agreeing to the fact that because of these diverse answers coming up it'll help them broaden their understanding of the topics that were on the document and 67 percent said it helped increase the depth of learning so I was interested both in the the worth and the the depth of the topics and their understanding so and about 54 percent agreed to announce them a question that does it help you reduce your exam anxiety and only 22 percent said no and these are I'm thinking the people who were high achievers and there will be another another 20 who who didn't benefit in that same way but look 54 percent have said that they they agree that the exam anxiety has been brought down so that's a noticeable number 79 percent said that they will use it again and only about seven percent said that they did not use the examopedia service during the examination revision period so with those highlights you know I approached my colleagues in my own school and I said well would you be interested in doing this as a pilot in your practice maybe some of you are thinking right now maybe that you want to use it maybe you think no this is not for me either is fine but this is what happened when I when I ran the first pilot so we looked at modules across the the school in the foundation year in the first year in the second year and so on and what we find that look this is this is what's happening each module is represented here as a block so that's one module that's another that's another and so on and all the way from level three to level six even level seven were were involved I'll talk about that later so you can see that they have each this is three days before the exam this is two days before the exam this is one day so mostly they are there one day before the exam this is remember the first time we are using it in in across the multiple modules and the students haven't used this before so they are unfamiliar with this and they they've been launched and the service one or two weeks before the exam because I've said to the model coordinator maybe that's when they're going to be most likely using it but in the future years this pattern kind of has changed not just stuck near the exam but anyways so the students who are much ahead in their in their course they are reaching a much larger number of engagement you know 90 in some cases 89 86 whereas in the foundation where I think we needed it to be working from day one the most only about half the population is is kind of using it in the first year of the of the of the pilot but later on it the internet increases so then this is just sort of showing some average scores of different groups of people people with high attendance people with the you know high with use of examopedia people with no use of not not using examopedia these are some differences there but they are not quite conclusive they are obviously in favor a little bit of examopedia being used but they're not statistically I would say or one great difference in some cases they are but not always so I'm not going to say that this is an evidence for but I'm saying that they are they are benefits to to to to students even who are high attendance attending in the class they if you use if they are using examopedia they tend to have a better average than compared to other scenarios when they don't but not a huge difference each time so a greater number of cases will be needed to prove any sort of significant effects and now this then compared to attendance in the last revision session this is another way of looking at it as to how many people actually were there in the classrooms versus how many people were there on the document using it treating it or contributing on it put together so that's that's a comparison only in one instance here this is about the same and I think Jeb showed some data as well which it was the other way around maybe in maybe in a couple of places so I didn't capture that one here because Jeb already had that and then in terms of averages on on modules where examopedia was used versus was not used this is not like a controlled experiment but we just said sort of some people used it some didn't use it and we then looked at there because we could track on google docs who are using examopedia and who are not so therefore we then look at their marks and see the averages so clearly again one module which is below others and this was a particularly bad module with a person we had a staff member leave so you know it's below past mark even so it's very difficult here for the for those students and I ended up doing half of the examopedia for half of the module and yeah it kind of is same but anyways you can see on the other cases it's mostly more average than otherwise in terms of student comments I did a thematic analysis of all these survey data and other data that I could collect from modules surveys end of module surveys so the first one here is about academic workload and the students recognizing the effort that the staff has been in this case it's Jeb's student and they are really appreciating his input and saying it would have been not that easy to give individual feedback to a class of 120 this is his one of the bigger class okay and the 120 is a student number I don't think it's a actual class size but he's hinting that this is a big big class students start to see it when you're using in this way in that exam revision period they start to see and they want it to then come further up in in the teaching weeks as well as a directed study opportunity or assessment for learning opportunity so example Peter came towards the end of the course assuming it came earlier it would be nice to that's you get the point there also some of these comments kind of aligned to this access to social and cultural capital of the class which is quite unique you know from coming from students saying uh collective information of the class gives various views we've seen that one before example it is actually a helpful tool where we can see each other's opinions and answers on the particular question or topic it is also a helpful tool because it gives you gives out or posts the question that we would not have expect in the exam so we're giving them all sorts of questions that they are very much prepared going in to the exam with less anxiety than than before so that we can learn from this from start and and so on right okay and then last one is about benefits for self-efficacy motivating motivation as well as lowering some anxieties in the students example to help me so much to prepare for the exams it gave me the chance to study more on my own and boosted my confidence there we go so confidence and self-efficacy and lowering anxiety they're related now we come to the uh much-awaited I guess part of it we're talking about open book as in as a disruptor as well as the chat gbt solutions that we are talking about today um so during the pandemic what did we learn we we moved from closed book exam to completely take at home open book exams where they would have full access to everything and that kind of changed a lot for us we thought that's already you know moving from assessment of learning to assessment for learning purely because the questions that we would be needing in the in the exam would no longer be recalled no longer be wrote learning type questions it will have some application orientation and I I released a document I'll share it with you the link and that was to how to design questions for for open book examinations and this document had been downloaded in excess of 1000 number of times at least from my records but also from Sally Brown who used it and then she circulated it it was about 3000 or so downloads in her document so the good guidance there for how to design an open book exam for the pandemic period that's when I released it but also benefits for people with disability and as it is open book has a research evidence that it reduces about 17-18% less anxious people because they can come in with their books and they can write from the book if they if if recall is the question but that's we don't usually call us as a style of question in an open book exam but still it students have its effect of reducing open book has the effect of reducing anxiety in students now we come to the negative side of open book exams academic integrity issue you know who's doing the exam is it been done by the student or has been outsourced to some some website or is it chat gpt and so on and they're using anxiety as a hinting earlier there has a downside is because the students come in too relaxed and sometimes they are then shocked by the type of exam question that they see and therefore using examopedia for training students for this formative exam is also a use case you know and we've been doing that the other thing so now I'm just gonna sort of do a question I guess I can ask you if you can post in the chat are you back to the traditional exam or are you still using open book exams but invigilating them and not giving access to the internet like we are that's what we are doing here in enforcement we don't give access to the internet but students can bring in hundred notes books whatever they want to bring in and the exam is designed in a very different way with the stair casing as well as you know application of questions and what what now with the with the chat gpt around I mean can you still continue doing take it home exams do you feel that's that's okay or or do you think that there's some potential maybe benefits of this kind of new technology and we've thought about it and we are as it was hinted earlier in both of our parts that we are using examopedia or exam video style document alongside these language models large 19 volts to tap in and not just the social and cultural capital of the class but also of this tool this tool is capable of talking back with some useful sometimes not so useful comments and really very that's very nicely with examopedia because that's what we are doing you know in exam period some answers are very good some are not very good we have access to the social and cultural capital of the class but we can then as academics come in iron out any any difficulties that the students are having to understand that topic but another thing that can happen if we if we use this tool alongside is it could trigger independent learning and that's kind of already happening one way that I've used example in the chat gpt is to basically have student after the students have given their answer I stick an answer from chat gpt and ask them to criticize it and sometimes it's so good the answer even though everything is wrong it is so confidently written that the students wouldn't find it to be any of you know it wouldn't be spotted by student and only an expert I would would be needed to really critically analyze that answer in in the case of multipath communication I teach telecommunications and it was beautifully written but there was some very fundamental error in the in the answer which which completely destroys the understanding it's not correct whatsoever so it gives the chance for students to criticize and if they didn't you could then highlight that you know the tool is not always right you can't really trust it it is it is still learning and it will probably get better in the in the long run but it's not always correct at the minute an alternative approach I use was to ask chat gpt to say not make the answer very good could you make it a less polished answer and in that it sticks in some mistakes and fantastic then then I can get the students now say I've used the examopedia with your answers now chat gpt is one of our you know fellow members of the class if you like now can you give it some negative feedback or critical feedback and that's a useful exercise to to to go through and that's you know students can practice the skills which they will be needing in a world where we are surrounded by ai tools and we are having to rely on ai based decisions if the students are able to criticize things right now that's a valid skill to develop and and make use of later on in their lives and one more way is to ask you know the students to find out sorry not the students yeah actually sorry ask the students to use chat gpt if they have access to it and tell get it to tell you the threshold concepts of the of the subject that you are that you are dealing with and in doing that that has you know okay it comes up with some answer but then you can ask the it to to elaborate on it further and that way the student can go on on a journey and of learning and making sure that learning is correct we're asking them to share that information back on on a google document just like example we just had a document and that seems to be working well as well because I've asked people to write some code I've asked people to write some answers theoretical answers and so on and that's been coming back to us via the other documents yet another way is to ask chat gpt to list cognitive conflicts it's quite good I've tried it myself in that sense it highlights mistakes that people have made in the past in my own exams in my own coursework and I asked it to make that prediction of what cognitive conflicts will students face in this topic and it's presented more than what I knew to start off and again they may not be all correct but they are I can I can verify them and if it if it makes sense to me I can then use it in examopedia and I post it and say okay well there you go which one is right and that's the kind of thing I'm anyways hoping for examopedia to present more to students because if students have two very strong views and they are conflicting only one of them is right it's important to get the wrong one corrected in the in the in the student's head before you know for for learning to really last I'm conscious of time but this is this is another way of what we are looking into well I'm not really gonna take any more of your time I have a read of this but if you feel that there is some synergies between your and our ideas and if you'd like to take part in some collaborative work you are more than welcome to reach out to us and if you have any questions now please feel free to ask in the chat or with the microphone okay so there are some I'll take the questions maybe in the chat first Jeb if you want to chip in please do we have a mix most are back to back say most are back to close book a few are open so they are responding to my previous comment you have one exam that's in a PC cluster okay manage I can actually perhaps give an answer to this question Sakiru has asked how do we make students to respond to the questions as they might see it as an additional form of assessment which they might consider too much also it says staff might consider it as additional workload if I start from the second part and the way we did initially the only thing I did I prepared some questions which is not too difficult you don't need to have the solutions because then we are preparing exam questions you need to have the solutions ready you need to have standard questions their casing etc but in this case you have into the freedom so you prepare just some questions and we had someone from our IT department who put them together in a tabulated form and put it on our visual learning platform so it wasn't not really too much work and then for the checking then it's not really too much as I said the students only a few days to the exam contribute I mean the majority of the students will contribute a few days to the exam so you will just check and provide some comments because you know the answer so it shouldn't be shouldn't take that much time of course now this year for example I did it myself I put it in and of course we had the template so I put the questions in the same template and put it up on the platform and the students could just and just sharing with the student or making making it available to the students and the first part students see it as an additional form of assessment it is not mandatory for everyone to contribute eventually so there is no they don't need to have you know to pressurize or to have anxiety to to doing things there that they and once a few students start contributing which could be perhaps good students at the beginning then later on the students will find easier to come and contribute and put their answers so it is not really I haven't seen any objection to using this this for the source for the students yes thank you Jeb I think it's right there in neither have I the students are either happily contributing to it or happily reading the reach kind of shows you that and once within let's say engineering and if you've got a question you have a certain way to solve it there'll only be three four five ways of solving it maybe making some mistakes I mean not make you know so once you have seen four or five answers anyways it's kind of job done for that question so there's no pressure on every single student to contribute but different students can contribute on different questions that way we can split the load as well but because it's one document it doesn't make any difference it doesn't have to be taken by every student okay so Tristan is making a question here as well would it be possible to use it them up in a style with non-subject specialist staff learning support so we have tried in the past to bring in academic support tutors who are actually they are subject specialists they understand the subject but with regards learning support probably feedback I don't see why not because it but that would be a more of a personal support isn't learning support so maybe maybe not you can you can try I think it's if depends on the the need of the student there if it's about academic writing perhaps that could be a general document where learning support staff could could provide the feedback and that'd be useful more than one student perhaps I don't know what do you think Tristan uh yeah yeah that sounds roughly what I was thinking okay thank you so yeah non-teaching staff can get involved but yeah thank you thank you thank you so Koan has asked if is is it really for exam as an essay type questions no with chat GPT yes perhaps for now but but as as Jeb has shown in his slides if you maybe I can show you some slides as well in that you can write equations and solve questions with the proper derivation in the step by step all of that is possible you can do that by hand and take a picture of your and then stick it in the document or or you could use the equation editor to write equations so it's not necessarily only for essay format exams but it is that that is one way or where it is also used and it can help clarify cognitive conflicts that that might be there in a particular topic that you're teaching does I answer your question Koan great thank you okay so I've put a link for the feedback for today's session if there are any more questions please keep them coming but if there's if that's it and that's it and please complete the feedback thank you for for coming along and and thank you for all your questions and contributions thank you I'm going to stop the recording one for coming