 now yep okay just jump back to my slides and then I'll do some introductions before we get going here we go okay so we've got a fantastic panel lined up today to discuss this we have Rhona Sharp who is director of teaching and learning at the University of Oxford and she was actually a key person who helped set up this group over 10 years ago we also have Ola Tundi Dura Wuju who is the faculty associate dean for diversity and inclusion at John Moore's University we also have Sue Attiwell from who is head of ed tech at JISC and she's been working in this area furiously over the past year and also Rob Howe who has been working alongside JISC as well from the University of North Hampton exploring some of these technologies in his role as the head of learning technology so I wonder if it would be okay to follow this particular outline here and ask Rhona to go first with about five minutes of just where her perspective is in terms of that Rhona are you okay to take over now just minimize this I'm very happy to do that Jim thank you very much for that introduction so as Jim said I've been involved in Elysig for 10 years or more and I do feel that I'm feeling my age a little with all of this fuss about chat GPT I wanted to present a bit of a kind of longitudinal perspective on it and also as I usually do to talk about the student experience but I'll start with a longitudinal perspective if you like and when my when my grandfather died I rather was very surprised to inherit his piano and the piano stool which came packed full of books and I wanted to share this one with you which is called music for the home and it was published in 1932 and I'm just going to read you an extract from the forward of this book music for the home and this book is so it's a piano book it's absolutely full of different piano music and it says statistics prove that the sale of music of every kind and description is so seriously on the wane that publishers are inclined to fight shy of bringing out new songs or piano pieces when they know there'll be scarcely any sale for them so these books these piano books had stopped selling what has brought about this state of affairs I feel there is but one answer the habit of listening to music instead of performing it this habit is to be so deeply deplored is so damaging to the progress and development of a beautiful art that it's up to all music lovers to do what they can to prevent it from spreading and I thought that was just such a lovely way of thinking about the position that we're in now that we're worried that these new technologies are going to come along and completely destroy the way things were before and we try and change them but of course they don't destroy the way things were before and we need to have a much kind of broader context and understanding than that and I was reminded of other things like Eliza I don't know how many people remember Eliza that little chat program before we even had the web where you it just presented back your responses to you in a kind of humanist therapist way and obviously we've got things that are still going like stack exchange for coding so it's really important I feel like I need to go around with these you know keep calm don't panic it's only chat GPT we'll get through it as my first overriding message the second thing I wanted to say so I want to say that historical context and I wanted to say a little bit about how chat GPT might be useful and I hope you've spent the weekend playing with it with your children and parents and friends I've been getting everyone to do it this weekend and watching their faces as they do and one of the things that we're we're struck by is its use as a productivity tool so very often when these new technologies come along we think they're going to replace some of the work that we do but actually they enable us to get through that first little stage faster and then move on and do more interesting things so productivity is a tool if you use it like you would a search engine it gets you to where you would like to be a lot more quickly than having to browse through the first 10 results which you know is what everyone does in using these tools and productivity is particularly useful in an inclusivity context so inclusivity and Sonya Oxford we talk a lot about inclusive education is where we're trying to reduce the gap between different groups who have to spend proportionally more time on the same task so you have students with different characteristics who need to spend more time on doing the same task because of their background their previous experiences their abilities their strengths their weaknesses so imagine a non-native English speaker putting into chat GPT can you help me compose a letter to my tutor asking me for this for example okay so we use these productivity tools to try and level the playing field a little bit to try and equalize the experiences for students and I think that's really important to remember but technological change is a challenge because it happens so fast and I think that's one of the things we've really seen with this so although the experts in my team tell me this has been around since November it's really only just hit very suddenly and we've already redesigned our assessment overnight for online open book exams during our periods of remote teaching and assessment and I am certainly sensing a weariness amongst the academic community of having to do that all over again so quickly in response to this technological change so it's a challenge because of the pace of change and it's a challenge because we feel that students are ahead of us so I mentioned Wikipedia in the chat and I certainly remember that feeling when we discovered the extent to which students were using Google and Wikipedia and being concerned that they're ahead of us so I just want to offer you one approach to help with these challenges really and I'm sure other excellent speakers will offer others and perhaps predictably I'm going to say that the one approach to help with these challenges of pace and concern about where students are is to engage with students directly to work in partnership with them on how education and specifically assessment can respond to some of these challenges of a kind of AI augmented world that we will be living in so we can encourage our students to use chat gpt to research their question and critique it and I know lots of lecturers have already started doing this to say well here's the task let's put it into chat gpt together let's see what it does and then your homework is to go away and say how your answer is likely to be different than the one from chat gpt those kinds of things have started quite quickly as ever our teachers are amazing and creative and innovative the other thing that I've I've heard is the use of a critique which isn't just about an academic critique but also encouraging students to think about how these kind of services might be commercialized and how they might respond to that so I've had the pleasure of talking to some students and I will finish by just saying telling you a few things that the students have said to me but it's I suppose always my approach to go out and talk to students as quickly and as often as we can about these things and the first one I'll just give you a couple of their quotes and the first one who's an electronic engineering student and he said it's a research tool and actually your own knowledge this extent of your own knowledge influences how well you interrogate it okay so if you're asking stupid questions you get stupid answers but if you're asking good questions because you know your subject you'll get better use out of it and it's just a research tool like that and that was particularly about coding and there's lots of really interesting things that chat gpt could do with coding particularly because it gives you the code and then it gives you the explanation of how it's written the code and another one said well we've always wanted model answers we always ask our tutors for model answers and so often we don't get them and I'm using it it's an economic student who said I'm using it to provide that initial structure to an answer so that I can then get started with it and I can put on the detail and it gets me up that kind of productivity tool up to the next level to really get started on the important bit of the of the work and finally Sam said international relations student I'm using it how to learn I'm asking it questions as I would ask my lecturer he did say if they ever replied to email and I'm asking them I'm using it to learn I'm asking it as if I was talking to an expert that's great as long as we understand some of the limitations of what chat gpt can do and I'm sure some other people were talking about that and and finally electronic student finished by saying he said we're not stupid give us some credit we're here how to learn we're here to learn and you know help us use these tools appropriately is basically what he was saying so I hope that's provided a bit of context and perhaps explained my approach to the way of thinking about the challenging situation that we're in I very much look forward to hearing what other people have got to say thanks rona that's brilliant gollatundi do you want to follow well brilliant thank you thank you very much jean um thanks rona a man I found that um very quite insightful um can everyone hear me okay because I'm known to have um microphone issues yeah yeah brilliant um yes and I I agree with um and I quite like the fact that you went and got data from students on on how they sort of utilize it and I think it's important um to do that what I've been doing is being I've been following you know responses of of academics um to this chat gpt disruption because it is a disruption but it doesn't have to be um what we perceive as a negative disruption which many people are seeing it as um and what I've sort of gathered essentially is people have sort of them had you know various reactions to it some people have said to see um because this would enable students or make it easier for students to cheat therefore we need to go back to exams um and of course that that is not I don't think it's a right response to it because you just have to look at historical data in terms of how different student groups perform when it comes to exam and of course um of all these student groups exam as an assessment type is one of the areas where students perform lists in and that gap is even wider when you start to look at different student groups for example when you look at student groups in terms of ethnicities um so that that wouldn't be um the go to that shouldn't be our responses what I would say to to that um some people have us express the fact that you know you don't need to worry you know ignore trans GPT you know um if students want to cheat let them cheat now we've got you know new technologies coming up you know with 97 percent accuracy or 99 percent accuracy and they can tell us um with great you know degree of certainty whether our students has produced work that has been generated AI or or trans GPT so you know the so the responses and we don't need to panic we don't need to worry but then again of course that that's I don't think is um is an attitude we need to sort of sustain because one that would then mean that we also want to get into this habit of you know because this new software that can determine you know whether students as um uh you know utilize content from from from the AI because then it'll be challenging universities that I want to presume you know so that's another on top of what we already have in terms of turn it in then we're also you know thinking that you know that well be that'll be okay for us to sort of pay for the services so that we can um catch you know students that are cheating using AI and of course we need to be very um strict about academic integrity and and that's that's absolutely important but I don't think that we should you know um just think that all right you know we've got those cheat cheating um uh softwares or applications therefore we don't need to we don't need to worry we don't need to do anything what I think trajipiti has brought to for um is um each around our um over reliance on written work um and I think that's that's the cause of of this panic when you look at a program for instance on an undergraduate degree for instance and again it depends on which um the discipline and some of them lend themselves naturally to you know more written work while others really don't but for those who don't really need a lot of written work you still find that the composition of assessment types involving writing is you know significantly higher than than other types and that's led to the next reaction that I've that I've found which is you know people saying like we need to rethink how we um assess students um so you know rather than just you know thinking about you know assessment relating to just writing for example we can say you know you know create a poster for example you know mix things up a bit um and and that has been some of the um reactions that I have seen and but again I would also kind of get that with you know again you need to ensure balance in all things so if you've got to recent assessment if you've gotten maybe presentation or presentation for example or maybe you've got scenario analysis because those are you know different assessment types that you could potentially use the idea is for you to you know strike a good balance so I think it's sort of pretty forcing program leaders or module leaders to think about you know how um writing heavy are the assessments and I'm just trying to make adjustments in that regard so I think that's been another response but what I would also say is there's a need for calibration just as Ron has said you can see that students are responding in different ways so am I using it as a as a um as a for example me as a critical friend for example or some are using that as a sort of research tool so how can we help students curate content and then based on that see how they reflect on you know what you sort of curated it could be maybe critic you know comparing their work with what an AI will would produce so those are some of the things that I I believe that we should really be talking about so how do we rethink our assessment strategy and whether we're not overly reliant on just written text and how can we use it as a tool to help students to sort of curate content you know to to think differently and also maybe in the sort of incremental way as well and I think that's something that is worth and discussing and be very keen to hear what others have got to say with regards to that the only thing that I'll also um and talk about with regards to you know inclusivity is um we know that many of the data you know that um chat GPT spews out again is based on you know the training set my question then would be you know how do we ensure that a training set is one maybe devoid of bias or you know cautious bias how do we clean those data set because if we're going to be using it you know how do we achieve things like decolonization for example of the curriculum within that how do we ensure that students are having access to you know a wide array of literature from you know from different cultures from different you know and places how can we utilize that and integrate that students you know to develop a more robust learning for our students rather than you know the usual text you know that our students we recommend to our student so those are some of the areas that I think as academics as well and also technology providers um we can start to sort of have this conversation on how you know we can sort of achieve some of these some of these things I think I'll stop there for now because I'm mindful of taking a lot of time yeah and then we'll take questions later on thank you very much thanks Tundie Sue do you want to follow up no sound from Sue sorry is that better sorry I haven't realized I had two mute buttons um so I'll start again um so I'm currently the national centre for AI and tertiary education alongside Michael Webb and in the centre our aim was to accelerate the responsible adoption of AI across the sector thanks to GPT we no longer need to focus on endless raising as it's now very much topic of the day but I think reflecting on that the speed of change has been phenomenal and it continues to be and it's hard to keep up so sharing information and best practice is one of our focuses but also providing practical guidance and support to try and make life easier for our members and we're looking at a range of things at the moment including the impact on academic practice and student experience I think one of the things I really wanted to cover is that it's important to understand that AI is much wider than just generative AI and certainly wider than chat GPT and it's already been embedded effectively in many tools we use all the time so chatbots recommend daters Microsoft insights things like that and I think even more important than that AI is embedded in most tools to support accessibility such as transcription tools so Grammarly premium users AI and that's used by many dyslexic students and discussions on how to use GPT to support accessibility are already underway but we there's a lot of concern about the immediate rush to not use AI to provide guidance and inadvertently disadvantage in students that have accessibility needs and I think with the generative AI tools it's not just chat GPT Google with Bert, Lambda and Bard and Facebook's have their own tools and there's many more tools being developed and in coming to the market all the time increasingly chat GPT is being embedded across Microsoft Office products I think somebody referred to that in the chat earlier so it's starting to come to things like PowerPoint Word and moving usage into the everyday and that makes it harder to make black and white decisions particularly around inclusion and accessibility so from our point of view and what we're trying to help with it's essential to have a clear understanding of how these tools work before making decisions around implementation to understand what you need to think about to ensure equity and adopt responsibly it's a rapid moving complex picture and often first thoughts ignore the complexities and ripple effects and it's important to take the time to reflect and look at everything in a holistic way I'll just give you a very small example of summarizing tools you know as soon as I saw this thought oh these are great it's going to make going to meetings and things really easy because even though I make notes I can never find my own notes but I'm reflecting about this particularly from the academic point of view you know my favorite revision method was to create my own summaries so there's both an opportunity and a risk involved if one at first was like that's really positive so for me I think very much encouraging discussions like this I think the more people try things out see how they work understand how they work and use them and share that knowledge is how we're going to learn and particularly for me I hope that chat GPT impacts in a similar way to COVID and acts as an accelerator to reflect on practice and ensure that we as a sector can provide the best education experience we can for our students and that's where I'll hand over Jim thanks Sue Rob do you want to follow up yeah thanks very much yes my sort of starting I think with AI was sort of in some of the very early sort of university size I mean I was following it when it was sort of being proposed really in the 40s and the 50s 1940s 1950s so it's nothing particularly new and I think what we've been spotting really is just the evolution of the product and the integration of that into more and more of our systems now when I was in my when I was in the 1980s I was sort of doing programming little bits of AI into basic on an old Atari sort of system and I think Rhona mentioned about Eliza and actually that was built into into some of the work that I was doing at that point really and then in the 90s it was expert systems and so we were sort of starting to develop and more recently over the past couple years I took the maturity model that actually came out of the National Centre for AI and I actually mapped all of our processes at the university against that maturity model just to see where we are and if you haven't done that it's a good process to go through because it gets you thinking about all the different systems that and processes that you have across the institution not just for students I mean obviously we focus a lot today on on students because of the ELESIG sort of steer but actually across the whole institution and a lot of our tools already have got AI built into them or very soon coming actually into that some of the the role that a lot of us have I think is about steering our institution in a in a sea of technology really and you know we've seen in the chat I mean Leo mentioned about Wikipedia Manage mentioned about essay mills are we actually looking at anything radically different at the moment with chat GPT yes it's it's obviously got massive publicity and that's done open AI the company behind it sort of a lot of benefit in terms of obviously commercializing it and moving to the next stage you know in a multi-billion pound company at the end of the day they are sort of textbook actually in terms of how to raise the value you know of the product and and other products actually around it lots of other as Sue mentioned lots of other products already exist across the sector I mean some of you already will be using various other tools that have elements of AI in it so AI actually isn't isn't that new from that perspective what we did with the maturity model is we looked at the various developments from both the student side and I think we've heard a lot about that we've looked at it from the tutor side so looking at what tools like chat GPT can do in terms of helping tutors and save them some time potentially so actually it's not just students that can use it it's the tutors themselves and also at the institution level so you can actually use chat GPT and various other tools to develop strategies for you as well and actually it gives you a bit of a baseline that you can then develop on that's no different to what some of the students will actually be doing they'll be using it as they as basis as they have with Wikipedia and various other tools already and then build their assignments on there as institutions we already have lots of tools and techniques to help us try to spot when students are doing that and obviously we have academic and integrity policies to actually sort of set down the guidelines and I think we've heard from various people about the the importance of actually just reinforcing to students to say you know we expect you to create original work or at least a reference when you're using other bits of work students I've spoken to students around the institution some students are more honest I think than others in terms of when they are actually using third party sources and then quoting those and we will continue having people using contract writers but they'll probably use chat GPT instead of a contract writer in some cases but there's probably not a huge difference there people that are trying to deceive will probably carry on trying to deceive that's you know at the end of the day a lot of people are very goal-focused there is a particular thing that they've come into the institution to do and their motivation is to achieve that at the the easiest possible way so there is again no real difference from that perspective so it is for us to actually look at our types of assignments I think Tunde mentioned about an over reliance on written work and we need to think about moving to more authentic assessment we need to make things or some of the assignments think about making those more personal so draft and outline embrace AI it's there we can't ban it it's just going to be around it's going to keep on developing we need to think about people potentially using some of that and then building on the output so doing a critique on it for example or as tutors we might want to think about actually providing some modeling for our students so actually use some of the output and explain about how people can actually build on that but I think the idea of banning it I think is a is the wrong path to go down to recognize it to talk about how people can bring it into their work in a in a sensible way I think is probably a much more positive discussion to actually have from where things have been previously but certainly you know we've spotted all of the panic that everyone has now in terms of of where things are so lots of lots of areas for discussion I think as far as that goes so I'll hand it back to you Jim thanks that's a lovely whistle stop tour of lots of different perspectives on this before because there's lots of questions out there so if people want to queue up the questions and Emma I'll come to you in a minute and just in case you've spotted any good ones in there I wondered if just to expand upon how we understand that student experience we've had those early examples of just talking and having a conversation with students about it but where do you think those key conversations should what what what are the most interesting things to address in there when having conversations with students or thinking about researching in this area anybody got any particular insights on that I mean I can just say something very quickly jimmy if that's okay yeah I mean yeah so so at lgme we conducted a study and trying to understand students experience of the of the assessment process and some of the things that were coming out of that was that if you look at and the various sources of information that students go to when they have issues with your module and the actual module leader or tutor is actually stored on the list so the first on the list is fellow students which of course might not necessarily be you know the the best way to get information but there is this problem where of access sometimes students and what when students study usually is in the very late hours and there isn't any module leader to answer any because if you send me an email outside of book times you will get a response you know and and it's important for students to have this sort of access not I'm not saying that they need to have access to us all the time but I'm just saying that if there was a tool that it can potentially go to all right yeah and if they've got a question you know well you know it can turn out an answer well close to as as reality as possible again one needs to be very careful with and the output of them of chat GPT I think that's that's a useful thing for for for students to sort of think about we also know that chat GPT can also be developed into this sort of conversational and AI where you know student can have you know conversation with them and you know I think there is an opportunity for us as academics to use some of the elements of chat GPT to really like chat bots you know where we've got some ready answers all right and then you know it can sort of utilize some of the ready responses to answer any module specific questions that students might have I think there's an opportunity here and I think it's something that is welcome we can't be there for students all the time but we can create tools you know or or or adapt to existing tools to help solve that problem yeah so you might see chat GPT beating peers in terms of assisting on assignments very soon so did you want to jump in I did I'm like coming probably more from the kind of business angle than the education angle but I reflecting on a few comments we've had with conversations is you know this is big employability issue for students you know AI is is out there in the world and I know that's a phrase often used so but preparing students and using the tools that are being used in the in the outside world I think is an important part but I think it highlights some skills that I think I'm missing at the moment so things like you know AI does give you lots of information how to assess if that's the right information that you need for your purpose is a real skill that only academia can give to students and that will help them both in their study but also post-study as well you know it's it's it's a it's a crucial skill that that is embedded in some other countries in their education but isn't in ours I think so a missing thing for me yeah and really interesting as well to start talking to employers as well to find out where they are in terms of the skills and those sorts of things rona I was thinking about the value of student experience research in understanding academic integrity generally and remembering when you know we all started working on plagiarism and the role that student experience research had there in shifting our view from students are dishonest you know students are fundamentally dishonest to talking to students about what's really going on for them and understanding why students cheat is because we have bunched the assessment tasks all together we have presented them with a single opportunity for high stakes assessment rather than doing continuous assessment we have not provided the criteria clearly and given them opportunities to engage with them um we haven't explained why the task has been set and why it's relevant to your next task or to your longer term career so I just think student experience research you know has the potential to really reframe this this this time that we're in and we can go back actually and look at how that happened in the plagiarism literature and the importance that it had there yeah and also links through to Sue's point which was don't rush into this get some they'd get better understanding before before making big movements Rob I don't know whether it you wanted to jump in or whether we want to I'm just going to build on I think what people said about different uses really for some of the the chat box so what we've been spotting over time is that pastoral support actually sometimes it's out of ours I think tunday also mentioned that as well able to provide support when a tutor is not always able to do so and we've been looking at the degree to which institutions can actually train some of these chat box to become more relevant and we're tracking some research I think there was actually some in in Europe I don't know if anyone's on the on the call already who's involved in that but they were involved in looking at chat box to support students that had potential mental health issues as well out of office times that heads into quite a risky territory for me in terms of where are the limits about where the chat bot then hands off to a more sort of qualified professional in some case but it was interesting about what the students were reflecting already in terms of they were starting to feel more supported in some of the area and I know that's something that affects a lot of our institutions where some of our services now are completely overrun so actually providing or triaging I think some of the support to some of the chat bots may be sort of more feasible now than it than it's been previously but obviously it needs to be done with a lot of caution as well brilliant thanks Rob Emma I don't know whether you want to come online and because I've found it difficult to keep up with the rate of the chat I don't know whether you've been a bit more successful I know some of our speakers today have been offering yes ideas and they Victoria Victoria's asked to ask verbally so yeah a presenter and then I've got a couple more say that I'll read out once Victoria said hers okay brilliant Emma thank you thanks so much Emma thanks for letting me speak no I just my I don't know if it's a question more something maybe that could add to the conversation or take things in a different direction I'm not sure but I guess my concern is as a tool for productivity there's a danger of sort of being pulled into this kind of harder faster quicker mentality and it's and a lot of the things I'm finding with my own students is what I'm needed to provide them is ways to actually sometimes switch off or to slow down their process and to when I heard Sue that was very encouraging to me that Sue sort of saying yes these technologies are coming hard and fast but actually the challenge is to to be able to retain that kind of critical take on them and to not easily kind of be co-opted into that end because we are all part of a system that kind of encourages eternal productivity sometimes to the detriment of health well-being and I think I think maybe that's my question is how because a lot that is very cliched isn't it because some of the technologies do help students slow down or they you know they can support well-being but I guess maybe my interest in is how chat GPT might form part of a balance a broader balance more balanced approach so that actually some of the issues in the sector have been around you know not the mental health stats for students and so on not just as a result of COVID but you know partly as a result of COVID but to think about a kind of not a pedagogy of catharsis but in some ways some way of kind of you know not sort of being pulled into the latest thing but as you said as you're just trying to do here establishing what is key and what we want to take from it as part of a broader more holistic approach sorry that's a massive comment rather than not it's a really important point Victoria and I'm glad you raised it have we got any comments from our panel sorry Rona well I was just going to invite Dominic to speak to this who's assistive technology officer in my department this link between digital productivity and inclusion very well just making a presenter while we're while we're waiting Sue did you want to raise a point yeah I did in answer to Victoria I think really and I think it's that you know I think chat GPT and then all the AI products and the wider ones you know they have the potential to help with things like staff work and things but I think the other thing is it's it's within outgift you know it's there is a there is a rhetoric to you know this is here you must use it you must adopt it why I think you know I'm saying I think for now it's about understanding it and seeing where the opportunity lies rather than thinking you have to do something and you have to do it now you know that's concerning a lot of people and I had I'd rather not increase people's you know mental load really yeah Dominic I think I'm here now yes I just took a while to unmute myself and get the camera on so yeah I think this is this is all great I posted it in the chat right now is is a link to or earlier a link to an article by Stephen Mofferman he made a really important point about we're at chat GDP isn't just this tool that's that's out here you know for cheating or whatever it is actually kind of a new empirical fact it requires almost empirical study because it is based on technologies that we never quite imagine would be possible working in ways that are actually counterintuitive and so we often do not know what follows what's what really possible so it's you know it is writing poetry in a way that nobody thought you could you could make so I was actually using it to translate Czech poetry into English I was thinking chilly that would be really awful at it but it was really good as a process not not giving a finished product but all of a sudden it can enable translator studio things I mean poetry is almost never translated by a single person always in two people to do it so all of this stuff is all of a sudden happening in there I gave another example in this you know I was trying to compare two different translations and and this is really cumbersome and I was able to just quickly create that quick software tool in JavaScript and HTML using chat GDP to allow me to sort of highlight differences within the paragraphs and students are kind of picking up on these things so and we kind of so for example chat GDP cannot do anything with numbers you're gonna do add multiply you'll be on the most basic things but it will give you it will write you an excel formula to do it for you you know it will it will actually create a little Python program to you know to do things that you may find otherwise difficult than you will not be able to do for yourself so so in a way it's kind of assisting people in different ways and and almost the thing it's almost worst at is writing an essay even though it produces relatively bland essays it kind of has a very sort of formulaic structure in the way it always you know has that to try to tell us tell you what to do and then say it and they say it again and repeats the question and so on so it isn't it isn't very satisfying often and I would say sometimes the contractual use will probably still do a lot better but some of these little things like that supporting your work you know structuring things destructuring things and as long as we're kind of studying all the things it kind of cannot do and sort of reminding ourselves of what things to keep an eye on I think I think that's kind of what in the future so in a way I'm seeing this almost as this new phenomenon that requires empirical study as in the way that you know that's sort of a new emerging things like the emergence of cities obviously we have to study what how people live in cities and the same way this is kind of the same thing so we're at the very beginning of of that and so I definitely kind of would encourage people to look at as an opportunity to learn something new about unexpected emergencies yeah and that links with the idea of understanding that student experience of using it kind of that that connects quite nicely with they are our experimenters for us that could actively help that what about going back to Victoria's point though about overproduction kind of yeah I think that's that's a very big yeah that's very often mentioned but in a way I had a very similar experience recently I wanted to find out differences between uh between powerpoint and and keynote as for recommending to people whether they should use one or the other and I found hundreds of articles written by content producers they were entirely free of content repetitive might as well have been written by chat gtp we're already in that situation I could have found a single place they would explain like a detailed feature by feature different in those yeah so software and so I actually tried to use chat gtp and it it actually gave me better results that I that I can have to use my own knowledge to interpret so it wasn't by any chance perfect or detail but actually kind of that iteration the process of iteration going back and forth checking using it kind of as a process of generating hypothesis was in a way was in a way more beneficial so I think that is true that people are worried that there's going to be more content out there but unfortunately there already is a lot of content out there and and a lot of the people who actually offer hire themselves out as contract cheaters they're also being hired by companies to produce content for their websites and and often that content is there just to have content and it isn't very good so I think I'm not sure if in practice it's actually going to be that much of an impact because sadly we're already living in that world brilliant thank you Dominic Rob you've been patiently waiting I think yeah I think it was just to to reinforce some of the things that have been said already is that a lot of this isn't new and I think as Sue said it's a case of taking a measured approach to everything that's going on make the decisions I think with knowledge about what's actually out there what the capabilities are try things for yourselves and then build on that rather than anything that is more sort of knee-jerk type reaction really and I think as we've said already work with the students on the development as everyone is I think has been saying actually students are very concerned that they want to maintain the academic integrity of their degrees and the qualifications generally that that we are actually providing to them so actually they're very keen to to help us in this process so that would be my my thoughts at the moment Tundi you've got your hand up I think okay Emma do you want to pick out any questions from yeah I've got there's quite a few that have come through I think some of them have been answered indirectly or directly in the session I think one of the ones that has come up I think from Tanya which is a valid point I'll read three of the questions that have come up and so we can sort of combine them together because I'm conscious of the time so Tanya asked do we need to test students on knowledge memorised though for me it's it's more about seeing how well students apply what they've learned in jobs you typically don't need to have memorised everything so that was one quote from Tanya which I think I'll digitise into several other things other people said about assessment there was a bit of a discussion about AI detection in turn it in but I think that's kind of been picked up somebody's also raised a rather interesting question that towards the end as AI tools are developing so quickly is students used to support their learning and assessment might those that work on the assessments of the tools at the last minute have an advantage over those who've got going earlier because the tools are advancing so rapidly there was one other question that I have myself is you know with people more and more people getting students or working with students or working with AI has anybody actually yet run all of this stuff through their institutions dpia or ethical approval or privacy approval or whatever it's called at your institution so that's three questions really but I think in a lot of ways they all link together yeah let's go back to our panel and and see if any anybody wants to explore those ideas anybody want to jump in yes can I yeah I'll go after Rob I was just gonna say we've started looking at it from the point of view of where the data is actually being processed I mean obviously a lot of us I think forget when we use some of these tools or when our institution signs up to to anything particularly we don't always know where the processing is is taking place and what is happening to the the data so you can imagine the millions of people that have now signed up to open AI to get access to chat gbt what's going to happen essentially to all of that that information and obviously all of the the questions of that people have been asking it you know to what extent can open AI actually do a lot with that that information not just refining the tool and it was interesting I think how as a company they actually were looking at what the press was doing and talking about them so they I think they've avoided some of the the potholes they did a lot of testing prior to releasing they didn't enter the pothole that that google hit a couple of weeks ago and I think you know people saw the the problems that occurred that when actually the tool is not properly tested out but I think a lot more is starting to come out now about the way that chat gbt actually generates its responses and the training data that actually was used and the way it was actually gathered and the ethics of that as well there's a lot more I think that is starting to become people people becoming aware of now in a way that they weren't of previously so lots of things I think are emerging and will emerge over the next few months that we probably weren't aware of previously and that might change our relationship with some of the tools and the way that you know we actually pick up on them in the future yeah yeah good point Rob Tundi yes thank you Jim yeah and thanks for that Rob I suppose the issue with ethics is always going to be there and I think it's worth you know continuing that sort of dialogue what I would say is that we need to think about this disruptive technology because I know that you know we've been talking about some of the challenges that chat gbt currently has and I know it's a matter of time before it's sort of them of our constant challenges so what I would say basically is as academics or as them technology providers or as educators we need to start thinking that is a technology that's come to stay and we need to think about it as a technology that would continue to improve the question is how do we sort of you know with it how do we ensure that we are able to sort of utilize it for example in that sort of adaptive learning system because I come from this sort of inclusivity space and I think that this tool is one of the tools that would help address some of this inclusivity issues but we need to start thinking about how we can use it directly to address some of those issues is what I would say and to answer the question I can remember the name of the person who asked it with regards to you know forcing students to sort of recall knowledge and so to speak of course I agree with you that you know I don't think that that is right I think it should be more about them application and that's another perspective that we can adapt to chat gbt so rather than just you know coming up with content should be more around you know how do students that apply this knowledge that they've sort of curated or you know gotten from chat gbt in a different you know different setting or all together and that that's all I'm just going to say for now thank you thank you rona I think you've got your hand up I was going to just say a word about developing students digital literacy before we finish this is in response to a question from Geraldine right at the beginning and where she said um it's a bit of a worry that students might assume it produces correct answers these AI tools produce correct answers um and I suppose my response to that would be is that a different from students reading web pages and assuming the information presented on the web pages are correct and think of all that work we've done largely with the support of librarians about teaching students how to assess the credibility of their sources digital or otherwise and all that work we did getting students to write wikipedia pages to understand how wikipedia works before they trust it as a source I think it's exactly the same issue with the AI tools as it is that we've already encountered with some of these other things so we need to develop the digital literacies and for the AI bots that means I think as Phil has just been saying recently in the in the chat with this kind of weird data sources it means we need to understand the data sources that the AI tools need and our professor one of our professors of AI here at Oxford was telling us last week that some of these AI bots have run out of the internet as a data source and are going to social media in order to to gather the vast amount of data that they need to run their algorithm so it's really important that we develop students digital literacies in this case to understand what data the bot is drawing on in order to create its answers wow uh sue I wonder if I could move over to you to summarize perhaps a little bit because we're running I think we've got about three minutes left okay leave us leave us with some encouraging words I'm going to actually because I'm really pleased to see there's lots of comments around you know talking to students including those in in you know how AI is included I do see AI as an opportunity rather than a threat but just about looking at all sides but I think one of the important things um we captured it in the recent uh blog post which somebody posted earlier is that in a lot of these cases we did a lot of research around guidance and whether new guidance and everything is needed and we looked at a lot of policies and things that already exist and you know one of our main conclusions was that there's actually no need for extra guidance to come down focusing on the tech because the existing academic policies already cover most of these areas you know so if you just use what you already have and think of that then then a lot of the time is covered and this is just another tech brilliant thank you Sue so we can I think um what I've gathered from this is uh this don't panic don't move too fast let's take a breather and reflect and use our students to help us understand what's happening before we make massive changes um uh that kind of we're we're down to the last couple of minutes so if you do have a question which we haven't addressed and you'd like the panel to um have a look at it we do have a link that um I just shared and I think Emma just reshared you can put in your comment in there um I think we can save this chat as well so we might go through that and see what we can find we will write this up as a blog article on the alt site so you might see some of your questions addressed there as well I want to thank ever so much the panel for taking the time um to think about this dwell on it and share that and thanks ever so much as well for everybody coming along and having hopefully a really rewarding chat and the chat space um so uh we will be having more events soon keep an eye out for Ellie Sigg and um and we hope to see you all again soon so uh thank you very much I think I'm getting a lot of pings in my ear at the moment I don't know whether everybody else is but I think that's a