 All right. For our beginning, we have two sessions coming up. And one of them you can already see a cheat or a treat the making of legal practitioners. I'm going to turn it down. We'll be here presenting for the partnership today. And then we are looking at XR and biology and Brad will be here presenting for his team today. So I'm going to immediately turn the floor over. My colleague. My colleague take the floor. Yeah. Oh my goodness. Good. Good. Actually had food. So my collaborator couldn't be here. Can you hear me loud and clear online? I don't know. Hopefully they'll hear. So it's actually a collaboration I've done with a lecturer law lecturer at Essex University and also we do some other collaborations on with migration and all sorts of other law things. So a cheat or a treat. It's actually a presentation around the use of tragedy, which have been the talk of the, yeah, among higher education and which probably will be the talk of the next few years because people are scared or very happy about it. A bit of mix of feelings around there. And you do also some collaborative work right with meds medical team. Right. Yeah. So here you go some collaboration work. So I'll actually be talking you through for you to understand the Russian is behind the work. I'll talk through a scenario with you. A typical scenario that gets taught in a law first year module. And then I'm going to tell you some of our primary findings. And it's, it's taking a teaching and learning approach so it's not just technology we didn't go and investigate but we used it in a learning and teaching context. And then I'll discuss briefly the problem based law assessment paradigm shift. So, actually, in law assessments there are, oh, I can be here I can see the slides here. In law assessment there are two types of problem that usually we ask students to respond to reply in their assessment it's a problem based question. So we tell them an issue so you know if well hopefully none of you have had to go to a lawyer. But if you've ever purchased a house or things like this you probably would have gone to a solicitor or some notary or something like this. So you're basically going to go with a problem. Okay, you want to purchase a land. Okay, and then you tell them and what you would like to do is you would like to be given good advice right, not just someone who's run something for charge PD and told you something. So that's basically what we ask, we usually ask law students to do, give them a real life problem, real life scenario like some of you in UK in other European countries as well you probably have this policy where you buy something and then you have 15 days to return it. So there's a lot of law that goes behind that. Okay, when you take your receipt back, and if they refuse to give you your money back that's, you know, you can then go to customer complain and things like this. And you need the person who's giving you the response to know what response they are giving you whether that's correct. And there's also it's a type which is like, I don't know, please tell me about how a contract is formed. Please tell me about how VLE is used please tell me about something you know very generally, a bit theory you can just copy paste from a book literary and put some nice references so that nobody says you plagiarized it. Now, Iraq is something that we use, and that we expect all law students to know, we tell them a problem that is the issue. Oh, there's like a gap in that screen there I hope you can still see. And then there is a rule that the students need to follow. They need to make an analysis of your problem. And then they need to make a conclusion. And for the conclusion they obviously need to check law statutes and all the previous you know common law in UK and all and then tell you according to this law this law this law. This is what I conclude and this is what you can do or cannot. And then there are five elements now of a law of to follow in a contract law so the module I'm going to I'm going to specifically talk about is a problem based contract law module. So there are five things they need to do will look at which is an offer and acceptance, a consideration and intention and capacity. Now the example that I gave you earlier if you're purchasing your house. I picked two people, you two. So you go, you go to Eileen, say Eileen I want to buy your house which Eileen is selling okay you're selling a house you have an extra house right now that you're selling. So you go, and you say Eileen I want to buy this house. Okay. And then she says yeah I'll sell it to you, you look like a nice job I'll sell it to you. She comes and says Eileen I'm giving you 10,000 pounds extra. No dollars. She lives in the US. Okay, $100,000 extra, and I want to buy you a house. Eileen say of course $100,000 I'm going to take that and I'm selling to you. Now you need to resolve that because that's a big issue. He'll go to the court and drag Eileen. So this is a problem, a typical problem that we actually give to students in law, and we also give things like, for example, okay, if a 13 year old goes and buy a cigarette with mom's and dad's card. If they are quite tall, nobody sometimes asks for ID. And then if they want to, okay. Not necessarily cigarettes, let's take something very pricey. I don't know. Something pricey, luxury, a big watch or whatever something like that. And then they want to return it because mom and dad said you were not allowed to do that. That's also a problem and you need to know how to resolve it. I suspect the students to obviously use statutes and arguments and then engage with a scenario. Okay. Now our research questions. What were the assumptions that we took that we, you know, had we had to obviously, you know, this essay type problem type problem based law assessment, and then what's the impact, because we were looking about it from a teaching and learning perspective, and then gender biases. The reason why I've, I've particularly said gender biases here, I'll go for it later on. And then the impact on academic benchmarking, you know, when we grade, we've got a table that we have to follow grading criteria, right. We can't escape that and then what will be the cheat or treat thing of making legal practitioners. It's a typical scenario. I've done some little graphics for you to understand it because the text can be lengthy and takes time to read. Now, this is apparently a pointer. I don't know if you can see there but I'm pointing. You obviously will not see there. So this scenario one little businessman guy here Philip, he contacts Margaret, who's a who rents services context Margaret who's the sales director at and says, Okay, I want to purchase your services. What's all your offers trial trial. Okay, Margaret is a sales director now, Margaret goes back. The little arrow is going to cost you 3000 pounds, and it's going to be installation staff training, plus the staff training and the price, you know what it's valid only for one week. Now Philip actually was looking for in the scenario was looking for a software. To do some accounting see and stuff like this. Now, at the same time, Philip goes back to his computer. Okay, he's multitasking. And he receives an email from cost plus so cost plus we don't know who was sitting there somewhere and send an email to Philip. And since they email email advert, you know, marketing, typical marketing. Here we are offering these services will cost you not more than 2500 pounds. Now immediately Philip says 2000 500 3000 I'm going to go for the 2000 500. So Philip goes for the 2000 500 calls cost plus calls and make an order. And then he realizes what they're telling me that the installation cost is going to exceed 4000 pounds now. Now if you look back in your life. That's typically some of the scenarios you probably have come across but you know more miniature thing. You make a decision and then you change your decision rate. And then what happened is, oops, Philip panics and cancels the order. Now next day. What you would see here is next day Philip grew in size on my slide became a bit taller, much taller than me. Now Philips now I don't know what happened he had a dream I think he calls tech Mac. He says, you know what I, well he actually didn't didn't get anybody. He goes to the answer answering machine. He says I'm accepting the offer that Margaret made last year, last day 3000. And please can you tell me if there is also on site training for new stuff and for the next six months. Now going back to your example of buying and selling a house. Sometimes you'll probably go and say I'm buying this house and then you'll go back to your notary or solicitor whatever and say, is there school nearby by the way. You know you've already kind of say you're interested in that and then oh you have other questions that you keep answering. Now this usually typically things like this we call that a counter offer. Now, again, Philip, I don't know what's wrong with him, but he likes to panic. Okay, he panics again. He calls, but this time he couldn't reach Margaret. He reaches the secretary. Okay. And the secretary said, okay, I've taken note that you want to cancel the offer and the earlier messages, obviously the security had not listened to them. He said I'm passing on your earlier message to Margaret the CS director. Now the reason why I have actually explained this taken time to explain this is this is a problem. It's actually not so complex that we give to law students. And in real life, when we want them to become a lawyer, solicitor, not real whatever legal advisor, immigration advisor or whatever. When you go to these people you want them to tell you things that will work for you solutions not go to charge you pretty and say hey, by the way, how can I resolve that right. So, now the question that goes to the students is, please advise Philip, whether he's concluded a contract with any of the two companies. Now this is our grading scheme I'm not going to go through it is the typical thing like all you other students have got to identify legal references have got to identify the problem have got to understand the problem have got to reference the right statute, the laws and applied and follow the Iraq I or AC that I had in an earlier style slide, the structure and then language now I've got referencing in red, and I'll tell you why. That's usually the by the way the grading that we use okay. Two things we had to do is one we had to put that scenario into charge to be the, and then we had to modify it and try to see also what else modify the question slightly. Now our primary findings was, when we send that when we put into charge to be the question, as it is, as it's presented to students charge to be did say in the beginning that okay I'm not a lawyer. You still need to kind of consult a legal legal professional, and that's what you should do. And technically even I'll actually go for that later on my slide and then charge to be the though did not know what is Iraq. Because in the question. There was nowhere where we asked the students please follow Iraq, because we expect them to know they should do it. Like how we expect you to know I don't know you've somebody had just gone to the toilet with just expect the person you know they need to wash their hand and do you know flush the toilet all these things like this right you don't tell them. So we expect the law students to know what they should do, obviously charge to be did you didn't know that, right, because it's just taking from the pool of information is got from the Internet, just making sense of the wordings the words that we've given it and we never said Iraq there. And then charge to be the also could not identify the valid parts of a contract which I had earlier on a slide which is around consideration acceptance often all these things did not reference even a single relevant statute. We don't engage at all with the content with the context problem. Use contraction like you know, sometimes you know people will say isn't is it of is not. This is not something we expect students by the way to use in legal. When you're writing you wouldn't be using these things, it's out, whatever I would you would write I would not I have to stop tea and things like this. And also, basically we've used the marking grade table. That was a full fail for these scenarios and the times that we've repeated this a fail as in a strength almost 200. Okay, that's not terrible. Going back to our research questions now. So instructions, what we found is basically what what I did then was add some more instruction, saying, please, could you follow Iraq, okay, and then at some point in time I also had to add further question further inference to say, follow Iraq and also please use relevant statute and laws and things like that. Now, the second research question, which was, we saw that chargeability now. Okay, it did a bit better because we were telling it, we were giving in more guidance, what to do, because well it doesn't have a mind of its own as such to know what was taught and what is expected of it. But there was still minimal engagement with the context, and it was still a fail. I failed this time, a 30 out of 100 and still no reference to law. Now what you don't want to do is to go to a lawyer who doesn't know the laws. Okay. Now, the reason why I put earlier, if you have seen Philip was a guy, Margaret was a lady, the secretary was a lady, cost plus, just say gray guy, no gender because it was not specified. Now that's a reason why in law, we often have issues which is if a guy or a man or a woman goes to, we know that the gender bias and pretty much everything right. And even if we look at, I think on the next slide, I've got it. So what I don't later on is, I swap Philip for Alex agenda neutral name. Now we, you probably heard about this that chargeability gets confused with gender neutral names and he's saying that nurses or women and things like this. So, this is actually coming from an, from a previous scenario previous assessment that was given to students where agenda neutral name was used and although it was she she she she she everywhere, students were just saying he he he he he everywhere. And when you look at their conclusion, you can see gender bias. Now, this is actually an example that I got off the off Twitter, which was the paralegal married the attorney because she was pregnant. And then chargeability came with this brilliant idea that, okay, pregnant before pregnant there was the previous one is that Tony but it was still saying paralegal. So it got a bit confused, because they are Tony had to be a guy according to chargeability paralegals. The woman and then it was anyway just arguing and correction and then it was like, oh, how have interpretation does not make logical sense because pregnancy cannot be cannot be for men. You've seen probably there's right on social media. Now, one example that you would probably be more familiar is is to famous people here. We know what happened. Okay, there was a UK and a US case, a UK case where John was a wife beta. His case was just in front of the judge only and he lost. And then he had a fair trial in us. And he was found to be a domestic abuse victim. And that was a fair trial in front of a jury and he won. And I wanted to give these example is obviously genderbasses have we do this even I'm not even talking of AI here. We people do that as well. And of course we could expect. It's obviously AI would do that chargeability would do that. There was a review on the abstract which was why did I only consider gender bias and not the other ism which is not racism, not disability or something like this. That's because the assessment questions that we were using at the like this year actually had a gender bias issue. We didn't have any example that had a bias around any other. That's why we just focus on that at the moment, as I said it's preliminary search and would like to see how this goes as well. Now problem, the paradigm shift. One thing we've found is obviously there's a C type which is please just go when regurgitate everything you know from the book. Okay, which is easy for chargeability to do actually. And then the problem based thing which is more life, real life scenario type. Okay. And we need to obviously be mindful of using an encouraging AI in education, because it's not. I think you guys are saying it's not you know these, we can be talking of technology, but it's not that technology is the ideal solution in every scenario. And then we need to also be mindful of how senior management and force technology. I think you've heard it from those students earlier before and there was another another session earlier from Peter Brian and all. I did tweet about it that you should watch the recording if you haven't. And you should be mindful also of how educational technologies and force technology. We had the story I'm keep, keep giving this example because you were saying, you know, you go and then people were like, Oh, it's all technology. Was anyone listening to what is needed or were we just enforcing technology because we should, we think we should. So if you go on on this module and you enforce technology and you say oh yeah the students can use no problem. Guess what we're going to actually generate a situation where we'll have graduates who actually don't know the law. And then we'll see the results afterwards. When you'll go to buy a house. You you're losing the house then you know she's going to win because she offered 100,000 dollars more. Okay. And we should also be very careful of not setting precedence now why setting precedence we have common law in the UK. So if precedence is a big deal, we still leave and abide by laws that were made, even in trading and commerce every day which we made hundreds of years ago. Okay. And how am I on time I mean, see you're looking at the. I mean how am I on time. Oh, okay. Monday. I could have still stayed in bed to for a career by whatever you want to go into laws or work as a learning technologies into I don't know, business and law, or start to do around that there's still scope for that because AI is not going to make this disappear teaching and grading law I can tell you. Now, I'm going to actually, I have some examples here that I want to tell you through. When you read because a lot of people read information off the net, which very often can be misleading. I have two example here on this slide which is. Chargivity but please U.S. law exam. Yay. Actually, I've had some people, you know I've had people teaching law who are like, Oh, but how about this, you know, judge a bit he can do that now. And then like, did you go and read the research paper. Okay, what says here. In bold it says, uh, chargeability often the bot struggle to spot issues when given an open-ended prompt, a core skill of law school exams. Okay. But you know there was a title which was like chargeability bot clears US law exam and people started to get panic. And then the other example is chargeability passes law school exams despite mediocre performance. It seems that the bolding do not quite appear or not very distinct. So anyway, so the first line is chargeability cannot yet outscore more students on exam, but it egg out a passing rate. You know the grade I was telling 2030 all probably could get is 14 out of 100 now. And chargeability average is a C plus performance fell below the humans B plus average. Alone, chargeability would be a mediocre law student. Now the word that's important here is alone. So if a student goes and uses chargeability, that will still be pretty much a close to a fail or a close to passing, you know, 41 kind of thing. And do you want to deal about purchasing a house or if you have liability, you've had a road accident, or you need to be represented in law, do you want some in court, do you want, you know, students, legal practitioners advising you when they don't quite know what they are advising you about. Now I have three other examples on this slide, which is media misinformation in law application. Now this was actually at Allen and Overy, they use, they use a software so chatbot call Harvey. Now in legal practice, in legal practice, you can't have just anybody give legal advice. It's actually not allowed by law in some sectors, like I do some work around immigration and I'm not allowed to give advice. Okay, because I'm not a registered advisor, as simple as that. So however, what it says is Harvey, the chatbot needs to be supervised by license legal professional. Now it's important now, you know that license is important, not just a legal practitioner, but the person has got to be licensed as well. And then there was another, there's an example. So Judge Galsha, who's from the first circuit court in Colombia, you've got different levels of court in America. So, and he actually used Chargipiti, but what he used Chargipiti for, it was in a court judgment was only basically you'll see the example he used was actually to check if, where is it written, so whether mine is diagnosed with autism or exempt from paying for therapies. So this is a type of question, which is yes, no. It's not something that we expect the students, well, the judge or anybody to give a lengthy, a say type response with logics, things like this. It just basically would look from Google, from its search engine and different texts of the net. Yes or no, kind of whatever is from the net already. And there's another article that says AI is actually in no way replacing the decision of the judge. Okay. What it did here in this case is the judge basically just check if mine is, it was a yes or no answer. It was not expecting more than that. Okay. And actually he likened so white who wrote that he likened to the work done by Chargipiti to that of a legal secretary. Now that's a huge, huge, huge difference because a legal, a legal secretary is not a legal advisor. That's actually, I've also given you some example of the earlier text that I was saying about about Chargipiti clearing exam. Okay. So this is actually a question that you will see it's scenario base, but it's still a multiple choice. So you've got a man was sued, railroad, personal injury suffered. Typical thing that you may hear from life, right? The third, how do you do that? Okay. And then you've got ABCD. That's it. You're not expecting in this question for a huge answer, huge logical, you know, response and things like this. So that was actually the type of question that was used for the bar exam, for Chargipiti on the, on that model bar exam. And that actually, now these are actually elements taught contract evidence, things like this, these are actually some of the elements that you expect someone who is going to clear bar exam to know. And the module that I was telling you the example of Philip and all Margaret and all actually is a contract is in contract. And you would see that there were 25 question of each type. And what it did was it was very interesting actually. So it did well in torts and civil procedure, but it kind of didn't go well, do well in constitutional law, real property contracts and criminal law. And the reason being, I mean, criminal law, if you've heard like a murder case, sometimes that's why there are lots of detectives sometimes working in a case because things can be so difficult to figure out. And contracts, real property and things like this, these are actually real scenarios. You're very, very complex to resolve. And you need deep understanding. And here, sorry, I'll leave the slides as well later on that you can check these details, but these are actually the research. So the title of media is saying, yeah, chargeivity is clearing bar exam, but we need to look at what type of bar exam, what type of question was fed to it. And some of my initial notes, like I was actually looking into, okay, maybe should I look for other things other than chargeivity? Should I try other, you know, but then I drop that, because I've kind of already come to a conclusion based on what I've done is, yeah, as I've put in the slide, there's still a lot of opportunities out there. You can still have a career pivot if you want and go into working into that. And I would like to also keep analyzing other modules as well as you saw earlier in the earlier slide, there were like tours, there were criminal law, there are other things. I would like to see how it goes, because media keeps telling us that chargeivity is going to put people out of work. Is it? And then I would like to also engage the students into understanding, because from observation, we know students have used chargeivity in the assessment. We know it. But you know, if you put it for turn in it, you don't get anything, right? So that's the end and I've put lots of reading. And I've also put two pieces of blog that I wrote. I'm very US in terms of law. That's actually two pieces of work around gender equity in the US law system, which also kind of explains why chargeivity is very biased against a particular agenda. So end of the presentation. Question time. Am I taking question now or later? Yeah, question. Sorry. Sorry. This you're supposed to speak in the mic. What is your institution's advice to undergraduate students on the use of chatty GPT when they're doing these modules then? That's yeah, okay. The last bit that you added, that's the answer. Because institutional advice in general, that's why I was saying we need to know where we're enforcing it. That's why now I want to go and look at other modules as well, because you cannot give a generic thing because there are modules like if I go to the earlier slide, this one, right? So there was there was civil procedure and taught that it did fairly well. Okay. Obviously these are modules at year one and then there's module at year two, there's module at year three, there's module at master level. So we should be very careful. That's also why I was saying we should not just set precedence. We should be careful also because each university teach different syllabuses. You have one more question. And this you personally then, do you advocate the use of chatty GPT as an example of these modules? Are you telling students that it's a good thing to use in specific situations then? Or would you just, would you take a step back and say, ignore it? Do you find it useful for students, I suppose, if it's, if some things are good, but something's not so good, if it's the waste of their time to be. In this one, I would say no. If you were doing a history module, like please tell me, like when I was doing my life in the UK thing, there's something chatty GPT could tell me about the history of royal family or whatever. Okay. But if I want to not create problem later on for my kids and generation after when they'll go and buy the, buy the house and I'll use the house because of your 100,000 more money, I will not create the problem now. There are modules and there are specific type of assessment. So I said type of assessment, they can use it. Okay. But if they are going to rely on chargeability for problem-based assessment, we all, the society is going to be in a big problem because law runs our everyday life in some, in some way. So you had a question? Question for the break. Yeah. Move on to the next Okay. But I'll give you. So keep, take your question, mate.