 If there's, you know, what we tend to do with our recordings is we'll edit them down so the discussion will be edited off. So once we start talking there, please feel free to talk and then we'll take that little bit off. And we'll share that recording with everybody if you wanted to share it with other colleagues or people who couldn't make it today. My name is Jim Turner. I'm currently chair of LEC, which is a special interest group that's connected with the Association for Learning Technology. Let me just share our little web page here if you've not come across as before. We are trying to support learning technologists and other people in trying to improve the research around learning technology. And that's one of the reasons why I was excited to have these two speakers today. Yeah, so we're part of that larger old group. I'm saying I'm the current chair because we are about to launch our elections. I'm standing down in September. So if you like what we do, you might want to get involved at some point and get into that. We also have a whole selection of events. I'll share with you in a minute. Here we go. So we have other events that we're running this month. We've got a couple of events turning up. But I'm sure you didn't come here to listen to me go on about stuff. So I am going to hand over to Clay and Matt to run the rest of the session. I'll be on hand clay just in case, you know, I'll keep an eye on the chat and these sorts of things, but you take it from here, you guys. Thanks. I'm just going to share my screen. We don't usually use collaborates. So bear with us. So yes, I'm Dr Clay Grandston and my colleague over here is Dr Matthew. Hi, Marsh. We both are at Liverpool John Moore's University. And we also teach on the sport business course. And I hope today we're going to, we're going to cover a few things and when Matt and I were going through what we should show around around mastery paths. We thought we'd change this up a little bit. So we're actually going to do one from a student view just to let you see what it actually looks like. Because every time we try to explain what one of these are, we end up making it sound far more complicated than it actually is. But essentially is a flexible learning pathway and we'll discuss about some of those other types of pathways that you could possibly use, I guess. But also come at this from a teaching and learning point of view as well and get some justification to why we've used a mastery path. We'll also go through a little bit about the process and how it actually works in practice. We'll also show you some of the results of the mastery path. I think currently, Mattie, you might be able to correct me here, but we've got data from over a thousand students now. Yeah, and it's across. It's across three different courses as well within the business. So it's a big cohort. So we've been doing this for around three years, but we'll show you what a mastery path is at the moment. They're just words, aren't they? And we'll give you some reflections on the different iterations that we've done around mastery paths to hopefully by the end of this you'll have a good idea about what a mastery path is. And hopefully we'll be able to show you the impact that it's had on our students learning, which is a tricky one to measure. But if we just start now, I think Mattie, did you want to go through this first and then we'll jump in at the end? Yeah, I guess I think just first of all, I don't know if your slides weren't moving on there because they seem to be frozen for me. Okay, I think you might be sharing the wrong screen. If you've launched a slideshow, it might be running. Okay, let's try it again. If we just do that, I guess we'll have to, right? Yeah, yeah. Go on, Mattie. Yeah, so I think, no, it's not quite clear. I think, yeah, so when we usually start these types of presentations that myself and Clare have done previously, what we always do is try and I guess bring it to life and give you a sense of why we come to implement and how we come to think about the use of mastery paths. And thinking about the start of this journey really comes from a couple of different strands of thought that myself and Clare have had based on our reflections whilst teaching over the last, well, three, four years in particular when we've been collaborating with each other. And I'm sure you're all aware that particularly at the start of first year, student transitions are probably one of the most important times in a student's lifespan whilst at university. And often what tends to happen is that how they settle into university often impacts upon their success moving forward, not only in the first year, but also moving into the second and third year. And there's an increasing amount of research that's coming out that talks around the fact that actually there's an increasing number of students that are finding the transition from whether it be from higher education to higher education or alternatively coming from the world of work into higher education is something that is increasingly difficult. And that can be for a variety of different different reasons from moving away from home and therefore kind of trying to settle in and get accustomed to to that particular city or area or town in which they are residing in. It could be to do with the actual academic success as well in terms of being a step up potentially in what is expected of them from from what they previously done to the higher education setting. So I guess a multitude of factors that impact on that increasing number of students that are finding the transition difficult. And what research is also highlighted is the the within that transition period. What tends to happen is that you find yourself being in a position whereby students are not necessarily aware of what's expected of them until their first form experience of assessment happens and occurs within the higher education setting. And from this what what we found as a as a team, particularly myself and clay is that when looking at that transition period what we've found is that often students are inundating those with emails and overloading those emails about certain things we might potentially covered within the induction period that every student goes goes on within the first week couple of weeks whilst at university. And within that what tends to happen is we cover policy around different areas of the institution and students often find that quite overbearing and therefore constantly asking questions through emails in terms of university policy, or any other course matters or particularly in what we're covering the mastery path as well in terms of referencing. So we find ourselves kind of being in a position whereby a lot of we're having a lot of time spent elsewhere actually having to deal with with these concerns that students are raising around things that you potentially already covered inside the classroom. And I think what's important when thinking about mastery path which we'll come to in a couple of minutes time is that students often have different paths to learning in the sense that they all have different skill levels, they'll have different expectations and experiences and they'll all have a different way in which they learn. And it's about trying to cater for that individual need, which is becoming more prevalent in terms of trying to make sure that students are at the fore of all the teaching and learning that takes place. However, with our course in particular, and I know Clay will test to this, we've seen within the last five years our student numbers go from being anywhere between 40 students all the way up to now as an in-court in intake for the first year exceeding 100 to 110 students. So it's thinking about how can we create that individual learning need and make sure that we're catering for every student, but do it in a way whereby it's not overburdening our own time because of that fact of trying to need to support each student through that individualised process. So I think that the key thing here that we're trying to get across is that we actually started to think about that and go, well, from these student reflections, we're getting to a position whereby maybe some of the things that we've potentially previously done as a team might not necessarily reflect what needs to be done now moving forward. Thanks, Matty. So we, I guess we're getting closer, aren't we, to what mastery paths are, a little bit background there. One of the ways that I sort of came up with this, I guess, is I was doing, as most of us have to, training video. And as I'm sure you're aware, doing lots of training videos over the years, I thought, how could we use something like this? And I thought back to this, sort of choose your own adventure books, and I don't know if anyone remembers choose your own adventure books, but I mean, I'm not that old, I'm only 40. 40, but I mean, I did, I did use choose your adventure books. You haven't played them. Essentially, you choose a page and you go down a different specific pathway. And every time you read the book, you could essentially have a different path, but get to the same goal, which is the end. And so we were thinking about this and we were thinking about how we could do something for, as Matty has mentioned already, some of the tasks that perhaps the students should be learning outside of the classroom. And certainly one of those would be referencing. And also we've done a mastery path where students have to look at the systems when they first get here. And we'll discuss about sort of the suitability of mastery paths as various points of assessment as we go along as well. But essentially what we wanted to do is create something that students could get on within their own time. And obviously we would incorporate it into a synchronous session, live session. But these usually start with, here's a resource, go and read this resource, and then there's a quiz. And once the students have done the quiz, then they move on to the next sort of layer. Once they've mastered that particular area of knowledge, then it unlocks another quiz or assignment for them to do with some more resources. And essentially what we've created are sort of, I guess, more linear versions of mastery paths. We'll discuss how we've started to move away from the linear a little bit, and some other aspects that we've included as well. But I think as well, again, I said at the beginning, one of the things we'd like to do is actually show you one. Because again, we've talked around it for a couple of slides. And don't worry, those slides down the slide, there's not a long set of slides. We are going to whip through some of these. So we thought one of the best things to do would be to actually show you what one of these looks like. So we've had a little play around and we've created one. Hopefully you can see this. Everyone see that OK? I can see it now, yeah. Excellent. So this is in student view in Canvas that we use. And essentially, this is the LJMU systems ones. And we have a nice sort of preamble, I guess, to instruct the students on what to do. And then the students would then click on one of these, which is setting up the LJMU systems telling them exactly what they need to score to be able to move ahead. And there's a little bit of blurb at the beginning. And then this one is simply all around the academic calendar. So students would have a look at the academic calendar, which is a very important part of them being with us. And then there would be a set of questions that they would have to answer. And once they've got three out of three correct, then unlocks the next part, which is setting up the LJMU systems. OK, so we've created one to have a little go at. We'll have a look here. So again, we've got some instructions or phone instructions, I guess, more than anything. And with my profile, we've got Matty's profile and obviously the course profile. So if you were a student, this would be about who are these people and what do they do. So we'll just show you how it would look. Again, the instructions stay at the top. And then we've got some questions here. So I wonder if anyone's been listening. What is my second name? Pop it in the chat if you know, or we'll click on the one that gets the most comments. Anything coming through yet? Who cares maybe? Or Clay Pot, if you want to be amusing. Anything coming through? We've got two. We've got Jim and Laisley that's commented. OK. What have they said? They've said Gransden. OK. And what program do we teach? Do you remember what we said right at the beginning? Again, just type it in. Anything coming through, Mat? Not as of yet. We have Chantel said Gransden. Oh, Mike said B. So we've got B. So I guess that would be sport management and business. Jim said sport business. So we've got a combination. We've got Chantel that said B as well. So that's sport management and business. OK. OK. OK. Here we are. So then this is what the students would say. Unfortunately, we didn't get this right. However, what we can do within this as you know, if you've played around with quizzes, we can put some feedback on some of these. And that's what we've done here. So we don't partly correct. Go and have a look at the links to our profiles again. So hopefully the students would go back and look at each sport, but it's actually sport business. So once the students have seen this, they can take the quiz again and have to keep taking the quiz until they demonstrate that they haven't mastered this particular whatever questions it it's on, I guess, but mastered this particular area. So if we do it against sport business. I'll add I put Klingon on Beyond. That is actually a course or was a course in America. Klingon language, just to throw that little tidbit in there for you. So what we have here is we have the attempts and beauty of this is we can have a look at each one of these attempts and we can see how many times the students attempted it, how many time, how long they've taken to attempt it. Certainly we'll talk about gaming of the system a little bit on a little bit later on, but there we are well done. And we click next and what happens is it now shows us the next one, which is the life and times of Matty. Don't worry, we're not going to carry on. I'll just give you an example. And what happens here is there would be another quiz with some materials. I'll just come out of this now just to show you what these would actually look like. Certainly one of the larger ones that we've done. So you can see this is the one I showed you before, setting up your LJMU system so the students would have the academic calendar. And once they've scored three, they would unlock this and they would unlock all these materials as they move along. So this is all around, you know, students getting used to the systems here at university, hopefully with that transition as well. So I'll just switch back. Hopefully that gives you an idea and certainly hopefully that makes sense about what a mastery path is. We always find it difficult to sort of explain to this until you actually show someone how they work. And as I said before, these are some interesting materials. But we will talk about how you could use them and how we're moving away from a bit more of a linear use of this. So when Matt and I sort of came up with this, we found some interesting papers and just some interesting materials out there on flexible learning pathways. And most of them start, as I say, with the quiz I've shown you. Now this one from Gordon we had a look at. But one of the issues with this was that depending on how the student did in the initial question, say if they showed mastery of a particular area, then they would be given harder questions, whereas others who perhaps didn't do so well in the initial quiz would go to the easier questions. But that would mean, of course, as you can see as you work down, the students would be disadvantaged and would end up doing only a threshold or the easiest sort of assignment. And we felt that was a little unfair. But nevertheless, it's interesting to see how you could use something like this. And I'll show you a couple of others as well. So this is one that we certainly, Matt and I had a long chat over, which is in a really flexible pathway where there would be different assignments after the initial one. And depending on how well the students done, they could navigate their way through this. And what will happen is there would be, it looks a little bit like blockbusters, if anyone remembers that, but they would, the student could go through this and different students might have a different journey, but they all end up with the same equivalent assessments towards the end. And one of the things we've done is that the summative assessment is the one at the end of these sort of layers that we've created. So the students have to go through, certainly for the referencing one, they have to go through three quizzes before they unlock the actual summative assessment, which is worth marks, as we know, students cover quite a lot. So this is essentially what we did is we got students, as you've seen, to engage in content, to undertake the quiz, to hopefully go back and engage the content if they don't understand it, which unlocks the next layer where students again engage in some new content, undertake a quiz and move down all the way through to that final quiz. And again, we've picked, or we've chosen so far, should I say, areas for students to do a mastery path on things like referencing, things like we'll go and have them come back and read through them and tell us, you know, what time the libraries are open or what's the telephone number for the wellbeing department in quite a linear fashion. We've started, we've created, as I say, a referencing one and sort of setting up LJMU systems and we've managed to get this across the business school in about three different subjects and we'll show you some results as we move along. But hopefully that makes sense. I hope it does anyway. We always, as I say, we always struggle to be able to explain what they are until you see one. And this is the sort of back of house. They're quite easy to set up. I said they're quite easy to set up. They take some time to set up, but once you've set them up, and again, this is in Canvas, you can quite easily add. So this year, for instance, we have the architecture for a mastery path. So we can go back to them and add different bits of feedback into them. We can update certainly any of the referencing, for instance, or videos that the library has updated. Make sure that we have a working mastery path, but it is quite easy after you've set one up. It's quite time-intensive at the beginning. Again, the architecture setting them up, discussing with your team about what you want them to actually go and do. But as Matty said, right at the end of the first slide he talked through, with numbers growing larger and larger, we can get these sort of, I don't want to say they're menial tasks. They're not the very important tasks at university, but we can get the students to go away, learn outside the classroom and apply what they've learned from inside the classroom to something like referencing, and it gets marked automatically through the system. So that's obviously really useful for colleagues, but it also means the students can go back through and obviously come and see us afterwards. But for a skill that's so important as they go through university, we found it's worked quite well. So Matty, is it back over to you now, I think? It is, it is. And I think adding on to what Clay's alluded to there, one or a couple of the reasons why we actually engaged in the use of mastery paths are the fact that, as Clay already made aware, whilst in the short term it might be a bit longer to undertake and put that architecture into place, actually over the long time period, it would save lecturers or any teachers or many members of staff, vital time in terms of marking because what happens as we saw within the example that Clay alluded to, it gives automatic feedback depending upon how well a student has done with a particular question. And it gives you that immediate automatic feedback that the students can see it tangibly, but it can then go back straight away and have a look over the work again at a earlier period of time. I think on top of that, obviously we're in a situation now whereby learning doesn't necessarily have to happen just inside the classroom and we're at a stage whereby students might potentially have other commitments that may mean that they might not necessarily be able to attend every single session and mastery path, what that does is it enables students to engage with the content at a later stage, whether that be at 3 o'clock in the morning or 9 a.m. in the morning, whatever time shoots them to be able to 5 a.m. even yet be able to learn valuable content that they can then take into the classroom and develop their work moving forward. I think on top of that it also gives a personalized teaching experience because ultimately that feedback that you get given is immediate but also relates to and corresponds to the type of answers that you've been given. So say for example you've got one particular answer wrong or a student got one particular answer wrong and got another one right that would completely change the type of feedback that will be given to them as opposed to somebody else that got the first question wrong or the second question right. So it gives that personalized teaching experience on top of being able to do it within their own time. I think on top of that it's actually quite interesting to see there's been research that has been done previously with other members of staff and then alongside ourselves that highlight actually that the pass rates and failure rates on these shows massive massive differences and discrepancies and we'll talk through what that looks like in our case in a couple of slides time. What's also important here is that we actually can use the data analytics that are already embedded and built within the virtual learning environment of Canvas itself to be able to see what the students are doing in terms of how long they're spending on certain elements of that mastery path. Being able to understand what they're actually understanding and not understanding to then be able to tailor our approach even further when we come into inside the classroom. So there's a plethora of reasons through literature that talk around why implementing something such as a mastery path would be effective in this particular situation and scenario. So I think looking at some of the results that we've had in our in our study that we've done so far and I guess I'm not going to go into detail of all of these but what you can see is that there are three courses that within the first year implemented mastery path. Unfortunately we don't have any of the data available for business with program or the marketing program for this year but it's still something that they're continuing to run to this date. However, what's what's interesting that we can see is that really regardless of whether the assessment is summative or formative particularly the LJMU systems there was a equal number of students that were really taking taking part and engaging with those with those mastery paths. So sport business is 86% with business with it was pretty much 89% so we can see that the students are actually engaging with that content regardless of whether it would count towards a particular assessment or alternatively was something that was there just to support them moving forward with other alternative pieces of pieces of assessment. I think in terms of referencing as well and I know it's something that we're going to come on so Clay wants to go down to the next slide is that what we've actually found and this particularly rage to sport business data is that whilst referencing marks for the first essay has remained similar what has actually changed as you can see in the table below is that those that would complete the mastery path compared to those that don't there's a marked difference in terms of the number of marks that are associated with referencing so to give you a little context around this we get our students to within their first assessment have to obviously reference in which that accounts for 20% of that overall assessment and we actually found that all those that completed the reference mastery path particularly in this year there was a there was a five mark difference between those that did and didn't complete the mastery path so you can see that this implementation of this program and of this overall process is actually making a mark difference in terms of students level of understanding on how to reference just to add to that Matty we can see that before we implemented the mastery paths that we were seeing sort of similar marks with the students so we haven't taken anything away from what we're delivering by implementing a mastery path what we're actually doing is we're sustaining the same level of marks for those that have completed the mastery path against those that would have perhaps done a two or three hour session or longer multiple sessions on referencing in the classroom should we move on to the next one Matty? Yeah, absolutely so yeah I think just some other notable results that need to you might want to be aware of in terms of why do you think this has been suitable to continue for the past couple of years is that the module of evaluation surveys that we have that have implemented the use of mastery paths in terms of the LJMU systems which comes and as part of our academic digital skills module and the referencing mastery path that comes as part of the first module that they do within sport business which is which is all to do with kind of commercialization of sport but we've seen overall is that actually found that the the digital resources that were employed which obviously the mastery path forms part of was something that was very easy to use and that the actual assessment tasks again which connected to the mastery paths were clear for them to be able to have a good level of understanding to be able to do this independently outside of the of the classroom itself now that's not to say that we don't provide some content inside the classroom it's not as Clay said this notion of replacing what we've already done it's supplementing what we already do inside the classroom to allow them to engage and get a more in-depth understanding of that within their within their own time and I think the quotes that provided both in the 22, 23 and 23, 24 kind of module evaluation surveys actually actually epitomize that so I think the 24 comments is something that is particularly interesting to note in that actually they found it challenging and something that really stretched them intellectually to be able to do but at the same time as a result of being challenged they felt that they've actually got better with that particular concept which in this case is referencing which allowed them to be able to then I guess have that as a stepping stone to move into the other assessments that are part of the diet that we have in the course overall I think a bit of reflection here Matty I mean we always try to reflect on these as we make them every year I think one of the first issues we had was when we made the first iteration is we didn't put enough feedback in really so we've had to go back and put some more feedback in and try to really as I showed you in the example there try to point students in the direction so they don't become too disheartened and actually you know tell them that they're engaged with the materials that we've told them to engage with so that was an issue I think one of the biggest issues we've had though is the second one is students gaining the system so certainly in the first iteration what we found is that students were spending oh I don't know 5, 10 seconds on each one of those quizzes and then they were marking down they must be marking it down remembering which answers were wrong and which answers we've had to look at and try to find the tools and the materials that we provided with them provided with the questions I should say so we've sort of changed what we're doing or changed the way that we've constructed the questions I guess is we've moved to creating a much larger bank of questions so for each one of these sort of resources that will get them to go and they're picked randomly. So there'll be three questions that are randomly selected. And for the final quiz, certainly for the referencing one that we do, we actually get the students to type in the answer, which has certainly changed the scores from being very, very high to being a little more challenging and seeing that students are slowing down and having to actually engage with the materials. They realise that they can't gain the system as much. So that's certainly one of the biggest changes and biggest pieces of reflection that we've had to do, certainly from the statistics or the analytics, I should say, that you can see in Canvas is seeing those patterns of behaviour that the students are adopting. We obviously reflect on the, go on Matty, go on. I was going to say, sorry, on top of that, I think that then goes back to the comment that was made previously around the differences in marks in that we can see that by having those blank questions that actually that has made a massive difference in terms of students' level of understanding and that those that have engaged with the mastery path have then been able to reference more effectively than those that haven't because they haven't been able to just kind of gain the system as you've already alluded to, which is what happened in the first iteration. We certainly did reflect on the accessibility as well, didn't we Matty? We're quite lucky here that our students can pick up a laptop or perhaps use their mobile phones or any device that they need to, and we tried to make sure that it was mobile friendly as well, to which it is, and we haven't had any issues so far. So accessibility seems to be, I'm not going to say we've cracked it, but certainly it's not an issue that's been highlighted to us from the students. And the last one was all around time resource and understanding really. As I mentioned before, there is a little bit of time and labour that you need to do beforehand, usually over the summer, but once it's set up it sort of runs on its own. And it's difficult to control the learning experience once it's published, and this is something that we reflected on and certainly this year we're going to have more sessions where the students sort of can do or start the mastery path and ask questions. We've had colleagues who've done them in sort of programmes of 300, and the students really enjoyed it, and there was a sort of bit of competition in the room, which was nice as well. And I think one of the biggest, one of the biggest sort of reflections we've had is the idea of using these mastery paths, perhaps in other subjects outside of, let's say business, and certainly it does lend itself or seem to lend itself to the sciences a little bit more, or you can imagine asking questions that have a specific answer, perhaps in similar English language, and I've seen examples of this used in English language, it is very different and it is certainly less linear, shall we say. But there is that, I don't know what sort of programmes some of you teach on or taught on, but some of you might be thinking, well yeah, this could work for my students, some of you may be thinking, well this just wouldn't work at all. But that's something we've reflected on, sort of the adaptability of something like this across other programmes. And one of the things we've done is, as I say, we've created a more robust set of questions as we move forward, one of our colleagues this year had our students read a report and then the students had to type in the answers and pretty much all of them were type in answers and we saw a much wider broader range of results than perhaps the sort of more multiple choice questions that we've used in the past. And just really to sum up some concluding thoughts I guess, we've been doing this as a say for quite a while now, I think three to four years now in various guises. And we've been lucky to be able to move past our programme and into the different departments within the business school. We've also done some work with Vietnamese University, the Hanoi University of Technology. And I guess some of our concluding thoughts that Matty and I always have around the use of something like this or even I guess technology and learning is does it facilitate the learning inside the classroom and in this case it does and it certainly facilitates learning outside of the classroom and where those two things take place, we try to sort of, this is more the hook I guess for academics or colleagues, does it same time and resources where appropriate. And this seems to be the big hook for our colleagues. Certainly when we've shown colleagues what we're doing, certainly in the 300 plus business with course, they've been, they've loved it because it means that they can apportion a little 10% or a little 5% to a mastery path. And it's automatically, automatically marked, but it also gives them a little bit of space in those sessions. Whereas I said before, you might do a couple of sessions on referencing or you might have it sort of a recurring theme. It gives a little bit more space to perhaps do some of the other aspects that you want to teach the students. And I think Matty, that's really all we, we didn't want to go on too long. I feel like we have gone on a bit. But you know, I guess we'll open up to some questions in a second. But if you're interested in any of this, we have a paper that we've just published probably September, wasn't it? Oh, it's on the screen, September. There you go. And you know, if you're interested in any of this or you want to collaborate at all, or you want to know more, or I don't know, just get in touch with us. We'd love to help more people and share, share this as far and wide as we can. Matty, do you want to add anything to the end there? No, I think, I think as Claire just alluded to, it really is something that we're really interested in, in developing even further and making it into something that is used cross universities, because we've seen the value in it for our students. And it would be great to kind of allow that to be transferred over to ordinary institutions as well. Thanks, Matt and Claire. That's a fantastic presentation and a really interesting example of trying to use the tools at hand and trying to also measure the impact that they're having. So obviously, if anybody's got any questions, you can drop it in the chat or stick your hand up, but I'm bursting with questions too, okay? So on one of the slides, or here we go, really interesting, we are doing something similar using quizzes to develop student confidence, your skills in life sciences. Leslie, do you want to collaborate? Can you come on the mic now if you do or share your email, if you don't mind sharing it? I'm sure there might be a crossover here that might lead to something interesting. Leslie, you're on. Hi, can you hear me? Yeah. Yeah. Hi, Leslie. I'm actually the e-learning officer for our life sciences portfolio at the University of Glasgow. The principal academic is Dr. Kirsten Knox. And this is something that we've been developing just over the last few months. So it's brand new to us. And of course, in life sciences, it is a new subject. And we find that students are, it's confidence, they don't have confidence in their own university skills. So we're looking at using RVLE as model. So functionally, it's the same as Canvas, really. But so we've developed a bank of questions, and we've designed some quizzes, I going from an easy level to a more difficult level at this stage. And really, we've just gone through our first run of it, and we've still to evaluate it to see if it has impacted on students. Exams are coming up. So we might actually have some data around the results of the exams and compare them with the logs and the completion of the quizzes to see if it has had an impact on the students. That sounds really interesting, Leslie. Clay, do you want to, or Matt, do you want to get back on that particular? I was just going to say, I know we use Canvas and master paths is obviously something you can turn on and off in Canvas. But I know that Moodle, you can do something similar. I didn't mention it because we've done lots of iterations of this, but we've done an iteration of what I've done, iteration of my paths where the students, if they didn't get the questions right, that we created remedial materials for them to again help with that confidence. And obviously, the students that did well enough move on, but those students that keep getting the questions wrong and are really struggling, we created extra videos and resources for them. And again, that's that more kind of less linear, I guess, mastery path. But as I say, you can do that in Moodle as well, but it is obviously the setup is slightly different. Yeah, we're kind of basing it on the completion and restriction. So complete this, we move on to the next kind of thing. I've also been experimenting with the close question type. I've been thinking that the close question type offers us the opportunity to build out worksheets, online worksheets that would also maybe help facilitate students working through and getting immediate feedback in their own time. Yeah, I think that's one of the beauties of it, is if you can work it into a session, students can go away and they can get that feedback nice and as Matty was saying before, automatically. And it's certainly if they get the questions wrong in a closed question format, then you can, if you went into real detail, you could say where they've gone wrong or or point them to other resources, perhaps to help them get the correct answer. So yeah, I think it's probably, as we were saying before, it works well in sciences and with certain objectives in mind, I guess. Sorry, we also have an induction course around digital skills and having seen your presentation today, I'm thinking, well we could really use that as a template in a way to redesign that course because at the moment, for me, it has a feel more of a document repository and it's not really guiding students through the digital skills that we feel they need to be successful in their courses. Sure, Leslie, we'd be happy to share the architecture of what we've done for sure. I'd just like to go back to one of your previous slides, which was the difference between the programs. So you had a diagram of a number of different, you don't have to show it now, but two were incredibly positive. You had two courses up in the 80% of students engaging, much higher than I expected, and then there was one that was more, what I expected, be more in the early 50% and stuff like this. So have you reflected at all on the difference between those two adoptions there? Yeah, I think when Matty was talking through these, we didn't want to go into too much detail, I guess, but some of them were formative. So when we originally started doing mastery paths, we incorporate them into courses and they used them as a formative assessment. And then the following year, it became a summative assessment. And so we saw some differences there, obviously, as you'd imagine you would. However, I know that there was some, I don't know the best way to put it, but there were some issues in one of the courses. I think there was some staffing issues, et cetera, which potentially impacted the percentages on one of those programs, I think at the time. That's right, Matty, isn't it? Yeah, and I think one of the key things that I probably missed out on that is important, as I said, is that with these mastery paths, it wasn't just a situation whereby it was, okay, here's a mastery path, go away and do this by yourself. There was an element of that these were embedded into the classroom itself, so that we would provide some content that related to that and then said, okay, as part of this now, here's some more information on this mastery path that actually, for us anyway, and I know for business with accounts towards part of an assessment which due, whereas I know for the one that potentially had the lower percentage, they didn't integrate it as much into the classroom setting itself, which thereby kind of just left the students to their own devices that was, oh, by the way, here's a mastery path, go and have a look at it if you want, whereas the other two in terms of sport business and business with, wear very a bit more hands on with it and go in, you need to do this, we're going to start this now, but then you need to finish it in your own time, so there was a little bit more kind of directive teaching there, as opposed to that kind of, okay, go away and do this by yourself, which happened with the small percentage group. Jim, just to add to that as well that these numbers aren't the students that have completed it at the end of the course, this is just the first time that they attend, first attempt numbers as well, so those numbers actually go up in the referral period as well, but we just wanted to put up the first attempt ones, because we found them to be more interesting, I guess, obviously with a referral then they do jump up by sort of 10-15% in some cases, so the ones that are in the 80s jump up into the high 90s. Yeah, because it is a well-known sort of trope, I suspect, that when we try and develop something sophisticated like this, there's academics that will just say they're not going to do it, and in some cases that is so, so there's this magic key or part of the solution is the level of integration that you're putting in there. I think with that, Jim, is that we can just cut and paste, once we've created it in our own canvas console on VLE, we can just cut and paste it into another, and as I say, the fact that it marks itself certainly is a big hook for other colleagues, certainly with the largest sort of 300 students are. And I know you're both really busy academics and very busy developing this as well, but with the further research of this, the opportunity to try and learn what the students are struggling with, you know, from this type of thing, is that something you're going to look at, or is that something that offers itself very easily through the analytics? So a mixture of analytics is being reviewed at the moment, but we're waiting for our ethics form to come back, and the next step is to do focus groups across the business school in the different subjects that have done it. And again, each subject or program does it in a different, slightly different way. So that's the next step really, is to do some focus groups with students, and also sort of discuss how it helps with their transitions from higher education to higher education. You mentioned about game in the system as well. Speaking for somebody who has to complete various different online modules around bribery and these sort of things, this is perhaps common across higher education and most businesses, you know, that you get weary of that kind of interaction. Is that something you're concerned with? Because, yeah, is that something you're concerned with? Well, I know it's Matty and I were talking about this just the other day. Our students only do two of these at Level 4, so our first year. We only have one across the whole program in our second year and we don't do them in our third year. And I think that's something else we could have spoken about today. But again, we would have ended up with 300 slides, but the appropriateness of this particular assessment with the levelness as well of that particular year. Obviously trying to get students to be critical of the top end and sort of Level 6 or third year. This perhaps in the iterations that we have already doesn't lend itself very well to that sort of level. Whereas at Level 4, we're asking them to go and sort of explore and answer questions on. It lends itself very well to that and where our colleague who in the second year has gotten to read a report and to, you know, do a little bit of maths and be a little bit, you know. Go that little step beyond. It seems to work well as well, but we haven't yet done it at Level 6. So as Matty mentioned before, it's part of a sort of varied diet of assessment that we have. But I think, yeah, there would, if we had this in every module, I think, or even more than the amount I've already, I think there would be a little bit of fatigue over this type of assessment for sure. Yeah, but it's a lovely experimental scenario, isn't it? You've created for yourself and one way of exploring that is that idea of gaming and you'll see possibly you can, through looking at the data, you might be able to spot those types of behaviours. I don't know. Whenever I've sort of clicked the buttons on the bribery module, I've always thought, do they know do they know whether I've done this or whether, you know, and I'm sure it's perhaps an interesting area to help the whole wider world of learning technology understand when, what trips people into that sort of behaviour. That's an interesting one, Jim, actually. I mean, it's one that we've looked at a lot and we have sort of looked with, certainly in the first and second iteration, really looked into some of the students that have taken a very small amount of time, but actually have maybe done eight attempts on three questions. And that's what got us thinking about, well, how do we stop this from happening? How do we stop students from essentially just clicking every button in a random order until they get the right mark, until they get the sort of answers? And there's no learning taking place there or very little learning taking place. So that's one of the reasons we've created the banks. And that's certainly one of the reasons that we've, we're moving towards students typing the answers in or, you know, and that has certainly stopped a lot of the gaming issues so far. Not all of them, but it is a really interesting, and it's certainly actually something we'll probably add to, to the focus groups, I think. Yeah, yeah. You know, that's a reoccurring theme, isn't it? And I have to admit, I've never looked at the literature, so there may be papers already in existence about the topic, but getting further evidence on what trips people towards that type of behavior. And, you know, I don't mind having an interview with you about my experiences in the bribery module. Yeah. But it's that, it's that kind of thing, isn't it? Yeah. Okay. Yeah. So there's three more minutes to go. I think maybe this is the right time. Oh, yeah, mastery paths. Okay. A little tip for you there. We will be creating a blog about this, which will be shared on the alt site. So you'll be able to catch up with Clay and Matt there if you want to hook up with them again and find out their details. And yeah, and hopefully we'll see a lot more research in this area coming through and helping us understand that. So thank you very much, Clay and Matt. And I don't know if you want to stay on the line for a little chat afterwards. I don't know. Yeah. Oh, Leslie, maybe in contact. That'd be good. Thanks, Leslie. Yeah, if we need any help with anything, just just reach out to us for sure. Yeah. I'll do. Thank you. I really enjoyed the session. Thanks Leslie. Thanks. Thanks, Gav. Cheers. Gav's just going and let's turn off the thing of me. Hold on a sec.