 Thank you very much and thank you everyone for coming to hear us talk. So I'm Tim Hunt, this is Chris Nelson. And so yes, we wanted to make some changes in Moodle, well we originally thought Moodle 4.1 and in the way of these things we haven't finished them all so some of them will come later. The way this talk is going to work is first of all I'm going to explain to you why we're going to be doing these things but it's all going to be a bit mysterious because then only in the second half of the talk Chris will explain what it is we're actually going to do for these reasons. And even before any of that I just thought I'd make sure you understood what sort of organisation the Open University was because I think the context matters. So this is the Open University scene from above or at least it's the central campus of the Open University but this really isn't what the university is because we're a part-time distance learning institution. So all our students are really studying from home all around the country wherever they happen to be and of course most of that study in these days takes place in a really big Moodle site. So in our Moodle courses we have online teaching texts but they're not really texts because these days they often contain a lot of multimedia, a lot of interactive activities. Some of those interactive activities are questions from the Moodle question bank that have been embedded in the learning material. We've got a plug-in for that that you can get from the Moodle plug-ins database. And when I say students are studying alone at home that's not really true because also students are put into groups of about 20 people with an assigned tutor who will grade their work, answer their questions, give them some online tutorials. But in this context something like the Moodle quiz is really, really important because for the most of the time they are studying at home so either embedded interactive quiz questions or quizzes are really important because they're one of the few ways that students can do something and then get immediate feedback about whether that was right or not. And I think it's very well known and documented that feedback is one of the most important things when you're trying to learn something. You think you may have got it, you want to try and you want to find out if that was right. Moodle and online technology can do that. So we've been using Moodle for a very long time since Moodle 1.6. We've seen it involved over the years and the growth has just grown and grown these days. Next month all is the big period for new modules starting for their next academic year and during that each day we'd expect to see about 60,000 people log into our Moodle site. Over each year students answer about 9 million questions in our quizzes and that makes up about 700,000 quiz attempts. So it's one of the reasons we want to invest in these technology. We use it a lot and if we can make it better for our educators that's going to help our students learn and succeed. Now more specifically why we're doing the particular changes that Chris is going to tell you about in a minute. Well we have a very personal institutional reason why we need to do these things now but then as we'll discuss we're not just doing these weird things for our own weird reasons. They're actually if we do these things hopefully everyone will appreciate them because I think they're generally useful. But before we had Moodle and there was life before Moodle we had developed our own in-house assessment system. We'd actually developed it and redeveloped it over a number of years. When I first joined the university it was the world where we were putting software on CD-ROMs and sending it out to students and quite a lot of that software was sort of question packages as we called them. Because again that was questions, student answers, it gets immediate feedback. We've seen the benefit of that for a very long time and then that got redeveloped into a server based system, an online testing system, there's a screen grab there called Openmark, and it was extremely flexible. We've got online teaching material sort of production specialists at the Open University who work with the academic staff to produce the learning materials. So we have the ability to produce these very flexible questions but each question had to be kind of individually scripted by someone with some developer skills. So very flexible but quite a high barrier to entry to creating questions. But you know it was a very powerful and potent system we liked it and already over the years we've added many of the features from Openmark to Moodle. I'll come back to that in a minute. But it's getting to the point where Openmark is getting very very old and we really need to get rid of it now. So the current push of the last, these are the kind of last few Openmark features that Moodle can't do. So I said already this is not a new thing, sort of taking Openmark's features and putting them into Moodle. So these are some of the things we've done already in the past. One of the big ones was in about, was it 2010? We Moodle 2.1. We added the interactive with multiple tries behaviour to Moodle. That's the one where if a student answers a question and gets it wrong they get some feedback immediately and then they get a try again button and they can try and change their answer to that one question and then they get it wrong again. They may get a third try where whether it's two or three or four tries or have many that's up to the teacher. And then if they get it wrong too many times rather than just leaving them frustrated will eventually just show them the work solution. So that multiple tries behaviour originally came from Openmark in fact it goes back way further than that. The first time we did that sort of thing was in the 1970s running on computers that looked like that, sort of dumb terminals attached to our mainframe. So that kind of pedagogic functionality in computer software we've been doing for a very long time we just keep redeveloping it in new systems most recently in Moodle because we know it's effective. A lot of the question types we've developed and put in the plugins database like our multiple response variant and our numeric question type variant and the combined question type are all inspired by things we knew people did in Openmark that were useful and we redeveloped them as Moodle question types so you could create them now just by filling in a form. And that also includes all the drag and drop question types we've developed and added to standard Moodle about five years ago now I think. And in the space of question types I mean some of the things that used to be unique to Openmark we haven't had to do anything about because other people have sort of developed amazing things like the stack and code run of question types ordering question type. So obviously from this great open source community we've got a lot of functionality we kind of needed but it was there anyway. And I've mentioned earlier we have the ability to embed questions in our content Openmark had that first and that was part of the inspiration for doing it in Moodle. But although we've put a lot of the Openmark features into Moodle there are still some gaps that Chris will tell you about in a minute. But the point is we're at a point in our history where if we can get Openmark switched off there are a lot of the modules that use it a lot of the courses that use it are ending in a few years so if you can get these features into Moodle now then when they redevelop those courses we can wean them off Openmark and finally turn it off and no longer have to maintain it. But I've said this is our personal selfish reason for wanting to do these developments right now. The main reason we want to put these developments into Moodle are that from reading the Moodle forums from coming to conferences like this and talking to people about what they need in Moodle all the things that Chris is going to tell you about are things we think everyone wants or at least most people want. And as I've said we use the quiz a lot that's why we're prepared to invest in it. And I think I'll stop there and hand over to Chris to tell you what these mysterious things we're going to do are. Hello everyone. So I'm Chris Nelson I'm a Product Development Manager for Digital Assessment at the Open University UK. Apologies, I can't be at this author. I thought it was better. It flew in yesterday and the flight just did something to me. I assume we've got a bit worse. Apologies about that. I'll try to be as clear as I can be. So good news. We're building loads of really cool stuff for Moodle and we're getting it. Thank you. So the first one is for Moodle 4.1 and this is to do with language override now. This is the score of languages at the Open University that we're really keen for this in sort of level two which is year two or level three which is year three and the postgraduate courses they want to have French exams that are fully in French. Obviously we're a UK university so most of our settings are for English UK but the ability to flip assessment or forums or anything into the language being taught is a really important thing for us. One of the things to note is that as soon as you set this setting in the common module settings and press save it transforms the entire activity including the settings page into that language so you should be fluent in it if you're messing around with this setting. One thing to notice is of course there's nothing clever behind this it's not auto translating content so you have to make sure that it's French, Spanish etc. that's being created by an expert. This is a big one I know loads of people have been asking about it on the community for years now. The ability to override the question number within the quiz navigator. This is thanks to our colleague Mammoud who might be in the audience. You're going to raise your hand Mammoud so everyone can see it. There we go. We've embarrassed Mammoud now. So thank you very much and we've sort of designed this so that you can just add one or two levels to it. It's not quite done at the moment so this is a dev screenshot but it will be finished very soon. One thing to notice is that if you have shuffling turned on it just ignores any overrides that you set. We've got no hard limit on the amount of characters but please do be sensible with what you do with them. As I show here we are sort of expecting people just to use it as question 1a or question 1a2 in lower Roman numerals. It does wrap though so it still looks alright if you've got more characters. As Tim's already hinted at one that we're working on for Moodle 4.2 is sectional weight in. When we get back to the office next week we're really going to have to put our thinking hats on and actually start designing this but essentially again this is called languages and if they've got a quiz that is broken down into reading, writing and speaking Spanish we want to be able to assign a certain sectional weight in to those three sections. So now to talk about the question gaps within Moodle quiz. I'm just going to bow through these because of time. So this is the one that we're just about to release. This is a crossword question type. It's quite basic because it's version 1.0. Usual warnings about first releases but there is a preview screen within it. It's very similar to drag and drop in a lot of ways and that makes it very easy to create crosswords. We will hopefully add in extra whizzy features later on in version 2.0. Obviously this is designed for those diagnostics or consolidative quizzes we weren't expected to be in more formal assessments like exams but it's also designed or at least we hope it's going to work well when we start testing it with the embedded question filter which is how we embed a lot of quiz questions into our online learning materials at the Open University. This is record audio video. This is one that we released a while ago. Originally it was audio only, then we added video. Then we added the ability to have customisable widgets and screen capture, screen recording as it's otherwise known, it's the next logical step. That tech already exists out on the web so we're just going to basically add it into our quiz question type. This is an example of how we use it at the Open University as an audio recording tool for the OU. We already have language override on the embedded quiz questions so that is Spanish content within a embedded question that has then got a Spanish language override on the strings. Because it's based on record RTC it's completely compatible with all modern browsers and even works on mobile devices as well which has long been a problem at the OU. So this is one of the newer ones that we hope to start working on next year. We are unsure if we could have a sort of all-in-one back-end and then you have a choice of how the front-end presents itself to students or maybe it might be easy to split into distinct question types. We also are considering whether or not we want to allow single or multiple choice modes. But this is something that our faculty of STEM and School of Languages in particular are really dead keen on especially the sorting. These are examples of current questions that have been produced as bespoke HTML5 activities. So we want to be able to create this in Moodle quiz. This has already been shown on OpenMark and we just want to build a Moodle equivalent. You can sort of do it in the stack question type but we want something that's much more simple for tutors to use. Then finally a name three things. It's a special form of pattern match which would be more holistic when it would be per input. And then finally there's auto graded essays. We're not going to make an AI for this. Just basically scale it up from pattern match which is a sentence. We're also going to hopefully bring ordering into Caw as well. Gordon Bateson had been in chat with him and he's very happy for us to work towards adding it to Moodle 4.2. There we go. Dead on time. Thank you. Any questions? Anybody raise a hand, we will get a mic to you. So hi, my name's Joost. I work in the University of Applied Sciences in the Netherlands and one of the questions we get for quizzes is that we want to do with detail analysis on which quiz questions have been answered properly and not based on labelling, tagging, metadata. So you can group them according to maybe reception, application one, application two, etc. Is that in the pipeline anywhere because that's what competing resources and tools are offering right now? That is the kind of thing we're thinking about with the sectional grading. Is it based on the sections, headings in the quiz? Is it based on the categories of the questions? Is it based on the tags? We know roughly what we want to develop. We haven't worked out exactly what the details are. So that's what Chris says. We need to go away and actually work out those details. But that's the kind of thing and hopefully what we will at least partially meet your needs. So that follow-up question was about the sectional scoring. Would it work with randomised questions? And yes, that's the kind of thing we need to consider. Again, I don't have a clear answer yet. Hi, this is Abdelhadif, Khalifa University of Abu Dhabi. Qeises have been used for assessment of reinforcement learning in this mode, I assume. Do you know any attempts from instructors on your side who would use the quiz-as-learning resource to convert a whole course into quizzes rather than using the quizzes just for reinforcement or assessment? Sorry, that was quite hard to hear. We're standing right by an air conditioning duct. So can I just check that? I think your question was that very roughly paraphrased, are quizzes just for assessment or are they also a learning tool? So, I think both Chris and I almost regret the fact that quizzes are associated with assessment. We mostly feel that actually if you're thinking about the learning, as I said at the start, it's that ability to immediately give someone some feedback about some skill they're practising. With feedback loops, the faster the feedback loop, the more powerful it is. That's an area where technology... It can't grade everything and it can't conceptualise the student and give guiding feedback in the same way a human tutor could. But where it can provide useful feedback to students really quickly whenever the student wants to study, that is such a powerful learning tool. That's how I most like to think about the quiz. It's also why we made... During this talk, we mentioned various plug-ins, including the record audio question type, our embedded question system. They're mostly in the plug-ins database and you can grab them now if you scan those QR codes or just search for them. It's why we made the embedded question system so you didn't have to go and do a quiz of 10 questions separate from the learning. You could put an individual question as a bit of interactivity and feedback in the middle of reading and learning about something. So, thank you for asking that question. Who next? Mae'r microphone is just coming towards you. Thank you. Can we limit the quality of the audio file in the record audio questions? If we can, limit the quality of the audio files in the record audio question. Well, we've already set to MP3. Yeah, so... Yeah, so the site administrator gets an overall quality control on the bandwidth used. I don't think teachers can control it individually for individual questions, at least not yet. The other control you have, and the teacher does have control of this when they're setting up the question, is what time do you want? Is this meant to be a 10-second thing or is it meant to be a two-minute presentation? So, you have that control. Thank you. Can you guys hear me okay? I think so, yes. So, one of my favourite features of the quiz activities is the overall feedback where you can define great boundaries and you can suggest resources and other quiz questions according to that. You can get some kind of adaptive learning going that way. Do you guys envision any more development for these kind of features to make the quiz activity a little bit easier to get some kind of adaptive learning going? Or do you think this is enough and that's the best that the quiz will do in terms of adaptive learning? Thank you. Do you want to answer this, Chris? So, we don't have anything on our backlog at the moment for the next year or so. We do have the idea after we've done sectional scoring of maybe looking at linking embedded quiz questions to a great book. That could be quite tricky and we don't know how long it's going to take us. The adaptive learning thing is interesting. The quiz is really very complicated and I know we've just talked about making it even more complicated by adding even more features but we have to be careful and in Moodle you do have the option of different activities. Rather than trying to add too much adaptiveness into the quiz activity a better solution and that actually people have made adaptive testing plugins, activity plugins in the plugins database. Possibly using the same questions but in a different activity for combining things. One of the things that keeps the complexity of the quiz limited is it is a fixed number of questions and all students get the same number of questions which becomes important for the reporting and the statistics. I feel the way to meet that requirement is to develop new activities that use the questions and sequence them differently and some already exist if you look in the plugins database. One day someone should update the lesson activity to use questions from the question bank as well but that's never happened. I'll just add internally that the OU is quite interested in student self-assessment behaviour so we've developed a small widget to use the record audio and students play back against the model and rate themselves. We are thinking about expanding that into other question types as well. Hello, I would be interested how deep are you working together with people who are researching on education or actual educators on how you implement these features and how effective they are. How do you measure this and how do you find goals for this process? Within the Open University we have what we call scholarship centres. The first one was in the STEM faculty and then that was so successful that they set them up in our other four faculties and the idea there is to allow give academics a bit of time to run a project to evaluate something they're learning so giving a framework for people to do these kind of scholarship projects about how effective is the techniques we're trying in our teaching and how we can provide the framework for doing the project. It also has things like an annual conference where people can talk about what they've done and what they've learnt and disseminate it. So we have those sort of processes and they find some really interesting things and we listen to them and that's another thing that guides the things we choose to develop. I'd also add that we've got the learning design team at the OU as well. They work with the course teams generally to develop good practice modern scholarship.