 yn y bobl yng Nghaerdydd, yn y gawith. Yn ymdudio'r cyffredin�� l designers i'w gweld fel yma, eich ddweud yn ym mhwanol byddai, i fynd ar ystod i'r unigion. Wrth gwrs ym nid oedd ymddangos a'r gwbl i'ch dod ymddangos ymddangos a'r unigion, ac ydw i'n ddweud ymddangos a'r ysgol yw'n ddweud am fyw ddweud i gwaeth yr unigion yn y prynsibl humaidd ac yn ddweud hon am y dyde. Rhaid i'w ddisgrifennu. Yn ystod y university, mae gennym, mae gennym, mae gennym'n allan i'w ffacultau ac mae'n bwysig i'w ddechrau i'r gwrs, ond ar ychydig yn ymwelwch, mae'r cymdeithas i'r cyfunol hwn. Chris yw'r ymddiol i'r lleoli, yn ymddangos i'r iawn i'r newid, ac yn ymddangos i'r IT yma. Rwy'n gweithio i'w ddweud, rwy'n gweithio i'w ddweud eich model sy'n cyflwyno. Chris help to work out what sort of moodle system we should build and go out and get users using it well. I want to have a map, but you can't really draw the open university on a map because we're a distance education institution. All our students study from wherever they happen to be in the UK or beyond. The one thing you can draw on the map is the main campus in Milton Keynes, it's where Chris and I work and it's where the moodle servers are hosted, so that's quite important. We're quite a big place. Typical OU, so OU study is modular and this kind of thing to take away is modules are quite big things. They're several hundred hours of learning and they run for six or nine months student study than part time. I said student study wherever they are, now they're not completely on their own, they're in a group with maybe about 16 other students and an assigned tutor. The other thing to know is really we prepare our courses up front which might take a year or two and then those courses get presented with only minor modifications for whatever their natural life is that might be average seven years but could be more or less. We have a very nice campus that is actually in the middle of Milton Keynes, it's a very nicely designed town with lots of parkland and we've been using moodle for about 11 years now since moodle 1.6. We've contributed a lot to the community in that time and we think that's important and our main moodle site's pretty big. There's just a quick graph showing our use of quiz over the years. First it took off a lot, you could ask what happened in 2012 and in 2012 we had a government that didn't really believe in paying for higher education and that hurt our student numbers and what's happened most recently actually is that sort of badged open courses have taken off on our open learn platform. Just quickly we use the quiz a lot, do our students like it, do they think it's helpful and so this is just some survey data from one course but it's pretty typical. So yes students think they think it helps them learn and even they think it can be a fun activity. So that's just a bit of background to the OU. Now Chris is going to talk about what goes into making one particular quiz and then I'll talk about how quizzes fit into the overall course. So over to Chris. Thanks Tim. Yes so very complicated, you can spend as little as five minutes or as much as five months quite easily and I could talk about this for an hour. But just to get a OU-ism out of the way, what we call an online test is made up of unassessed quizzes and assessed ICMAs or interactive computer marked assessments. So there's a lot of considerations when designing a online test and it depends a lot on the assessment strategy where it fits into our modules, what the online test is trying to achieve and ultimately obviously the questions that the module team would like to ask the students and therefore what sort of question types best fit that question itself. So here's some stats that we recently ran and one of the big things that I hope you can see on the pie chart is that multiple choice accounts for almost 40% of all answered question types at the OU. Good reason for this, it's very straightforward, it's accessible and it's easy to mark. One less obvious thing and it's to do with colour in here is that the green segments they represent the selected response type. So that's essentially where the options are presented and the student only has to select a response. Obviously there are some issues around this, for example a student might simply recognise one of the responses or they'll work out which one's the wrong and then just take a random guess. So maybe it's not always the best method to use for testing. The blue segment is the constructed responses and these are where the student will construct essentially their response. It's open-ended and this is where you'll see the typed essays, the short answers and pattern match for example. It's generally characterised by a blank input area with no clues but these questions can be much harder to plan, design, implement and test as well as mark. So moving on, the next consideration is usually whether or not to draw randomly from a large question bank say 10 questions in the online test and there's 50 that will randomly be drawn from the larger question bank. Whether or not there will be set questions and those set questions will have variants within it or right down to the very simple design where there are just simply say 10 set questions. And then there's also the behaviours. So interactive with multiple tries is where a student will answer one question at one time, submit it and then they will be given some hints if it's wrong and given some feedback which will allow them to try again and the possible marks will be adjusted. The alternative is deferred feedback and we tend to use this mostly in our assessed ICMAs and that's where students can go through the entire online test, answer every single question and then amend their responses prior to the final submission. And once that is submitted the feedback is in provided either immediately if it's formative or after a cut off date if it's summative. So there are a couple of best practice points that I'd like to highlight here and the big one for us is to always clearly communicate to students in advance the online test availability, what it's testing and its scope so whether or not it's the first unit or the entire module and then how the online test will behave. So if a student gets one variant and another student gets another and there's a question then there might be some confusion because not everyone was getting exactly the same question. What will happen after completion is very important especially if for example it's a diagnostic quiz and that might feed into a tutorial later on. So if there's an area that all the students in the tutor group didn't do too well on then you can actually have that as a tutorial segment. What I'd really like to press home though is the use, review and improve for next time especially if it's assessed. So that's some of our key points and now over to Tim to discuss use in practice. Thanks Chris. So yes that's the considerations for building one quiz and then how do quizzes fit into the overall learning design of your module. So this is some more of the data analysis we did when preparing this talk and I was quite surprised by this one. This is the number of different quizzes in courses that use the quiz and I was expecting to find that like if you're going to use the quiz you probably have half a dozen across your module. But actually if you're using a quiz the most common number to use is one so let's look at why people use different number of quizzes. So turns out there's no particular reason why people use one quiz it just turns out that somewhere in this module you need a particular learning activity and quiz fits the bill. So at the top you've got there's just been a tutorial on data analysis in a sort of nursing course or something and after that tutorial there's a chance to come back and do a practice quiz to consolidate your knowledge. The next one's similar they've just had a session online about team working online and this is a sort of quiz that I suppose is meant to prompt reflection a whole bunch of true false things to think about. If you were doing group work face to face you'd do this is this relevant in an online setting. So those are sort of formative uses at the bottom we have a summative use a module I studied was the introduction to social science which obviously you might expect is mostly assessed by essays. But towards the end again they had a data analysis bit that they needed to assess and they did that with a Moodle quiz because it fitted the bill and that's the only quiz in the course. But what is kind of I would say more typical for a course that's using big use of the quiz. So here's one of our level one maths courses. The course is split into 12 units each takes two weeks and at the end of every two week block there's a practice quiz so you can see if you've taken in what you meant to have learnt. And then you have some for assessed quizzes throughout the course and the strategy here it is that OK so we hope the students will do the assessed quizzes because there are marks up for grabs. And then because they have to do the assessed quizzes hopefully that will drive them to do all the practice quizzes to sort of consolidate their learning first. And then there's a written quiz again so let's have a look at does that actually work. Here's some data about that. Slightly complicated graph which I apologise I stole it from one of my academic colleagues. So the first four bars show the assessed quizzes and what you can see is that there's a drop off through the life of the module which is a bit regrettable. But the red the red line shows the equivalent sort of handwritten assignments that the tutor marks and there's also a drop off. And it turns out that if you're doing open entry maths courses that's quite hard and you do lose students and we'll come back that in a minute. And then the other bars of the practice quizzes and you can see there's quite a lot of use of practice quizzes that does also drop off. And with the practice quizzes there can be multiple attempts which is why the bars are stacked up. It's also very interesting to look at the when students do the quiz which is what this is showing. And you can see very clearly guess where the deadlines for the assessed quizzes are. But what you can see there when you have a spike for one of the assessed quizzes also the lines for all the corresponding practice quizzes dip up as well. So it seems like that assessment strategy is actually working the idea that the assessed quizzes will drive use of the practice quizzes. And of course at the end you can see look where the exam revision is. So again that's sort of what not unsurprising when you look. So I said open access is difficult and the thing is trouble is open access is absolutely fundamental to the DNA of the open university. So really the issue is how can we prevent students from kind of biting off more like they can chew and sort of committing themselves to a personal mistake. And it's a big worry so they've tried various things but the sort of summary is of all the things they tried the thing that was most sort of effective was to get students to do a diagnostic quiz before they sign up for this course. And then based on the mark they get on the diagnostic quiz we might try and direct them to sign up for a more appropriate introductory course first. And it's not correlation but students who did the diagnostic quiz and acted on it, if they go on to the course they're about 60% chance of passing whereas 40% of students who didn't. So that's another use of the quiz. The previous graph of how many quizzes in a course was cut off at 20. So what is the most quizzes in an OU course? The answer is 114. Our Beginner's Chinese course is a 36 week course and every week there are three practice quizzes quite short. Wow but you know it's not just this one all the beginners language courses do this it's an effective pattern that's worked for them. I was short of time so I'm afraid I'm going to skip this but this is basically this was a quiz without a hard deadline. This is when they put in a hard deadline you must do it by the deadline or you can't do it later and total completions went up. And this is just very quickly all our different subject areas and whether they use quiz and in the past it was kind of only stem but now really all subject areas use the quiz in some way. This is formative when it's summative it's still not all faculties but even now like W is law the law faculties jumped on board with assessed quizzes. So now back to Chris to talk about some of the practical issues of running the quiz at scale. Thanks Tim. So just another warning this is another oversimplified diagram. Essentially we use a workflow system and the amount of use depends on whether it's a brand new online test if it's being viewed or if it's an older test that is simply rolled forward with every module presentation. But essentially we have the academic author or authors who will devise and create the questions and then other members of the academic team will test it and they'll be backed up with editors who will help proofread the questions. And responses and our online services colleagues who help administer. Once it's approved it's then copied across to a live site. And then when it comes to administration of in this case assessed ICMA the main point is that it requires a human expert who must always check the score in practice should always be continuous improvement after the online test is closed. There should be a view of the actual responses given and general improvements to the question hints and feedback made if and as necessary. Finally it's always worth adding new questions dependent on form feedback or areas the cohort may have struggled with in the other assessments. Now back to Tim to wrap up. Right thanks Chris. So this is not an academic presentation so we shouldn't really have done references but actually some of our colleagues. I mean all this talk is based on things that have been found out by teachers around the OU and some of them have written it up in papers and they are really quite interesting papers if you're an assessment geek like us. So we're not expecting you to write this down. We'll put the slides on slide share and you can follow the references if you're interested. So just to wrap up there is no one right answer for how to use quiz. It really what the right thing to do is depends on what you're trying to achieve educationally. However it does seem to be the case. I mean effective education often it's as simple as time on task and then active learning is better than passive learning and you know quizzes are a good way to scaffold some active learning time for your students. So basically give it a go you're unlikely to do any harm for your students unless you build something that's just a complete waste of time. But also don't expect and don't worry about getting it right first time. After your students have done it have a look at what happened and you can probably adjust it and make it better. I said there's no general advice but actually generally I would say little and often or even little and once in a course is probably all you need. Just you know you don't have to build huge quizzes no one will enjoy that. And finally sort of summarising use at the OU it seems it's much more it's not really about the marks. All these quizzes really are about the learning they inspire even if they happen to be marks floating around. So that's it. Thank you very much.