 Hello everyone and welcome to this Moodle Academy webinar on creating quality quiz questions. I'm Mary Kuch, Education Manager and I'm going to be going through the presentation with you. And I have alongside me, Education Advisor Anna Kraser, if you'd like to say hello Anna. Hello everyone, nice to see you. A lot of people are still coming into the session today so I'll keep an eye on that and of course your messages in the chat. That's great, thank you. Okay, so I hope you're going to enjoy this webinar today. There is a course attached to it as well and we'd love you after the webinar to go through the course and get a badge if you want to a badge of completion. We're going to start by giving you an overview of what this webinar is about, and also what it's not about. In other words, what we're going to cover, and what we're not going to cover. So in other words, we are not going to look at the technical aspect of how to set up a quiz in Moodle. We're assuming you either know how to set up a quiz in Moodle, or if you don't know we do have courses and videos showing you how to do it. So it's not that kind of a webinar. This is more on the pedagogical side, the teaching side of how to phrase your quiz questions, so that they are both challenging, but clear and unambiguous. Nor are we going to look at how you can give feedback for each of the responses in quiz questions. This is really, really important, but in fact we felt it's so essential that you give good feedback that it warrants a whole course or webinar in itself and we didn't want to dismiss it by just trying to slot it into the one hour that we have for this. Even though we might have a webinar on on feedback itself at some later point, but this is not for today. So what we are going to look at is first of all we'll explore some of the more popular question types that you can use in quizzes. And we're going to ask for your opinions of them. So then we're going to focus on what I like to call the question traps. In other words, what to avoid when you're creating questions are main focus I have to say will be on multiple choice questions, they are the most popular and the most versatile. And then having looked at some question traps. Then we're going to look at some what I call question tips. In other words, as you're developing your quiz questions, what to think about to ensure that they are what we like to call quality quiz questions. So I hope that's okay. And we're going to start by looking at some popular question types. Now, the advantage of using the moodle platform is that we have many different question types. And if you're creating your quiz, and you click to add a question, you'll see, as in this screenshot here, the question chooser with a range of question types and moodle offers quite a few as standard. But there are also if you're admin allows there are also what we call contributed question types extra ones that you can add. What you may not know is that actually your administrator can order the question types in that list, they can put them in a specific order, for example, in order of popularity. So in this list here, we've got multiple choice is at the top as the most popular question type. And also, if desired your admin can hide questions that no one ever uses or they don't want to use, just to simplify the question chooser. What we have here is our top four, I think are generally accepted to be the, you know, the most popular question types, multiple choice, matching short answer, and true false. But we would begin by just exploring these a little bit and looking at their pros and cons and asking what you think of them. So shall we do them in reverse order of popularity. So let's take a look at true or false, first of all, and I have to confess that I just don't use true or false question types at all. I don't ask primarily because the quizzes that I do, people can take them more than once, usually at least three times. And so it's pointless because if you get a true or false question wrong the first time, then you're definitely going to get it right the next time. However, I did ask in the discussion forum in the course attached to this webinar of what people thought and people were saying they like to use it as a very, very quick quiz. Do it once only quick knowledge check, perhaps through the time limit. So that's interesting. I'd like to hear shortly your opinions on those, along with short answer short answer question type I do use I do like it. Because unlike the other questions, for example, multiple choice or matching where the students sees the possible responses, because you have provided them with the short answer question type they've actually got to think they've actually got to use their brain and their response and type it in. That's good. However, it does mean that as the question creator, you have to think very, very carefully. All possible answers. Do you include spelling mistakes, for example. Well, obviously not if it's a spelling test. It's important to have lower case uppercase capital letters punctuation is one word okay would you accept more than one word. All of these things you need to think about. Otherwise, and again I speak from experience you'll have students complaining their answer was correct, but it wasn't marked as correct. You can ask to risk known as a wild card that can help you. But again if you're going to use short answer I think you really need to think it through going up our hit parade or chart of popular question types. It's a matching question type. And as I was reading around this subject I learned that, as well as being easier for teachers to create, because you're basically having a statement, which you split into two to get your two halves. It's like a multiple choice where you have to think of some possible answers which are incorrect, as well as being easier for teachers to make. It's apparently is also easier for students to answer they like matching questions. And that doesn't mean you can do it quickly without thinking, you need to ensure if you have several statements to match that it's not obvious, which is the which matches which, and also that it's not ambiguous perhaps to there are two possible answers. And yet you will only accept one of them. So once more, you really need to read it through ask someone to check it for you. So we are coming to the officially most popular question type multiple choice. And I have to say, just because something is popular doesn't mean you have to dismiss it or you have to use alternatives. Maybe it's popular, because it's, it's very, very good. It is certainly versatile as a multiple choice question type because you can use it for summative final examinations as in many places, but you can also use it for formative knowledge checks continuous assessment as you go along. And of course in moodle's multiple choice question type at the moment you can have one single answer. That's probably what we're looking at today, or you can have multiple answers and a variation of grades. But the problem is, because multiple choice question types are the ones people think of first of all, and they and because it's so easy, they don't give enough thought to the careful creation of the question type the questions, and it might be. Again, I speak from experience. It might be that because you didn't think enough, your question is too obvious, or your question is ambiguous, and students don't know what the answer is. So that's what we're going to look at today. Question traps, and then some question tips with a focus on multiple choice. But while we're still talking in general about question types, I would just like to pause myself for a little while and pass it over to you. I'm going to ask you if you'd like to type in the chat and I'm going to ask Anna to give me some comments there. Do you have a preferred question type. Are there certain question types you avoid other quiz question types here that are not standard but that you'd like to share because you think Anna. Sorry, I was muted. What I want to say actually a comment. As you said I'm not using a lot the short answer questions like you, and especially when I'm working with Greek, Greek have a difficulty in writing and also there are accents, and it just makes so complicated things you have to be very, very precise on what you describe and what you want. So, yes, that's the question that I'm usually avoiding. And let me check we had a question from fire rules, who says that, is there a limit of the number of correct answers in short answer type. I don't think so no I mean if you think you're going to have a lot of variations you can use that wild card can't you that covers that could cover several possibilities. Exactly. It depends how you will build it I mean if you offer a specific options then the correct answer will be among the specific options, but if you provide the wild card then yeah. Yeah, thanks are generic. I don't okay. Well, we have a lot of comments. Lauren says that I do have I do like my choice, especially when you can allow more than one right answers for that's interesting. Kim says that it depends on the purpose and that I will agree with that. Jones says I use sort of answers a lot to test a different level of knowledge wild card is useful and students don't know if or where it is or where it will be used now that's sneaky clever. Jones says that I like to use a true and false in a question page just to emphasize the fact multiple question types are difficult one incorrect one correct or nearly correct and one nearly incorrect. He passes drag and drop into text is my favorite crease our vocational departments like the dragon drop on three much questions technique to level diagrams. I see even I like my choice as like multiple questions. Yes, look up. I like multiple true false and K prime, but they are both not in standard their plugins. Alanda says that calculated question is a good type of questions for mathematical questions. They've formula math q type Kathy gapfield very versatile and can be used for asking questions in different ways that MVQS can do. Yeah, I like the open university pattern match to enable long text and three questions with a model answer. I like to usually and score but sometimes with scored keywords. It depends what I want to test. I like to use close for having questions in context, context, I guess. Richard, I've used simple calculation questions in multiple choice question for topics in economics like price elasticity. But my bath talk says I like my choice question. I hope I said correctly the name cut says also votes for the dragon drop onto emails and can be very useful but not accessible. True. Yes, that's that's a problem. And I think it says that I use one of the questions the most but I often find it difficult to come up with alternatives besides the correct answer and not obvious wrong answers. Yeah, I would like to use the calculated questions more but I think they are difficult to make. I struggle with calculated and calculated multi choice and simple but that's because it's not my area of expertise really I think and what you were saying about struggling with possible answers is something we will look at during this webinar. Let's move on. Because I think I'd like to now consider these question traps. What to avoid, and I was trying to think of the best way to do this. And I thought, well, why don't we have a quiz, a fun quiz, which is actually a really badly designed quiz or with some really badly designed question types. I'd like to think of a topic that you wouldn't be familiar with or most of you wouldn't be familiar with, because the idea I want to show is that even if you don't know the topic. The quiz is so bad that some questions you can get right anyway, and some questions nobody could get right, even if they are experts on the topic. Hence, I want to demonstrate question traps by using a very bad quiz. And one of you might smile here. I tried to think of something which I thought very few of you would know to use as our quiz, and I thought, let's use the black country. So the black country. I'm in the UK now I'm in the north of England, but I was born here. It's an area in Britain in the West Midland so that's the centre of England around Birmingham. 200 years ago it was a part of the Industrial Revolution the early days, and you can see from this graphic here chimneys, mines, mills and so on factories and it was called the black country, because of the black soot of the Industrial Revolution. It's now much cleaner, much more attractive, but it has its own distinct dialect, they speak English, but with many different words and grammar. And I grew up with the dialect because my father spoke it all the time. I had to translate for my family when I had a family because they couldn't understand him. My mother always used to say that my mother did not let me speak black country because she said, ladies do not speak black country and I had to learn standard English. But our quiz is going to be on the black country dialect. And I'd like you please as we go along there are seven questions, keep your score just out of interest. And for each question, I want you please to type the letter ABCD that you think is correct into the chat. And then I'll ask Anna just to tell me which which answer is the most popular. Okay, we'll try to move through quickly. But here we go. Question one, black country dialect. Black country, the black country term for a road is an stripe, or shroud, why, or round. So please type your answers now ABC or D the black country for a road is an, and as soon as people start typing and just let me know, yeah, and stripes. I think that most of them are a's, if I can account them correctly. Yep, most of them are a's the black country for a road is an straight and fast road, and why, or on road. Okay, so most of them are a's. That's really interesting. I asked my husband that he's not from the black country had to translate he said a to this is very interesting because if you read the question carefully. And this is an English language question this would be different in other languages, I speak other languages so I know but if you read the question carefully. What can only be be there is only one right answer, because in the English language, if a word begins with a vowel, a, I owe you, you have to say, and not, so you cannot say, and straight, you cannot say, and way, you can only say, and off road, because, is a vowel. So, the answer to question one is be just a little bit of extra information for you off is black country for a horse. So you can see how old the dialect is, because it actually means the road where horses used to go the horse road hundreds of years ago. There you go, a little bit of extra information. Let's try question to the black country word for it isn't is it ain't tie, which is actually a shortened form of it ain't it's not, or it war. Okay, please type your answers ABC or D the word, or it isn't. Okay, I think this time we'll go for more bees. D, D, D, did you say be be. Oh, sorry be. Okay, so the black country word for it isn't is be you are correct if you said be two reasons first of all the question asks for a word and tie in be is the only one that is one word the others are two words. And often you'll find as a teacher you are so keen so enthusiastic to explain the correct answer that you sometimes give more information for the correct answer. So the lengths of your responses are different. So I really wanted you to know that tie is actually the shortened form of it ain't which means it isn't. And that is a kind of a clue as to be being the correct answer. D and C I just made them up D for your information only it war is actually black country for it wasn't but that's not relevant to this. Okay, keep your score two out of seven coming up with question three. What is the black country term for a horse. So please type your answers here. I think that we, everybody's paying attention. Yes. Now, this is actually really important that I hope you knew it was a because I actually told you in the feedback of question one the us road and believe it or not. So incidentally, including the answer to a question in an earlier or a later question and a quiz happens quite often, even in official state exams, I used to teach languages at GCSE that's the UK 16 year old exam, and I noticed it on occasion in the questions of those exams. I took B two level Russian state language exam in March. There was a question I didn't know, but a few minutes later I found the answer in another question so I just went back quickly and added my answer there. So even in the highest most officially checked exams. This can happen. Be careful not to give away questions in other quite not to give away answers in other questions. Right. I made up the other three, by the way, question number four. In modern times, who speaks black country. Old people always speak it. Usually older people. Nobody ever speaks it anymore ever. Or Joe Biden lol. Okay. You might need to think but please type what you believe could be the answer. Old people always usually older people. Nobody ever it's data or Joe Biden. Well, everybody is mostly at B. Okay. Now this is an interesting one. For a couple of reasons. Again, you're correct if you put be usually old people. The thing is life and quiz questions are really absolute. So if you put always never nobody. It's probably not going to be the right answer and you see even then I said probably because things are not absolute. So if you see something that is so definite and so black and white. It's probably not the right answer. The other thing I want to point out here is the Joe Biden lol. I used to do this a lot because I used to make quizzes for younger students because I thought oh it makes it more fun. You know I could put in one of them or one of the teachers. The thing is, it basically it removes one of the options because he's obviously not Joe Biden. So the student doing the question only has to choose from three options here and not for. So although you might think it might lighten the atmosphere, it actually means that one possible answer is removed and it makes it easier to guess. Okay, question number five. All of these are one answer only by the way, I should have put it in all of them but I put it here. Okay, so where is the black country dialect spoken one answer only please. West Midlands area of Britain. B in the centre of England around Birmingham. See in the north of the UK or D only ever in Birmingham please type your answer a or B or C or D. We are battling between A's and B's I think. Right. Okay, now that's interesting because they're both right. And this is a serious problem here. So A and B are the same. And this is a good example of where the teacher didn't reread or ask someone to proofread the question. If you want one answer only you've got to make it clear which is the correct answer. Also, the fact that we have in D only ever in Birmingham that's an absolute things are really absolute. Now, what would you do if you had this in a quiz you and students had answered, you'd have to go in and edit this question either to remove it from everyone's answers score or to give everyone the grade for it to make it correct. This is a faulty question here. Number six out of seven I hope you're keeping your score just out of interest. Question number six, which of these statements is not true. A black country is not spoken in London. Older people be older people in the region do not tend to use black country terms. Black country is not a modern dialect or D questions with double negatives are not easy to understand. I can't even remember the answer I put myself for this. Okay, so please type what you think. Lucas says I'm leaving that one blank. Well, you might as well put an answer Lucas because you might guess and get it right. That's what I used to say to my students all the time just to guess. We don't penalize you for guessing wrong. Okay, anyone like to what's the general thoughts. People is confused now way I don't think we have a clear view, perhaps a little bit of the A's around. I don't see C, but AB and the part. Okay, right. The correct answer is B older people in the region do not tend to use black country terms that's not true. It does tend to be older people in the region who use black country it's less used now. And the other three are true, but the problem is, and I am a native English speaker, and I struggle with this and I wrote the question. I'm trying to put a question, which is not true, and then your quest your answers has have negatives in them. It's really difficult to get your head around it. You have to take out the negatives and try and make it affirmative and then work it out which is why Lucas gave up. So my advice here is to avoid negatives in your questions and responses, and try to make affirmative statements. The game is D. This is very true, but also it's irrelevant. This isn't anything to do with the black country. So it removes a possible answer there. So, our final question of one very dear to my heart actually question number seven. Assumed origins, if we assume it's origins in PIE, quints BCD. Is it a more than 80% ASB quite a lot of OE, C, NF or D trick question is origins are not PIE. There may well be someone who knows this because there's certainly someone I imagine, surprisingly who knows the black country I saw so maybe someone does know this. Let's see. Well, it's really fun because most of the answers in the chat sees all question marks or a bit. I would say this. I know it, but this pants. Excellent. Right. Well, what can we learn from the question then. Well, we can learn the answer. The answer is a, as a matter of fact. But the problem is manifold. First of all, in the question following is assumed origins in PIE quints BCD BCD black country dialect, who says quints, quints is not used in standard English. So why use it in a quiz question just to make yourself sound clever use standard English French Russian German whatever your language is. Also, if you're going to use abbreviations, make sure your students know what they are. And I'm assuming you don't know so I shouldn't have used them. Use the full form and then afterwards use abbreviations. But how can you know that PIE stands for proto Indo European, which is the supposed original language that many of our European languages come from. Unless you learned it. Yes, stands for Anglo Saxon. So 80% of the black country dialect is Anglo Saxon English. Oh, he stands for old English. And look at this quite a lot of old English. And vague figures don't tend to be the right answers if you're going to give answers correct answers, they should be precise. Yeah, more than 80% that's precise enough. This is Norman French, by the way. And also D trick question, don't have trick questions. I know it's tempting, but you're trying not to trick the student I hope you're trying to find out what they know or help them to progress, rather than catch them out. Okay. So it was pretty difficult to guess or to work out so don't worry about that. I'm just interested before we pause. Would you like to tell me your scores. And let's see what the kind of approximate average was. Okay, five, four, five, four, three, four, four, three. Well, it's like zero. Yeah, that's also. Yeah, I figured that if you paid very close attention, you should be able to get four out of seven. And if you got more than four you were either lucky or very, very, very clever. Yeah, because the first four you should have been able to work out. But don't worry if you got less my husband actually got to, and he's a native English speaker so not and he got the first one wrong the off road. I'm just going to pause there. I'm apologizing in advance for this slide which has lots of text and I don't approve of slides with lots of text, but this is basically summarizing the lessons that we've learned from that quiz. And I've put this here so that if you get the presentation after the webinar, it will be available for download, or if you're watching the recording, what we've learned and what I've explained as we've gone through these questions is there. For instance, making sure your grammar doesn't give away the answer. It might be a different kind of grammar in a different language, but it's still possible that you could accidentally give away the answer. Avoiding absolute statements, avoid ambiguous questions, trick questions, abbreviations, joke answers and so on. Okay. I'm just going to pass it over to you for a few minutes looking at the time. Do you have any more question traps. You see this picture you remember the picture of the graphic of the black country. If I were to put that into a quiz question, a question with an image and ask about it. Would I need to think about a potential trap there. Okay, over to you Anna, for some comments or thoughts from people. Well, we have some comment already from Kim says always do a spell check on. Yes. Yes. Yes, is there an option optimal number of options to offer. Yes, well we're going to look at some tips in a moment and we'll look at the optimal number of options there. Okay. So then, none of the above all the above types are very bad practice indeed reaches is avoid using yeah none of the above. I wish I thought that you know is they always forget something none of the above, especially as I've discovered, if you use none of the above and then you accidentally shuffle the answers, none of the above could come in the middle or at the top. You know it's best not to use it at all. Very good points. Okay. I would also say the specific questions like. A and B are correct, because again with the suffering it's messy and again it's, it's quite pointless it's better to make it a multiple choice multiple answers. Hi Russes, how we can estimate necessary time to answer. How we can estimate the necessary, the necessary time. I think, basically by trying it out, you can look at the statistics reports from the quiz and see how long people took to answer, but on the other hand that doesn't necessarily tell you if they're just going through and guessing, you know, so that's a very I can tell my tip to fibers is that I'm usually calculating one minute to one question. The time is huge, but the pressure that timer applies over to the people who take the quiz is also huge. You don't have to worry just small like you creating the feeling of a rush of you have to be prepared for that. So I wouldn't worry that much. And about the question you asked Mary regarding this picture. I would avoid to use the picture and use an option like black diet, how this that, how this dialect was called and have a black inside because so black. Yeah, don't don't say picture of black country or something no correct yeah. Okay, anything else. Yes, we have more comments. He says that all the above and all none of the above, obviously, they validate partial knowledge. Someone says that I don't like soft, soft link answers rather than, rather, a short term. Ah, now that's interesting. You've obviously been looking at my slides ahead of time so we'll just keep on that one. Shortly we'll look at that shuffling to shuffle or not to shuffle that is the question. Yes. This is if not suffering the answers we tend to place the correct ones not on the top or the bottom. And when students are guessing they won't choose the top or bottom option either. Kathy says that shouldn't really suffer answers unless line length is the same for the ability avoid the tragedy layout. Yes, and Richard says that what are your thoughts on negative marking. Well, um, well we could have a long conversation about that. In fact, negative marking comes in in one of my tips here that you might want to consider or not. I don't have a problem with it myself personally. I mean, I use it for example if you have multi answer questions so that students cannot just think oh I'll mark I'll select everything because some of it must be right. So then you have to give negative marks in order to penalize them for simply selecting the wrong answers. But let's let's hold on that until we look at our question tips actually. Are we okay to move on Anna. Yes, yes, absolutely. Okay, building on what we looked at during our fun quiz. These are some some things of what to consider. And I'm hoping that you have some other thoughts as well at seven again seven is the magic number today. In terms of moodle when you're creating your quizzes. If appropriate, you could shuffle the answer choices now as I say if appropriate. This really does depend on your own circumstances and what your quizzes about, but something I found very useful since moodle 3.11. If you are on a lower version you'll look forward to this is whatever you set in a question. So for instance here we've decided one answer only, and we do want to shuffle the choices. Moodle will save that. So each time you make a new question, you don't have to select one answer only shuffle the choices it is saved for you. You should shuffle the answer choices as well as the questions. And sometimes you don't even want to shuffle the questions, but to shuffle or not to shuffle. I would say, if it's important not to shuffle, try to list the options either logically or alphabetically. So in this question, which of the days of the week in English is named after the Viking God Thor. Of course it makes perfect sense whichever one is the correct answer to put them in the order in which they come from Monday to Friday here, for example. Alternatively, if theoretically you could theoretically shuffle this one, but I think it might be better to make it alphabetically. Which of these words expresses a relation time or place to another word or element in the sentence the answer is a preposition actually, but in order not to give away any kind of clues, just put the answers alphabetically. There's no way of guessing it's not in the middle. It's not at the top it's not at the bottom it's alphabetical. So those are some thoughts on shuffling. Now in terms of what's the optimal number of responses. I was surprised. There's some reason there's a research link on the slide here there's also some more. If you take the course afterwards, the obviously you can have as many as you want, but apparently there's no benefit in having more than one correct answer, and then to incorrect answers distract us. So you can by all means have more. So for instance, you will know that the fun quiz on the black country I decided to have four I just happened to like four. I actually choose four, because I think if you have three, you tend to pick the one in the middle sometimes, but there is no benefit. So you've done experiments on three, four, five, and it doesn't make much difference beyond three beyond one correct, and then to incorrect. It's useful to know that simply because if you're one of those people who really struggle to think of answers or responses, you don't have to have a lot. And this is close to my heart as well. Do make sure that if you're going to have a question which has one answer which is 100% isn't a controversial topic ambiguous topic, which is the longest river in the world. There's some debate on this how do you define longest largest deepest tributaries and so on. So do, do think carefully when you phrase your questions if it's 100% answer. And of course that leads on to what I will say shortly about having someone check them for you. This should go without saying, but I've seen it for me really sometimes when I'm creating a quiz, I think I have 10 questions have got two more to go what can I make a question about always ensure that the questions match your learning outcomes keep them there in front of you, as you create your questions, so that there are no irrelevancy so that you're not making question number 10, because you need 10 questions, and you need to find something quickly. Okay. And going back to that the ambiguous controversial always ensure that someone else, preferably a colleague can proofread and check your quiz questions. If you're sharing a course in moodle for onwards, then in the question bank there is the option as you create a question to set it as draft, which is basically saying I'm not ready to use this yet. I want my colleague in the course to come and check it, and your colleague, and you see the other column in yellow here comments, your colleague can look at it. If they can comment on it I think you need to rephrase this it's a bit ambiguous. Then you can go back when you've had your feedback, and you can select it as ready to use in your quiz. So, in moodle for onwards, this is a good way of collaborating on creating quality quiz questions. I'd like to say, if you have done your best to create the best quality quiz questions, you've had them proofread and checked yet you're still afraid that your students are just going to get lucky by guessing and get the answers right by guessing. What could you do. You might like to consider certainty based marking. This does have issues with the gradebook I have to say so perhaps the formative assessment really, but what certainty based marking does is it means that if a student guesses correctly or rather let me explain. Every time the answer and you might like to answer this for me by the way. Every time the answer you have to give your answer. And then you have to state also you see where it says certainly whether you're unsure one I'm just guessing I don't really know whether you're reasonably sure to or whether you are quite sure three because you learned the answer. And so then you will be graded according to. Did you get it right because you were quite sure you get extra points. Did you get it wrong, but you are quite sure that you lose a point. If you get it right. And you guessed well you do get one point you get the correct point for getting it right and guessing because okay it's right it's it's correct. But if you get it right and you said, I'm sure because I learned the material. You get a benefit you get a bonus. So you might like to consider this. And before we move on, would you like to try to answer it and maybe you could tell me as well what your certainty is. Is this sentence correct. Thanks for inviting John and I to your party. Is it a well it depends it depends on the situation. We need more information. Is it be that's correct. Yes. Good English grammar or is it see no it's not correct. So I'd like you to type a B or C that's your answer but as well put one two or three if you're guessing. If you think become fairly sure or if you know the answer because you are a pedant. So would you like to tell us what people are saying. Okay, some have used CC. I think it's C three. Whoa. But that's one case to online. B's C three C two B two. Okay, so we don't have any a but we do have some B's and C's the answer is actually C is incorrect. It should be thanks for inviting John and me to your party it's a common mistake called hyper correction. But if you're interested in knowing more you can look at the presentation afterwards. But the point is, if you guessed, if you guessed correctly, you get a point. If you got it correct, but you were very sure you will get more because well done you. So it's something to think about basically. Okay, so we had seven tips from me, but I'd like to finish off in our last few minutes. If do you have any more tips, do you have any comments or any questions, just as we head towards the hour, especially more tips please. Let's give some minutes. So people can type in Sonya provided a link fun examples of bad questions. I'll link into a blog. The forum actually the discussion forum afterwards in the course if you like. Yes, please that would be useful. Are there any other new quiz questions features in mood for Chad is asking. Well, there are some new things in the question bank yeah I'm actually after this webinar I am going back to updating the documentation and preparing presentations about what's new in Moodle 4.1 and there is some new question bank stuff. We've got we are doing a webinar on the 15th of December for anyone who'd like to know what's new in 4.1 including the question bank yes. Joe says that I always do the quiz as a student to test. Yes, research says avoid inconsistent number of responses to questions. This is a very good versioning viruses using random question is good for quality quizzes like that as well. Alex says there should be an opportunity for students to discuss the quiz after it is over. And Chris says that good to write the question set down separately before adding into the mood please. It allows you to organize your thoughts. Good because I never do. I always do them straight into the quiz, but I know a lot of people who do write them separately what do you do, Anna. Well, I prefer writing them separately and especially when it comes to multiple choice questions. I think a little bit because I can use the icon format and import all the document at once, or the VLE tools if you are familiar with that. And I can get the quiz XML file and upload them all together and because I'm not a native speaker, my English are always need for a review so I prefer to have a document. I will review them, fix the language, have auto corrector while I'm writing questions, and then finally upload them. And one thing I love doing is what I would say about using random questions. I love random questions, not only for the quality of the quiz, which is the main point, but also for the flexibility that it gives you. Absolutely. And we do it in Academy, don't we? Yeah. Exactly. And one tip when I'm writing the questions directly into moodle question bank. I never, well, I always use meaningful names for the questions. For example, I never use one, two, three. Well, because I never remember what was about it and it just takes me so long to search what I'm looking for if I want to update something. I have inputs from people. Let's see what they do. Dave says use the random question function with questions with slight changes. I will say that close questions are good multiple types in the same question. Kat says that I would say always write a storyboard, master answers document first. When build questions in moodle always preview each one to test that is all okay. In fact, I'm in a company that I was working on and we had quiz quiz made with random questions. And we want to review the quality of the question I was doing like what virus and Luca suggested Kathy suggested I had a quiz for students to take it and a quiz for teachers and subject matter experts to review. And in the quiz of the subject matter experts, I was using all the questions, no suffering, none anything. So just all the questions so they can quickly go through them and fix any troubles. Yes, Kathy says I have a recommendation to use a number with the title as well. Yes, that's also an option for sure. And I think that was all the inputs we have. Please do if you have any more comments, please come into the course afterwards and add your comments or questions in the general discussion forum would like to get a discussion going there and do the course, which should be opening up very shortly. And I would like as we finish just to remind people, and if you're also watching the recording how to get involved so we can grow Academy. We'd love you to go to our get involved course it's linked to on the front page and suggest topics which we could cover in future webinars and courses. This is an example of something which was suggested and which we then set up. Perhaps you would like to volunteer to present a webinar and would look at me on your area of expertise you could get a presenters badge. And if you would like to help us, we'd love you to help us contribute to creating our courses. And again you could have a course builder badge for that. Please get in touch with us and help us to grow Academy. Also, not mentioned here, but we are would be delighted if people could translate our courses as well. We have a translate Moodle Academy course you could do, and then start translating the various courses that we have. And if you enjoyed this webinar, please tell your friends and colleagues about this about Moodle Academy. Come get a badge tell others. If you're an educator, and you haven't already explored the Moodle educator certification. So let's go and try are you ready for the MEC course. And if you are, sign up with one of our certified service providers and take the MEC and be certified it's a certification program for people who have experience and knowledge in using Moodle. And we'd be delighted for that. I'm going to stop there now and I'm just going to say thank you for joining. Thank you for all of your useful comments and contributions. Thank you from me, Mary Cooch. Thank you. Nice to see you all here. Thank you. Goodbye.