 Like Troy said I'm a chemistry teacher and so I'm a walker. I'll just have to you know just bear with me on this My I am I'm the chemistry teacher, but I'm also the Moodle server administrator and the Moodle trainer I kind of fell into this do everything all at once mode. That is the story of my life right now But I got into Moodle explicitly because I wanted to do something with my homework I was Staring at kind of what I'd seen my class do the last year and realized that They just weren't doing what they needed to do and I need to figure out a way to make that happen and I'd seen WebAssign before and I knew that WebAssign was kind of cool and could it really forced some of the things I wanted to do, but I also knew that I wasn't going to be able to spend money on this sort of thing either and So I came to Moodle With the express idea of building something like this and it is kind of morphed over the years into the the current form here If you will if y'all are on Twitter, I've got my My screen name up there if you want to use the the make homework work hashtag I'd love to see what you have to say and I'd love to hear your thoughts during and after While this is going on Before we get started here. I wanted to kind of tell you why I why I created this chemistry is very focused around solving problems It's very much like a math class in that way you have to be able to practice and perform Those sorts of calculations over and over and over again And these are skills that get built into every single problem that Happens for the rest of the chemistry class one of the earliest things that I teach is how do you name a compound? Well, then I start using those names all throughout the rest of the course of the year and it automatically gets built in But how do I build in that? Comfort level so that I can take in all of these little disparate topics and put them all together Into making one fairly complex question at the end of the class How do I build them from just barely knowing any chemistry to being able to solve complex problems? and So as as Diana mentioned earlier automaticity is the real key here the way that we learn is that we learn how to do something and We practice it over and over and over again until we don't think about it anymore We don't want to have to think about how we drive a car anymore And I remember my first day is behind the wheel I remember doing all sorts of things when I was there where I had too much brain overload Trying to process all the different things that I was going to do I mean parking a car is not the easiest thing in the world to figure out how to do when you're doing it the first time Getting it right in between the lines and thinking about all the motions you have to do But these days we don't think about those sorts of things because it's become automatic We're proficient at it now that's what you want students to be at and so my design on homework is all about building that automaticity in Building that structure so that students can take those little pieces Practice all the little pieces and then be able to pull them together into one big thing at the end So I've got a bunch of different assignments that I use that talk about that I used to talk about how I do all of these different sorts of things and I I The purpose that I use these assignments for is not necessarily the name that I give the students Because quite honestly, they don't really care exactly why I'm picking all of the different things that I do They just want to know what they have to do and then move on So to try to help you guys a little bit understand my nomenclature and understand The what I use with the students at the same time I've got this slide here and I will try to use both terms along the way, but I didn't want to Didn't want to not throw this out here When I'm talking about just homework with my students This is just kind of generic practice that I want my students to do to practice to get things better when I'm talking about What I call quizzes with my students. These are really timed practices One of the things that we know about automaticity is that the better you get at something the faster you get at it You don't have to think and back in and out of a parking lot a parking spot anymore when you're really good at it You can just do it and whip right into the spot and you're done So when you can do things fast You know that you're not sitting there trying to figure out along the way to do it And so that's the purpose of the time quizzes and then my students Well-written multiple choice is hard If you do it right, it's very hard to do well on a multiple choice test And my students regularly complained to me about how my multiple choice was structured and how they didn't like how difficult it was And so I've put in some of this multiple choice practice along the way to give them practice on the multiple choice Yes, but in my mind is more practice on the concepts that they need to be doing and Then my students also love the test review everybody wants a test review right before the test so that they know What's on there and these are really just problems that they've seen already But they want need the focus to know kind of what's important So I'm going to walk you through each of these different assignments and I set all of these assignments up as Moodle quizzes But I set them all up in slightly different ways from each other And so this is the first this is the practice. This is kind of the entry-level piece into how these things work This is a really early question on in my class. We look at how we convert this number into this number Using a step called a process called factor label And so one of the early questions might be convert 9100 seconds into hours and I want my students to be able to do this method Consistently without thinking about it so that when I actually introduce chemistry with this method later They're worrying about the chemistry and not the method and so I use These are really short homework assignments typically just four or five questions at a time Students have an unlimited number of attempts to get these things right my goal here is for students to practice these until they figure out how to do and So I use the adaptive mode no penalties grading scheme the quiz quiz behavior Because what that does is it puts this little checkbox that you see up here and A student can hit that checkbox as many times as they need to to get it right And there's no penalty for however long they do this because at this point I'm worried about making sure that they figure out how to do it rather than what they're This is completely formative as opposed to summative I want to make sure that they know how to do it before we move on to checking that they actually know how to do it Everybody ends up getting the same questions for all of these You could build random questions in here But I don't want to because I want my students to be able to talk to each other about it Ask questions of each other and figure out how to do it. Does that lead to so a little bit of collaboration sometimes? Sure, of course But at the end of the day, they still have to take my test and they can't collaborate on that So I'm not terribly worried about it There is a deadline for all of these things But the real key to all of this is to make sure that you put in good feedback If you run these questions with students for a semester, and you've got my case 80 to 90 students a semester taking these questions. You see a whole big range of answers I mean that went over there on the end I'm not sure if you guys at the back can read the number up there But that is a ridiculous number of hours in 9,100 seconds And yes, that does appear in people's answers I mean if you look at that you say there's no possible way that that can be right and yet students will still put that in Because if you do the math wrong, if you multiply by 60 twice Instead of dividing by 60 twice, you will get that answer And so you'll see this in the logs and you start putting in on the questions feedback for that specific wrong answer Because what you want is you want students to be able to work through this type in an answer and see What they did so the answer that the feedback is down there at the bottom above the incorrect button It says check your math and your conversion factors. Does this answer make sense? That is a numeric question and so it does support the units in there as well And then this middle part here you'll notice is mark partially correct I don't know how many of you remember any chemistry or the concept of sig figs If you don't remember that or don't want to remember that then good for you It's basically that's a little tiny piddly thing that I it needs to be checked But it's not really important and so I get it marked partially correct so that students know. Oh, I've done this, right? I just need to fix this one little thing All right, so that's the that's the practice piece of this the next piece is what I call the quiz These are really just time practices is exactly the same sorts of questions. They'd see on their homework only now I'm slapping a time limit on it These are really short one or two questions And at the same time quick time limits are really short I'm trying intentionally to set the time limits long enough so that a student who knows what they're doing Has plenty of time to maybe make a mistake and then fix it But a student who doesn't know what they're doing isn't really sure is having to look through their notes to figure out What to do isn't gonna have enough time Because remember my goal here is I'm promoting that automaticity and my message to my students is Oh that time limit on that quiz seemed too short You don't know well enough yet If you know it better you'll be faster at it And so if you consistently find these quiz time limits too short Then what you need to be doing is practicing it more to make sure that you're there And now I'm building in that extra practice to make sure that the students are getting that automaticity These because they are Grated and there's short time limits I want students to get a better sense of how well they know the material and so I do use random questions here Because I want a student to see something fresh and have to work it out in the time limit Rather than oh, I just saw that one on the last quiz Let me do it again and and fix it from there So I want these random questions built in so that students see new things I let students have two attempts at this because I don't expect them to be perfect at this point Oftentimes that first attempt is kind of a wake-up call I thought I knew what I was doing and then I see this time limit of four minutes And I realize I don't really know what I'm doing And so these quizzes Range in time from about this is actually questions from I think my shortest quiz This is two minutes long. There's three questions on it. I have I think my longest quiz is like eight minutes Is they're designed to be exceptionally short just quick little checks to make sure students know what they're doing. The next piece is The the multiple choice part of this So again a well-written multiple choice question is incredibly difficult to grant to answer correctly If you don't quite know what you're doing because those wrong answers on those multiple choice questions are Examples of how you might do it wrong You work the problem wrong intentionally like a student is going to do and put that answer on there Because at the end of the day, what are we really interested in we're interested in figuring out Does the student really know how to do this or do they sort of know how to do it? And I don't want students who sort of know how to do stuff. I want students who really know how to do stuff and so That building in that practice on the multiple choice and letting students see what's going on there is really fairly important And at the same time I'm also trying to do one extra piece here in the same at the same time This happens right before the test in my class. It's kind of the final review before they go in to take the test This is the concept piece of it. We'll talk about the test review part of it later Which is more of the problems But this is kind of making sure right before you take the test that you've really got it down And so you'll may notice here if you're familiar with the the quiz interface that this looks a little bit different than normal There's this piece down here on the bottom part of this question right above the check button That looks a little odd and then there's this breakdown by certainty bit over here on this side Which is also very odd. This is the certainty based marking behavior the way certainty based marking works is That you have to answer the question, but then you have to say how sure am I that I know the correct answer and Your grade is determined as a combination of those two things So we don't want students who say I am absolutely certain that two plus two is 48 That's crazy. Number one. They're wrong and number two. They think they are absolutely right. We absolutely do not want that We also don't want students saying. Ah, two plus two. Is it four? I'm not really sure We want students to know what they know, but also know what they don't know We want to develop some metacognitive capacity here and One of the things that Moodle does with certainty based marking is it helps them do that a student who correctly answers a Question and says that they're absolutely certain about it with certainty based marking will get three points out of one point for that question So right now. We're already talking about extra credit flip side of this is student who gets The wrong answer but says that they are absolutely certain. It's right He's going to get negative six points out of one point now That changes the game a lot at this point because now students are having to say, okay, do I really know this? How sure am I about this and then there's various levels of how sure you can be about it? If you pick the bottom level you say I'm really completely unsure and you answer it, right? You get one point out of one point if you answer it wrong at the bottom level of certainty You get zero points out of one point. So it doesn't penalize you for saying I don't really know this and you got it wrong Okay, that's kind of to be expected It doesn't help you any if you guessed But if you start guessing on the higher confidence levels, it can really mess you up and Basically the way the scoring works here is you get the best score possible When you know the most material obviously right and you know how well you know that material It's the combination of those two things. So when a student is done with this They're looking at this breakdown by certainty report at the end after they're done with it and it helps guide them on you know what? You're a bit overconfident in that C3 category there. You only answered that group of questions accurately 71% of the time You are using this level. You're saying you're overconfident here basically you don't know as much as you think you do You need to study some more on those things that you think you know well Now at the bottom here, there are six questions here It says responses three next to C2 and responses three next to C1. There are six questions here that you said you didn't know very well Clearly you need to go back and look at those two So this is helping guide their studying based on how well they did how well they assessed how much they know what they're doing too Because of all the complications of grading I have not yet decided to incorporate this actually into my course grade. It's more of an instructional piece for them There's no grade for this activity because trick figuring out how to do grades that range between 300 and negative 600 Try to explain that one to a parent right my child got a negative 600 out of 100 on this. How is that even possible? Well But at the same time even though that's really hard to explain in terms of a grade that really helps them see You know, you don't you're not doing nearly as well as you think you are You need to do some more work here. All right, and the last piece here is The the problem type review. So this is your kind of generic test review here. These are questions Again that are very like what they're going to see on the test again. It has the same sort of feedback that we've had before These are longer These tend to be you know, 10 to 15 questions I'm trying to cover most of the things that they would generally see on the test Everybody knows there's some problems that you're just going to expect to see on the test If you've been paying attention those problems are on there The the different piece here that is Different kind of from that first homework is instead of using the adaptive mode no penalties behavior I'm using the immediate feedback behavior So the difference on immediate feedback from adaptive mode is that when you push that check button You notice it's grayed out up there You can't check it again during that attempt The only way that you can go back and get that question completely right is to do an entire extra attempt Now That carries a bit of cost to the student at this point and what I found was students were Getting through their test review and saying oh, I finished the test tree. I got a hundred on it. They go and take the test the next day They don't do so well they come up to me and like how did in the world did I get a 63 on this test? When I got a hundred on the test review You and I know the answer to this How long did they have sat there with that test review trying answers checking over and over and over again until they finally got it right? Uh-huh. That's exactly what I'd do But when you finally get it right that builds in a false sense of security You have if you've tried it five times to finally get it right you're not going to get it right on the test We are what we practice and if you've practiced it wrong more than you have practiced it right You're going to do it wrong on the test And so I switched to immediate feedback behavior on this actually started with adaptive mode No penalties when I first started doing this I switched to immediate feedback behavior on this Because I wanted students to see their first grade Not the grade after they tried it five or six times to finally figure out all the answers I wanted them to see if you were doing a test on this What would be your grade and by doing it as immediate feedback behavior? It forces them to see a numeric score for that first attempt And if that numeric score for that first attempt is you know a 63 That's a much better predictor probably of how they're going to do than whether or not They got all those questions right at the end unless they take the time to go back through and practice it the right way Because if we keep doing it the wrong way if we've practiced it more the wrong way than we have the right way We keep doing it the wrong way I mean I talked to my students about Tim Tebow and his throwing motion all the time right those big news about how He's never going to make it in the NFL Because his throwing motion was wrong and he practiced it wrong for so long it took him a long time to fix it You have to practice the way that you actually want to perform it the real key to this is how you do feedback and Moodle is very helpful in terms of giving you the information and statistics you need To write this feedback if you have never taken a look at the Moodle quiz statistics It's buried in a menu somewhere. It's easy to miss, but dear God go look at it There's so much useful information in there that is incredibly helpful This is one of the pieces that you can see so this is a particular question From one of my practice assignments These are all of the wrong answers and the frequency with which they were put in And so this is again a numeric question so you can see it supports units And if you look at a couple of these you'll see that it's different based on whether they put a space in Between the number in the unit or not So like if you look at this negative seven point eight four atmospheres over here and this negative seven point eight four atmospheres over there That's the only difference between those two But we're already seeing a couple of these that appear multiple times well What's the student doing to get that I mean clearly there are multiple students that are getting this wrong answer How did they get this wrong answer? And so you sit down and work the problem wrong you try to come up with wrong mistakes in this particular case No knowing the chemistry and knowing which question this came from any chemistry features out here anybody Okay, so it's in atmospheres, and they've getting negative numbers They're using Celsius temperatures that are negative It's a gas law problem, and that's where they're getting that number You have to use a Kelvin temperature in order to get a gas law problem, right and so This is this is now easy to fix to write some feedback for you type in that specific answer You say hey the reason why you're getting this problem wrong is because you're using Celsius temperatures instead of Kelvin temperatures And now the student sees that and say oh my goodness I'm supposed to be and they know this they just forget about it And then they go in and they fix it and they do it do it again And so students are not sitting there blindly bashing their heads against the wall at night trying to figure out what they're doing Wrong on this problem. They're saying oh this is what I've done wrong here And as long as you cover most of the common wrong answers you're gonna hit most of your class You look at most of the rest of these There's not any commonalities on them. It's probably a one-off calculation mistake Somebody transpose a number somewhere or put in something crazy or just really has no clue what they're doing You're never gonna figure those out But for all the ones that come up frequently you can hit, you know 90% of your class 95% of your class with those common wrong answers and a lot of these One-off mistakes will become common mistakes later when they keep trying to fix it So a few other a few other things that you need to kind of consider if you do something along the lines here You have to really be careful about how you scaffold the question difficulty along the way You want to start with the easiest questions and build them up from there You want to make sure that you're building that process through if you're looking at the quiz times like I do you have to Very closely watch how students are doing on the quizzes and how much time you're you know Good students need to crash through a quiz How much time your middle students need to get through a quiz and where the kids who you know really are they don't quite have it yet How long it's taking them to do a quiz and you adjust those time limits accordingly so that the time limit itself can be feedback I Hope y'all are familiar with the concept of space versus mass practice and how you spread out Practice you don't want a whole big pile of practice on the same thing You want to break it into little assignments and I mentioned before that I've got four or five questions in each quiz Because I'm deliberately breaking them into much much smaller assignments so that it splits these things up I use progress bars to help them the plug-in that Diana mentioned earlier to help them see Along the way what they have done what they haven't Just a quick little bit of advice progress bars changed my homework completion percentages from about 75 percent to about 90 percent Because they see a big red X staring them in the face when they don't do something and they don't want that They figure out all sorts of ways to make sure that they don't see these big red X is staring them in the face And it's just it just works and then I build in some forum support where students It's basically a homework help for them where student will post a question there. I'm having this problem I don't know how to work this and I'll get other students to answer them there Takes a little bit of the the load off of me in terms of supporting them and answering questions all of the time so What questions do you have? Like I said, I'm a chemistry teacher. I like to blow things up Fire is very important to a chemistry teacher. And so, you know, this is this is me lighting things on fire. I Sorry, I can't light things on fire here I can't take flammable things on an airplane to bring them up here and light them on fire for you But this will this will have to do Yes, sir some of it is Well, so before I got to Moodle I've been doing chemistry for about Well, I've been teaching chemistry before about four years and then tutoring chemistry for many years before that and that helps a lot Stealing from other people who have created devilishly clever multiple choice questions before you is good I mean, it's one of those things where you look at an exam bank that somebody's given you before and you read through it And you're like, well, that's easy. They don't even need to know anything throw that out Don't even worry it with it pick something that there's a couple of things in there that you know Somebody's gonna make this mistake because that's really what you're interested in you're not interested in figuring out the students who? Like sort of know something from the kids that don't know something That's not a that's not what I'm trying to build with my class I'm trying to build students who really know something versus the students who sort of know something And so I kind of gear all my stuff that way That the the one from them the CVM that I showed you was a real student attempts from last year And their average score their accuracy percentage on that was like 60% or so and that's you know for a multiple choice test That's a standardized test. That's normal I mean a standardized a standardized multiple choice test has an average score about a 50 because it's doing exactly those sorts of things and It's where you follow along that bell curve in terms of you know, did you get the 50 or did you get the 75? Because that's now starting to get into a standard deviation away from me So that's that's the goal there But yeah, it's hard to it's hard to do the nice thing about the Moodle quiz statistics is though It'll tell you you know, nobody's picking this answer choice You can throw if nobody's picking that answer choice then get rid of it Pick a pick another mistake the students are likely to make and if you've been teaching for a while You know you've seen a hundred ways to work the same problem wrong And you know that 90 of them are going to be the same things from semester to semester What other questions you guys have? Sure, so the CBM is just a behavior like all of the others So just like interactive with multiple tries like Diana mentioned earlier just like deferred feedback Just like adaptive mode no penalties. There's two different CBM behaviors There's deferred feedback with CBM and immediate feedback with CBM And so I was using the immediate feedback with CBM behavior on there So it's built in it's baked into core Moodle if you go into a quiz In the question how questions behave setting in the quiz settings. It's one of the options and the drop-down Actually it does it all for you. You don't even have to worry about any of that so the certainly based marking behavior was The guy who looks after it is Tony Gardner Medlin who's out of He may have retired at this point But he's out of medical school in the UK and they use this for all of their Exans for the medical school because if you have a doctor you definitely want the doctor to know the difference between something They sort of know and something they really know When it is somebody's life that you're holding in the balance You want them to be aware of how sure they are about something and he's done There's if you're a data geek like me and you want to see some really cool graphs So I set up one of the courses on the on the Moodle page I've got links to the the dots about certainty based marking if you want to see some really cool graphs on how it all Shakes out and real like lots of student data on how it shakes out in practice Then you should go take a look at that page. It's impressive All right. Well, thank you very much. If y'all have any anything else, let me know