 All right, I think while we wait for the couple of people to come in, we might as well get started, so good morning and welcome to the first H5P symposium here at UBC. To start things off, I'd like to acknowledge that UBC, which is hosting the symposium, is located on the traditional ancestral and unceded territory of the Musqueam people. As we are here virtually today, I'd also like to acknowledge that here in the lower BC mainland, we are often also on the unceded territories of the Squamish, Slewa Tooth and other Coast Salish peoples. You may be joining us from many different areas and I'd like to take a moment to appreciate, consider and give respect to the lands in which we are situated. I appreciate the land where I am as it provides me with many opportunities. When I acknowledge being on the territory of the Musqueam people, it is rooted in an understanding that I, as a resident of Vancouver and a member of UBC, am privileged to be learning and working on territory that is not my own. If you'd like to post a land of acknowledgement from where you are, please feel free to do so in the chat. Hello, my name is Simon Lollier and you'll be seeing a lot of my face and Kaylee Johnson over here, so I thought we'd just do a quick introduction. I'm an assistant professor of teaching in the psychology department here at UBC. Hi, I am Kaylee Johnson, an associate professor of teaching in the Department of Chemistry at UBC. Now, we're not the only people here, you're going to see us a lot today. But Will Engel, Lucas Wright, Novak Rogik and Renumber have been huge important members behind putting together this symposium. So we'll be meeting them and interacting with them as we go along. Now, why are we doing the symposium? Well, H5P is a flexible, adaptive and open source and free tool that can be used to provide formative assessment opportunities for students. Now, we're also able to create these opportunities that follow the best practice guidelines gleaned from cognitive psychology and learning and memory. But more simply, Kaylee and I have been presenting the projects that we've been involved with. And at conferences, one of the most common questions we get is, well, how do I get started? So we thought we'd throw together a symposium that goes through the different types of H5P content elements, best practices for them, and then actually provide opportunity for you to start practicing developing elements for your own context. So a little bit of a roadmap today. 9.05 to 10am, which is now we're going to be hearing from our keynote Professor Mark McDaniel, which is being recorded. From 10 to 11, we're going to be doing some project showcase around tables, which will show different ways in which H5P has been embedded into classroom context to help spark some ideas of your own. From 11 to 12.30, Kaylee and I will be taking you through getting started with H5P, how to log in, how to do multiple choice questions in the life, before taking a short break, and then having a consulting clinic where you can actually practice what we cover today and have one-on-one interactions with people who have used H5P and can help. Roadmap for day two, we're going to start with a panel which looks at creating H5P projects with students, which is also going to be recorded. From 12.30 to 12 tomorrow, we're going to have a studio session where we can look at some more complex ones, interactive videos, branching scenarios, fill in the words, and the like. We'll have a break from 12 to 1, and then a second consulting clinic. Now, I would like to introduce Professor Mark McDaniel, our keynote. Professor McDaniel is a professor of psychology and brain sciences. He received his PhD in cognitive psychology in the 1980s at the University of Colorado and has held positions at the University of Notre Dame, Purdue University, University of New Mexico, where he chaired the psychology department for a few years, before landing at Washington University in St. Louis, where he is the founding director for the Center for Integrative Research on Cognition, Learning, and Education. Professor McDaniel's work is focused on prospective memory, encoding processes and enhancing memory, retrieval process, mnemonic effects of retrieval, functional and intervening concept learning, and aging and memory. Basically, pretty much anything that has to do with memory and learning, Professor McDaniel has researched it, and he has the publication record to show for it. He has published prolifically across eight books, over 275 articles, which have collected over 38,000 citations, 100 of his articles have 100 or more citations, and he also holds research grants from NASA. His hard work and dedication and insight into human memory has been recognized by his peers and colleagues, as he is the co-recipient of the Lifetime Achievement Award from the International Conference of Perspective Memory. He's a fellow of the American Psychological Association, the Association for Psychological Sciences, and the Society for Experimental Psychologists, and was the president of the American Psychological Association. One notable contribution of Professor McDaniel's work is that he has tried to bridge the gap between research in the laboratory settings and practical applications within classrooms, which happens to be one of the major goals for this symposium. Reading through Professor McDaniel's work, I could spend the whole 15 minutes of the keynote rattling off his accolades and contributions, but I'd much rather hear from him what he has to say about learning, memory, and how we can use these H5P elements in our classroom to best effect. So without further ado, Professor McDaniel, I hand the speaker over to you. I'm really, really thrilled to be here. I appreciate the introduction, Simon. I appreciate you and Kaylee inviting me to this symposium. And I appreciate it because I give talks regularly to middle school teachers, high school teachers, college instructors. And the thing that I tell them is that formative testing, or what I'm going to call retrieval practice, practicing, recalling things, remembering things from memory, I think, is one of the most effective ways that we can increase student learning in the classroom. And we can do it, I think, with minimal cost in terms of preparation and in terms of integration into your current curriculum. And you'll be learning about how to do that during the symposium. So what I want to do today, what I rarely have time to do is to tell teachers why retrieval practice, why formative testing is so effective. And that's what I want. Simon, can you advance slide, sorry. So it turns out there are many positive benefits of retrieval practice, and many of those turn out to have fruitful consequences for learning and retention for students in classrooms. And what I want to do today, Simon go ahead and advance, what I wanted to do today is talk about those benefits. Tell you what they are. For some of you, this is all going to be familiar. For some of you, it's going to reinforce what you maybe already intuitively feel. And after I talk about, as I talk about a selected set of effects, I think maybe most important for the classroom. I'm going to give a brief summary of an experiment that demonstrates that effect in supports, scientifically supports its value, its merit. So at any rate, there are two kinds of effects, just generally there's a direct effect of retrieval in which retrieval itself modifies or alters memory, that makes memory more robust. This is a kind of effect that's most often demonstrated in laboratory experiments, almost all laboratory experiments are oriented or focused on this direct effect. But then there are a host of indirect effects. And I'm just going to cover again once I think it really could be very important in classroom learning. These indirect effects are effects that affect cognition and behaviors of students, but they don't directly affect memory instead they mediate other processes that impact memory. So some people call these mediated effects of retrieval, but most often they're called indirect effects. So I will touch on some of those and show you some briefly some evidence that in part two, I want to move on to selected classroom experiments that show the effects of retrieval practice. I also will call this testing. So that is if we give formative tests, what we're doing is requiring students to retrieve information from memory. So I use retrieval practice and testing interchangeably. So part two that I want to talk about some classroom experiments and I want to use those as a vehicle to bring out some practical considerations for how you might use quizzing how you might use HIP to implement quizzing and ways that are effective in the classroom. These are kinds of questions that teachers often ask should I do this should I do that so we'll touch on that a little bit and break in if you have questions Simon's monitoring the chat. My goal is to try to finish in 45 minutes so there'll be time for open ended questions. So I'm going to try to stick to that if I can. Simon can you go to the next slide. Now I'm going to, I just want to handle something dispense with this very quickly. When we've written about and talked about testing as a learning mechanism is a learning device learning tool you can use in the classroom. We've seen pushback from educators of written history have seen articles written in op-ed pieces that they say, we, you know, you're going in the wrong direction you're advocating testing they're already too much test. So Simon anyone shows side and if they're talking about this or talking about this is a cartoon of a little girl who's woken up from a terrible nightmare, and she tells her parents, she dreams she was being chased by a giant standardized test. That's, that's what people are talking about when they they're saying there's too much testing there's too much standardized testing maybe too much summative attest testing. We're not advocating I'm not advocating that we're talking about testing that's low stakes or no stakes that's used in a as a learning opportunity. It's a learning tool. It's not a summative or standardized test. And in fact, that I don't want to get too far ahead but in fact, probably what if you give lots of form of a test and you're probably lowering this anxiety about the standardized test. At any rate, this is why sometimes we've tried to stick to the term retrieval practice to refer to what we're talking about because it doesn't. It doesn't mislead people and they're thinking we're advocating lots of standardized testing, but I'm going to use it to interchangeably in this talk and so that hopefully everybody be comfortable with that. Okay, so let's get started. So the first that the direct effect of testing is that is that retrieval makes information more durable it alters memory, it modifies memory. This is this is a little bit different than the way educators have thought about testing for a long time and even psychologists are learning there's and which idea was you give a test that kind of it's like a dipstick in the head, and you pull it out and you see how much people remember in it assesses what you know. So, the thinking it always meant that testing a retrieval is a neutral event. It's not, it's not at all neutral that we now know that alters and alters and changes and memory trace. So, there have been a lot of work on that with word list, but in 2006 Rotiger and Karpicki published a paper that I think now has become a seminal classic paper in which they looked at classroom type materials. And they had students study prose passages these are short passages about page ones on sea otters ones on the sun. But Simon, and there were one group, either they all everybody read these passages for five minutes and they kept reading. Then one group restudied the passage they did more study on their own or another group recalled the passage. There's no feedback I need to stress there's no feedback on this recall, they just recall it. Then half of, sorry, a third of the participants in each of these groups had to test five minutes after the end of the session, two days later, or one week later. And if you just minimally think about it might say well, I don't think recall is going to recall is going to be at a disadvantage because you're not going to recall the whole passage. And you're actually restudying it's going to be much better because you're, you're, you're, you're, get to review all of the material of the passage and recall, you're not reviewing all the material because you're calls not going to be perfect. So let's see what happens. So, you can see, off to the left is the five minute retention interval group and the dark blue bar study study in the purplish bars study test, and you can see after five minutes. Repeated study does produce a little bit better performance on the final recall test five minutes later than the study test, but things changed dramatically in just two days. And just two days, if you've taken the recall test with no feedback, you're recalling substantially more than if you repeatedly studied. And a week later, things even augment that is the testing recall has now produced even greater advantage over repeated study. And this is a finding that's now been replicated over and over, which is that recalling or retrieval testing produces good long term retention, especially relative to simply repeated study, which is what students like to do repeated study. So we can say it another way Simon you want to advance. We can say this another way at the. Yeah, so there we go. So another way to look at it is another way to say it is testing helps attenuate forgetting, it helps dampen forgetting. So if we look at the forgetting rates in the study study group, by the end of a week, they've lost 35% of what they could recall after five minutes 35%. Whereas if the if you simply did one recall of the passage, that's it one retrieval you you've kind of helped create a relatively permanent memory over a week, you've only lost 16% over what you can recall five minutes later. So testing is an excellent way to study it's an excellent way to get students to to review material and to make that material more durable and less forgettable. It helps long term attention. That's the direct effect. Okay, Simon. I want to talk about indirect effects which aren't generally discussed in the lab because we're not studying that much of the lab, but these indirect effects could have great consequences in the classroom and I think that they do. But what I'm going to do is I'm going to talk to you about laboratory studies and try to isolate these indirect effects so one thing that retrieval or testing does. We think is that it increases meta memory accuracy. So what am I saying, quite simply what this says is that if you take a test on something, you know pretty darn well, what you know what you don't know. It's kind of an asset test. And we also know that students met a memory that is there. They're awareness of what they've learned well and what they haven't learned well their awareness of what they can remember for a later test, basically, and what they can. That's poor. Well retrieval increases that meta memory accuracy. So let's see that Jerry little a postdoc of mine, who's now at Cal State East Bay, and I looked at retrieval effects on meta memory accuracy in text and classroom type materials. Most of this work have been done on wordless. And what people did is they read six passages about a different region of the world and each passage talked about the geography the climate and the people of that region. Okay. And the critical manipulation was half of the, half of the participants reread the passages, kind of like students do in the classroom, and the other half versus a group that recalled the passages with the topics listed so they were given Norway recall as much as you can about the geography, the climate and people, and then so on and so forth. Then, this is the new part. Everybody was asked to give a judgments of learning for each passage and topic. And it was exactly this, how many of the eight backs, we remember in two days, and they were asked that for the about geography of Norway climate of Norway people of Norway. And then those three aspects for each of five additional countries so the participants are making 18 meta memory judgments, 18 judgments of learning. So it's a pretty detailed survey of what they think they're going to remember about the passage. Okay, so go ahead, Sam. Two days later then everybody recalls the passages just as they were told how much we remember two days, two days later they have to recall the six passages. Okay, go ahead. So we're not interested in what happens on recall. Well, we're not interested in our levels of recall two days later what we're interested in is their meta memory accuracy that is what I'm going to show you is a correlation for each participant, each participant, you compute a correlation than the average across participants between the number they said they were going to remember about particular content geography of Norway is actual number they really did recall. So it's an association the higher the association, the more accurate the student is about what they're going to remember better and what they're going to remember worse. If you get a correlation near zero, they have no idea what they're going to, how they're going to do on the test now they're not. So let's see what these meta memory correlations look like. So an experiment one, the re regroup the correlation was about point two three not that good, kind of low, getting towards zero. And, and that's, that's about what you get in experiments that have looked at meta comprehension and meta memory. Now they haven't looked at the results of testing on meta memory, but a lot of experiments have found it's about that it doesn't give much higher look to the right look at the meta memory actually after people recall the passage. Now their judgments of what they knew what they could recall later what they couldn't recall gets substantially better it gets more accurate. So we, we wanted to make sure that this was a stable finding we read did the experiment with a minor twist here there, and it's the same result. So it's very clear that retrieving really does increase a learner's shot accuracy of what they know about what they can recall later and what they can. All right, so that's a that's an indirect effect that's not usually it's not really important in the laboratory because in the laboratory people were given tests and then they're given a final test and you look at the direct effects of retrieval. In the classroom thing this could be really important because in the classroom students, they're getting their quizzes or form a test and then they're probably restudying. So what does this mean about restudying Simon go ahead. Next slide. Well, it turns out that lots of research has shown that meta memory affects study time allocation that is students use their judgments of what they know about X and what they know about why to guide their study time, guide their focus of study. And in particular, it tends to be the case that students will tend to focus their study time on information they don't think they're going to recall very well. That's a very rational study policy. The problem is that if meta memory is not very good. This rational study policy is not being implemented in a very effective way. So, the idea would be that if meta memory actually improves study time study should become more effective study time allocation should be focused on things you don't know you actually don't know. So, let's see about that. Okay, so I'm going to go to the next one. This is something that's almost never studied in the laboratory for educational type materials that is for word list or parent associate this because usually students are brought back to do restudying. It's a, it's a, it's a more complex paradigm and people just don't want to go to the trouble to implement this or they're maybe not interested. In 2015, we did a study where we looked at, we looked at valid lab classroom materials, and we asked the question, does retrieval increase study effectiveness and is it helping people focus their study more effectively. So, we had people read a research methods text this is a standard research methods test from our experimental psychology course, although we did the experiment in the lab, things like reliability validity random sampling so on it's technical it's full of complicated stuff. Then. Okay, so I'm in two days later, they come back to the lab. And there are two conditions one condition, restudies three times. So they're spoken up into three segments said you can restudy for about six minutes, then another than another, and then another group gets a quiz on the research methods text. And then they get to restudy, and then they get another quiz. I've put the, go ahead, one more click Simon, the, the middle study session is in red, because on that study session, we told participants to highlight what they were studying. We didn't highlight to study we said, as you're studying certain things go ahead and highlight it. We want to see what you're studying. Then, five days after that study study study or test study test session, five days later, trying to approximate a reasonable class trying to reasonably approximate a classroom situation. Everybody was given a short answer test. And by the way the quizzes are multiple choice. Application and definition questions five days later. So, let's now look at. I want to tell you about how people studied in that middle study session the one that was read. First, that's going to tell you about okay, so let's go on Simon. We looked at the highlighting patterns. And, okay, stop right there. And in the test study test condition, we were interested in the portion of items from that first test that first quiz that were correct. That were then highlighted during the study phase. And we looked at the portion of items from that first test that quiz that were incorrect that were highlighted during subsequent study by the way we gave people right wrong feedback we didn't tell the correct answer. We just said, no you got that wrong, or yes you got that right. So people know what they got wrong they know what they got right they don't know the correct answer. And we want to know the proportion of things that got right. What proportion of eyes they get right match content that they highlighted, and what proportion of eyes they got wrong match content that they highlighted. Okay, so I mean let's see the result. So, first of all items they got correct 44% of those items had content that was highlighted or let me say another way. The highlighting showed that 44% of those items that were correct were reviewed, essentially content was reviewed 40 so less than half. What about the incorrect. Let's see this Simon incorrect dramatically increases, almost 80% of the items that people got incorrect on the first question almost 80% there focusing on that content when they restart. So this is just what we want. This is what you want formative testing to do. You want people to learn what they don't know, and then revisit that information during the study. And in this experiment that's exactly what we found. Now, the next question is, well students can focus on that incorrect information, but are they are they really learning about it are they improving. So let's look at the next slide Simon. And on the, this is the, this is the short answer performance, five days after that we study period. And I'm sure you the white bar is the study study study group, and the dark bar is the test study test group. And you can see that if you have it if you get quizzed and then you have a chance to restudy, you're doing significantly better on that final short answer test five days later. So both application and definition questions. So, across the board, you're doing better. If you're given a test to get to restudy. Now, you, what I want to say is well this was the effectiveness of formative testing on guiding people's restudy but what you could say to me and you should say to me is, No, maybe what all you're showing here is you're showing this direct effective restudy had no benefit at all these two little quizzes maybe they were produced all the benefit. Well, we had a third group that I haven't told you about that third group had test test test on that on that restudy day so that all they got was three quizzes, plus feedback that it was right or wrong they didn't get a chance to restudy Simon let's go to the next slide. This is performance on that group on the gray bar the hatch bar and what you should see is that. Yeah, that group's a little bit better than study study study not much. And they're worse. Three tests are worse than allowing a restudy after the first quiz. So, these data suggests that that restudy period is producing additional memory, and it's above and beyond the direct effects of quizzing. To my mind, a lot of what can be happening in the classroom in terms of retrieval practice is is more than just the direct benefit of altered memory. It's also this indirect or mediated effect of being able to more accurately know what you don't know and then directing your study at that material. Okay. Next, so we'll move on to the next point and that is, and this is something I never would have expected. This is really a pretty neat finding. Well, no. This idea has been around for a little longer than 2011, but we spent it all really had the seminal study that shows that it works for classroom. And what it is is, if you take a test, let's say I take a test on a research method section. It's about reliability and validity, and then I move on and I read my next section on whatever it is. It turns out that taking a test on the previous section increases the learning of this new section, this new material you're reading, more than if you don't take the test of old material. So I'm not talking about testing improving memory for studying material. I'm talking about testing improving memory for new material you're just now encountering this related to previous material. So let me tell you about the wisdom and all study just to show you what this what this means. Well, wait a minute. Let me tell you about the study and then I've got the result. So what's been at all did was they gave people text say on greenhouse gases and people read about greenhouse gases describing what it was what costs and what the solutions might be. Some, some people just read, you could consider the three different chapters or three subsections of one chapter. They read all at once. Another group read a section or a sub chapter, then they had to do retrieval practice on it. Then they read another section recall, and then they read the third section. At the end, everybody got a test on the third section. And we want to see did the tests on the previous sections, potentially new learning on the third section solicit. So these are the number of ID units recalled. And they did four experiments. I've just got three shown here with various conditions different text reviews. Different, you might say, well, the people didn't get any interim tests, maybe they were tired and they needed a little break so little breaks were given to those students they did something else, but you can see it doesn't matter. What happens in all experiments, giving an interim test, potentially learning for the new material relative to no interim test. So why is that not sure exactly and there's still debate on it but two reasons seem to stand out. Most probably one is that it may be that if you take a test on preceding material, you, you have that material more accessible as you're reading your new assignment. And so you've got more knowledge that you're integrating in with the new material we know that helps memory. The second idea would be, if you take it to an interim test, you are, you may understand, boy, I thought I knew that, and I thought I was reading it in a way that would help me learn it. But gosh, I found out that that's not the case. I'm not learning as well as I thought. So on the subsequent section, you may, you may up your game. You may do more effective studying and reading, you may read in a more active way, and thus learn more. We're not sure why yet. But this, I think this is also pretty tantalizing provocative for how formative testing could improve classroom learning. Okay, next one, Simon. So another indirect effect that in the laboratory doesn't matter at all, and isn't even looked at, but in the classroom could matter a lot, is that frequent formative testing throughout the semester say weekly could just encourage students to keep up more with the class and study more. Instead of waiting to the midterm, and they study just all at once. Instead, they're keeping up in material on a weekly basis. And I'll show you just briefly a study. Go ahead, Simon, a study done by lemming. This study was my lemming in which he implemented very frequent testing. He had what he called an exam a day first 10 minutes of every class period. And he gave some multiple choice questions and some short answer questions every 10 minutes. And then he looks at what happens compared to another course he teaches, where he just gives three midterms, and he looks at end of the semester retention just to make sure this exam a day. Some people would say if you give a quiz a day you take the quiz, and then you forget the material. So that's not going to be good for long term retention. Well, go ahead, Simon. The retention test at the end of the semester, the exam a day is producing somewhat better retention significantly better than the standard class. But more to the point of what I want to show, go ahead, Simon, is that in student surveys that were taken at the end of the end of these exam a day classes and lemming had about two or three of them in the study he'd like this procedure. Go ahead. So they're asked, students are asked, I studied more in this class than most, and that's talking about the exam a day class, 91%, almost all of them agreed in this learning and memory course, and 85% agreed in the intro site course. So they're studying more, they're following along. That's, that's important. That's what we want our students to do. I kept up better than the most classes. In the learning and memory class 70% said, yeah, I kept up more than I did most classes intro site 65% agreed. So that's an indirect effect. We never see that you never look at that laboratory in real live classroom settings. This is important. This is a positive kind of benefit from frequent quizzing. Okay, one more, just one more quick one, and then I'm going to move on. We don't have a lot of support for this, but we've got some preliminary support. We surveyed over 1400 middle and high school students in our, in our quizzing project, an eight year quizzing project in the middle school. And we asked, how did all these little quizzes in class making feel about the exam 72% said it made me feel less nervous. I was less nervous. And only, only 6% said I got more nervous. So all this quizzing doesn't make people more anxious. It tends it makes 141400 middle and high school students said, I'm not anxious for the exam. It stands to reason, but I don't have time to go to those reasons you can figure it out. I need to move on. So there's time for questions. Okay, part two, testing effects in the classroom practical considerations. Okay, number one, go ahead. Do you need incentives? Or can you just tell students, take these quizzes are going to be good for you. We did this in a by at my center, we work with some biology instructors and we did this in biology, but you can you should take these quizzes are going to be good for you. So that can be great. They're not going to get credit. So we did an experiment, and I can answer this question. An intro psychic University of New Mexico where I was chair Simon mentioned that our intro site course is known as a killer course 30% of the people failed the dean said change that we're losing a lot of students from the university because it can't even pass interest site. So I said, the instructor saying what I'm not going to dumb down the course I said well let's do this. He said he said it my great. So there were two sections of the class go ahead. One quizzes were voluntary. These are weekly quizzes presented on the web students can take it as much as they want. And the other quizzes were required by required I mean students got maybe 2% of the entire course grade was corresponded to taking the question so they get a little incentive a tiny incentive that they are required. What happens on the exams. Go ahead. The light blue bar is the is the same course where quizzes were voluntary less than fewer than 10% of the students took the quizzes, even though they were told these are going to help you. Even though 30% are failing, whereas in the quizzes required, everybody was taking the quizzes almost everybody, and you can see performance on every single exam went up substantially when quizzes were required the quizzes produced learning benefits in the class, it dropped the failure rate by 50%, but the key is, you had to require. If you didn't require nobody was doing it. So that's the first thing I would say, I, you do need some incentive, you've got to have centers. Probably all of you know that, but this reinforces it. Okay. Second type of format request questions. Well, the recommendation from lab studies is go ahead. Is it you want to use quizzes that require more retrieval effort. The short answer makes it is a is a challenges recall effortful to recall multiple choice recognition. Some recall recognition. Glover did a great study back in 89 where he had free recall quizzes short answer quizzes multiple choice quizzes, the effects of testing diminished as the quiz challenged retrieval less and less, but there was no study allowed so that was all direct effects. So the recommendation is use tests that require more material effort so let's see. Let's see about the classroom in the classroom might say maybe not because the classroom allows all these indirect effects I just talked about. So let's see what happens. We've done a couple studies on this. So in, we did a web base in a web based brand and behavior course taught by Janice Anderson University of Mexico. There were weekly assignments which turned out to be quizzes. Go ahead Simon. So, facts were presented on these weekly assignments. Go ahead. And some people. Well, everybody read facts to review everybody got multiple choice quiz other facts everybody chance short answer quizzes. These are counterbalanced across weeks and others, the other facts were presented. So go ahead. And then there's a course exam multiple choice exams go ahead. And I need to say the quizzes are ungraded in the in the introductory site they were great at these are ungraded. I just provided but they're not great. The students know they're not great. But go ahead. Go ahead. But the students were we got credit for just taking the quizzes. So you don't have to give a grade on the quiz you can just tell them, yeah, we'll give you a little bit of credit for taking the quiz do the best you can but you're not going to get graded on it you're just hopefully going to learn from. Okay. And students had to take the quizzes four times from Mexico credit. Okay, go ahead. And here's the main result short answer essay and multiple choice quizzing of the facts produced great final exam performance relative both to reading that's qr rereading the fact, or in X is no additional exposure. So this is a totally controlled within subject design, where every students exposed to every condition, but but the contents completely counterbalance other. All of the possible compounds you could you could worry about the end is low because not everybody did all the quizzes and thus the counter balancing was wrong we couldn't include them in here. So we did it again. Okay, and a little bit better here. Same thing. It doesn't the point is it doesn't matter whether students got multiple choice or short answer quizzes. They still got the benefit. They still got the testing effect benefit. In the classroom, it doesn't seem to me, and I think it's because of all the indirect effects of testing, they need to give short answer to get the benefits you get multiple choice. By the way, the end is very low here, but at IU, they're doing a mini classes so called many classes experiment where hundreds of classrooms are precisely experiment where they're replicating this across all different kinds of content. So in about a year, maybe two years, we'll see if this holds up, but I don't see why it would. Okay, go ahead. Sorry, I'm rushing. I want to give you time for questions. What about when exams are also short answer in this class exam for multiple choice. Same thing with short exams are short answer in middle school and high school. Multiple choice quizzes are just as effective when you have two multiple choice quizzes. Okay, go ahead. So, I don't think you have to worry about giving short answer questions in informative testing in the classroom, because of all the indirect things, but even if you do the other thing about the about the Anderson study is that you don't have to grade that if you give a short answer format test, you don't have to grade it necessarily just give the students feedback. But if you give them incentive just to come just to do the quiz, they will do it and you'll get effects. And I say that because a lot of teachers say to me I don't want to give short answer questions as my quizzes. I think they could be more effective, but I don't want to grade them. We don't have to. Just give feedback. Okay, types of quiz questions is my last point. Try to make it quickly. So I'm going to go ahead. All right, so here's one criticism that's been thrown at testing effects or retrieval practice is that oh yeah it's great for helping students remember facts. But that's not what our class is about my class is about getting students to this what a teacher would say my class to best students to use the information to be able to analyze and integrate and apply it higher level thinking in the blue taxonomy. I'm not doing that. I don't think. Well, let's see about that. It all depends on the type of question. So in the middle school science in our in our big eight year project one of our studies was we gave two types of questions. These are in class questions on both quizzes exam definition questions. Here's a couple examples of the quiz. What's the struggle between organisms competition on the exam. It's a little bit different. It's not the exact same question on the exam. It's a different phrasing on the exam, but it's a again a definition question. Then we also gave application questions. Go ahead, Simon next slide application questions. This would be one. It's a fox and raccoons are a long eyelid pheasant what type of ecological process is this illustrating and the exam question. It's also an application question but a different content. So just want to show you that there's, there is transfer from quiz to exam. It's not identical. Okay, so the question is, do these application questions do they have a benefit. Go ahead. So look at the left part of the screen. I've got the label concept term. It's a definition question. I did the study had different kinds of definition questions. So just call that's a definition question. What you can see is that if you're given a definition question quiz. Yeah, you do real well on the exam. But even if you're given an application question on the quiz, and you switch to a definition question on the exam. You still get benefits over no quiz. That's that point 83 versus point 75. So application questions actually produce some benefits on definition, but it's a symmetrical go to the application exam types on the right. And what you can see is only application quiz questions are producing significant benefits on the application questions on the exam. So, if you want to, if you want to produce higher level kinds of acquisition of higher level knowledge ability to use it since size and so on, then you want to focus on a different kind of you want to, you want to be concerned about the kind of one more, one more example because this is really, this has been a paper that's really gotten a lot of attention and intro biology at Brigham young with Jamie Jensen. She teaches an active learning biology kind of flip faster but one semester. She did this neat thing. One semester all our quizzes all our exams were lower level question types these were retention questions. And this is an example, the humans the haploid number and equals you just have to know this. Just to me as a consequence of this another factual question. One section had all all exams and quizzes were like this. Okay, go ahead. In another section, they were had a steady diet of these application analysis questions so you get a vignette, and then you've got to answer questions that you're reasoning about the vignette answer the question so you can look at those slides later I don't, I don't want to go. I don't have time for that so one section is a steady diet of these kinds of questions, then on the final exam, go ahead, this is my last slide. Final exam, people are given a final exam and both sections with low level and high level questions. And you can see on the high level final exam questions, the high level, the section of the course that has all high level questions outperforms the section of the course that has all low level questions, not a surprise we kind of saw that for Columbia Middle School, go to the next one Simon. But this was maybe a little surprise, even on the level retention questions, the high steady diet of high level questions throughout the semester produces better performance than does a steady diet of low level questions. And this kind of reinforces the blue taxonomy that, in order to do analysis and application, you've got to learn, you've got to learn the material. It's, it's a, it's a, in order to, I don't want to say it, it stimulates answering a high level question stimulates consideration of those basic facts that you got to use and help you learn about. So this is, I think, a very tantalizing finding in terms of how we might formulate our questions, but of course it depends on the course objectives. Okay, that's it. Well, I almost got done a 45 minutes but Simon took a lot of time when he introduced me. So, was language proficiency of the students considered analyzing results. No, but in those, in those classes, we generally evaluated people's science reasoning skills with an instrument, and I can't remember whether we had SAT scores in their verbal SATs, but in some of the studies I don't know if that's okay. Well, I have a quick summary slide. I don't think I need it. I think every go ahead and put it on Simon. Let's, let's open it up to questions. And sorry, it was a bit of a hurry at the end. But, oh, next slide, Simon, please. I just have to acknowledge our grant support. Next slide, grant support. Yeah, Institute of Education Sciences supported our work in the middle school classrooms and James S. McDonald supported a lot of work in college classrooms. Questions. Thank you very much. That was fascinating. One of my questions that I had you've already answered about giving questions that aren't exactly the same as you'll find on the final exam. So, thank you very much. That was fascinating. I learned a hang of a lot. I have a question over here. Recent article Nature highlights despite benefiting more from interleaving practice students tend to rate the technique learned less from it. Other studies also such as those by Bjork reference students misperceptions of how much they learned after session as well. I think this relates to your discussion of metamemory accuracy. What strategies do you use to bridge the gap perception and evidence of the impact of learning for students. So, you're absolutely right. The, the, the, the, when I talked about metamemory actually I was talking about the accuracy of people being able to know what they knew and didn't know. But you're talking about a metamemory judgment on does retrieval help you learn more. That's the judgment that Bjorks and other people asked. Doesn't help you learn more than we studied. And you're absolutely right. Almost everybody says don't reread and I learned more. Even my students say that. So what do I do to convince students. I'm an educational advocate of giving them a demonstration so that they can see for themselves that retrieval practice is producing dramatic benefits to get them to believe in the technique I've been I've been writing articles now for the past three years So I actually do in class, the experiment that I presented at first the car pick in the road of garden car picking, where I present them, they see other passage, I have reread it, then I present the sun passage and I kind of bounce across the passion. And I have them read it and then I have him recall it. And then I asked him what do you think you're going to learn more from, and most of them say reread a week later I test the recall, and everybody is recalling more if they were nearly everybody. And if they had to do a recall initially than if they reread and the, and I got to tell you the, the light bulb that goes off. And when they see that is just really reassuring and gratifying and we have small discussions about it afterwards and students say, I never ever thought I'd do more in recall, because it was so hard to recall. I didn't think I really was learning anything. And then they see they do and these are good students, and they start to adopt a retrieval strategy when they're studying on the wrong. I think if the if the instructor is implementing the quizzes and giving it to students. And what we heard from a political science professor law shoe, who knows about our work and started implementing quizzes was there was a lot of grumbling and a lot of pushback, and they didn't like it at first and by the end of the semester students are saying, This was a great course. I learned so much. I, the quizzes were very helpful. And so I think if you're implementing the quizzes, I, I would still advocate your demonstration, but a lot of people don't want to take the time to do that. And if you tell students why you're doing this, and you say, give it a chance. I think, mostly what what instructor signed up by the end of the semester students you're really on board. That's great to do. I was someone echoes the sentiment actually doing the experiments in class with them. You know, we're trying to embed all these questions into textbooks and Canvas and using h5p but actually telling students that they work is different from experiencing how they work. So that's a great idea. Absolutely. Absolutely. I'm a real advocate of that. It works in my class and I'm really advocating that. Fantastic. We're getting a lot of questions here. So did you look at other forms of motivation beyond grading a few of our faculty are experimenting with conditional release of materials. So take the quiz before you can move on to the next content, etc. Well, I think that sounds like it should work. I haven't looked at that. I don't know any evidence about that. But it seems to me it should work. I don't have. I think anything you can do that gives students motivation or stimulates them to do the quizzing. I think it should be just fine. The problem is when teachers think that if I have it available for students and tell them that will be useful. If teachers think that's going to help is going to get students engaged in doing the retreat practice. I don't think it will. Yeah, we did an experiment son experiment. Sorry, we just study in biology that we just published last year in CBE life sciences in which in biology students were given quizzes but then they were allowed to access the quizzes to study for the exam. And it was optional. And what we found was that people weren't accessing the quizzes and studied, but those who did were doing lots better on the exam now it's correlation you don't know cause and effect. Again, I mean to the biology instructors that that advice was this would be good if students did this you ought to give them a little incentive to do it. They're not going to many students are going to do it on their own. So I think you have to give a little motivation. I don't have any feel for how the different motivational policies, whether one will work better than the other. But I'm feeling as students. If you set up a little motivation students will respond to it. We have a program here at UBC where first year students come for the week before their first year. And we talk through what's it like to be a university student and one article, you know, I sort of talk them through all of retrieval practice and into leaving. But there's one article that always makes me a little bit hesitant or sad that students often know the best practices but they don't always do them. And I really liked your idea of even just giving, you know, you did this you get a grade and it's nominal it's 2%, but it's that little nudge to get them to do it. So we have another. We have another question. Oh, sorry. I was going to say, I think you do need to give them a nudge. They have to somehow learn that this is working. And then they embrace it. And I do it with a demonstration in class and also just quizzes. But I think if you give students incentive and they have to do these quizzes, I do think they start to appreciate the value of the quizzes. So I often find that we don't explain or show students why we do the things that we do in teaching. There are often just things that are inflicted on them. And this is bad since we need students to buy into engage with them 100%. We absolutely do need students to buy in. I try to make them collaborators. I agree with you. You can't skip students interleaving and retrieval practice and expect I think a positive response. I think you have to say, where I'm not trying to torture you. That's what I'm not trying to torture. What I'm trying to do is set up a course that gives you the best opportunities to learn as much as you can and retain it. And these are the techniques we know work from the, from the learning sciences. So, in fact, there, you can do other things to maybe make these more palatable. And this is a simple example. In my class, I, I said, well, given the science and you've seen the science we're going to do quizzing little bit of quizzing every day in class. And I, students were saying to me, I'm quizzing. I don't like the word quiz. It makes me anxious. It makes me unhappy. Can we change the name of this? And I said, sure, these are learning opportunities. So every day in class we're going to do these learning opportunities. So I don't label them quiz one, two or three anymore. I label them learning opportunity one, learning opportunity two. Just that change has created a different vibe in the classroom. Just that one change. That's fantastic. That's, that's very smart. I like that. We've got another question. Was the impact of doing this in all student courses considered so preparing for one course weekly quizzes maybe manageable but doing six of courses might be overwhelming. I'm sorry, Simon, I kind of missed. In one course doing a quiz a week would be fine, but doing more quizzing or if you've got six courses you're teaching. Is it hard to implement that? I think from this, from the student perspective that a student's got six different courses and each of them offer weekly quizzes. Does that become over, overbearing for this from a student manner perspective? Well, I have a definite, it could be, I guess maybe it could be, but a couple ideas are that one in the brain behavior course, it was a web-based course. And I think a web-based course is students have told especially during the past year to the pandemic that they'd fare better in those courses if there are assignments that they're having to do. And so the assignment can be the quiz instead of some other kind of homework. It can be, it can be a quiz. So that's, that's number one is that substitute if you can, if it makes sense, other assignments you might have substitute a quiz for. That is the same. That's the learning assignment. A second thing is that if students feel they're being graded on the quizzes, maybe it does seem overwhelming. But if students are told you're just doing, I just want you to do it, just complete it, just try it. I think it's a little bit different feeling for the student. It makes a different sense. It's not overwhelming. It's just, great, I'll log in and I'll try this. See what I know. They don't have to do any extra preparation. They don't have to sweat it. And so in a sense, I think that if you do it that way, you'll get credit if you try it. But we're not going to grade it. We'll give you feedback. I think that's much less of a tour for students. And in fact, I think, I mean, students have told me that's kind of fun to do these. It's fun to test themselves. It's fun to see what they know and don't know when there's no evaluation of it. They don't like being evaluated, but they like checking their knowledge. That's right. I've embedded a lot of questions into my textbook and I give practice midterms and I thought embedding a lot of questions using H5P into my textbook would do away with students asking for more questions to practice on. But they're always hungry for more questions. But I like the idea that you mentioned about just changing the way that it's framed and saying, well, look, you know, we're not marking this on right and wrong, but whether you did it or not. And so it's low stakes. It's not a quiz and you'll get some participation. It's called an understanding check. Let's check your understanding. Here's your assignment, an understanding check. I'm going to do two more questions and then we need to wrap up. Unfortunately, the second to last question when you mentioned correct answer feedback. Does this include feedback on why other responses were wrong? No, that's we don't provide that. But some people think that you should provide that. I can tell you that we just published a study and it does need replicated in which we were interested in the amount or the elaboration of feedback that was given to the quizzes. And then the effects on re-studying. And what I can tell you that we found is that you can give minimal feedback. That is, you can just tell people how many they got right on the quiz. 8 out of 10, whatever. And if you allow them to have the quiz back while they're re-studying, the effects are every bit as good as if you gave students correct answer feedback, students in another group correct answer feedback. And if you look at their study behaviors, again, we did the highlighting thing, students who are just given minimal feedback, they also focus on items they got wrong. So students seem to have a pretty good sense even without you telling them that I didn't, I didn't understand this very well. I didn't, I don't think I got this one and they focus on that information on re-study if they have the quiz in hand. If they don't, they don't, they don't tend to remember the questions and they can't focus on re-study. So a lot of the faculty are saying, I don't want to give all these quizzes. I don't want to grade them. I don't want to have to be responsible for keeping that in the codebook and so on. Our laboratory study found that you don't have to. If, if, if it's, if the, if this is just told you got eight of these right or seven of them right, maybe they don't even have to be told that. I don't know. But that minimal feedback, if you allow them the quiz, if you get the quiz back and they can use it to re-study, our evidence and one experiment shows they are just as good as you can get correct answer feedback. So our interpretation was more and more amplified feedback doesn't matter as long as the students have the quiz and they're able to use it to help in hand while they're re-studying. But, but I have to say we haven't really examined explaining the reason why the wrong answers are wrong. And I know a lot of people advocate that and I'm not saying that it shouldn't be done, but I don't think it has to be. I'm reading through a number of them. None of the comments are sort of elaborating on the points that you've been, that you've made. We will probably head on with the conference soon. We'll probably start at about 10 past 10 with the project showcase. Professor McDaniel, I'm sure if we were here in person, there would be thunderous applause. Thank you so much for taking the time in your busy schedule to come talk to us today. I'm sure that's given us a lot to think about as we go ahead and try and create these formative assessment opportunities for students and embed them wherever possible. And this has given me a lot to think about as well. Thank you so, so much. And there's been a request for the references that Professor McDaniel has spoken about. We will go about and mine the PowerPoint for the references and we'll post it on the symposium website. Simon, if you can't get the full citation, let me know. I can send the full citation. Thank you very, very much. Thank you very much. We'll be starting again in four minutes at 10 past 10 with the project showcase showing how H5P has been used in various contexts. We'll see you in a couple of minutes. Okay, I enjoyed it. Thanks. Have a good symposium. Okay, bye bye.