 Hi everyone and welcome back to the session on providing feedback by myself and Dr. Kaley Johnson. What I thought we do just to kind of ease us into the process, hopefully we are all used to in some capacity giving some feedback. So what I'd like you to do is in the chat over here I would like you to imagine that there is an evil instructor who is grading student papers and what is the worst piece of feedback that you can imagine this evil instructor giving their students. So take a minute type it in the chat and then I will say enter and we all press enter and see what we come up with. So we've got a minute to think about what's the worst piece of feedback you can imagine giving a student. All right, I see a couple of coming in if we can start putting in here we go. You clearly didn't even try are you kidding and oh dear great fantastic amazing you're the best unclear I think you should go to the writing center no feedback at all disappointing should have listened better yowza a very general very vague and very bad. We're quite familiar with what that feedback is and part of this was inspired by some techniques that Lucas Wright and Will Engel and the team spoke about but part of it was also inspired by this rated post from our UBC which was about a month ago and they said what's the harshest feedback you've ever gotten on an assignment and here are I think some of the top contenders. One student said that they just got a series of question marks on one of their assignments. One of the students said reading your introduction was like being a blindfolded hostage led through an obstacle course feels particularly harsh. I had a profit my last college who had just write awkward beside any sentences she didn't like not particularly harsh but it was probably some of the most useless feedback I've received. The teacher literally wrote system dot out of print line maybe computer science is not for you. I dropped out of the class and now work as a professional movie watcher. My problem told me that my final paper was disappointing and said I disregarded everything we had learned in the course. I didn't think so. I won't say the name but it was a first year course and so we see over here that sometimes assumptions are made and feedback that are go against what the students actually the time and effort that they put in over here and we see that this feedback is not useful at all and apart from it being just writing awkward and not being useful it can be at worst demotivating. One student who said that they dropped the course and now professional movie watcher. And so now you know we've got this question well how do we give the right type of feedback. When is it appropriate to give feedback when is it appropriate not to give feedback. And so we find generally within the feedback literature there are several questions but they can kind of be lumped under the umbrella of what goes into feedback like what it makes good feedback in terms of content and when to provide feedback. I'm quickly going to look at this question of when do we provide feedback because there's a whole literature so this Kulik and Kulik is a meta analysis saying that it's best to provide it immediately. At the same time this is a narrative literature review saying that it's best to delay the feedback and then this is some results from looking at many classes experiment and experiment where they had a handful of classes looking at timing of feedback and it says well it doesn't really matter. So even over here within when is best to provide feedback you can find literature saying best immediately delayed or doesn't really matter. So where does this kind of leave us with when to provide feedback. Harrison Rothenfeld sort of in 1985 already stated that well feedback timing is important but what is perhaps more important is focusing on the content rather than the timing of the feedback. So then this is what I'm going to focus on for the rest of the session is what is the content that goes into providing good feedback. So to be able to do this I would like us to be able to answer these questions by the end of this session what are the broad types of feedback that exist which are best of these broad types. Is feedback only effective in summative assessment? What role does or what role can feedback play in formative assessment? Could you use text feedback or audio feedback and do you always need to include rich feedback? And so hopefully by now we'll have a good understanding of what makes good feedback when to provide it and how to provide it. Now I am a psychologist and so I am inherently data driven and I'm going to be showing a number of things called Cohen's D's and Cohen's D's are what are known as effect sizes. And these effect sizes tell us about the difference between two groups in terms of a standard deviation unit. So if you've got one group that doesn't receive feedback and one group that does receive feedback this Cohen's D can tell us about and we see that there's a difference between the two group. This Cohen's D can tell us about how big is this effect size. Now typically some benchmarks Cohen's D.0.3 is considered small a Cohen's D.0.5 is considered a medium effect size and a Cohen's D.0.8 is considered a large effect size. Now one thing that I particularly like about effect size especially if we're thinking about it in terms of formative assessment but especially in terms of summative assessment just looking at the impact that this can have on students and student performance. My midterm that I had two weeks ago had a standard deviation of 14.5 percent. So the standard deviation talks about the spread of the data and so there was a standard deviation of 14.5 percent. Now let's say that I go and take a approach to providing feedback and the Cohen's D associated with this is 0.5 and so what this means is that I can expect there to be 14.5 percent times a Cohen's D of 0.5 a 7.2 percent difference in my class who received this particular intervention versus those who didn't receive that intervention. So for me these Cohen's Ds are quite nice they're quite interpretable if we know what the standard deviation is we can kind of see well what does that translate for students and I think it helps contextualize the size of the effect. So feedback is it effective? Well we see that yes it is one of the most studied which made a lot of literature review to comb through for today but it's one of the most studied pedagogical tools that we have and you can see that by the number of meta-analyses. So meta-analyses are sort of statistical summaries of all the studies that have been done on a topic. And so for instance Kuga Indonesia in 96 published a meta-analysis of 131 different studies into a meta-analysis and there were 12 000 participants in this meta-analysis and they found that feedback interventions typically had a Cohen's D associated with 0.38 percent. Cohen's D over here we go to 2020 with the Wisni ski using another 994 effect sizes with 61 000 participants found a Cohen's D of 0.55 meta-analysis from 81 a Cohen's D of 0.6 another meta-analysis from early 2000 the Cohen's D of 0.71 Hatti has done a lot of work and there's a lot of really good research that Hatti's done and provided a number of really good outputs on various pedagogical innovations. One of Hatti's books that they've published is a review of 800 meta-analyses on pedagogical innovation so if you're looking for that that's a very good way a very good place to start and they found that sort of these effect sizes for feedback range from 0.7 to 0.79. So we see at worst providing feedback has a small to medium effect size and at best we have we see this feedback provides a rather large effect size for student learning. Now this Wisni ski paper say that there's you know as you see we go from 0.38 to 0.79 we kind of cover the gamut of small effect size medium and large effect size and they take note of the substantial variance in effect size because there's different types of feedback that we can give and we can break this down into two basic buckets the one being a valid of feedback and the other one being descriptive feedback. So in a valid of feedback is receiving a grade back on something so quite often with summative assessment you get a grade and that's what that gives you some feedback as to how you did it not a lot of information but it gives you some feedback. You can have written praise and we actually saw at the beginning one of you said great fantastic amazing you're the best and this is written praise and this comes from a behaviorist point of view that if we provide if we provide a positive reinforcement then students will want to do the thing that got that positive reinforcement. At the same time we can look at criticisms why did you even bother you were wasting my time what are you doing here print line why are you studying this class and also judgments and then also just sort of evaluators also just sort of giving right or wrong giving correct or incorrect and all of these fall under some sort of evaluation of performance that what you see is that they don't really give any indication as to well what why was it right or why was it wrong and so this is when we get into descriptive feedback that provides information about what was done what has been considered what was right and then where does improvement need to be made and we can see that this descriptive feedback kind of aligns with the main goals of feedback because feedback is trying to fill this gap between what is understood by a student and what we want a student to understand and not only students you know we have these student evaluations of teachings we have peer reviews of teachings we want to get feedback from our colleagues or from our students because we are interested in where is the gap between what we do pedagogy and how can we improve pedagogy when we look at students they express a desire for feedback so Higgins found that students absolutely expressed a desire for feedback and but then this in a really interesting paper from 1986 say that students prefer having descriptive feedback rather than value of feedback and so when we look at this type of elevator the value of the feedback praise great critique we see that this kind of feedback is not informative to the students it may have no impact on the student beyond the realization that they got the answer right or they got the answer wrong so evaluative feedback while does contain some information you heard our keynote yesterday Cynthia Brain talking about corrective feedback it doesn't really sort of give any indication as to where students can improve so now looking at the Winnicki paper they kind of delineated feedback into three types of feedback punishment and reinforcement which falls under evaluative and this comes from sort of behaviorist principles in psychology that if you praise someone doing something then they will repeat that thing that got the praise if you criticize someone or you do some averse of consequences to that person they don't like averse of consequences so they'll stop doing that thing and so feedback was was sort of designed or can be designed according to these behaviors principles you also get these corrective feedbacks and that's just information about task label and that's what Cynthia Brain was talking about yesterday in part of her keynote where it's just you got this right you got this wrong and then you can either provide the right answer alongside this or not and then you get high information feedback and this falls more towards this descriptive feedback where you talk about some sort of description about the performance what was done right what is good what is correct about it and then where improvements can be made and this is corrective it corrects any misconceptions and it stimulates reflection on the learning process for the students now why I mentioned this punishment corrective and high information feedback is because this Winnicki meta-analysis that looked it over I think what was it 61,000 students with what was it I forget how many effect sizes there was they looked at this effect size from students who had punishment and reinforcement compared to those who didn't have those who had corrective right wrong versus those who didn't and those who had really good high quality information versus those who didn't and we see this punishment and reinforcement giving praise or giving some sort of a verse of response to feedback does actually improve students performance but we see it's a rather small effect size when we look at corrective feedback just right or wrong we can see that that then almost doubles this effect size where we start going from small up to an immediate effect size when we look at high information I would like you to just look on the graph and where do you think on the graph with high information feedback for what effect size would you expect to see because when I read this paper and I saw this effect size we see that it's almost students perform a full standard deviation beta when they get this high information feedback compared to when they do not get high information feedback I see over here Tamara saying that many students are focused on the grade they often don't even look or consider the feedback and you'll be talking about that a little bit later and rare that is one of the issues when it comes to summative assessment that where I think formative assessment has something to say about that so so far we've looked at one of the broad types of feedback we've put them into these two buckets evaluative and descriptive we looked at evaluative being things like punishment reinforcement and just corrective feedback versus descriptive which contains a lot of information about what was done correctly and where improvement can be made we see that this high information descriptive feedback is based it can lead to a standard deviation improvement in outcomes for students and typically that is reflected in an improved grade because they take the feedback they integrate the feedback and then do that for the next thing that they're next assignment that they hand in what role does or can feedback play in formative assessment so this is where Tamara kind of hit the nail on the head because when we look at summative assessment getting a final grade we saw already that students really desire descriptive feedback and students not only want sorry they really desire feedback but not only do they want feedback but they want descriptive feedback however when you provide in a summative assessment descriptive feedback with a grade students actually don't read the feedback presented with the grade they just look at the grade and take that on and so within summative assessment research has shown that providing descriptive assessment feedback with a grade for summative assessment does not lead to improved grades on a subsequent summative task we see Sinclair and Clareland looking at medical students and they had a paper graded they received their grades back electronically and then looked at how many students actually came and collected their paper with the descriptive feedback and this is within the field of medicine and they found that 50% of the students didn't even come and get their paper with descriptive feedback and then as I sort of alluded to earlier when you're providing descriptive feedback with a grade on summative assessment it doesn't need to improvement on future tasks so you can see how this becomes immediately a little bit tricky because how do we provide feedback that is effective that is unaccompanied by a grade and that's rather difficult to do within summative assessment and so what do we do we look towards formative assessment and we've been couching H5P during this symposium as a keen tool to provide formative assessment for students now you know me I enjoy data so I went and looked at there are some new meta-analyses ArgoVol who does a lot of really good research and pedagogical innovation Phelps having done two meta-analyses from 2012 and 2019 found that formative assessment is amplified when you include feedback so over here once again trying to what to contextualize the size of this feedback we see that the testing effect so just providing practice questions like we know is really good practice we spoke yesterday how this can lead to 10 improvement in grades just providing a testing effect without anything more than corrective feedback in a formative assessment leads to a huge improvement in summative assessment later on however when we I didn't punch it all but that was really great that you asked the question exactly when you did so thank you for not ask the question when you when you gave us that insight however when we give this descriptive feedback in the testing effect and formative assessment and we make this license descriptive we see once again a huge increase to almost a standard deviation improvement and when your summative assessments for instance my midterm has a 14 percent standard deviation I'm sure you asked any student if they want a 14 percent higher grade or some equivalent of that and they will say yes please and so I see something's immediately gone wrong with my my what you call it animations there the feedback in formative assessment is inherently powerful because it tends to be task level feedback rather than sort of summary feedback on they've handed something in and this is a summary of what went well and what didn't go well with no chance for improvement but if you're performing a task and this is where sort of some timing comes in because task level feedback is inherently immediate in nature we see that students are able to correct misconceptions or see misconceptions in the here and now this tends to be educative in nature because you're able to find out what these misconceptions are and then correct those right there and then one of the other problems when looking at this literature where we look at feedback within summative assessment opportunities and if we solve some of those reddit posts earlier on is that students don't often find the feedback useful because it's written in a way that's super convoluted that's not easy for them to understand and so when we are looking at feedback in formative assessment we need to make sure that the feedback is useful written in a way that the student can understand another thing that we found within the examples that you all gave thank you very much for giving those examples at the beginning of the session is that any type of feedback that has these kind of behaviors principles where you are rewarding or punishing something we need to make sure that it is not the student that we are rewarding research shows that if you have a student and they perform well on a test and you say well done you are very smart you are encouraging something that is essentially seen as unchanging I am already brilliant so why should I try for my next assessment and we see that this type of praise it increases what's known as extrinsic motivation so this motivation that comes from external I am not doing this because I want to get good grades but I'm doing this because I want to get praise but if you're praising the thing that is seen as somewhat stable I'm really a genius I don't need to try subsequently we see when you praise the child or the student rather than the process performance decreases subsequently but if you praise the process wow you studied really hard what were the study techniques that you used I see you're working hard on this this then shows and signals to the student exactly what they are doing that is working and then they know to repeat goes and we see that this type of praising the process increases this intrinsic motivation which is what I think we are all trying to cultivate a love for learning in our students and understanding and the search for knowledge and a lot of that comes from intrinsic motivation so praising the process does indeed increase intrinsic motivation at the same time I know that there is research published on this I'm sorry that I couldn't find my my references for this part but we see that also you don't want to criticize the students that was a stupid mistake what are you doing in this class are you even even trying really demotivate students to the point where they may drop out so if you do need to level some sort of criticism make sure that it's focused on the process I see that there is some trouble understanding s and one reactions and so focus on looking at understanding this point of that right so you know often when students come to me and say how can I improve my performance on the midterm I ask what are the study techniques and they say I read and reread and I say you know well that's not the most effective technique while we do need to read our assignment I think our what you call it the the pages to get the information and we then need to make our brain work and so here are some things that you can try out so once again I make a very big point of trying to say well look you know this is not a reflection of you this grade is not a reflection of you but rather the processes that are going on and those are easy to fix and then this is something I sort of you know in developing a session with with Kay Lee spoke a lot about and this ties into what we spoke about yesterday as well if you've written really good distractors these distractors should be based on common misconceptions that you know exist and so if you write these good distractors you should be able to predict why a student chose the answer and then be prepared to write to that common misconception and so this is hopefully this is just a couple of bullet points that can help us orient ourselves to writing feedback that is so do you think AI can create good distractors I think AI can create good distractors but as with everything it needs to have an eye cast over it by by the instructor and that's sort of what I was thinking with in terms of feedback which I asked Sven Torre is that you know we we can you know there's a lot of literature on providing good feedback and I would have tweaked some of the feedback in the northern lights or in the sun saying that um you don't find a lot of carbon in the sun it's not abundant then they went to say say you know like helium and hydrogen is abundant and for me well I think that that's fine providing these hints as to what the answer might be I would have probably stopped at saying well um yes we do find carbon in the sun but it's not in abundance which element do we find in abundance the distractors must be plausible to students at a particular level that's right and so I think a lot of these things with this AI generated stuff and Sven Torre kind of Torre spoke about this as well is um you need to go and review this stuff and so you need to review the distractors as well and then you know sort of when I've asked I've been playing around with chat gpt and how it can help me write better distractors um I might ask chat gpt to include a distractor about this misconception and then it's pretty good at integrating the distractor into the question for these formative assessment pieces so I think AI can create good distractors but one of the problems that we know with AI is that AI is based on a corpus of knowledge and um billions and billions of paper and there was an article released I think in science early this year looking at how stereotypes against different groups while the stereotypes may have changed about what stereotypes are held towards white individuals or black individuals or men versus women the stereotypes the words that have used have changed over the past 200 years in written texts you look at the valence whether they are positive or negative that has remained constant over 200 years and so we're getting this AI to learn from these texts that are written by humans and humans have these biases that can lead to um uh um sort of creating these distractors that Lily was talking about that can be threatening to certain students and so I think it's um of paramount importance that we read over the distractors and the feedback with a critical eye with a critical eye towards um EDI issues so that we can catch any of those and before they get make it out to the student very good question thank you for asking I hope I answered it and I spoke for long so now we've looked at the role that um assessment plays within formative feedback um it plays a really important role once again boosting up um uh the effect side of um these kind of practice tests that we give students the next question I'd like to start looking at is do we should we use text feedback or should we use audio feedback and I've included this in because you know it was mentioned in Sydney of Brains workshop yesterday this um you know sort of the theme to this symposium is creating interactive videos um um I you know sort of interactive videos allow us to create the opportunity to provide verbal feedback and is there anything over and above providing verbal feedback or written feedback that there's something special in verbal feedback or is written feedback enough so we can see these interactive videos can allow you to build on prior knowledge one thing I really like about what the videos that Kaylee has done is that um especially for sort of naming uh um uh naming molecules and chemistry using nomenclature you need to understand certain parts before you build up to the next part and having someone sort of talk about feedback is almost like another mini little um education and so you know getting something wrong then provides another opportunity for students to learn and so as I said earlier I personally don't like providing the correct answers I don't like providing very strong hints towards the correct answer because the literature shows the more we can make our students think the stronger the um the connections are made and and the greater learning takes place um but rather I like to tell them why the chosen answer is incorrect and so for instance um in the sleep disorders video that we looked at yesterday if I may quickly show it over here I'm quickly going to forward to this one hey doc I was taking the history of patient number one and they fell asleep am I really boring and so over here which sleep disorder is characterized by suddenly falling asleep now the answer is narcolepsy but let's say someone says somnambulism they click on somnambulism and over here the feedback that I've given is the etymology of the word somnambulism gives us a good clue as to what it is somness is the Latin for sleep where ambula is the Latin for to walk the current patient seems to be doing some sleeping so that's acknowledging what they've got right um but they're not doing a lot of walking walking try again so then students can click on this they can try again REM behavior sleep disorder check REM sleep behavior disorder refers to something that happens during REM sleep the patient cannot seem to stay awake which is the issue over here try again and eventually they hit upon narcolepsy and they check that now what I also like to build in um to this feedback on a correct item is um other sort of cognitive psychological tricks that need to greater learning so over here instead of just saying correct and having them move on I ask a probing question what is the process of suddenly falling asleep called and hopefully the student will say oh that's called a sleep attack or if they're not sure what it's called when they relearn the stuff they know oh I need to look at what this um process of suddenly falling asleep was called and then they move on oh narcolepsy so there's an example does that mean they work for there's an example of um written feedback when we go look at spoken feedback um I won't show this too much because I know that you daily showed it yesterday but for those of you who aren't able to make it today uh hey doc your e um EEG machine thingy is beeping what is this mean so this is a bit of a roundabout question I felt a bit mean putting this in because a this is a um sleep spindle and so students should be able to identify that that's a sleep spindle and know that sleep spindles occur during stage two the student clicks on stage one when we're awake and alert our brains exhibit beta waves in stage one of sleep we would expect to see alpha waves which are lower in frequency and higher in amplitude than beta waves as the patient continued in stage one we would also expect to see theta waves which are even lower in frequency and somewhat higher in amplitude in the EEG that Preston showed the patient does seem to exhibit theta waves but there is also a burst of high frequency high amplitude activity that is not consistent with stage one sleep which stage of sleep involves theta waves as well as brief bursts of electrical activity so what I like about this is that you can see that the feedback acknowledges what was you know where they might have thought it was stage one that it wasn't theta waves and that they do see theta waves over here but then without giving the question that gives a little bit more information a little bit more of a learning opportunity um before providing the opportunity to take again so I've just shown you two videos one is what we refer to as like a traditional interactive video that's where pauses and asks a question and then you can give your answer and you can move on if you get it wrong it gives text feedback right which one do you use this interactive where it gives text feedback all this what we call this branching interactive where the question is wrong it branches off to a mini explanation um but which one should you use is it really worth putting in all the hassle to make a branching interactive or do interactives work well and so uh Kaylee said that we would um speak a little bit about lab study that we ran last about two years ago and we were looking at this um uh sort of the seminal paper by Rudecker and Karthike in 2006 showing the doer effect lasted one day five days uh two days to five days later people who had done practice quizzes out performed those who just studied and re-studied the material so we built our study um on this where we had students come in and read the material first then they did a two-minute distractor task and then they either reread the material they watched a traditional video so the the one on stages of sleep we made that into traditional video that just played out and gave all the answers didn't stop and plot for information versus an interactive video that stopped asked question and then gave text feedback versus branching interactive video that then would move on and give um a little explanation we had 408 students come for part one and then we had a 339 students return a week later for part two as I said all students read eight stages of sleep a passage on stages of sleep from the OpenStack textbook we then got them similar to the Rudigan-Karpike study to do two minutes of math tasks um which was a very popular part of the study by all people and then we either got students to reread the same passage or they watched this interact this traditional video or they watched the same traditional video but with some interactivity embedded with stopped and also questions with text feedback or they watched this branching interactive video we then got them because we're so popular we got them to do another two minutes of math tasks and then to half of them we gave them a test a multiple choice question test and then a write down as much information as you can remember and then we asked them how did they find the study the other half of them we just gave them the survey one week later we called all the students back all the participants back and we got them all to take a multiple choice question on a test on the the topic that they've done a week earlier and they did a multiple choice question and then they wrote down as much information as they could remember we then showed all the students you know here this is we had some of you reread the material again we also had some of you watch a traditional video here's the traditional video here's the traditional video some of you watch this interactive here's this interactive some of you watch this branching interactive here's this branching interactive and once again we asked which was your preference which one would you prefer the study from which do you think is better which provided you with most control for the first time that they came in remember we said we did the the manipulations and then we got them to fill out a survey we asked overall the video helped me understand stages of sleep the video was enjoyable the video was effective those of the who read the text twice said the text was effective the text was enjoyable i'd control over the pace of my learning when watching the video or reading the text the pace of the video the pace of the text was appropriate i was able to master each concept before moving on to the next one i'm going to show you a lot of results here this is better understand was enjoyable was effective had control appropriate master these are students who read the text first and then read the text again those who read the text watch the traditional video those who read the text did the interactive video those that read the text and did the branching interactive video and then higher numbers refer to better understanding more enjoyable better effect more efficacious so here when we see better understand students felt at least students who had read the material twice thought they were able to better understand the material from reading a text which makes sense because the text has all the material there when it came to enjoy ability they found that they surprisingly enjoyed the traditional video the least they enjoyed the branching video the most but you can see it's not a lot of gains over those who read the text in terms of how effective it was at helping them learn we see that these branching videos and these interactive videos um outperformed both a traditional video and outperformed the text now what's really interesting is that obviously if i give you something to read you have control over how quickly you read so i was expecting text to have the highest control that here we actually see that it was the branching video and the branching interact the interactive video that had the most control students felt that most control over the page much more so than the students that read the text here we see that those who had the interactive video found that it was appropriate and when it looks at mastery students in the interactive and the branching interactive felt that they were able to master the material more and this replicates those three studies that Katie showed yesterday during during her workshop on the effectiveness of interactive videos now this is the second when they all knew that they were text there was a traditional video interactive and branching video and we can see that when students knew of all the options that were available they showed the stronger preference they said that interactive and branching interactive were the most effective the most enjoyable the most engaging and their preferred style now one thing i would like to point out is that for three out of the four of them we can see that this branching interactive video is statistically higher than the interactive video but not by an astounding amount right and i'll return to this a little bit later but now in terms of feedback in terms of what i want to get across today the feedback in the interactive video which was text feedback and the feedback in the branching interactive video which was spoken were near identical we tried as much as possible to have verbatim feedback so that it wasn't something that one group was getting different feedback and that's what led to feedback so over here the interactive video and the branching interactive video the feedback was near identical the only thing that differs was did they read it or did they listen to it and we want to know did the students perceive the feedback as more personal when it was during written in text or when it was spoken to them in like a mini lecture and over here we see a cohen's d of 0.7 difference and that it was almost verbatim the same feedback but when it was spoken to them they found it so much more personal than when it was interactive and this supports what mayor calls the personalization principle and mayor was put forward the model that Cynthia brain spoke about a lot yesterday that when students feel that there is this personalized feedback has a large effect because they feel that there's more of a social partnership with the narrator and that needs to increase in motivation and greater effort so over here we see a clear winner that the branching interactive video is superior in terms of feedback in terms of how personalized it felt for the students we then asked students to give some feedback and so over here they said it's really interesting it actually made me laugh and kept me engaged the integration of video based feedback also seemed like an extra practice situation instead of the process of elimination since you got a question wrong it seemed to provide more information and context about the right answer so here students you know we didn't ask about feedback we said how did you find the videos and this one voluntarily offered up that the feedback was particularly important in providing more information extra practice and context as to the right answer even though it didn't give the right answer right the feedback seemed more personalized when the answer was wrong and I was able to pinpoint exactly where my thought process went wrong with an in-depth video explanation of the correct answer and quite often these explanations were left in 40 seconds and they found this quite in-depth I was able to learn more information each time I made a mistake in a way that helped me learn and remember so here we see that in terms of feedback these interactives do give good feedback and we see that it has these beneficial effects but when you are able to make it spoken and more personalized that seems to have benefits on students so we see these clear differences in personalization of feedback but we don't see these clear differences in effect that we see both interactive and branching interactives were more effective more enjoyable better mastery we see some gains in branching interactive videos but which one should you use we see that whether you choose branching interactive videos or just plain interactive videos they perform significantly better than text and then traditional video in several metrics students felt that these interactive forms were best especially showing a preference for branching interactive but we see the gains of branching interactive was significant or often small but the power was really lying in this personalization and so when we're kind of trying to figure out well which one do I use I feel that it's important to bring up some other feedback that students gave and so over here it feels like the interactive videos are the best combination of being engaging and time efficient so some students found it a little bit laborious going through this branching interactive that then stopped to explain and kept them as a captive audience now whether we want to do that remember in Cynthia's talk yesterday Cynthia Graham's talk yesterday she said the more you can sort of force students to ask answer questions the better the effect is I'm not sure what you know forcing them to listen to feedback versus read feedback but even though students might have a preference for being more time efficient it might be better to to have the less time efficient one but you know we see shorter time than branch interactive video and can do some extra practice questions and another student said it's more engaging and involves more recall than simply reading the text or just watching a traditional video that at the same time not as time consuming as branch interactive videos and so you know students I think often have to do these hedonic calculators as to where they are going to put their time when they're studying especially when you know it's midterm week when reading week starts ends next week we're going to go into the second wave of midterm week and they're going to have a lot of things pulling out their time and so these branching interactives might be really good for the first time or two or three going through some participants anecdotally said that they actually got questions wrong so they could see the explanations so they actually purposely got things wrong so they could have these learning opportunities but then that might frustrate other students and that it's time consuming so I don't think I think branching interactive that's what I generally have a clear preference for but I don't think that it is useful in any situation in every situation I should say not useful in every situation or not more useful than interactive videos especially with some of the time investment it takes to planning it out as Kaylee showed you yesterday you can copy and paste things pretty quickly and once you get into the flow of things it goes by pretty quickly but it's often the planning outstage that takes a fair amount of time where interactive videos allow you to just pop questions in and still get those benefits did you look at whether they actually did spend more time on branching videos than interactive or they perceive that they did because video is longer since include branches that's a very good question we don't have any sort of because h5p the way that it functions at the moment we were not able to get data for how they interact with the video that is possible to do using things called xapi statements or various other ways of going and sort of pulling interactivity so we would be able to get which questions they chose how many times they hit pause rewind play that's what we are hoping to do for a future study but we weren't able to do that for this study per se and so you know it was an anecdotal students during the experiments have told me or the research assistants that sometimes they were getting things wrong so that they could actually see you know more of the video but I don't have any hard data that I can show that to you now sometimes we might not want to provide feedback when might we not want to provide feedback um mark mcdaniel who was the keynote for the symposium two years ago told us about a study where he got participants to come in and read a section on research methods two days later they brought participants back in and they got some of their participants to study the material for six minutes do a little distracting task study the video for six minutes again do a distracting task then read the material again so this was just study study study the other half the participants did a multiple choice question then they studied the material and then they did some multiple choice questions again now you see that these center studies are highlighted in red that's because what they did is that when they studied the material in the middle section they gave highlighters and then they looked at which um uh terms students were highlighting so this is sort of if students study and then study again do they know what they don't know and does this bear out in their patterns of highlighting key terms versus if you do a multiple choice question before studying the material again you it points very clearly to where your lack of understanding is and then does this bear out in how they read the material now these multiple choice questions just provide a corrective feedback they didn't do any of this nice descriptive feedback and as I said in this read that they highlighted five days later they came back and wrote a quiz on the material again so over here we can see the proportion of content that was highlighted um correct during the first multiple choice question oh sorry yeah so over here what we see is that students who got the multiple choice question um they highlighted about 44% of the key terms that they got correct but the students who started out for the multiple choice question and then we studied the material they highlighted 78% of the material that they got incorrect so McDaniel talks about meta memory how these multiple choice questions at the beginning of the study question can point to gaps in student knowledge which then makes their studying a lot more efficient and when we look at their test scores five days later you can see those who in the study study study group didn't have that prompting of meta memory scored 63% where those who took the test then we studied the material then took the test five days later scored 10 the same higher so sometimes we want to get this descriptive feedback but sometimes we can think about using videos or interactive videos in ways before reading a section so if I can show you an example of what that might look like um over here this is chapter eight analyzing findings students then go to correlational research and before they read all the material on correlational research um over here I get them to watch a video now they can choose do they want to watch the interactive video or do they want to watch the non interactive video if they click on the non interactive video it goes to a video on correlations but over here if they start the course and they click on interactive video this goes to a video that then stops and asks them questions you can see this is from a video that my my colleague steven barns put up on youtube so this is one that i pulled off youtube and retrofitted some interactivity so this is what this is what it looks like so it starts off right at the beginning before we begin which are the following options best describes what a correlation is so we hear a correlation of forms about how a participant scored in the test this does give a little bit of feedback this is incorrect you can use a correlation to find out what is related to test performance but it doesn't tell us how someone performs so they can retry informs us about a correlation relationship between two variables yes the correlation tells of two variables are related to each other and how they are related and the strength of the relationship you'll learn more about this next time so here this is hopefully creating some meta memory and then they get so let's talk about correlation here is a scatterplot graph this may look like a very organized view of the night sky but it's actually a treasure trove of information with a graph like this you can figure out the correlation between two variables be it positive negative or none and you can make predictions about data first if your graph looks like this then you have a positive correlation as x increases y also increases but then i pause over and i say what is another possible interpretation and this would be over here a decrease in the variable on the x-axis is associated with decreasing the variable of the y-axis so this is trying to get students to read a positive correlation in both ways just because a positive it doesn't always mean there's one increases the other one increases it's just that they move in the same direction if x decreases then y also decreases if they get it wrong there's no feedback and it tells them what it is and then it asks for another example of a positive correlation and so this is the idea to get them to think well what is wrong oh why is that wrong and then to listen more and then to read on find it out so that's how i used this to provide feedback to try and serve this purpose of negative memory and you can kind of see i try to talk about that providing high quality feedback is time consuming for the person giving it and i think hattie mentioned that in the some of their papers and that's what i quite like about h5p is that it does allow you to provide bulk feedback to students so for instance if you look at multiple choice questions supports detailed feedback multiple choice questions yes it does single choice and image choice it doesn't provide detailed feedback but it does provide say for instance corrective feedback where questions say it does allow the provision of detailed feedback you can look at short answer style questions fill in the blank provides corrective feedback advanced fill in the bag provides some more descriptive feedback i think that they are workarounds in which you are able to provide more detailed feedback there hotspot style questions supports detailed feedback image hotspot provides corrective feedback but find those hotspot and find multiple hotspots provide these detailed feedback and then flashcard styles once again this is only corrective feedback within interactive videos we can see that multiple choice questions crossroads navigation in some sense provide the ability for descriptive feedback whereas image choice true and false and statements single choice they provide corrective feedback unless you do these branching interactives so there are all of these decisions that need to be made and you know this h5p offers one opportunity to doing this especially if you're trying to think of how we can create these things that then provide this type of feedback for students that we can provide good detailed descriptive feedback but in a way that is manageable for everyone so the summary feedback is important about summative and formative shouldn't say research that should say research shows the summative informative test we should aim for descriptive over evalative feedback feedback needs to be understandable contain correct information and focus on the material formative is especially important because uh encourage the read it encourages reading and taking in of feedback uh branching interactive videos can be easily effective at personalizing feedback but they do come at a significant time cost sometimes no feedback beyond corrective can also enhance learning if it's used in um correct ways so that's the end of the talk i know that we've got about another 25 minutes here and so what i thought i would do is that maybe i would head into h5p and start looking at well how might we create some of this feedback uh within um h5p especially if you might be using something like um but spentorio is saying this generation how can we go and change any of the feedback that we might want to do so over here this is some of the materials that we're going to be using those of you who want to go into your h5p instance are welcome to follow along here and uh then what i'll show you to do is how you might be able to go and copy over here we've got our question over here if you click on an answer it shows incorrect we want to put some feedback how do we go putting feedback over here so if you click on this now uh once again i'm in firefox um sometimes cultural videos don't like working in firefox so i've taken this over the chrome and there's the interactive video you'd like to use this interactive video to follow along you can click on this reuse button and you can either download as an h5p file which then you can upload or you can just copy content so for now i would um encourage you to click on copy content and then go back to your instance of h5p you will then i'm just going to pipe in for a minute i think that if somebody is using um that to use that copy button you that will work for you simon because you're within your own um h5p but i think it might not work for others um until unless that h5p element was their own in their own uh like where they were authoring h5p and so instead they need to click reuse and click the download and then they could upload it to have a their own instance of that there the copy button should just work within when you are the one editing your own h5p items within your own author it within the same authoring tool so it might not work for others okay excellent so then if you click on reuse and you download you can see in the download history that's where the interactive video has gone uh we then click on add new in our instance of h5p and you will click on you see this h5p select content type create content you can click on upload click on upload a file over here you double click on that and click on use interactive video was successfully uploaded and there you have that um available and there's that one interaction that we were talking about over there so there are two ways in which you can um go about doing that if one doesn't work try the other because we use copy content so to add new you could also just click paste over here and then that's it and then you can correct my spelling interactive video remember from a edi perspective we might want to give more of a description over here um this one covers material on various sleeping disorders uh text and tracks over there we see that there's the web VTT so that the closed caption should be there for you already as well so we're going to click on interactions we're going to click on the little bubble that thing calls us up and we are going to double click on this so it pulls it up you see the pause video we've got the poster interaction and here are all of our questions with the correct one being narcolepsy over here we want to add the feedback nice this is what um what is the process of suddenly falling asleep call you click on tips and feedback message displayed if answer is selected we will put that in over there you'll see that there's no chance for doing a heading over here um that is because it's less important for the answer to have a heading but more important for the question to have a heading to make it um match honorable for keyboard shortcuts so if you would use the same title of hpp that i've already used i imagine if you don't have that instance in your um h5p instance already if you don't have that name it will just call it um interactive video with the spelling mistake so i don't think it will override it it might have a copy of it i'm not too sure um what i will do if there's a second instance of it but if you do already have that name i don't think that it's going to care what the name of it is because it will still have a unique h5p id which is how h5p identity it gives a little id number to each um different element that you create so i don't think it would care whether the title was repeated because it would have a different id element an id tag so you can see over here at the end of this um where it's being hosted you see id equals three one six eight if i were to change that to something different um then we might find a different h5p instance that you can see that that's the id tag that katie was talking about all right so over here you can also do message displayed if correct answer is not selected um this can be quite powerful sometimes you know if you i find it easy to think about it in terms of like if you've got um a true false and you've only got two answers and then you might be able to say well done or not falling for the red herring so which one is not a psychological myth that we only use 10 percent of our brain or um or that uh writing material in bold and italics makes it more difficult to read which promotes learning and so um you know if you select that second one that would be true that would be the correct answer and you could have well well done for not falling to this myth um and then have a little bit of an explanation a tip text that's what sventol was showing earlier um uh you can type something like uh something to try and and give a hint which will appear as that little speech bubble beep apnea over here uh so it's beep apnea i don't have that uh the feedback copied over here ram sleep behavior disorder ram sleep behavior disorder copy and paste um nambulism is sleepwalking over there uh night terrors over there copy and paste what you can also do is that you can sort of say feedback if the students score somewhere between 0 percent and 100 percent you can give them some feedback you can also add a range 0 to 20 percent you could say um uh i would focus uh uh um focus on reading section xx on sleep disorders and then try again and then add another range from 20 to 50 saying you have uh are you trying to avoid the value of one over here um uh once again you oh you might actually just say why make things difficult to myself 0 to 50 50 to 70 you could say you have an understanding of the material go and read the material again to bump up your focus and then if it's uh between 71 and 100 percent so there are ways in which you can give this i find this particularly good when you're using a question set where you can have like multiple multiple choice questions there um you can give them feedback on the overall task once you've done that you scroll up to the top and you click on the done button and you click on the create button i'm going to put up one more thing that i'd like to show you before i'm going to stop talking and open up to questions and so then when we press say we move across to there which sleep disorders characterized by suddenly falling asleep if we go on to some nambulism there is the what you call it the the feedback now try again this is a narcolepsy check correct and there's your feedback and so this is now just a nice way of providing sort of like personalized feedback if they get one wrong you're able to tell a little bit of specifics about that created into a learning opportunity one thing that we don't want to do is that you want to be careful on the language that you use because if you have any of this evaluative that has this judgment sort of negative with this criticism that can demotivate students to actually try um at all and so um it allows you to personalize for these different ones and as i say if you've created these types of distractors you know why the student might have chosen them so you can say what they got correct but then focus on the part that maybe they um they haven't got through one more that i would like to show you and i um this one is particularly powerful uh and i would encourage you to play around with and i didn't i know kailey is very good at using this uh but you get this h5p widget which is um an essay question i believe kailey yeah so over here having just learned about sleep apnea so here they read about sleep apnea having just learned about sleep apnea try and write down as much information as you can answering the following questions so very similar to Cynthia Brain that was kind of telling students what is expected of them um in this one and over here you can say sleep apnea describes the condition whereby you can give students a sentence to start off with so sleep apnea is a condition whereby a person stops breathing for 10 to 20 seconds at night um is they more than one type of sleep apnea well there is central and uh obstructive sleep oopsie apnea now let's say that's all i can remember when i click check it says over here well done stop breathing well done this is a defining feature so you've got the defining feature 10 to 20 seconds you mentioned another defining feature or how long it lasts for congratulations did you get the description of central sleep apnea correct well done did you get the description of corrective and then there was some other things that they didn't fill in you seem to have missed out on defining diagnostic feature of obstructive sleep apnea what could it be so you can give students hints as to um where they might be lacking helping them focus re-reading the material or maybe even trying that again and then what you can also do is have a show solution button where you can write for instance this is what a sample solution would be like and students can see what that might be they are multiple ways in which you can provide feedback on these individual ones on the task as a whole and we see different h5p widgets have these different abilities to provide correct feedback this is descriptive feedback and uh we see that some of these widgets can provide a real nice bit of feedback as to points that students have trying to build in as many of these good practices as possible and so that has brought me to the end of what I plan for today