 It's a great pleasure to be here and to be able to talk about a subject very close to my heart, education and in particular how to be more scientific about it. Now some of you may already heard me speak before or watched my confessions of a converted lecture on YouTube and most of my talks on education, I start them with that picture on the screen and ask the audience what is it that is happening in that picture that was taken of me when I taught my introductory physics for pre-med students at Harvard? This is a very old picture, it's BC, before computers. You can see I'm using an overhead projector. The point I want to make with that picture is that the traditional approach to teaching focuses on information transfer. And what is important is not what the instructor does in front of the students, but what happens inside the brains of the students. And it turns out that's not very much. And I want to back that up with some data that were published recently in an article that was published by my colleagues at MIT in the Media Lab. They developed a very interesting sensor that you can wear around your wrist that measures electrodermal activity, which is a measure for brain activity. Actually, most of this paper goes to great lengths to point out that the electrodermal activity measured at the wrist is correlated, very strongly correlated with what happens inside the brain. The great thing is that you can basically wear that sensor for a week, then download the data and analyze how brain activity correlates with different activities during the day. So they had a couple of students wear them for a week, and they published in that paper without, I think, commenting very much on the actual activities. This graph that I'm showing here that shows on the vertical seven days at the bottom day one, day two, day three, day four. And you can see the electrodermal activity, the brain activity basically, as the trace that goes up and down. And labeled in color at the bottom, you can't read it, the letters are too small. Oh, incidentally, on the horizontal axis is time for a reason that I don't exactly understand. It starts at 16.00. I guess at MIT today starts at four o'clock in the afternoon, I don't know. And then it goes all the way to three o'clock in the afternoon. Now, you can't see it, you can't read it from where you are, but I want to draw your attention to the parts of these trace that are labeled lecture. Actually, they're labeled class, but, you know, they're typically lectures. The trace goes flat. In fact, it's very interesting to contrast that with the brain activity during sleep, which is highlighted there. In other words, students are more asleep in lectures than they are when they're in bed. You know, the French writer Albert Camus once said, some people talk in their sleep. Lecturers talk while other people are sleeping. I wish he could have seen this trace because it would have made the point much more salient. Now, there's only one activity. There's only one activity that matches class. You can't read the labels, which is good because I want you to guess what other activity does the brain go flat? TV, exactly. Here's TV. And of course, that's something that people knew a long time ago because people had done, you know, using electrodes in the brain, scans of brain activity of people watching TV, and it's been known for probably close to 40 years that people are more asleep in front of their TV than they are in bed. If you stop to think about it, it would be so surprising that class or lecture, including this lecture here, is not that different from watching TV. And I think that's a very important message, especially as we move a lot of information transfer out of the classroom, onto the web, onto computers. Why would people be more engaged by watching a lecture online than watching it in real life or watching TV? What matters is to get the brain engaged, and you don't engage the brain when you just have a flow of information at the person. Today I have a slightly different message here. And the message is on the screen there, is that we should not abandon the scientific approach when it comes to teaching. A lot of decisions in education are made based on completely anecdotal data. And in fact, it's not that different in my own department. We often talk about teaching. I'm in a physics department surrounded by physicists, and you would expect physicists to only take decisions based on evidence, but it's kind of surprising to see that even very distinguished physicists, Nobel laureates, completely abandon the scientific method when it comes to teaching. My students like it when I do this or that. Lots of decisions get taken based on simply anecdotal evidence. I want to quote here Lee Schulman, former president of the Carnegie Foundation. He once said, and I love rubbing this quote into the face of my colleagues, he said, the plural of anecdote is not data. You don't use anecdotes for evidence even if there are many. And I know that in a sense because I based my approach to teaching on my own experiences. When I was an assistant professor, that picture is from when I was an assistant professor at Harvard, I basically started doing what my teachers had done to me. And I really thought that's how we learn. And I really thought that's how I learned. So I taught this win for many years. I thought it was a good teacher until I discovered that my students were not learning anything. So ever since then, I've basically used my classroom as a laboratory to obtain data. And I'm going to show you today three little projects that I've carried out in my classroom that show how we can use the classroom as a laboratory to obtain data to improve teaching. And I want to urge all of you to do the same when you have access to a classroom. It's not that hard to come up with nifty little ideas. I'll talk a little bit about these are three separate parts. And feel free to interrupt me at any time and ask questions we don't have to wait till the end. I'll talk a little bit about gender issues and especially gender issues pertaining to science, education, a particular physics education. Incidentally, I'm a physicist, so I'm going to put this in the context of physics and you'll hear a few physics examples. Do not worry if you cannot follow the physics because the physics is irrelevant. The message transcends the field. So I'll talk a little bit about gender issues. Then I'll talk about lecture demonstrations and sciences we love doing lecture demonstrations. And again, this is, I think, very relevant for online education where a lot of people think seeing is believing. The same is true when doing demonstrations as a physicist and particularly as an experimentalist, I thought what better way is there to teach students how nature works than by just showing how things work? False. People don't see what they should see. They see what they thought they should see. So I'll talk a little bit about that. And then lastly I'll talk about confusion and the role of confusion in the cognitive process of learning. So let's start with gender issues. For many, many years I have administered and so have many other faculty across the world a conceptual survey called the Force Concept Inventory. It's a 29-question survey that tests students' understanding of the concept of force which is one of the first concepts to be covered in any physics course. And it's actually that Force Concept Inventory that prompted me to completely revise my approach to teaching because I found out the first time I administered it that in spite of my high evaluations as a teacher and in spite of my students doing well on what I considered complicated exam problems they failed to do well on this concept inventory which I thought was actually pretty trivial. But that's not to the point. I want to highlight another issue with this Force Concept Inventory. Basically the way it's administered typically is once at the beginning of the semester before physics instruction and once at the end of the semester so pre-test, post-test to see what the difference in score is and to measure how effective the teaching has been. Well the post-test scores show a disturbing trend namely that men outscore women by a significant amount about 15%. These are data actually from the University of Minnesota which is a large public institution in the Midwest. If we compare for a couple of institutions the size of this gap here about 12% of the University of Minnesota in my own two classes that I've taught at Harvard two different introductory physics classes which are similar and Woosta Polytechnic Institute which is a small liberal arts college also has a similar gap. So the question I'd like to ask is what causes this gap and can we do anything about it? First of all is it cultural? If you take all of the data from the US and put them together you get about 11 to 12% very statistically significant gap between male and female post-test scores. Now I have the benefit of getting a sabbatical every couple of years and whenever I spend that in a foreign country I try to push my host to administer the FCI in their institution. So a while back I was in Belgium at a pretty well-known engineering university and I decided to see if I can convince my colleagues to first of all translate the FCI into Flemish which is not that hard. I was educated in Holland so I'm fluent in Dutch so I could at least check the quality of the translation and then administer it and two things were striking. First of all I having been educated in Europe always thought that in Europe things surely had to be better than in the United States they weren't but secondly the gender gap was actually significantly larger than that in the US. A little bit later this was still in the mid-90s I was in Taiwan the whole first generation of my graduate students ended up becoming professors at Taiwan Dashi which is a national Taiwan university their flagship university in Taipei I spent half a year there and tried to convince the people in the physics department at NTU to administer it there. They looked at the FCI and they said no way we cannot administer that to our students they thought it was so simple so trivial that they refused to to even consider testing their students I said but look we even do it at Harvard and the students don't do that well well their respect for me immediately plunged and I was unable to convince them but fortunately near Taiwan Dashi was another institution National Taiwan Normal University which focuses on training high school teachers and there they were very interested in this test we translated in Chinese I had my graduate students translated back from Chinese into English to make sure that I could vet the translation and we administered it and not only did the Asians do significantly better than both the Europeans and the Americans on this FCI but on top of that there was no statistically significant gender gap now I was never able to really research what the cause was of this but there was one variable that stood out at this National Taiwan Normal University about half the faculty was female in Belgium at that university I'm not allowed to disclose which one it is after the dean saw these results he said he called me to his office and said I never want you to disclose these results so I won't tell you which university it is but at that university there were no female faculty at all in the sciences it was a completely male dominated department and in the US the number of female faculty hovers around 25%, 20 to 25% so maybe the environment has an influence on the gender gap what about if we go back to the US data where I have many many more data tens of thousands of data points what about the effect of pre-college education you would expect if a student has had some high school physics then that is going to affect their understanding of Newton's laws in fact I learned Newton's laws in high school when I went to university to study physics we didn't even discuss Newton's laws anymore the concept of force was assumed to be already understood well there is some correlation between students who've had between the score on the FCI on the vertical scale and students who've had no high school physics whatsoever students who've had a course in high school physics and students who've had a college level advanced placement test but it's not very strong which is kind of surprising right you would expect that to be a much stronger correlation especially students who've had an advanced placement this is for the female students here's for the male students and you can see that this gender gap persists across the pre-college level of education and it's actually even worse if you look at the numbers because women are underrepresented at the higher levels there's about the same number of students who've not had this is in one particular year in my class at Harvard who've not had high school physics 15 and 18 about the same number who've had one year of high school physics but then the advanced placement course is taken by twice as many men as women so what can we do is there's something we can do to mitigate this well there's plenty of literature that shows that women thrive in an environment that is more collaborative not competitive and also that focuses more on verbal and visual representations as opposed to purely mathematical representations and when I read about that I thought well what I'm doing in the classroom actually mirrors that let me tell you what I did in the classroom after discovering that students didn't learn very much from my highly regarded lectures what I did is basically abandon lecturing altogether instead of focusing on information transfer which is kind of ridiculous in an information age right ever since the biggest invention in information technology people have been saying that we should no longer lecture referring here to the invention of the computer or the internet I'm referring to the invention of the printing press and the book back in the 18th century I think it was Samuel Boswell who once wrote why are we still lecturing we now have books but for some reason people kept focusing on information transfer as I said before education is more than just information transfer so I like to see education as a two step process one information transfer you need that otherwise nothing happens two assimilation of that information or in Piaget's terms accommodation of that information it's what the learner does with that information in his or her brain that matters you need to connect it to the knowledge that's already there you need to connect it to the experiences I've often asked myself where did that happen for me did the aha moments the insights the connections did that happen while I was sitting in a lecture hall not that different from this one here listening to my instructors I don't think so I think most of that happened outside the classroom so if you think about it completely pragmatically education being a two step process one information transfer two assimilation of that information and you ask yourself which is the hard part I think we'd all agree it's step two so it's kind of ironic that in most of education we focus on the easy part information transfer leaving the hard part to the students on their own that's true in the information age two most uses of information technology focus on information transfer not on the hard part the assimilation thereof anyway when I realized that I thought what I should really do in my classes is throw out the information transfer completely ask students to watch a lecture online or ask students to read the book and in class I teach by questioning rather than by telling I walk into the classroom I put a question on the screen student think commit to an answer the whole clicker craze started basically in my classroom about 20 years ago and then I tell them now turn to a neighbor who has a different answer and try to convince your neighbor of your answer complete chaos but in the process students teach each other and then I have them vote again so there's both feedback which as many educators knows crucially important in education active engagement you can't sleep in my lectures because every few minutes your neighbor will start talking to you right and the message has shown to give much bigger gains in learning but I'm not going to focus here on this method what I want to point out is that it's a collaborative approach to teaching students helping each other and it's one that very much focuses on the verbal individual rather than the mathematical so I thought could that method help mitigate the gender gap because it increases both collaboration and interactivity so I decided to take the many many years of data I had collected both in the traditional lecture setting and in interactive lecture setting and to compare three different pedagogies which I had done for many years before switching as T then the first thing I did was make my lectures interactive by using this approach to teaching that I just described and which I later in my book called peer instruction I labeled that I and then a few years later I decided not only to have the interactivity in the classroom but also enforce that I don't like determined force but also encourage that outside of the classroom so in discussion sections in homework I encourage the students to work together increasing the collaboration there as well so let's look how these three different activities impact the gender gap so let's first look at the pre-test course for women for these three different treatment groups T, IE, interactive engagement and IE plus interactive engagement plus and there's no difference for the pre-test because they've not yet been exposed this just shows that over the years because we're talking about the 10th year span here the type of students that get admitted to Harvard performs equally which is quite reassuring otherwise this research would have been very difficult I put a little triangle on the far right side of this diagram to show you the average pre-test score for women here at the men there's a little bit of fluctuation there if we take the average of these three different groups we get that red triangle and the difference between the blue and the red triangle is the gender gap on the pre-test that's how the students get delivered to me in my classroom now the question is can we close that gap now let's first compare how women do across the three different groups so traditional interactive engagement interactive engagement plus and you see that women gain significantly more as the interactivity gets increased the men do too but the women gain disproportionately more and in the interactive engagement plus the gender gap is no longer statistically significant so that's good news that means that women don't discriminate this gap which shouldn't be there in the first place simply by increasing the interactivity in the classroom if you're interested in more data you can find that in the paper that we published about six years ago in the American Journal of Physics let me slice the data a little bit differently who are the low gain students I'm showing there a data point for one student who started out at a post test of 70% and ended at a post test of 90% I put the pre-test on a horizontal score and the gain the difference between the post test and the pre-test on the vertical score you want your students to be as close as possible to that diagonal line you start at 90 you gain 10 you start at 20 you can gain 80 that means you have a 100% post test score here are for a class I taught at Harvard all the female students some of them on top of one another so there are actually more women than there are data points here and here are the men and if you look at those who score below the mean you see a gender imbalance there are far more women than there are men this is in a traditionally taught class after implementing interactive teaching all of the data points move up which is great it gains more but if you now look below the mean you see that there is actually gender balance there are as many men as there are women so I think the verdict is pretty clear the interactive teaching actually helps overcome this gender balance so the points I want you to keep in mind for this first little part of my presentation here today is that in the sciences this gender gap comes from both culture and background you have to do something about it no question but even if there is such a gap it can be eliminated by introducing interactivity in the classroom I'm actually happy to take any questions now where we can wait till the end but feel free to raise your hand and shout out a question because we're switching now to something different namely demonstrations and I chose that topic because I think it's particularly relevant for people who are doing simulations using technology or putting lectures online in hopes of actually promoting some real learning I'm an experimentalist as I say a moment ago I always loved to do demonstrations especially if they're kind of spectacular so when I started teaching I would have this rocket car which was basically a fire extinguisher propelled car where I would shoot through the lecture hall I would hoist myself up to the ceiling of a lecture hall that was as big as this one using a set of pulleys if there was a cannon to be fired those were the best demonstrations and students would remember the demonstrations vividly but then I read this that was published at the University of Washington in Seattle by a student who had gotten a PhD in physics education research and as part of her research she had interviewed students and in one of these interviews she discusses with a couple of students who have taken introductory physics but not majored in physics their recollection of demonstrations if you've ever taken an introductory physics course you may have seen it it's called shoot the monkey or sometimes it has a more politically correct title but anyway you aim a gun at a suspended monkey it's not a real gun it's a spring loaded gun and then the moment that the projectile emerges from the barrel the monkey stuffed monkey not a real one of course is released and falls under the action of gravity but miraculously the projectile that is aimed exactly at the monkey still hits the monkey the reason is that it too is subject to gravity so they both fall so no matter how hard this projectile goes if it goes faster it hits the monkey higher if it goes more slowly it hits it just before it hits the ground well two months after seeing this demo the student correctly remembered that the monkey was hit was convinced that the person who had done the demo had adjusted the aim of the barrel in order to allow for this falling of the monkey which is not what happened because the point was just to show that you don't need to adjust the aim so I thought that's weird you show something and then several months later even though they've seen it with their own eyes they remember it incorrectly so I thought what is actually the purpose of demonstration is it entertainment or are people actually learning from demonstrations so I decided to do a study which we carried out over two years where we basically carried out first seven and later thirteen demonstrations in four different modes I had about 250 students in my class so I split the class into four groups and these four groups would get rotated each week through four different modes of presentation the first mode is very simple we would not show the demonstration at all that was our control group the second presentation mode we call observe we ask the instructor to do the best possible job showing and explaining the demonstration but it was completely show and tell yes the students were allowed to ask questions but nothing else in the third group before showing the demonstration we showed a graphical representation of the demonstration described what was going to happen and we asked the students to predict the outcome of the demonstration using clickers so we had a number of different potential outcomes and they basically had to click in their answers after thinking we call this the predict mode and then the last one is discuss we would give the students a worksheet the top of the worksheet would describe the demonstration then it would ask the students to predict the outcome free response not multiple choice so they had to write down this and this is going to happen then we did the demonstration and they had to record their observation so we were actually able to check whether they had observed it correctly or not and then they had to check a box my prediction does or does not correspond to my observation and then we gave them a few minutes to resolve any inconsistencies between prediction and observation with their neighbors in a discussion of course there is a different investment in time here no demonstration takes no time at all the demonstration the observed takes whatever the demonstration takes predicts adds another 2 minutes to it and discuss adds another 8 minutes to it so we would rotate the four quarters of the classroom through these different modes so if you were in my classroom one week you would be in the no demo mode another week you would be in the discuss mode in a totally random order and we actually never even told the students about this study they didn't know that we were part of a study we followed up with a free response test at the end of the semester we only took the demonstrations in the beginning of the semester so we had at least a month for the memories to settle in and we added also a number of exam questions where we would take the demonstration but put the same physics idea in a totally different context now I'm going to show you an example and again don't worry about the physics the physics is irrelevant although I'll explain it briefly but you'll still get the idea without getting any of the physics here's the demonstration demonstration consists of two scales one on the left one on the right and there's a plank that rests on both scales and there's an object in the middle so that each scale reads half the weight of the object then the demonstration consists of moving that object to the side and noticing how the scale readings change a lot of students, a lot of people think that it doesn't matter where you put it because the plank evens out the load this demonstration in a physics class is used in the context of teaching a concept called torque which has to do with forces making things rotate and the point of this is it makes a difference where the object is because you have to balance not only the forces but also the torques you can forget all of that just remember what is shown graphically on the screen here if you move it to the side one scale goes up and the other one goes down so that was the demonstration then two months later on a web-based test which was a free-response test we put this question we changed it into a numerical question so we said we had this drawing there and we showed that the two scale readings were 10 and 10 when the block was in the middle and the students had to say what the scale readings were when the block was on one end which is what they had seen should be 20 and 0 and then we extrapolated to something they had not seen in order to test their understanding, namely what would happen if the block were not all the way at the end but halfway towards one end this is something they had not seen in class well let's look what the result is 24% of the students gave the correct answers and correctly in their explanation which was free-response mentioned the concept of torque which is exactly what we wanted another 38% had also had the correct answers but used proportional reasoning rather than mentioning torque we'd rather have them use the concept of torque than proportional reasoning so we could debate about what is right and what is wrong here if we're generous we could say both groups are good if we're very strict we can say only the one on the left is right 20% of the students said it doesn't matter it's always 10 and 10 when I saw that 20% so that's one in five students at Harvard University okay I thought those must be the students who did not see the demo but we attract the ideas of students so we could exactly see in which group they were we had taken attendance so we knew exactly what had been in the group that had no demo or had been absent which means they're effectively in the group or been in one of the others no they were students who had actually seen the demo in fact some students actually wrote down and I'm quoting here as shown in lecture it doesn't make any difference where the block is placed I got so excited I went straight to a friend in the psychology department and I said I think we found something really really exciting here some really new exciting psychological effect he said really and I said yes and I told him about it and he started smiling he said Eric we've known all along about this he said why do you like physics? I said well I like physics because I don't have to remember that much he said exactly the brain stores models not facts so imagine you're a student who has the wrong model the model of it doesn't make any difference where you put the block now you come to class and a professor does or teaching assistant or whatever does the demonstration and you see contrary to your belief one scale go up and the other down you have what Piaget calls a cognitive dissonance you go whoa this is different from what I expected maybe you quickly scribble it down in your notes but and this is the key point before you have time to think about it the professor continues to lecture you have no time to readjust your mental model and what happens for many students especially students who are discipline after class to sit down and think let me sort this out what happens is that instead of adjusting the faulty model to the observation which is what we want to happen what happens is that the memory gets adjusted to the faulty model and you know psychologists have known this a long time because advertising agencies try a lawyer they all make their living on this principle now notice that if you're a student who will actually write down as shown in lecture the demonstration is actually rubbed in the wrong model right now you're using your adjusted memory as evidence that your faulty model is correct okay ten percent of students just used qualitative reasoning they didn't give numbers they said up and down and another six percent of the students the forces don't balance so if you add the two numbers all of a sudden the block has become heavier or lighter and two percent of other incorrect answers if we look at how this correlates with the presentation modes this is what we find and I want to draw your attention in particular to the observe mode those are the lowest scoring only eighteen percent got it completely correct meaning they had the right numbers and they mentioned torques so is just presenting harmful I'm going to quickly whiz you through an exam question and again ignore the physics here it's just so you get the picture and here this is exactly the same principle but a very different context I would like to argue that the real hallmark of learning is being able to take something learned in one context and applying it in another context we do very little of that as anybody who is teaching here knows after exams all these students coming to you saying professor we've never done a problem like that I'm telling my students well if you had done this problem it wouldn't be a problem anymore by definition right anyway here's the same problem in a very different jacket a plank suspended by two ropes and the tension in one rope is given in essence the scales have been replaced by ropes well you know automatically or you should know automatically that the tension in the other rope because of symmetry also has to be 150 Newton and therefore the weight of the plank is 300 Newton but now the question is if you move that rope from point Q to point R what are the tensions in the two rope they should of course still the sum should still be 300 but how the magnitudes depend on those torques which are not that easy to calculate this is a little bit harder problem 36% of the students gave the right numbers 100 and 200 was the correct reasoning then different amounts of students gave wrong answers based on incorrectly calculating the lever arms for the torques interestingly enough as I was scoring this this was a free response exam I saw lots of students who came up with very curious numbers 112.5 and 187.5 I had no idea where these numbers came from this shows that you know when you're a teacher and if you know the right answer it is almost impossible to wrap your head around the wrong thinking of a student which is why it's so hard to teach students who have the wrong model because you don't understand how they're thinking in fact I remember sitting there scoring this exam and each time I saw in total there were something like 16 students in my class or 20 students because I had 215 in total I kept seeing that same number I said where does it come from then I figured out 112.5 is 3 eighths but where does the 3 eighths come from and 5 eighths is 187.5 and then after probably scoring 180 exams finally there was one student who actually explained where those numbers came from look if you take the plank between the two points P and R and take the left half and you say the left half belongs to the left rope the right half is a right that belongs to the right rope and you also add the part RQ you get those numbers so they just you know imagine the plank being consisting of two parts I would never have imagined this and then of course they are here about 2 or 3 students who had 112.5 and that's because the part RQ does not need to be supported at all anymore it sticks out who would have thought right so interesting it is something normally when you grade an exam or when you look at incorrect work you look for a mistake and you stop thinking the point I want to make here is that often it's very interesting to reverse engineer the thinking of the students because it gives you a glimpse into their thinking and unless you are aware of their thinking you cannot address it anyway over all the demos that we did 13 demos you see that as you invest more time the number of correct remember the first column is how well they remember it goes up oh no no sorry this is the exam problem I'm getting ahead of myself here so here's for the exam problem so you can see that as the the discussion increases from no demo to the bottom the number of students who actually correctly explains it goes up dramatically now the aggregate for 7 demonstration shows the following and I'm going to show it graphically rather than in a table if we use the no demo group as a control group you find that just observing on the vertical I have the improvement in the students ability to correctly predict the outcome well in that case it's usually rather than a prediction right and the observe is barely statistically significant in other words the observe group does barely better after one month than the group who had no demo at all there's an increase with prediction and discussion which becomes even more dramatic when you ask for the students to explain the demonstration observe not statistically significant predict a significant improvement and discuss an even bigger improvement so the lesson that take home lesson for me was here if you do a demonstration at a minimum you should ask the students to predict it before they do it takes just you know a minute or two and look at the enormous improvement over the observe the same should be true for anything that is done online if you have a simulation have pause before you show it have the students predicted before they actually see it it makes the cognitive dissonance so much bigger that students are more likely to adjust their mental model to the observation rather than their memory to their faulty mental model so points to keep in mind before I wrap up here was my last part demonstrations without engagement are just not very helpful I always thought seeing is believing I think this is very clearly proven here that seeing is not believing what you see depends on what is already present in your head and the results can be dramatically improved simply by having students predict what they're going to see before they actually see it so anybody doing online visual simulations online demonstrations and so on should think really hard about incorporating such a step in their approach any question yes why don't we wait till somebody brings you a microphone this way people online can hear too so when I was doing air level physics we had to show our working out on the paper and we had to write them by hand our scripts and I don't know how you do them nowadays maybe they're online or on the computer or something but I know that must have been really helpful to expose misconceptions to teachers and to inform them as to why we got the answer wrong and I know we just didn't get a mark for the question if we didn't show our working out it was that strict so is there some way that that could be what's your ideas about that maybe using technology to reintroduce that into assessment I was educated just like you writing things out and in fact I still teach that way students have to work things out on paper and I look at it but frankly I used to usually not pay attention to the mistakes you end up spending disproportionately more attention to the students who get it right than those who do not get it right you see a student make a mistake stop reading the rest you stop thinking about why the mistake was made so it really takes some effort to analyze for misconceptions and wrong work and I would argue 99.9% of faculty do not do that they should but they don't do it now online distortion gets worse because most online systems only check for the answer not for the steps that lead to the answer I wish I had time to talk about assessment because I've asked myself lately a lot about what role does the assessment play and I think that the type of assessment that we have that we use mostly you know summative assessment high-stakes summative assessment is often the silent killer of educational innovation you you pushed a button here so I'm going to get sidetracked for a few minutes what is a real problem a real problem is knowing exactly where you want to get but not knowing how to get there you have a product you want to make it successful we have a problem in education we want to make education better in both cases you know exactly where you want to get a successful product or improved education the question is not the outcome which is known the question is how do you get there if you stop to think about it every real problem fits that pattern you know the outcome you don't know how to get there instead most assessment certainly in the sciences does not fit that pattern it's the opposite you apply a known procedure to get an unknown answer and the answer is checked I would argue that that's actually an inauthentic assessment one that does not really teach an interesting skill and one that actually a type of problem solving that can be done by computers and just as assembly workers have been replaced by robots so will any procedural problem solving be replaced by a computer so I think any assessment that checks the answer and not the procedure is one that is inauthentic and I would argue that presently most online systems you know mastering web CT you name it there are many different systems they focus only on the answer and therefore you know they don't expose any misconceptions they don't really put the emphasis where the emphasis belongs that's very unfortunate and I'm not sure there's an easy technological solution that's an online question let me see if I can read that does this have implications for the Khan Academy type of presentation is that the question yes so you know the Khan Academy to me and I know Selkan I've met him and he's an incredibly charismatic person there's no question about but frankly let's face it it's old wine into new bottles it's old wine that's not the hard part of education I said before education is a two-step process one information transfer two assimilation of that information that's the hard part I've been focusing on that second part because I think that's where we need to put our emphasis just taking a lecture and putting it online why would it become a better lecture than than a live lecture anyway any other question on demonstrations and the ineffectiveness of demonstration could we have the microphone can I ask the person there's somebody in the back I had my hand up I saw Selkan's keynote at Blackboard 2012 this year in New Orleans and he was very clear that his new wine old wine in new bottles is intended to be used as part of a more interactive environment in a school that's my first point the second point is that I think the fact that you're lecturing at us shows why the lecture is still a popular mode of information transmission because it's a way to introduce information to large numbers of people in a cost-effective way particularly when these ideas are new and I wondered if you'd care to respond to that oh absolutely I'm not trying to teach you physics here I'm not trying to do something that is conceptually very difficult so I think that's not the same I think and you know there are plenty of examples online there was one that was actually posted by Michael Pershan a few months ago look for M. Pershan P-E-R-S-H-A-N and there's another one by what is his name again is his Twitter handle Derrick Miller in Australia critiques of the Khan Academy and also ways of improving on the Khan Academy the problem is a lecture including my own lecture here or a lecture in the classroom or a lecture online is that it holds the mind captive there is no time when you're in the audience to pause and think there's no time to let the mind wander and the mind likes wandering you need to have your mind wander in order to make the necessary connections so I think the big problem is the continuous stream of information and you saw the trace that I started my talk with both in TV watching and in listening to a lecture the trace goes flat because there's no time to think and reflect that could be improved by breaking up things in questions but sorry we're breaking up presentations into small sections which the Khan Academy is doing in a sense but it could be improved even more by having speed bumps in between those sections where you actually engage the mind of the learner by asking for input and questions actually Sal Khan is starting to implement that too you had a question too just a quick observation in the early 70s I was at Sussex University and with colleagues we started learning to use programming to teach a whole variety of things linguistics, logic, mathematics and so on and by setting exercises where the students had to design things to do things they got feedback all the time because their program didn't work and they learned from that and they were asked to write about it and so on and I think that that is a way to get feedback which is highly productive absolutely I couldn't agree more about the importance of feedback I think the feedback is so crucially important where do you stand as a learner and that actually leads very naturally to the last part of my talk which deals with this topic of confusion in general instructors are praised for clear lectures in fact on most teaching evaluation it's a question that Professor Mazur give clear lecture on a like it scale of one, two, five and confusion is generally seen as something extremely discouraging nothing is worse than having a confusing lecture but on the other hand to wonder is to begin to understand we don't learn by just opening our skull taking the information and closing our skull you learn by thinking why is it that way and reflecting on it and then going oh yeah now I know that's the real learning process not just listening, storing and regurgitating later although I would argue that much of our assessment just as lectures are regurgitations of information so is a lot of our assessment regurgitating back of the same information one thing I noticed when I switched from lecturing to peer instruction was that there were no fewer questions but more all of a sudden you know there were lots of students who were asking after I had them think about a question and talk to each other there were lots of questions saying Professor what about this, what about that so and another thing I noticed is the following before each lecture I give students a reading assignment they have to read a chapter in the book and they have to communicate to me before class whether they found anything in the reading difficult or confusing and the night before class I press on a button on my website and I get the pictures of the students with their names and after it says what is difficult or confusing normally when you teach a large class you see a lot of faces but there are no names attached to the faces and when you check work that people do you see names but there are no faces this was an opportunity for me to connect names and work to individuals but the interesting thing was to correlate the students to their perceived confusion after reading the textbook I would have some students would say what is so confusing and I would look at the picture of the student and think this is not one of the best students in the classroom let me look up his record I am sure enough was a struggling student and then there were other students who said I am confused about this I am confused about that I am confused about this these whole essays this is one of the best students in the classroom I thought that is kind of strange it looks as if there is an anti-correlation in their actual level of understanding so does confusion indicate a lack of understanding or maybe the converse does lack of confusion indicate understanding so I decided to actually after using this approach to teaching to go back and carefully analyze the feedback I received from students on their reading assignments now if you are interested in more detail of that technique which is called just in time teaching there is this book published by Pearson Prentice Hall in 1999 called just in time teaching blending active learning with web technology and the idea is you give students an assignment typically it is a reading assignment but it could be a watching assignment you could watch a lecture too and that is followed by two content based questions difficult questions are not the type of questions where you can just look up the information you have to take whatever you have seen or read and applied in a new context most students cannot do that but they are told you are going to be evaluated on effort not on correctness you have to show me through the thought you put in the questions that you have done the reading or the watching and then the third question feedback question what did you find difficult or confusing I am going to show you one example out of a fluid dynamics fluid statics class and again you know there is some physics but don't worry about the physics and the idea was to analyze the understanding expressed by the answers on the first two questions and correlated to the confusion expressed on the feedback questions so let me give you an example I am going to give you the context here too so here is the first question consider the capillary rise of a liquid in a glass tube where you take a small capillary you put it in the water rises in it how does the pressure at point p at the surface of the liquid compared to the pressure at point q at equal height now let me give you the answer and I am not here to teach you about fluid statics but it is the same because if there was a pressure difference then there would be a flow of water into or out of the tube because that same height in a liquid the pressure is the same by the way this was a class taught to medical students capillaries are important right blood vessels understanding capillarity the textbook that we used was one that actually applied physics to the context of medicine and it made a very strong point of always having examples out of the medical world here is question number two you have a tube with a valve in the middle and you have two balloons identical balloons but blown up to different amounts balloon B is inflated more than balloon A what happens when you open the valve the intuitive answer would say the big balloon empties out in the smaller balloon but as anybody who has ever blown up balloons known right for your kid's party or whatever you take them out of the bag, you put them on your lips you start blowing you must have noticed that the effort in the beginning is significantly larger than at the end the first few breath that you have to get in the balloon you have to work really hard once there is air in the balloon all of a sudden it goes a lot easier the reason is that the elastic membrane when it is folded can pull in much more easily than when it is blown up actually the pressure in the small balloon is much higher than in the big balloon and something absolutely surprising happens if you do this we actually do the demonstration in class if you open that valve the little balloon empties out in the big balloon and it turns out this is crucially important in medicine for the alveola in the lungs the alveola in the lungs are not really little balloons but they are cavities that are similar to the cavity inside the balloon the main reason that prematurely born babies before 20, 21 weeks of gestation cannot survive is because they lack a substance called a surfactant which coats the inside of the alveola of the lungs which is there to even out the pressure because if you didn't even out the pressure then little alveola would collapse into bigger alveola before 21 weeks you don't have that take a baby or fetus expose it to the air all the little alveola collapse into the bigger one the bigger one's burst with disastrous consequences the book actually mentioned that so I thought it would be interesting to see if students could take it from the alveola to balloons and then the third question please tell us briefly what points of the reading are most difficult or confusing if you did not find any part of it difficult or confusing please tell us what you found most interesting that last part is crucial if you don't add that then students could get away by just writing nothing is difficult if nothing is difficult they have to tell me what's most interesting so let me give you an example of two students responses to these three questions one capillary action is due to the cohesion between water molecules and the adhesion of water to the surface of the glass tube this is a sentence just pull out of the textbook cut and paste more or less it's true I don't know how relevant it is to the answer but it's true negative pressures can result from the cohesive forces of water that is another sentence out of the book it's also true I don't know if it's logical but that's okay then at the same height the pressure inside the tube is much less due to negative pressures I can't follow the logic I don't know but it's wrong anyway so wrong then the second one the air flows from high pressure to low pressure that's right the fully blown up balloon has higher pressure than the half blown up balloon sorry that's wrong so the air flows from the fully blown balloon so wrong wrong what about the last question about the feedback nothing was difficult or confusing the sections on the surfactant in the lungs question 2 and the heart as a pump were interesting because they relate physics to biology okay so now for a very different example the water rises because of an interaction between the water and the walls of the tube that's the same idea that the other student said but now this student is trying to put it into his or her own words rather than using the words from the book the interaction creates an upward force which causes the water to rise the force is due to the surface tension between the water and the walls of the tube the pressure at the point inside the tube must be the same as the pressure at the point of equal height outside the tube because if there was a pressure difference then there would be a net flow of water into or out of the tube and the pressure difference is equalized fantastic the students own words well argued correct question 2 Laplace's law tells us that it requires a greater pressure difference to maintain a small sphere than a larger one so the pressure in the small balloon must be greater and the air will flow from the small balloon into the large one correct now let's look at the last question I found the explanation of Laplace's law and while I can understand the conclusion drawn I don't understand the reasoning which led to the conclusion so I thought what I should really do is take the first two questions for all of the 250 students in the class code them on whether they're correct or incorrect and then look in the third question for confusion expressed on the topic of question 1 or question 2 so this student would be correct correct confused about question 2 but not confused about question 1 and the previous one would be not confused about anything but incorrect and incorrect so let me show you the data which shows something really interesting of the students who said they were confused about capillarity 44% got the right answer to the question about capillarity of the students who did not say they were confused about capillarity only 25% got it correct and the same is true for the other question Laplace's law so confused students are twice as likely correct and the reason probably is that the students who say they're not confused have not even begun to understand they've just read the book passively without engaging their brain they are not even realized they're not even realizing the book is different from what they're thinking the same danger that appears in a quote-unquote clear lecture you walk out with a false sense of security thinking you understand it but in fact you haven't even begun to understand I'm sure there are many academics here and you go to colloquial seminars once a week from time to time you have these brilliant lectures and then an hour later you come across a colleague in the hallway and the colleague asks you did you go to the colloquial lecture yeah I went oh it was fantastic what was it about and that's when you realize that you can actually reproduce the subject right that's the danger of a clear lecture incidentally using a research-based text which is a text that prevents the students from reading through fast which anticipates their misconceptions and asks questions again shedding light on how we may want to do online education by just not just delivering information and having students reflect while they're reading it we actually are able to erase that difference between the confused and the not confused students but here's the key point there is more confusion among students who understand especially when they're not pushed to think so I think that we must recognize that confusion does not necessarily correlate with understanding it's also not necessarily the result of poor teaching you can have brilliantly clear lectures no confusion at all but there really is no understanding either because most people have not even begun to understand confusion is part of the learning process and should actually be elicited Socrates already said 2000 years ago we should teach by questioning not by telling why am I making this big point of confusion well you can go over the past 2000 years of history of educating people and you'll see plenty of complaints over the centuries of educators complaining that students are not learning and as I said 2000 years ago Socrates already said we should teach by questioning not by telling how come that we're in the 21st century and we're still mostly teaching by telling how is the move to Coursera Udacity Khan Academy and so on the focus is on presentation not on questioning I think in part it is because once you start implementing questioning you elicit this confusion and confusion is generally seen as something negative of course confusion in and of itself is not good enough you need to resolve it eventually right but unless that confusion is an essential part of the learning process it is going to be very very difficult to change the approach to teaching so the point I want to make by these three little distinct parts is to show to you how classroom data is vital to actually improving education and as I said at the beginning of my talk I see my classroom as a laboratory in my real laboratory I collect data on the interaction of laser pulses with matter in the classroom I collect data on education in the hope of finding ways to improve education so I want to end by giving you a link to my website the slides are not on there yet because my hotel internet connection was not very good but I'll upload them very shortly and you can also find more information now no need to write it down just remember my last name go to Google and then hit this button and then you'll have a copy of this presentation thank you very much