 Thank you. We are very pleased today to have visiting us Dr. Peter Norton. Peter is a very distinguished lecturer. Many of us have learned from him, from his books on this programming, on artificial intelligence. We've all been very much influenced as a field by the work that Peter's done at various capacities. He was the chief of computational sciences at NASA before moving to Google, where he was director of search quality. He is now the director of research at Google. There are many other accomplishments and achievements. I will refer you to northing.com. You can't go there. There are a few moments you will not only be edified about what Peter's done, but you'll also be abused and interested by many things you have to say. You can't remember that URL while you can type it into Bing. And I think it's the second one. So one of the more recent accomplishments was kicking off the current movements of massive open online courses. And he's going to tell us a little bit about the future of education today. Thank you. Thanks, Mike. A couple weeks ago, I had a meeting with Eric Horvitz. Beforehand, we were talking back and forth. And I said, oh, that reminds me of such and such. And he says, oh, let me remind that. I want to look that up. And I said, yeah, Eric, you could Bing it. But he said, no, I want to find the answers. I'll Google it. OK. So I got into this online stuff because of this, this book that maybe some of you have seen. And when you write a book like this, periodically the publishers come back to you and say you've got to write a new edition because kids are buying too many used ones from the bookstores and we don't make enough money off of those. So it was time to write a new one. And this time, more so than the previous editions, I felt the limitations of writing on paper or even on the pseudo paper that you can get on the Kindle. And here's an example. So here's a section or a couple paragraphs that my co-author, Stuart Russell, and I both had a crack at. And Stuart wrote one. And I said, no, that's not quite right. Let me give it a try. And I wrote a version. But then I said, oh, that's not quite right either. And then I realized that the two of us were like the worst people in the world to decide which version was best because it had been a long time since we had been confused by this stuff. And the best people to decide were the readers. But we were losing that feedback loop. And we knew that there were readers out there and they were taking the previous edition of the book and they were marking it up with the little yellow highlighter saying this is really important or maybe they had a red highlighter said this is really confusing. I don't understand what's going on. And they were writing notes in the margin and maybe they got it right and maybe they got it wrong. But we didn't get any of that back as we had no way of knowing how effective this version was, let alone the capabilities to do an experiment to say, let's take half the students give them version A, half the students give them version B, and see which one is better. And even farther removed from saying, can we predict from some features of the students, this guy would really benefit from getting version A and this other person would benefit from getting version B. So that's the limitation of writing on paper. Here's another aspect. So here's a little diagram I made and I had to write some programs and generate some data and play with that program and then plot it out and then take this 3D surface and I rotated that 3D surface around to try to give you the best view which gave you the best outlook on it. And boy, I learned a lot from playing with that and now the poor readers don't get any of that. They don't get any of that experience of playing with the data. They just have to imagine what's there from this static set of pixels. And I'd like to be able to give that back and have something that was interactive that actually moved and they could press the play button and they could edit it and they could say, well, what happens if I move that point? How would that change the output? Or what if I used a different loss function or maybe I vary these things back and forth and so on? But they can't do that on paper. So I thought, well, one way to look at the problem is say, how can I make a book better? And the other way to look at the problem is saying, can I go back into the classroom? I had been sort of out of the .edu top level domain and as Mike mentioned, it's been a brief time in .gov before moving on to .com. So I hadn't taught for a while but I had the opportunity to go back into the classroom and see how that approach compared. So here I am at Stanford teaching a class and I hope that the subject matter of the class is up to date. But it occurred to me that the presentation style of the class hasn't changed much since this 14th century woodcut. This is University of Bologna, the very first university. And we got the stage on the stage, just like today. And we got the textbook. I don't know if that's the second or third edition or what. They didn't have LaTeX then, so it was harder to make a new edition. So that may be a first edition. And we got the sleeping guy in the back. So our class at Stanford was 9 AM, so I'm very familiar with that phenomenon. And I should say, there's this tension between the lectures on the one hand and the book on the other hand. And this isn't the first time that people have complained about this. So Boswell, in the life of Johnson in 1791, said that now when we can read and books are so numerous back there in 1791, lectures are unnecessary. And what's the problem with lectures? Well, there's no rewind button. If you miss it in the lecture, then you're out of luck. Whereas with the book, it's interactive. You can go back and forth. You can read the paragraph over again. It's an exciting new technology. So I wanted to say, we've got these online stuff just starting, but it seemed like they're approaching it from the lecture point of view. And then I wanted to come at it from the book point of view. And then I was wondering, do they meet in the middle? And if they do, what's the right result? So Hal Abelson, who's a distinguished educator for many reasons, happened to be visiting Google. And I said, Hal, I've been sort of doing a little bit of education. I've been a teacher before. I've been a textbook author. But I don't really know the theory behind it. And now I want to really understand that. So give me a reading list. And I was expecting Hal to tell me a big stack of books that I had to get up to speed to understand pedagogy and the educational literature. Instead, Hal gave me the best answer I ever got, which he said, you only have to read one paper, which is this paper by Benjamin Bloom from 1984 on what he calls the Two Sigma Problem, in which he says that if you give people individual tutoring with mastery learning, which means you keep tutoring them until they get it, rather than tutoring them until it's the time for the test and then if they don't get it too bad, then the results are two standard deviations above the average of the control group. So two standard deviations, though. The curve moves to the right. You can see here, 98% are above average, 2% away from Lake Wobagon. So a very exciting approach. We'll see later that not everybody could duplicate exactly that full 98%, but it seems the most promising approach. So if we know that that's what works, why doesn't everybody do that? Well, there's only two problems. One is there aren't enough qualified tutors for everybody. And secondly, we've decided we can't really afford that. Education is expensive enough as it is with a 30 to 1 or whatever ratio. We can't go to 1 to 1. But can we use technology to get something close to that? So this really struck home to me. This one-on-one tutoring approach seemed to work very well. Here I am with my first tutor, my mom. And that seemed to work for me. So I said, can we duplicate that? Can I bring that to the masses to more than one student at a time by having an online class and making it seem as if it was one-on-one tutoring? So I said, let's try to do that. And I'll take the place of my mom. And the student taking the place of me will be a video camera rather than a live student. Now, the question is, how can we make that work? How can we make that seem like a one-on-one experience? Well, Sebastian Thron and I said, let's give it a shot. It was our turn to teach the AI class. And we said, in addition to teaching it at Stanford, let's open it up to the whole world. And it turns out the world was interested and started to sign up for it. And since then, there have been lots of other classes. So we've got Coursera, and Udacity, and edX. And you've seen this familiar format. So there's some videos. But then the videos are interspersed with things for the students to do, with questions that are interactive, which seems much more engaging than the hour-long recordings of videos that were the previous generation. And a lot of these questions are really making the students engage. It's not just fill in the blank. It's making them think about the first principles. So what are these courses good for? And the first thing we'll ask is, who are the students? Who are we serving? At least so far, and then potentially. Well, in our class, we had 160,000 students from 209 countries sign up. And in the end, 23,000 of them made it all the way through. So that's a lot of people. Seems pretty diverse. What can we say about the population? Well, today, there's about 250 of these classes. And this deck is about a month old. So maybe there's 300 now. I don't know. Completion rates seem to be about 10%. And on one hand, that seems bad. University of Michigan does much better than that, right? On the other hand, let's compare to Stanford, where my online class, 99% of the students finished. We had 300 students, only three of them dropped out for whatever reason. So that seems like you're getting 99%. On the other hand, Stanford only admits 6.6% of their students. So of the people who potentially might have wanted to take that class, multiply 0.99 times 6.6. And that's only 6.6% got to participate in that class. So Stanford is filtering out potential students at the start, whereas the online classes are saying, you get to filter yourself out on your own. And so I think the 10% rate is really meaningless. What would be more meaningful is saying what students have these classes failed. So if you sign up for a bunch of classes, like I know I've contributed to the low completion rate, because I signed up for about 30 classes and I haven't completed any of them. But that's not a failure, right? I got exactly what I wanted out of it, is that I wanted a sample and see, well, what's this professor doing? Is there something interesting or new here? And I watched a little bit of it, and then I got what I wanted. And I moved on to the next thing. So those classes were a complete success to me, even though I had a 0% completion rate. And I think there's many other people that are in that same situation where, for whatever reason, they only want to see a little bit of it. But if there are other people who really wanted to complete this class and they aren't able for some reason, then that's where the completion rates become important. We should try to understand why are they failing? Are they feeling discriminated against? Do they not have the prerequisites? Are there parts of the technology that aren't working right? So let's focus on the why's rather than the numbers. Looks like there's about four million students all together. And now some of the providers have higher numbers. I think Coursera has said that they alone have five million students, but that's counting student registrations, not unique students. And they have many repeat visitors and so on. In the one and a half years since this all started, there's been at least a 12-fold increase in the number of students, but an 80-fold increase in the number of classes. So that means that the class size is going down and the completion rates is going down. But it's still very high growth, so I wouldn't worry about those numbers either. Looks like about a third North America, a third Europe, and a third the rest of the world. So some amount of diversity there. We joked that our class was biggest in the United States and then beyond that was in the typical BRIC countries, Brazil, Russia, India, and Canada. Those were our four biggest. And why was China left out? Well, because you usually can't get YouTube at China. And so it just didn't penetrate there. If you were clever and knew the right proxies to go through, then you could make it work. But for most people, they couldn't make it work. In our class, and this seems typical, 80% already had college degrees. So yes, it's been very rewarding for me that I get emails from some kid in the developing world in Pakistan or something who said, I never thought I'd have this opportunity to get an education comparable to Stanford, but I took your class and I was able to do it. And that's great, and there are thousands of those kids and it's very rewarding to be able to reach those. But most of the kids are not that. There's thousands of those, but there's millions of these elite, privileged early adopters. So this is a new technology, and the people that are using this new technology are the same ones that are buying the latest iPads and all the latest consumer devices and trying out new things. And that's what you'd expect with all new technologies. There's this curve where you hit the early adopters first. And we haven't shown that we can cross the chasm yet and reach the broader community. So why did the students decide to stay or go? So I decided to leave because that's all I wanted was to sample a class. Well, getting back to the book, when I wrote the book, I thought, what's my job? Well, my job is to write down all the information we know as a field and make it really clear and concise and easy to read. And if I've done that, then I'm done. And similarly, at Google, I thought my job was people type in a query like that and that's a hard query that another search engine doesn't get right. But when you type that one in, then in 0.2 seconds we come back and say, children of men, we know that one. And getting this hard information right, I thought that was the key. But then I started to realize that no, that isn't the key. And you know, Pablo Picasso said that computers weren't interesting to him because computers can only give you answers. And I think he was right, that the important thing is the question. And so, you know, I'm proud of what we were able to do in search to be able to answer questions like this. But really, the more important part of that exchange was the person who came to us who had something they were interested in, who had a question that they wanted the answer for. All we did was find the answer. That's the easy part. And it really struck me when I talked with a teacher from Teach for America who had a class, I forget what it was, I think maybe eighth grade class in mathematics, and they were two years behind grade level. And by the end of the year, she had them up to grade level, almost everybody in the class. So that's three years of math education in one year. And I thought, hey, wait a minute. She didn't do that because they had better information. It wasn't that they had a better textbook. And yes, she had to be a skilled mathematician, but she didn't succeed because she knew more math than the previous year teachers did. She succeeded because of motivation and determination because she made a pact with the kids that said, look, I believe in you. I'm gonna do this. I'm gonna work hard for you. I expect you to work hard for me. And together we're gonna learn this stuff and we're gonna progress. So is that determination, the motivation, not the information that mattered? And that really turned things around for me. So what makes you get through this? Well, willpower is the key. How can, it's not that we're just imparting information. It's that we're providing the environment where people can harness their own willpower to do things that they wouldn't otherwise do. Another way that should have been obvious to me is I taught this class and lots of people have come up to me and said, professor, thank you very much. It was a great experience for me to take your class. Lots and lots of people have done that. And in the whole 20 years or whatever it's been of writing the book, I think I had three people who came up to me and said, I read the book independently of a class all on my own. Now anybody could have done that. Anybody could have gone to the library at any point and gotten that book or got another book and studied it on their own. But few people have that willpower and it was the class structure that made them be able to do it. So how do you instill willpower? Well, I thought back to my college days and figured due dates, that was the key. Why did I get stuff done? Not because I had great willpower but because I would have flunked if I didn't get it in by Wednesday or whatever. So I think that's important. We focused our class that way in contrast to things like the MIT OpenCourseWare where they said, let's take advantage of the internet. We're gonna put this material up there and anybody can access it at any time. And that's great, that gives you a lot of flexibility but if you can do it any time, you can do it tomorrow and if you can do it tomorrow, eh. You never quite get there. So we had set dates. I'm not sure that's quite the right way to do it. I'd like to give a little bit more flexibility. I think it'd probably be better if we had the students make their own commitments, set their own due dates but then be forced to keep them. And then peer support was the important part of it. To say it's not just you who's doing this but there's a community of people and they're all learning it too. And look, they're studying hard and they're working hard and if they can do it, you can do it. And so we had discussion forums and people formed other outside groups. They used meetups to meet in real life and sort of every social networking thing on the web people were using. Faculty encouragement is really important so the fact that they have a relation and it's not anything like the one-to-one relation but it was still important and every week we would send out a message to people who completed the work saying congratulations you got this done, you did a great job, keep it up. And every week we would get emails back of people saying, oh, you know, a terrible thing came up at home or at work and I was just about to drop out but then I got your letter and I realized I had to keep going. And I felt like writing back and saying, you know that was a form letter, right? Yes. I didn't write a personal letter to all 100,000 of you but it worked so it's not a complaint. With the pride of accomplishment of them having done something and you know the association with Stanford and the fact that they would be recognized for it I think part of it was the authenticity and maybe that's why the email worked it wasn't just anybody writing to them but that they appreciated that Sebastian and I took our time to do this and put in this effort for them. I think the excitement of being an early adopter and trying out this new technology. And here's just a list of a bunch of the forums where people were hanging out and when we started the class, you know, Sebastian had this proto startup that consisted of a grand total of two of his students when we started and we said, oh yeah, can you build a website that will handle this? I said, sure, no problem. It'll handle like thousands of people and I said, okay, great, we got it covered. And then when 100,000 people signed up, they said, well, maybe we don't quite have it covered and we scrambled to make sure the website would hold up and one of the things we did to cut the scope was to say, well, we won't have the discussion forums ready to go when the course launches and that turned out to be a great decision because people made up their own and they went to Reddit and Meetup and everything else and formed their own discussion groups and I think it felt more personal to them to say, you know, we sought this out and we're a subgroup rather than to say, well, everybody's in this kind of anonymous class forum so that actually was probably better. The bad part is we weren't able to capture all that data. We weren't able to say, you know, hey, this person participated in the forum and then they scored such and such a score and now we can analyze that data. We lost it because these sites own all that data but in terms of the experience that they had, it worked great. Here's a timeline, I got this from Coursera. They called this a successful course. It's probably not a typical course. It does indicate all the phenomena that all courses see and I've seen, you know, probably 100 of these timelines now. So all the courses that have deadlines have these spikes. The spikes are before the deadline so the deadlines do work. They do make people work. They all have a little ramp up at the beginning where people are a little bit slow to get started and they all have an exponential decay. This exponential K is much less than most. That's probably why Coursera released this one and not another one. But you see that pattern of falling off, punctuated by spikes whenever the homework gets due. So what does that tell us? So we have these three components and I think what's easiest to manipulate then is the due dates. That seems to have the biggest effect and the great thing about due dates is it makes people work. The bad thing is it makes people drop out. That people say, well, something's due and gee, I just can't get it done in time because something else going on in my life and if I can't get it done then I won't get the highest grade in the class and if I'm not gonna get the highest grade I'm not gonna do it at all. So we know people drop out because of that. So there's these various models. There's the synchronous model which is the class meets at nine in the morning and if you don't, aren't there for the lecture, you never get to see it again. That hasn't really been followed online. That's mostly for just in classroom. There's the evergreen model of the classes. Oh, is there and you can take any lesson at any time. That works but it doesn't give you as much motivation and doesn't give you the concentration of people in discussion forums where everybody's working on the same problem. So if you ask a question you get an answer right away. So Sebastian and I and now many of the others use a semi-synchronous route where we say, okay, homework's due every week but during the week you can watch the lectures and do whatever you want as you want. I'd like to get to what I call the bus route model which says the course starts up and you can hop on the bus and take it and there'll be a schedule of when homework is done but at any point if something else coming up in your life you can get off the bus and then you can take a time out and then you can wait and get back on again and maybe the system will tell you, well you can't start today because we don't have a big enough cohort but we're gonna start up again tomorrow and then you can join that and maybe you can take the express bus that goes a little faster or the local that goes a little bit slower and we can find you the right cohort of people that you're comfortable with. Okay, now the big questions. So what is learning? Well one of my heroes and founder of our field of AI Herb Simon reminds us that learning is something that the student does not something that the teacher does and I said when I was writing my textbook I thought of my job as something that I did but that's the wrong way to think about it. I have, if I write the clearest explanation in the world I've done nothing if nobody learns from it and what I really have to do is facilitate their learning. Learning is something that goes on their head not something that goes on in my head and that I put down on people. So what does that mean? So we decided we wanted to have an approach where instead of explaining things clearly our goal was to get the students to do the work and in order to do that we asked them problems first and then we gave them explanations. So we said here's a new area, here's this thing how do you think it works? A, B, C, or D fill in the blank, whatever you've got something to do you've got to think about it. And this was important one because we just wanted to engage them at all we didn't want them to sit passively and two because we know from the literature people like Eric Mazur at Harvard that if students have misconceptions that just telling them the right thing isn't enough to rid them of their misconception. Instead you have to get them to commit to say well my view of the world makes me think that this is gonna happen and then show them that that commitment or that prediction is wrong and then they're willing to listen to something new. And it's important for them to fill in the blanks themselves so there was a great study where people took experimenters took passages from textbooks that were graded as being very clear examples of great writing had the subjects in the experiment read the passages and then answer some multiple choice questions and measured how well they did. And then to the other condition in the experiment they crossed out one of the key sentences in the paragraph and had the students read that. And it turns out the students in the second condition did better because what happened is the students in the first condition read this paragraph and said oh yeah this makes perfect sense first A then B then C I got it this is so clear no problem and then they got to a question they said huh I kind of forget what's going on but in the second condition the students say okay let's see well A then C oh wait a minute well that doesn't make sense how could that be? Let me think oh there must be a B in the middle and doing that on their own meant that they were gonna remember when the first students didn't so that's how we tried to organize the course so that the students would come up with the answers themselves rather than having us just say it. And I gotta say they didn't like that. And I also gotta say that most of that was our fault in user interface design because you know we would say okay we're gonna ask you a question we know you don't necessarily know the answer it's okay to get it wrong but we said it was okay to get it wrong but then when they gave the answer we gave them either a green check mark or a red X mark. And you know these guys have all been trained through their entire educational career that a red X mark is really really bad so we just blew it in terms of design there and I think that could be done a lot better. I'd like to have more experiments with more open-ended problems we mostly did the fill in the blank or multiple choice type stuff. There've been some experiments now with essays and other types of things using pure grading for stuff that's hard to grade mechanically. And now here's two more of my heroes Hal Varian, chief economist at Google and I guess he has a connection to Michigan as well and Richard Hamming and when they were together at Bell Labs Hal was writing this classic textbook on microeconomics and he asked Hamming how do you write a book? And Hamming said well first you get together the problem sets and exams that you want students to be able to solve and then you write a book that will teach them how to solve them. And I realized hey I've seen this before in software development we call this test driven development. Right so first you write a bunch of tests and then you hit the go button and they all turn up red and then you keep writing code until all the tests are green and that's how you know you're done. And so it's the same thing except writing a chapter of the book is like writing the code. I think that's mostly right that you wanna focus on a set of skills that the student has. I think you don't wanna focus too much on just teaching to the test has this sort of negative phenomenon so you have to be really clear about what the skills are. You don't wanna say well I want students who can answer these specific multiple choice questions then you're kind of optimizing the wrong thing. But if you choose the right set of skills then I think this is a good way to go about it. Okay now, so do we trust Benjamin Bloom? Remember he was the one paper that you had to read to understand everything there is to know about education. So here's some plots from his paper and you can see how far we've advanced that we no longer have to use pencils to write our graphs. We now have computers to do that. But look what's actually happening here is that you can see from this that it's not just the tutoring it seems like it's this combination of mastery learning and the tutoring that plays a role. And they both seem to be important from what we see here. And recently Kurt Van Lane who's now at Arizona State and he was at Carnegie Mellon during the 80s when this whole sort of intelligent computerized tutoring was getting started. He did a meta-analysis where he found every study that looked at this and here's what he found. So the conventional wisdom was that the human tutor gives you two sigmas of advantage and intelligence tutoring system gives you maybe one compared to a baseline of just these multiple choice drills which don't give you very much. But he found in actuality there's very few studies that really give you that full two standard deviations and mostly the human tutoring is about the same as the intelligence tutoring system and they give you about 0.8 or so standard deviations. And so the conclusion from that is one that it's more mastery than tutoring. And in fact one of the key studies that Bloom sites wasn't one on one tutoring it was one on three, three students to one teacher and what seems to matter more is the degree of mastery that you require. So if you let students go on to the next lesson once they've scored 80% then they do much worse than if you require them to get up to 90 or 95%. And then another thing that struck me out of this is so here are these lessons that have sort of been ingrained in us. Hal told me about the Bloom study. I've been quoting it, Daphne Kohler's been quoting it in her talks. And you go back and look at the studies and sort of the most persuasive of the studies has N equals 33 students, right? So should decades worth of research be predicated on what 33 students happen to do? Maybe a couple of them had a bad day or something and it could throw everything off. But now we have the opportunity to do experiments with N equals 33,000 or maybe 3 million. We are suited to gather much more data much more quickly. So maybe we can advance this field a little bit more. Okay, so how do we do that? How do we make it so that we have these two key ingredients of one-on-one tutoring and mastery learning but at an affordable scale through automation? Now, I should say we're at this point now where everybody's excited about this new technology. But that's happened a lot of times before. So these are all examples of new technologies that we're gonna save us and save the field of education, right? So movies, then no, but we got radio, right? And then television. Television was supposed to be, its main purpose was gonna be education. And well, we see where that went. And then, you know, video tapes and then we had the personal computer and that was really gonna be it. And then say, oh, wait a minute, I was just kidding. You know, personal computer with a floppy disk, that wasn't good enough. But once we got CD-ROMs, that was gonna revolutionize education and then through all these new devices. And it seemed just keep on sounding like the technology was there, it was gonna revolutionize, but then it never materialized. And this is Edison with Eastman. And in 1913, Edison said that this new technology of the movie was gonna revolutionize all of education. And he said, within a decade, the textbook will be obsolete because we'll just learn everything by movie. And you can substitute that within a decade quote for every other technology we've seen so far. So what's the hope that this will be different? And I think if there is a hope, you know, well, maybe 10 years from now, somebody else is gonna be standing up here telling you about the new technology that's gonna finally pay off. And that may well be. But if there is a hope for why is today different, it's that we have a feedback loop that we can close and because we can close that, we can do this individualization and we can complete that loop. There are many types of individualization. So the simplest is student control over rewind. They can just go back and replay the lesson. They can have a choice of where to go next. We can flip the classroom where they watch online and then they come in and work on it. We have peer instruction where they help each other, answer questions, peer feedback where they grade themselves. We have these forums. There's this possibility of collecting all this data and applying machine learning. We have what I call courseware engineering and we'll get to that. So quickly, this is peer grading and this is a correlation of student grades of other students with teacher grades. Looks pretty good. And in fact, if you did teacher versus teacher, the spread would be about the same. Turns out self grading is even more precise, which was surprising to me. People actually know when they're doing a good job. Of course, if you had the self grading actually count towards their grade, then you'd lose that correlation. So what do these four characters have in common? Well, you might answer two things, which is online classes and artificial intelligence. And I think it's not an accident that so many people involved in this come from this field in that we have this sort of religious trust in the power of data. And we think if we just gather enough of this, we may actually learn something. And what do we mean by that? Well, this is the traditional model of what the output is from the educational system. So you're here for four years and you get this transcript at the end. You can go take that to your employer or whatever. And depending on what you believe about grade inflation and therefore the entropy of an individual letter grade, this is something like 20 bits per year. Now, compare that to a consumer electronics device, which is 192 k bits per second. So that's a trillion times more. It may not have escaped you, that the one on the right is cheaper than the one on the left. So in bandwidth for dollars, it's a quadrillion times more. So maybe we could be doing better. Maybe we could be getting more out of this interaction than just reporting this. And so if we're tracking every click, then we have a much better stream of data that we can use to improve. And also that we can use to characterize who this student is. And maybe the student comes out with a portfolio of projects they've done rather than just a list of grades. Here's another example of you can use the data. So this was from my class, the exercises and how well people did. And they did pretty well on almost all the exercises, but there were two in red there where they didn't really lousy. And that was because I screwed up and we had the assignments wrong so we could go in and fix them. Their cases, let me just skip that. And what we're hoping here, so if you've seen me give my other talk about big data, I show lots of examples about how at Google we have problems that we work on where if you start out with a little bit of data, your quality on that problem is lousy and then if you get lots of data, the quality becomes good. And there's this idea of a threshold beyond which if you gather enough data, the same algorithm which seemed dumb over here suddenly seems like a smart algorithm. And the hope is that we can apply that to education. Now as much as people like Daphne Kohler and me talk about that, we haven't proven it yet. So that's a promise that's yet to be cashed in. Then the last point I wanna make is about how we navigate through these classes. And I tried to find the most complex branching structure I could find for an online class and there was this class in genomics and it had these four parts and they had a little bit of sub parts that's not shown here. And then there was a capability to link in to a prerequisite course and a follow-up course. So about this level of detail is about the biggest kind of branching structure you can get from one of these online classes. Now compare that to this structure. Can you all read that? Is this okay, is this font big enough? So this is Wikipedia's mammals. So there are 7,000 mammals and these are all the links between articles. So this is clearly a much more dense branching structure. But in terms of an educational experience, this doesn't really help me that much. Yes, I could just go sampling in and find something about one mammal or another but it doesn't really guide me. So I think what we need is to combine those two to say we want the dense information that you get from Wikipedia but we also want the guided paths that you get from a class where there's a coherent point of view maybe not just from one author maybe from two or three authors and they guide you through and there's some branching points and if you want you can go off that path and navigate through Wikipedia and go in whatever direction you want go sort of off-road but you can also follow their guidance and get that coherence. And so I have this notion of courseware engineering and the idea is that in the early days software was done by individuals or by small teams and it worked really well if you had the genius individual, here's Wozniak and he could put together a whole operating system by himself but now things have gotten more complex it's harder to do that and we can't rely on a steady stream of geniuses. So we built systems of course of software engineering where we could have teams successfully do larger stuff than an individual could do. So these guys built a video game together and we're pretty good at organizing for a team of this size to work together with the processes we have and we're sort of okay at working with much larger groups with the groups they get up into the hundreds and thousands some people think we're good at that something where we could do much better. So there are certain products where it's clear that the artisanal product produced by a skilled individual is better than the mass produced product, right? But there are other places where the mass engineered product is better than the artisanal product and I think courses are kind of a mix of those but I wanna see it so that we can get to the point where if we want to we can build these larger classes where we can take something that's more than just one or two instructors point of view and have lots of people contribute to it have a big infrastructure and have that available for everybody to consume. So I guess we're running out of time let me stop here and open it up for questions for a few minutes. Universities we're gonna go away when the pretty pressures are extended that doesn't happen. Let's say that you're right and we make courses like this 747 how do you think the university might change your response? Right, so great question and I don't know what the answer is. So what do universities provide? Well I think the most important thing they do is that they're a meeting place for people to come together and it's a meeting place for faculty and for students. So in some sense what Michigan offers is say hey faculty come here and we'll deliver really good students to you and hey students come here and we'll deliver really good faculty to you so it'll be a nice place to work together. And I think that works really well and it may be harder to do that online although it certainly can be done. It's also the signaling of just being in physical proximity of seeing somebody working hard and saying well that reminds me I should work hard too. So schools do all of that. Now we've come to a situation where schools do multiple things. So Michigan does research as well as teaching and that's all been bundled together. And I think that may end up being shifted out a bit and we've seen that in other fields. So in the newspaper business traditionally they were able to bundle together journalism and advertising and as long as they had a monopoly on advertising they were the only one that could bring a text advertisement to your front door every day then that model worked great and they could afford to pay for a lot of really good journalism. Once they lost that monopoly it was harder for them to do that and they saw the two parts split out and they started to say well maybe we can cut on the journalism and only cover Angelina Jolie or something like that. And so that's a net loss to society but I think it was inevitable because of their loss of monopoly view. And so universities are in a better position because both halves of what you do are useful to society both the teaching and the research but you may see them separated out a little bit more that people will be asked to specialize more in one or the other. So I don't know those are some of my thoughts but I don't know what's gonna happen. As opposed to a college and university standpoint how do you feel about open courseware being used in an elementary setting before we hit this data gap where you have enough data? Yeah, so I haven't really focused on that. My background is at the university level so I figured I'd concentrate on what I know how to do. My intuition is that the personal interaction is more important at that age and setting the role models and so on and having the social interaction with your peers. So to the extent that technology is used you wanna make sure that you don't interrupt or miss out on any of those social parts. A lot of my friends signed up for your class and other classes and they are the kind of people that are just motivated to learn how to read and speak and to bridge that gap. I think that's right. I think we haven't crossed that gap yet, right? So there's two questions. One is people who come from a disadvantaged community but have the self-motivation and that I think we're doing a great job with but the ones that don't have the motivation because it's been beaten out of them for whatever reason, I think you're absolutely right. We haven't addressed that problem yet. There's a great interview. Stephen Levy interviewed Bill Gates, came out recently and they asked Gates, well you're a big supporter of Salman Khan and how would your life have been changed if the Khan Academy had been available to you when you were a kid? And Gates says that wouldn't make any difference to me. He said, you know, I was self-motivated. I could find the stuff on my own. I grew up in a wealthy suburb of Seattle so we had a good public library and that's all I needed. He said, what Khan does is for the kids that don't have enough motivation to do that but they still gotta have enough motivation to wanna go to Khan Academy. So it's, you know, it's taking it one step at a time but there's another big gap of the kids that don't even have that much motivation that we haven't reached yet. How much of the discussion is like, we will change your hat regarding the accept of the you know, some of all the credentialing issues that we saw when I said we can't sort of the ability to sort of look at this and hope to read the credentials you've collected in this environment versus the degree. You had those discussions yet? Especially Google which I said already tends to look more at the individual's or a degree of experience. Right. So you know, we've gone through an evolution. We've hired a lot of people by now so we have some data on this, right? And we can see what factors matter and don't matter. And in the early days, sort of grades were a big factor because some of the founders and early employees thought that that was important. Now we've run the numbers and found, you know, that doesn't really matter that much. So we were looking for other factors and so I think we would definitely accept substitutes. You know, we wanna know, can you do the job? Not so much, do you have the degree? And I especially see how having something like this could help calibrate, right? So I mean, every year we have several examples of we get an application from some school in China or India that we don't know, right? We know the top schools there and we have those well calibrated. But if an application comes from a school that we don't know and it says this is the greatest student we've had in 30 years, we usually have to throw that application away because we just have no way to calibrate it. But if we got that application and on top of it, it said, and they took these online classes and they got these particular scores, then we could say, oh, okay, now I know how to calibrate this score on this online class versus what we know for students that we've been hiring. So yes, this is good or no, it's bad. So I think that would help. There is always this problem of verification of identity but we figure once we've made the commitment to bring somebody in for an interview, then we can quickly tell whether they really are that person that they claim they were or not. So you brought up the user journalism space as a potential example. But they're simultaneously there is a democratization because you have all these sources of news and information. At the same time, there is also a little box news in the concentration. So is there a similar danger if this takes up that there are say non-engineering fields in particular where there are points of view and aspects of these things that there are a few professors that end up teaching humanities to the world. Right. Yeah, so I mean it could definitely go in both ways, right? So it's just as journalism was opened up to successful bloggers. Most amateur bloggers don't have an audience but a few have a big audience. So people who otherwise couldn't reach anybody now can but we've also seen consolidation to where there are fewer sources. So I think you'll see a mix of that, right? There'll be less need to have lots of different classes offered at every university and there'll be some of reusing the same materials and maybe that's all you need is to take that online class. Maybe you have that as instead of using a textbook you use the online materials and then you have the in-clash discussion or whatever, we'll see some of that. And there you can have the bringing the diversity back in to say, hey, you know, the teacher said this but here's another way to look at it. And because it's cheap to produce these things you can definitely have more than one. You can say here's different points of view on this subject and choose the one you want or combine multiple ones. This is sort of the Coursera and Khan Academy approach which is just taking lectures putting them online and doing some amount of innovation using the lecturers with base. And then there's sort of like the smarter data that by heart are very tasking on YouTube channels style of doing it where it's used very short but high production quality if you will. Videos that sort of touch on maybe a very small topic or something that's not cool. How do you see those combining or which one do you see taking off more or are they completely different? I think they're both teaching stuff but it's a question of what the student wants. So sometimes you just want to know one thing and sometimes you want a more complete education. And so I think you'll see both of those flourishing. You're already seeing kind of a little bit of meeting in the middle and that you're starting to see these classes become shorter. So it was a shock for me to teach at Stanford which is on the 10 week quarter system. Say how can I say everything I know in only 10 weeks instead of 15 or 16? But I got used to it. And now you're seeing a lot of the online classes are similarly shorter. You're seeing the five and seven week classes. And I think that makes a lot of sense. I think we go by the semester system because historically it was a lot of pain to fill out papers and stand in line to register. Now it's a lot easier to register so we can do it more often. The units can be smaller. But I think you're right that there is still a big difference between sort of committing to one professor and getting their point of view and kind of understanding how they think and the vocabulary they use and the terminology and so on versus saying I'm just gonna watch one YouTube video from somebody I've never seen before. But we'll have a combination of both. You're gonna have to end the formal Q and A there but you're all invited upstairs to the reception where we can continue to talk to Peter and have a clear discussion as well. Let's thank Peter Orton. Thank you. Thank you.