 Okay, good afternoon. On behalf of the Red School of Information and Computer Science, I'd like to welcome you to this interesting panel this afternoon on MOOCs. And I'd just like to briefly introduce the panelist. The format is that Dan Russell, who's from Google, is going to start off with a little bit of background about what MOOCs are. But then the three people we have, Dan, Scott Clember from Stanford and soon S.D. and Chuck Severance from Michigan, have all taught MOOCs and they're going to share with us their experiences of what went well and what didn't go so well and so on. Each of them for about 10 minutes or so, so we have plenty of time for you to ask questions and for have some more general discussion. So a lot further to do it, turn it over to Dan to get things going. Thanks, Gary. So my talk is going to be a little bit of background about MOOCs in case you don't know if you've seen them in the New York Times or skywriting in the sky or somehow have this one in root wise. I'll give you a little bit of background and then I'll talk about my MOOCs, my experience, the good, the bad and the ugly. So first off, if you haven't heard the word MOOC, you've been living in a cave. I know it sounds like a word that would be spoken by Dr. Seuss' character, but what it really means is massive open online course and it's a terrible term. It will be a temporary term. Don't worry about it. It's one of those things that will be over shortly. You just have to sort of live with it as a transient term. But what is a MOOC? There are a lot of different definitions of MOOCs. We're going to be talking about our MOOCs. We're not going to try to cover all MOOCs across all space and all time. To first order, what it means is we've got some content in short video chunks that we present. You participate with massive numbers of other students who then all participate and you all kind of go through the content together. So what this looks like, schematically is you get a short video presentation, you get a question, maybe some stuff to work through, you post stuff to a forum, you get a test at the end, you maybe graduate, maybe you don't. It's a bunch of little tiny chunklets, all then organized in sort of little sort of macro elements that you go through with an order of a couple weeks, maybe 10 weeks, whatever. Obviously, mostly at this point, they're kind of patterned after regular courses. My class, for example, the course that I'll be talking about is a two-week class. So it's not a typical 10 or 12-week class that you'd see in university. But increasingly, we're starting to see small MOOCs, mega MOOCs, two-week MOOCs, five-session MOOCs, 200-person MOOCs, 100,000-person MOOCs. There's a whole bunch of different kinds of these things coming up. I'll make this clear in just a second, because we're going to show you some real examples. The big players in MOOCology right now are companies like Coursera or edX or Udacity. I put mine up there, Course Builder. This is an open source platform for building MOOCs from Google. There are others out there in the world. What's interesting is that there's a huge amount of sort of surge going on right now. Udacity makes a great move. They just announced that for 7,000 bucks you can take their classes online with Georgia Tech and get a Masters in Computer Science. You have to pass the test. But assuming you do that, you can actually do that. Increasingly, you're going to see more and more announcements like this as this kind of stuff becomes more and more popular. edX, of course, is the coalition that started with MIT and Harvard, another Ivy League school you might have heard of. Udacity started with the Sebastian Trunn and Peter Norvig class coming out of Stanford and Google. They're now a separate company. Those are the main players right now. All of them offer some sweet options, some sweet of classes like this. So here's a typical catalog. This is from Coursera. In the middle you see our buddy Armando Fox's class, software as a service class. So there's a bunch of different kinds of classes. And currently, it's student choice. You can take these things, you can sit in, and it's just basically free range learning. You can go, you can attend, you don't have to finish, you don't have to show up even. But it's interesting because I started this about, actually, about almost exactly a year ago. And I was teaching people how to search. I'm from Google. People, turns out, are not that great at searching. So I help them learn how to search. And I go and teach classes like this. I would go and teach a class like this. Okay, everybody get out your laptops. We're going to search for cicadas of Central America. And you would open your laptops and we'd go through a problem like that, right? So my boss says, you know, it's great that you're teaching 3,000 people a year. Good job. You need to scale. This is what happens at 3,000. You've got to scale. So I think 3,000 and he says, no, no, no, no, no. You need to think arena style teaching. So I think it's going to be like this. You know, the rock star model of teaching online. It's really crouchers to the max. You know, everybody is holding up their iPhones with pictures of Twitter feeds and so on. But instead what we did is we thought, well, how can we do this online? Because that's what we're good at. We're good at building stuff online that will scale massively. So in some sense, when we started thinking about this, there's really nothing new about online education. It's been around for a long time. So in particular, at a university far, far away, a long time ago, there was a system that actually had green on black graphics. Remember those? I know you guys in the back, the young ones don't remember that stuff. But at the University of Illinois, they had mainframe computing. And in the 70s, there was a bunch of work done here at Irvine by Alfred Dwork and so on. But there's a lot of stuff out there. Play-Doh had forums, online message boards, integrated video, email, all this stuff that we now think of as super modern. 60s. That's a long time ago. So now the modern version of this is to do it all in a scalable way using YouTube feeds or integrated group messaging and so on. So this is a class I put together, power searching with Google features. 32 lessons by me teaching you how to do better searching. It does things like showing you how to use tools like search by image. I assume you all know how to do that. And we go through lessons in these little chunklets and then every couple minutes I say, show me you know what I'm talking about. Show me that you actually are following along. You have to do these little activities. Now one of the cool things we stumbled across, a great pedagogical tip if you're building a MOOC, is to do this. You see that thing in red? For class 2, lesson 5, the activity is to discover something and post it to the forum. If you have 100,000 students in your class, that means you get roughly 100,000 people posting through the forum and it gets really busy, really fast. And it gets very exciting at that point because all of a sudden they start to see other students and they start posting questions to the forum. How does this work? And someone will answer. And so all of a sudden it becomes social. Now one of the interesting things that we did in our class, because we're Google, because we're a software company, we developed it like software. So what this means is we did usability testing. How many of you have been in a class that obviously had never been usability testing? Right? Yeah, me too. Most of them, in fact. And so we added this idea of actually running usability tests, bringing people in and then you see all the people in circles. Those are my people. They're actually watching the people, the participants, the users, going through the material and they're writing down every single thing they don't understand. Everything that went wrong. And we then fixed those bugs. We iterated rapidly. We did the whole line yards. Classic software development style. Don't you wish your teachers had done this? Now one thing that's really interesting is classically your professor comes with a PowerPoint deck like this and talks for an hour. And you have to refactor. Oh wait, that's a software concept. We have to refactor the content to fit into these different modules. So they fit and you present an interesting set of concepts in a couple of minutes, fits into a larger arc of a story, and it all becomes like a Dan Brown novel. You need the cliffhanger to get you for one module next. Now what's fascinating about this is that if you have lots and lots of people in your course, how do you know when you're starting to succeed? There are a lot of mentors. I'll show you some in a second. But one is when people set up Facebook groups for your class and they don't tell you about them. That means you're winning. You also start winning when you discover that Pirate Bay has a copy of all your videos. Fantastic. So this is a kind of sign that we're succeeding. Another sign of success is when you measure pre-class to post-class sort of classic tests and you get a curve like this. Fantastic. Me in here is about 40% improvement. Effectively we're doubling the student's ability to search. Rock on. We've done it. Now one of the things that you'll hear about from other speakers is when you have large end, you're doing real science. You can actually start to see how people behave, how people learn. And so there's a dirty little secret that I've never seen published in an educational journal anywhere. Which is that. That little red chime. What does that mean? It means that some people scored worse on the final than on the pre-class test. Right? Yeah. They were guessing in both cases. They guessed poorly the second time. When you have 22,000 students in your class you actually see that effect. If you've got 30 students in your class, physically you never see that effect. So there's a lot of stuff like that. And there's another kind of phenomenon. We put up our YouTube videos and after we closed the first class, we watched other people. And that's what the red line is here. These are people watching my videos without being in my class. And between October, December of 2012, we served up seven years of YouTube video. So that means there was somebody in a locked closet somewhere watching me for seven years in the last three months of last year. One of the fascinating things about MOOCs is that there's this little interesting community dynamic. And both Scott and Chuck will talk about some of that. But we did things like holding Hangouts on Air, which are the ability to have in-way sort of video conference. And we allowed people to send in questions and vote them up or down. And what that allowed us to do was to get thousands of questions, of which we can't possibly answer 2,000. We can answer the top 20, the top 20 that matter to the most students. Again, this is the worldwide class. So it's an interesting, interesting kind of space to be in. So for us, I've run three MOOCs now, the basic one twice and the advanced one once. And these are aggregate statistics. So since July of last year I've had 350,000 people register for my classes. Of which 77,000 finished. So since July of last year I've had 77,000 people finish my class. Yes, I know that's a big attrition rate. I don't care. I have 77,000 students who took my class and finished it successfully. That's a massive win. Me in my classic computer science professor mode will never ever teach 77,000 students. So this is a huge win from my perspective. And there's a lot we know from post-class surveys. There's a lot of reasons why people don't take the course. The first order, I understand. Life intervenes. Stuff happens. We also have things like the overall 28% is a total completion rate. What I call the first class to course completion rate. What that is, I don't actually care how many people sign up for your class. I get 20,000 Wombats to sign up for my class. Doesn't matter. I care about the people who make it to the first class and then finish. Because you always get a roughly 50% attrition rate between registration and first class attendance. So we also had things like would you promote this, would you recommend this to someone else? We got 4.5 out of 5. That's a great score. I don't know if you do that statistic in your classes or not. That's the kind of net promoter score you get for conjugal visits. For breathing. I mean it's a really high score. So one of the things that this means is when these large cohorts, classes, when you've got 100,000 students in a class you can get very rapid turnaround because students answer other students questions. It changes the dynamic. Look at this. Somebody posts this question at 4.07 and it's answered by someone else at 4.08. I was not up at 4 o'clock in the morning in India when that first question was posted. Another student answers it. There's a huge advantage when students are doing your TA's for it. Because I don't have 100,000 TA's. I don't even have 1,000 TA's. I think I got 10. Now when we run this class again in a couple weeks I predict we can do it with 3 TA's. Very different kind of world. One of the things I like about this large space of students helping each other out is this example. The question here posted by this person in the first case is something I cannot understand this Challenge Certificate blah blah blah blah. This person answers below in Arabic. Why would they answer in Arabic? Because they knew. We moved the names here. We redacted the names. But they knew that the poster was an Arabic native speaker. They cut through right to the point and answered in their native language. I can't do that. But other students can't. Now because we had students when they signed up for the class they signed up in terms of agreement that they would allow us to look at their searches and aggregate the class and for two weeks after the class which point we then deleted the data. But if you aggregate the data you get this beautiful chart. I've never seen this published anywhere else but let me tell you what it is. The red line here this is a measure, this is a proxy measure that we have of how many advanced searches per day these students are using. So they come in at roughly .4, .5, they come into the course and the course starts here and they go up. They also then transition up into this black curve. I'll talk about this curve in a second. These are the students who completed the course. The black people, the black curve is people who completed it. The people who signed up but did not complete the red curve they kind of stayed the same. They didn't finish the class, they didn't get much benefit. The people who finished the class go through this curve up here and they exit the class at what? Two. That's five times the rate that they entered on. And then of course the class ends, they start forgetting, they don't practice as much, but they still continue at .9, .9. So I have evidence A that they've learned really well and furthermore they retain that information. As far as I know nobody's actually tested chemistry 101 students a month after the end of their chemistry class to see if they still remember that stuff. Okay, let's wrap up here with a couple of big takeaways. Okay, first off, as you saw from that the MOOCs really work. So it's clear people are learning, they're actually in a pro-social environment, they're helping each other out. In the first class we ran 157,000 people signed up. We had to exclude three people for antisocial behavior. Three out of 157,000 that's a pretty amazing number. The term MOOCs, here's the good news, the term MOOC will disappear. It's a little bit like cellular telephone. This is my phone. So those sort of descriptive adjectives will wear away under the erosion of constant use and MOOC will just become learning for just a class, just education. We live, and the MOOCs are helping, we live in a remix culture, if you haven't seen this by now. I think the MOOCs with open content, open courses and new ideas that are flowing back and forth will constantly see this remixing of content, ideas, pedagogical notions, all this is going to be remixed and reblended and I hope improve all of online learning. With all the classes, with all the different companies, with all the different styles, with all the different media they're being produced the time of great wrestling is ahead. I do not know which company will succeed. I don't know if it's going to be Coursera or Udacity or Us or whoever. In some sense, the student wins, the culture wins the planet wins. But it's going to be a time when I'm not going to want to invest in any particular stock because I don't know what's going to happen. Formats, content, data formats, all that stuff is up for grabs. Malleability matters. One of the great things about MOOCs is that I can edit them in almost real time. During the course of my first class, we had 100 data pushes. We had 80 software pushes. That's like changing your textbook in response to people misunderstanding parts. We can see what's happening and we can change it. It is a malleable plastic substance. Your textbook that you learned chemistry 101 was not. You got a data push maybe once every four years. Lastly, I want to leave you with this thought, making a MOOC is not trivial. Scott and I are talking about this a lot. It's not, you're not going to save money by making a MOOC. You will save money if you think about it on a per student cost basis but actually creating a MOOC is as much work as creating, of course, maybe more. It's actually a lot of stuff to get in place. The video, the software, the assessments, the rubrics, all that stuff. It's not an easy knock them out of the park kind of thing. Alright, I'm going to stop and turn over to Scott. Thanks very much. We'll take questions at the end. This is my colleague Scott Clemmer. He was at Stanford at the moment but is in the middle of a quantum transition to UCE San Diego. The cat is not dead. Dan and I were at a, we ran a symposium together. Dan wrote me into joining him at the Higgs conference in January and somebody asked us the question. So we've been seeing a bunch of these ideas for two decades or more as Dan pointed out. What's new this time around other than the hype in the numbers? And the answer that I gave was the hype in the numbers. And so that's what I'd like to talk about today is how those small things are actually transformational specifically in terms of the role of peer learning. Currently I teach in the computer science department but my original training was in graphic design and I have a deep love and admiration for the studio culture for teaching. I currently teach a large introductory design class in a computer science department in a school of engineering. And over the six years that I've taught it the enrollment has benefited from the groundswell of interest in computer science and HCI. So it's grown from 70 to 250 students drawing from now 22 distinct majors. And this is a project based course featuring weekly critique and small studios. And to pull this off we have a dozen highly talented TAs who run those small studios. So when I started thinking about this, putting this online it was clear to me that we weren't going to be able to replicate any of that. And for several years at Andrew Inge's encouragement I've posted lecture videos online through Stanford's open classroom. And then Andrew encouraged me to put on one of the first Coursera classes. And I had no intention of doing the projects online. And I got talking with Daphne Kohler about this explaining that to do that would require massive scale peer assessment or something crazy like that. And she said well let's do it. And apparently I'm easily influenceable. So last spring I collaborated with Daphne and Coursera to launch the first massive online class to use peer assessment. And Coursera tells me that since then about 50 classes have used variations of this infrastructure. Featuring diverse media in diverse fields from sociology to nutrition to world music. And so I'd like to share some observations and experiments that we've run. And a lot of this is in progress. And so there are things that we almost have numbers for that I'm going to have to hand wave today. But in a few weeks we'll be able to put a sharper point on. The higher bit is that most peers do a great job of peer assessment. We're finding that 60% of a median score peer assessment is within 10%. And this is for online classes where the ultimate goal is either a binary pass-fail. And once you do a regression to the mean over a number of assignments this actually works extraordinarily well. And this Gaussian looks a whole lot tighter for in-person classes where you may need more fine-grained success in terms of assessment. And students really love it when we see, when we ask people how much do you learn from doing peer assessment. People reported an extreme skew to the high end in terms of the learning value of peer assessment. However, it doesn't work automatically. There are a number of techniques that we employed that turn out to be extremely valuable. And so in order to get peer assessments to correlate with some ground truth measure you need to have a number of principles. And so today for the first time I'm sharing with you the seven habits of highly successful peer assessment. And I'm gunning for a really schlocky title because I think a number of people have tried peer assessment and gone it doesn't work. And when we've looked at it it's always been because one and often more of these things weren't followed. So in order to get peer assessment to work you need to have a per assignment rubric. And that's because you're trying to get learners to do assessment. Which is not impossible but difficult. You don't have a general purpose rubric like the argument was coherent. You have to lay out what it means for an argument to be coherent in this specific context for this specific assignment. And the same is true in terms of training. So before people assess peers you want to have them look at a number of training assignments and when they grade them you can say okay we graded it this way and not that way and here's why we did that. And I think it's extremely important to have self assessment at the end because students are in a different mode when they're evaluating work than when they're doing work. And I think anybody who's ever been an author of a paper knows that the difference between I submit my paper and think it's great and I review other people's papers and I think they're crummy. And if you do self assessment at the end it helps make that gap clear in both directions. We're both more empathic when we realize that others are in our shoes and we can understand the other perspective. If you're going to do this you want to gather staff grades or some other measure of ground truth so that you know how well it's working otherwise you're flying blind. And while the graph that I showed was for taking a median of a small number of grades you can adaptively infer greater bias and skew grades to get much higher quality correlation. And that's being first demonstrated in a paper by Chris Peach that will be published in just a couple of months. We've been using that in the current run of the HCI class and it works fantastically. And then the other thing is that the ability of large amounts of data allow you to improve your learning materials in ways that wouldn't be possible for an in-person class. And I'll give you one example of that which is I've been teaching with rubrics for several years. We took the online class and we ranked every single rubric item by the quality of coherence of peer and staff grades. So we can see what elements cohere most strongly and which ones are totally all over the map. And that helps you look at some of those items and you see them and you say yeah that doesn't make any sense at all. I can see why people wouldn't be able to agree on what that means. And so you can use data to drive revisions. And we have the code for being able to do this on your own class open source online. The other thing that we realized is it's not just about grades. I think in many ways it's primarily about giving people meaningful feedback. That's what they appreciate most and that's what they get crankiest about when it's absent. A lot of feedback for projects is variations on a theme. And you can scaffold that by providing people with what we call fortune cookies of particular pieces of advice. And our interface for that is you can simply copy it with your mouse and paste it into a text field. And we encourage you to add a because reason at the end. And what this enables you to do is not only improve your learning materials but to build a broader theory. And I think the opportunity for science in this space is absolutely enormous. Two quotes by Kurt Lewin that have guided a lot of my research he wrote once that nothing is as practical as good theory seeing these as deeply intertwined. And the best way to understand something is to try and change it. And we've had extremely humbling experiences in terms of trying to move a needle on several issues. We've also had some really fascinating things that we've found with data mining. So for example my student Chinmai found that though all peer assessment is anonymous students rate peers from the same country 5 percentage points higher than students from another country. You can imagine lots of reasons for this and I'm happy to talk more about this offline. Also I put this in for Chuck. Here is our retention graph over the quarter. And Chuck and Dan have seen the same thing. This is for two runs of the class. It is identical across the two runs. So we wanted to move the needle. And we've run several experiments of increasing the sociality of the assignments. And what we learned is that the assignments are a lot of work and hard and life gets in the way. And so gimmicks are insufficient to making a real change here. You need broader stuff. But you can make some simple changes like turning each assignment into a checklist that don't have a big move of the needle but they do get you a couple of percentage points of increased retention. What I think is most exciting from a peer perspective is that students really adopted the class and made it their own in all sorts of different ways. Here's one example where a student took every book I mentioned throughout the class and put it online as an Amazon reading list. I'm sure they make a few cents on each transaction but it's also a really great learning resource. From Chuck I got the idea of in-person office hours. So in Philadelphia and Paris and in the computer science department at Stanford we'll invite people to come for office hours. And though I worry that nobody will show up every time, we've had almost exactly a dozen people show up for each one of these which has been really fun. And students put it on Facebook unbeknownst to us. We didn't know about this for six months. And Twitter where they talk about the assignment. And I think there's a real opportunity here because my worry is that we can have a lone together phenomenon where there's thousands of people around the globe but there's no connection between them. By contrast to a studio where I know what everybody else is up to kind of ambiently. You lose that online unless you add it back in. And so as you can see here we've got our studio. Which computer science is like by the way, when I was a computer science student there was one lab that had all of the computers. And that meant that I knew what everybody else was up to. And so to close I think there's three things that we're currently exploring. One is increasing the richness of peer interaction. We've built our own peer assessment server that talks to Coursera and other platforms. And it's available for anybody to use. Additionally we're really interested in supporting teachers as recalours. Being able to take a week out of dance class and a week out of church class and a week out of my class. And make some of your own content and put a new course together composing all of that. Lots of people around the world are doing this but the infrastructure support is slim to none. And we can do a much better job here. And I think there are interesting questions. If you value the sociality in terms of pacing, do you run it once a year and people fall off the wagon? Or do you run it all the time and maybe lose some of the cohort? So these are things that we can talk about more in the panel. Thanks very much. So I'll start by saying that nobody has mentioned what I think is the single most important impact that this movement is going to be. And that is before I started doing my move, I took Scott's move. I didn't do any of the assignments. I lurked. I watched. I saw what he did. I saw Scott Page from University of Michigan's move. I liked some of what Scott Page did. Scott Clemmer had this kind of very transparent open thing that he did at the end of his class where he wrote. He gave a lecture of what worked and what didn't work. And I just looked at every single slide. And I said thank you, thank you for both the things that worked and the things that didn't work. So Scott has borrowed from me. But Scott was really an inspiration. There is just no way that I would go to Stanford from Ann Arbor and sit in the back of his class and learn from his pedagogy. We are seeing the greatest experiment of collective understanding of pedagogy the world has literally ever seen. Some of the best teachers in the world are naked on the internet with everyone able to see. We are going to see so many wonderful things that come from this. So that maybe I'll start with the change in teaching as my first great thing. I'll end with another great thing that might scare you. So I've been doing this a long time. In 1996 when the internet came out I had a television program. It was the first nationally television program on the internet. And we interviewed all these cool people like when real audio came out and could send audio over 14 form motives. There was a suggestion that this 14 form audio modem could be used to teach around the world. So I built this teaching environment. I called Synchomatic 2000 and it put a really scary picture of me in the upper left hand corner and flipped JPEG slides about every 30 to 40 seconds while streaming 9600 bought audio. Through a series of intellectual property problems I switched universities I rewrote it in 2001 on Apple platform. I had scribbling, I had video, I had audio, I had a screen cam. I had this whole thing. I even went to Apple and demonstrated at Apple and begged them to take this for free from me and put this into Apple products. But I was just ahead of my time. And I took a brief detour building the Sakai open source learning management system thinking that that would, if I could just convert everyone to be using a single learning management system then I could finally deploy my video based teaching techniques in that. And then I realized that after 7 years we had a 3% market share and it was going to take well longer than the rest of my life to take over the entire market learning management systems. Then I briefly went into doing computer standards so I could plug stuff into Blackboard and Desire to Learn and Angel and all these other things called learning tools in and around for building that for a while. And then finally Coursera comes into existence and I get to go back and actually do what I really started to try to do well over 15 years ago where I was trying to teach online using this new thing called the internet back in 1996. And I picked a topic that I had unique ability in because I had this television program and in this television program we would walk around with really crappy old little hi-aid cameras and stick them in the face of Tim Berners-Lee and Robert Caillou and all these people that were using these things back when it was tiny. And so I had all this really cool footage from a long time ago so I'm like hey I should just do a course on the history of the internet because no one else on the planet has the same stuff that I've got. And I now have this column in IEEE Computer Magazine where once a month IEEE Computer flies me around. I'm like a reporter which is really kind of cool. And I fly around and do a video interview and I'll get all these famous people like Bob Metcalfe. There's a party like next Wednesday. Dan are you going to be at that? There's a party next Wednesday at part for the 40th birthday party of Ethernet? Can't make it. You going, Scott? Just sign up. It's really going to be cool. All the cool people in the world are at the 7th park, which is just kind of up the road for you. So I picked this class which is just like a fun class and so they talk about rock star teachers. I want to be a rock star teacher. I come from a television background. I'm picking a class that I'm uniquely able to turn the whole thing into a reality show as much as I can. So I try to make my class fun. I try to make my learning my class, taking my class fun. I'm trying actually the secret that I'm trying to do is convince people that technology is actually cool particularly for people who hate technology. So this is kind of a gateway drug for trying to want to learn more about technology and I'll talk about that. So Scott said about what's up new. Is there anything different other than the hype and the numbers? That's partly true but I think there's something also interesting. So if you're a University of Michigan student for $3,500 we will rent you one of these chairs for three hours a week for 15 weeks and I stand up in the front and I'm about this tall, right? And my slides are about that big and you can't hit the replay and if you watch me on start doing Facebook you lose track of what I had to say. A lot of my students watch me on a big screen TV which means I'm three feet away from them. I'm this tall and my slides are perfectly wonderful. My slides have higher fidelity in Italy than they do in the same classroom that I'm teaching. You would see my slides better if you were in Italy right now rather than here. So there is something different. YouTube, Netflix have made it so that I can send HD video to all corners of the planet or at least most corners of the planet and so yes I started out by putting a picture of me in the upper left hand corner and yes you can learn but this is a much more engaging environment. So I come up with this sort of teaching style where my cat walks in so I had this situation where the cat walked in and I tried to make it fun and the cat sort of walked around and I showed the cat and I introduced the cat and I tried to teach the cat something about cryptography in this particular slide and try to make it really fun for it but now I coach other teachers to try to be as open and fun loving and really enjoy it to make the students really look forward to the classes. Scott mentioned numbers. These are the retention rates for my class. I taught it twice and the first time, I mean the top line is the number of people who watched the lecture in weeks run through seven, number of people took the quiz weeks run through seven, second time around, number of people watched the lecture, number of people took the quiz. The weird thing about this and we were talking earlier that all these curves look the same. I thought that I had made lots of mistakes in the first time and that's why the curve was the way it was. I didn't make those mistakes and I think I did a much better job in the second time and the curve is the same and so I think that it really is some law of physics that just says if you're not paying $3,500 and life, you know, you have to make priorities and people just go away and they don't feel good when they leave but I think that we should stop as Dan said, kicking ourselves for this retention rate. I think another thing that's important is the age that I think it's correlated to retention rate and a lot of these people already have an education, right? And so they're not hungry. They've already got a bachelor's degree, some have a master's degree, some even have a PhD. Over half of my students had a master's degree or higher and of course they are all over the world and that's fun. You get to do some fun visualizing. Now one of the things that's kind of interesting about this is I try to create a class that's very personal and very engaging but I also feel very lonely. I felt this even in 1996 that the students were really, I was funny, I would tell jokes, they got my jokes, they would make fun of how I drank coffee all the time, they still make fun of how I drank coffee all the time but I was lonely and so this is where I invented the face-to-face office hours. They think I'm like the nicest guy in the world for coming to Barcelona. So I've done this all over the world. I've not yet traveled specifically but I've been all over the world because I get a lot of invited talks and go a lot of places and I videotape them all and then I send them all to the class. So I'm turning it into a reality show basically and I don't understand why like duck hunters are on but I'm not. I mean I should be on television. And so the funny thing about this is they don't care about the class, they don't really ask me about the class, they ask me about Coursera's business model. It's one of the things that they ask me about. So I just give you a little bit of what I tell them when they ask me about Coursera's business model. They're like what's going to happen? So there's this race to the tipping point. In a networks class like Laudas class you learn about things like network goods where the more people that have them the more valuable things are. Like if everyone has an iPhone it's more valuable because you can like borrow stuff from people. Coursera is the way ahead. Coursera, this is old, they're probably closer to 3 million now going at 300,000 a month. edX says 500,000 Udemy has 500,000, Blackboard has one of these things. But think about this. It's not unlikely that Coursera will have 10 million users in 2 to 3 years. They have 25 employees. Let's just say of their 10 million users they find a way to get $10 worth of revenue for each user. 10 million users, $10 revenue. Simple multiplication. That'd be $100 million. The simple division is 25 employees. That's 4 million dollars of revenue for every employee. Merely making $10 off of every student. This is why Daphne does not need to monetize right this minute. What Daphne needs to do is to get to a 10 million people as fast as she can. If she tries to make a bunch of money off of these people and then stops the growth, that's going to ruin things for her. This is an exciting time. The other thing that's cool about Daphne's business model of Coursera is that the universities are doing all the production and she just has to keep the servers running. She doesn't have to have 300 people. She can keep it 25 people and make $100 million without too much. It's not unlikely that she could make $100 million in 5 to 6 years and still have a staff of 25 to 30. That's a pretty comfortable way to enjoy your life, right? So the numbers go up. So one of the things that's a criticism of this whole thing is that there's a bit of elitism involved in it. I'm lucky I was at one of the first 4 schools. Cod was also lucky. He was at one of the first 4 schools and it was very elite, right? Even at my school they're going like, whoa, there's 70. I'm not sure it's so cool anymore. There's 70 in there. So one of the things that I wonder about is why is it that I'm lucky to be part of this? I'm lucky to be at a school that was selected. I'm lucky to be selected at my school. So I was really excited when Stanford started this thing called Class to Go, which is an open source platform and that's going to democratize the ability for everybody to give MOOCs. So I came back very excited from my meetings at Stanford about the Class to Go, which is now kind of merged with that X. And I went back to the University of Michigan and I talked to my Vice Provost and I said to her, we should have online.umich.edu and she goes, well, that's a great idea. We should probably have a faculty committee talk about that for a while. And that's really what administrators, when they really are going to tell you no, that's really loosely translated, is you'll hear the word no in 2 years. So I know that, right? So I went to my dean. I said we should have online.si.umich.edu. And he's like, I don't know, we got a lot of things in our plate right now. So that's another way of saying no, right? And so then what I said, as I said, okay, I'm really believing that the granularity of these MOOCs shouldn't be there's like 3 worldwide things that are going to do it. And so that's when I created the University of 1, Dr. Chuck Online. Online.DrChuck.com Okay? So I'm teaching a Python class. Now I won't let me teach a second class at Coursera because so many people at Michigan want to teach their first class, that if I got to teach my second class, it would cause a riot. So I can't teach a second class, not because they wouldn't want me to, but because other people are in line before me. So I got this class I want to teach pythons. My life's work is to teach Python to all humanity, basically. So I made my own MOOC and I've got my own students. I've only had 1,000 so far and I have little ribbons and they put up their Twitters and I use an open source management system called Moodle to run the class and I got a bunch of cheap free servers for my own Dr. Chuck University. And I give away, I don't give certificates, I give these things called Mozilla Open Badges. You earn a badge that you can put on your resume and stuff like that. Like I wrote this thing in like 3 weeks over Christmas break because I needed it by January 1st and there we go. The other thing that I'm doing in this class that is not really part of the core of these MOOCs is open educational resources. So I really want to move away from me being the Rockstar teacher teaching students. I want me to be the Rockstar teacher to teach other teachers how to be Rockstar teachers. Not only do I want those other teachers to be Rockstar teachers, I want to be those other Rockstar teachers to teach in Russian. I want to create Rockstar teachers teaching in Russian. And I don't want to use the examples of baseball batting averages when they have a sport of cricket in South Africa and they really don't care about baseball. I want it to be able to change it. So this course is all creative comments, all remixable, everything, talked about on video, everything. Now here's another interesting thing. I have a business model too. So I put all my videos up on YouTube. And I was making $65 a month off of advertising while I was teaching my Python class. I averaged $5 a month on my YouTube advertising. So I went way up. So like $120 I put in my pocket for teaching a free online class. I'm going to start moving my Coursera videos to YouTube as well. I can really start pocketing some real change here. So I'm a little entrepreneur. The other thing is people took my stuff and created a class on this thing called Peer to Peer University which is like the weirdest educational form of thing. You go to the end of the course and you have to post a thing that says, here's my evidence that I finished this course and you have to wait for two other people to get there and give you a thumbs up. It's kind of the ultimate peer certification. So my stuff goes up there because it's free and open education resources. I'm going to run a course on Blackboard in this thing called Course Sites which is free as online. And I just got asked by a guy from Russia if he can start doing a Russian class in the Russian National Online University like ok people are starting to get it. Ok so I will close there hopefully but I have one last thought. I started by talking about how this is going to transform teaching is the most important thing. A lot of people if you read talk about how it's going to transform the university. I believe it's not going to transform the university at all. I do think it's going to transform students and in a way that I think you might not understand. And I think this is things that we should worry about more than the change to higher education. And that is when we start admitting people in 10 years are we going to sort by the number of course error certificates that they have. So all of a sudden just coming in with a 4.0 GPA from high school won't be enough. To get into an elite school you're going to need a stack of these things too. Now think of what mom and dad are going to behave like from sixth grade on once it becomes the truth that just going to high school and getting a 4.0 is not enough to getting into high quality university. It's going to make life's kids pretty damn miserable by the time it's all said and done. And I'll leave you with this one last thing. I had office hours in Austin, one of my students, she's so funny. She's an older lady, VBS type, right? She was in VBS's 70's and had a bunch of kids and now she's nerd now again. And at my office I was with Jeff Young of the Chronicle Higher Education and she said that what she had done in Coursera, because she tries to hide her identity, I don't know quite why, but she tries to hide her identity so she logged in, she changed her name to Batman and Robin and printed out a certificate. Then logged in and changed her name to something else and printed out a certificate. Then logged in and printed out a certificate with something else. Jeff Young's eyes, they talk about plagiarism, I just got about this big and he went Rand Andrew Ning like within 10 minutes and they fixed it like within 10 minutes. So here this lady was manufacturing certificates. The other thing she said is I think there's a significant fraction of the students in our class that are actually the parents of the students, they're taking the class in the student's name so that they can get into the really best high schools. So think about that. Okay, well I will stop there with sort of some exotic notions to get us all thinking that it's not just necessarily what the people in the press are talking about. So thank you. Get all the panelists up here and announce your chance to ask them questions. I will even turn the light on so you can see. Yeah, yeah, why not? Fire away. There was an article a few months ago on the communications of the ACM that titled, little mooks kill academia. What's your, you said so a minute ago that you don't think it will change the university's mind, but why is that? But it seems to me that people in the other free education this way. Let's get it. Is that a question for checking for all this? Okay, so for the video the question is about will the ACM has this article about will it change education? So I completely disagree with you. I think it's going to radically change education and you mentioned one of the ways. I think in fact the interesting data point here is that Harvard Business School, you've heard of them have outsourced their accounting classes to BYU. So for example, this remix culture, this malleability the fact that one would like to take a week of years, a week of Scots, a week of mind I think it's going to actually drive education to be very diversified and very spread out over the planet in interesting ways. So for example, if I'm in a community college and I have a bunch of not very great instructors, or maybe I've got better instructors at XYZ Community College, why would I take a crummy course from a guy at Harvard? So I think in fact the effect you're talking about will happen, but I think it's actually going to have the effect of requiring great teachers to step up. And rather than taking a crummy course from a professor at my institution, I'd rather take a much better class and I think one of the big changes is going to be the credentialing mechanism is going to radically shift in the next 10 years. So I think that's one big indication. For me the most exciting part of the question, what is the future of a university, is that it forces you to ask the question what is a university? And technologies in general have a disaggregating effect, where stuff that used to be clumped together can all of a sudden be pulled apart. And it's easy to think about a university as a bundle of lectures. But I think at most top universities, the in lecture time isn't the majority of the benefit of a university. At Harvard for example especially, it's a co-located social space, and accrediting institution far more than it is a lecture hall delivery mechanism. And so the benefit of being able to be in a class sitting next to Dan and Chuck and Gary will be equally true to whether or not the video of that lecture is available online. So I think at the top universities, in the short term the co-location benefits will remain the dominant factor. I think that what Daphne talks about from Coursera's perspective is their sense is that it won't necessarily be top universities or bottom universities who are most impacted, but rather it will be ones that ignore the changes that are underway. And I think what's extremely difficult about any sea change is trying to figure out how to experiment and prototype without just running around trying everything under the sun. So I think we don't know where we're going, and the hard question isn't so much where are we going, but let's try something now but not try everything now. So I think I've probably been doing educational technology teaching online longer than anybody up here, and I've just heard this too many times. And X is the end of education as we know it. I will say that there are things that are going to change. Higher education is not going to go away, and this is not going to be a theme park, this is still going to be UCI 100 years from now. So UCI is not going to go away and there's going to still be people out here. I mean Scott, what you said is co-location is still valuable even if the information. So I think the place that will see the greatest change is in master's degrees. It's really difficult to do a PhD by clicking the next button a million times on a computer right? That's just not how PhDs work. And interestingly bachelor's degrees don't work that way either. And so when I meet a freshman and I'm their advisor, and then I go to our graduation dinner with them, and their mom and dad hug me and say, I can't believe what you did to my child. Because they left my house as a complete loser and I see them at this graduation as such a strong young woman with a great job at Cisco. I didn't think they would ever get a job, right? I mean they never did anything good in their whole life. And you made them this way and I go like actually I didn't make them that way at all. And I gave them a couple of classes and they weren't even that great of a class, but you're right. They're very different at graduation than they were as a freshman. And I think it's exactly what Scott was saying. It's that there's a certain togetherness, a certain social contract. And I'll be frank, I'm a lot tougher in face-to-face classes because I got you. You paid 3500 bucks to sit in that chair and all I'm going to give you at the end is a letter. And if you don't do what I say every single day, you're not going to get the letter either. So there's a certain kind of response to pressure. And they need to take five classes at the same time. And the consequences of failure, right, that I'm not sure we can exactly replicate. It's so little about knowledge. If I watch the transition from 18 to 23 and I'm watching amazing 23-year-olds graduate and lost and confused 18-year-olds show up, it's not just the lectures and it's not just the quizzes. It's not just the assessment. But I do think master's degrees could be dramatically harmed and in particular online master's degrees might completely be destroyed. Frankly, I wouldn't mind seeing that. Master's degrees fall into one of two categories. One is sort of a richening of understanding. Just go to school a little longer and learn more. I actually think that of course there is probably better than that kind of a master's degree already. And I've talked to students in my office hours that said I dropped out of an MBA as soon as Coursera came into an existence. And I've talked to students in Barcelona that basically say I'm not going to take a master's degree in Coursera. Our master's degrees are all crap around here and this is much better than anything I can get in Barcelona as a master's degree. I'm not saying PhD, just master's, right? And so I think master's are going, there's another kind of master's where you kind of took the wrong undergraduate. You've got to kind of fix yourself as a master's, right? Those I think are important too but those require face to face and that's a life changing event which is a nonlinear life changing event that's not just knowledge that requires cohort and peers and all kinds of things that are not just a classroom there. So I think there'll be a change but it's not going to hurt these buildings, right? It's not going to hurt the buildings. There's still going to be people in classes. There's still going to be paying for tuition. It's going to be different but it's not going to the gloom and the doom and I think what we're going to see and I think we're already seeing it is we're going to see which of the administrators at which schools can think of a nice way to say this. So I'll just say the bad way. Our idiots, right? These people are going to blink and the administrators are going to try to run the show not let the faculty run the show at these schools and you're going to see these. We're seeing just a couple in the last couple of weeks where the faculty are kind of like under the thumb of some horrible administrators I think and they're going to prove what's wrong about the fact that you can't replace it. So I think that there's going to be horrible failures where people try to follow the notion that education is going to change dramatically so we're going to change dramatically and it's like no you just broke yourself. That's what's going to happen. The rhetoric about the university business was common 500 years ago when the printing press. I don't exactly understand mooks as well as obviously a lot of people here. When I looked them up on the internet, take some of these courses. There's this whole array of courses and they're given at various times and it's inconvenient for somebody that's doing things to take specific courses and you can't get the courses in an organized way so you can learn a particular subject without a big time span problem. It seems to me the issue ought to be that these things ought to be recorded and that you ought to have them available to you whenever you want to use them online like some courses that I know of from MIT where you can get courses that one of their professors has taught and you can look it up anytime and take the course at that point in time. There are several wonderful ironies or paradoxes in online education and I'll share two of them. The first one is many years ago my colleague Jeff Ullman said that sometimes he gave great lectures and sometimes not so much. His conclusion was record every one of his lectures as well as he could, prepare a whole lot, give it great if you mess up, re-record it, just make every lecture possible. He walked into class the next year and at the beginning of each class he would press play on the video recorder and then at the end of class he would press stop and field any questions that no one else had and you can guess exactly how well this went. So I think there are some ideas that if you follow them to their logical conclusion become absurd. And so another one is many of us have had our lectures online in exactly the format that you described for many years and lots of people watched them to see the number that we've seen in the last year and a half and I think one more than anything else, the major reason why this stuff is all of a sudden on the front page of the New York Times it seems almost every single day is that Sebastian Thrun and Peter Norvig had the idea of having a deadline and having it be like a class. So that the classes ran and you had a whole cohort and there was a whole community and in having a deadline and weekly assignments and all of that you felt like you got to be a Stanford student in a way that just watching the videos doesn't have. Now all of that said one of the things that we've learned over the last year and a half is that university deadlines don't work for real people. If you're not on a college campus full time, weekly deadlines and weekly assignments don't make sense and it is awfully inconvenient to have classes that only start a couple of times a year. And so what do you do with this on one hand the social momentum and cohort effects are huge and on the other hand it would be nice to be able to tivo it and have it at your own time. I think what that will evolve into is still unknown. Peter Norvig has suggested the bus stop model where you can get on one run of a course, take it for a while, when life gets in a way get off and then get on again at the next run. That's one of many potential futures. So I was really interested in the curves that you showed that looked identical. Do you have any sense of how many of those will repeat students? But you know if somebody dropped off the first round and then started back on the second at all? Do you have any sense for... I bet Coursera will start looking at that. They're actually starting looking at those exact curves and they have more data because they can look across lots of classes. So they're doing a lot of really good thinking about this at Coursera. The first thing that came out of that was the shape of the curve is different if you paid $40 for the signature track. The shape of the curve is dramatically different. Again you gave me $40 so I'll give you back a piece of paper at the end but the act of giving the $40 and the thought that you're not going to get your $40 worth of paper keeps people in surprising. So Coursera is going to play with that and understand that. And I agree with the Norvig idea of the bus stop because pure self-paced leads to nothing I think. It's even easier to stop. I'll go back to your question a little bit and then come back to your question. The one thing that bothers me about it is I have a set of students that's finishing my 101 class. They are so ready for a 102 class. Today for them to take the 102 class is like four weeks after the 101 class ends. They're hungry. They're like what next? And because it's just this giant a la carte menu that is random start times I can't say this is what's next and we're going to stick together for another 10 weeks because I got you. I got you. You're in. I got my 5000 but they're my 5000 big fans. I could teach them a lot more if I could keep a hold of them. But that's what Online Doctor Truck is. I take all my Coursera students and sneak them into Online Doctor Truck. So that'll be my first experiment of moving a cohort from Coursera out of Coursera into my own environment so I can teach them one if not two or three more classes to see. Because that's I think the biggest problem of this is it's still there's no curriculum committee but I hate the curriculum committee but you need it but it's also a problem. I want to amplify on the curve thing because we've all actually seen exactly the same curve and it's going to draw you the MOOC curve. It seems to be exactly the same thing for all MOOCs and it's roughly this diminishing you can write the formula for it. You're going to start with X 100,000 registrants and it's going to do that. Now the one interesting thing I've noticed about it is if you send out reminders, I'm sure you've seen this right, you get this little uptake. So if you send an email reminder it goes up like that and goes back down. If you send it every week we'll see an uptake like that. I saw a 3% uptake every time I sent an email reminder and send out email reminders. I call that culture. So that's one thing. Second thing, it doesn't matter this is T. You can determine your retention rate by varying the length of the class. Imagine my surprise. So my class stops here two weeks. Smart guy. Scots goes out here at 10 weeks. Guess what? So I don't know if this is a universal law but it sure looks like it. I gave this talk at Berkeley and Armando Fox. Leap stuff and says that's my curve. So we're all seeing and I think it is like universal gravitation as we were saying. Really dumb question that I have which is why do we care about completion rates? But I will ask what are the three most important things about them to you guys. So Mark Sanders at Stanford's online learning office pointed out to me that if Amazon made all Kindle books free every day and then we watched what rate of books were completed it would be low. But the total number of pages read and literature absorbed would probably go up. And so what we may want to do is take the integral and just look at the area under the curve. For individuals. Look at all their learning. Not just yeah. Yeah. I see it as a challenge and I think that the key is because it's kind of this law of physics that's going to always get us. It just means that I'm questing for ways to improve it right. And I think that the bus stop model I think we'll find ways to improve the software. I think we'll find ways to make it more social. Social interactions are one of the things that get the cohort to form. But the software kind of doesn't help it very much right. And it kind of fights it. But the students form social groups, Facebook on their own. And so I think there's a lot to learn. And so to some degree we can change it's not going to change the shape but we can change the constant. We can struggle. And I think it's worth it for us to want to struggle to improve it. To say oh I'm not done yet. I mean if my curve goes straight I mean like okay I'm done. I mean I'm just going to hit the play button at the beginning class right. And so I'm just glad that I'm struggling and have so much room for improvement. And even a tiny improvement is like a major success. And I think that's neat. I think that's actually a neat thing. But in terms of Dan kind of said it in his talk. Who cares how many started. And there's also like 10% who just watch it and don't take a quiz. They think of it as entertainment. And that's really kind of cool all by itself. So these are really great comments. So I actually have a data point for you. We did the class the first time. And this was the curve right here. And then we went through the bug list and fixed every single thing I could find. Hundreds of bug fixes. And we basically ran the second course and it was like this. Right. I can add the constant. It's exactly the same. So we have basically the same constant. Slight variation on coefficient. But it's a universal law. And you're right. I mean I had 22,000 people finish that class. I'm sorry about the other 80,000 who had life emergencies or couldn't get into it for something. But that's a fantastic name. Okay so another place where there's broadcast style of information transfers. Television. Okay. And television production is very adaptive. And they do a lot of attention to market share. They reshape content based on focus groups and what have you. And educational television of course was tried long time ago and seemed mainly to target K through 12 rather than 13 through 18 plus kind of thing. So is it likely that we're going to start to see kind of a potential emergence or convergence of the entertainment aspect of education coming in by which I mean one way how University X will compete with University Y for their MOOC is they're going to have a bigger production budget. They're going to have if you will, more attractive lecturers. Whatever attractive me. I think taller, shorter, older, younger, smarter, whatever. So is there a barrier to the emergence of entertainment and MOOCs or is that really an area of opportunity and innovation? Yes. If I get Brad Pitt to give my class with the same intelligence and energy and passion that I do he is going to be Ben Stein giving that same class. So yes, personality matters, production values matter. But by the same token in a class like the ones I do at least where we have to churn them over every so often because the world changes. It's not like Chem 101 where redox reactions are going to be the same on reducing them. But in my world, in my search class I have to change the content every six months. So I keep the production values relatively low because I need to reshoot them. So that's, you know, we actually have a set of variables here. Age group is one kind of content you're teaching is different. So if you're doing immunology 101 do not. I would not recommend you spend a lot of time in production because it's going to change. Any other comments? I think that when I started teaching my class Andrew and Daphne emphasized to me the importance of low production values. Right. And so I actually made the production values on my class crummier than would have been my natural inclination. And I think there are two important reasons for that. The first is what Dan pointed out is not only does the content change frequently, but it's not at all clear to me that we know how to run these classes right. And as Dan sees these classes as a software development pipeline, through my own lenses they're a human-centered design process. And your first prototype should be quick and dirty, not too flashy. I think the other thing is that when the production values are low there's a media psychology effect where it conveys that the content is what's important. And though I believe that fancy production is inevitable there's an advantage to being shlubby with a webcam because it conveys that you're there for the passion of the content. So I think production values are low. Low production values that seem low but aren't low is really the sweet spot of the situation. So I have a light, simple light to make sure that there's not shadows under my eye. I have a really high quality microphone I make sure that the sun isn't behind me. I do all these things to make it look effortless but not look ugly. And so at the same time I've got a home studio and I've got an office studio so that I can just chill out and say you know what I'm going to knock out a 20 minute one right now. I walk in, I'm comfortable, I'm relaxed and I'm really what the media psychology thing you were talking about is really I'm just a person. I'm not CNN. And I tell people don't try to be CNN. Don't try to read a teleprompter. Some of those intros of Coursera, the most dreadful video ever because they wrote in scripts. Professors are horrible at reading scripts, right? They're horrible at reading scripts. And people are like you were great in that. 40 times to get the intro video right. It takes me one time to do a lecture. It takes me 40 times to write the intro video because I had a teleprompter. They're like oh you're great but I'm not a CNN anchor. So I tell people like sometimes you lose your train of thought and you're supposed to lose your train of thought because actually the students losing their train of thought at the exact same moment and when we pause we pause for a purpose, right? So don't even edit out the stops and the mistakes. That doesn't mean that like you get something completely wrong. You don't want to fix it. You're supposed to become a human and you're supposed to be a peer to the student rather than sort of like a god, like CNN or Anderson Cooper. Like I'm not Anderson Cooper. I'm not trying to become, I want to be more like Kim Kardashian. Somebody they think about is like just somebody they hang out with and try to create a peer like relationship and too high production values can get that. But you do want to fix little things like shadows under your eyes and bad microphones. And some of the Corsera classes that are most dreadful look like someone pasted the poor professor against the wall, chained them somewhere and it's not going to let them go until they're finished. They look terrified. It just does not create that kind of a relationship. But it then becomes entertaining just like reality TV is entertaining because it has full bad production values but it has high production values that don't, they don't you still see what their faces look like and they don't have shadows under their eyes. It just kind of looks like it's an informal interaction and I think that's it's got to, you know, I think that's important. The last thing I would say is you talked about who, right? Are we all going to have to become Brad Pitt? And the answer is not everyone who teaches college currently is really suitable for their own MOOC. But college professors at elite institutions are actually pretty funny, pretty articulate people. There's a lot of them that are really funny and that's what we're seeing, right? Oh, like, whoa, who's this Princeton person? Oh, they're pretty funny. The way this guy runs his poetry class, that's just like, that's inspiring, right? So we're the Brad Pitt's, we are. We're the stars already. We're good at what we do and our craft has been honed in classrooms for a long time and we're just, we got to not lose what we're actually already good at and we're already crafted. So these universities went out and picked the 20 or 30 kind of articulate, reasonably coherent teachers and so we already have done a talent thing and there are people that are never going to make it on these campuses. But that doesn't mean that they're bad teachers, it just means they're not suitable. So I think we're going to find thousands and thousands of existing faculty are perfectly suited for this. We don't need more talent than we already have, but not everybody is suited. All the way in the back, I saw a question. Yeah, no, I've got a question. I know you're here to promote this, right? But I was wondering, there are obviously certain kinds of courses for which this is not very well suited and I would like to have you reflect briefly on the kinds of courses that you already know that they would not be well suited. Very quickly, I, my hunch is that the gamut is larger than one would initially think, but that different domains will require different kinds of transformations. One lesson I did learn is that I think HCI design is a particularly good topic for peer assessment and I didn't realize that originally because its knowledge proposition is fuzzier than 3 plus 4 equals 7. But sharper than what constitutes success in modern poetry. And while there have been great successes in modern poetry, it's I think more challenging the aesthetic component of a field, the harder it is to do peer assessment as just one example among men. So last week, Gary, Scott and I were in Paris and I took a cooking class and I learned how to make croissants. And I, A, I recommend this to you. And I, B, I love MOOCs. C, it will not work for croissants. You need to know how the dough drapes just so, and so he lays it over your arm and you say, oh, that's what it feels like. You didn't know you had to test croissant dough in your arm, but that's, that's the truth. And you have that, so for that kind of high touch ceramics, you know, cooking, there's a whole bunch of stuff like that I can't see how to do it in MOOCs. I was actually impressed with your class. I'm pretty worried I would have said that's not possible, but Scott showed me, I was wrong, doomed. Yeah, that was exactly what I thought. So kudos to you for that. But croissants? I'll argue with that after check. Okay, so, so I think that there are some classes that change who you are as a human being. They are such old experiences that you emerge different. Like people at University Mission who take this C++ class called EECS 280 are different human beings when they're out than when they're in. Or they've changed their major, either one. But there's something to it. They're changed. That course changes who, it's not just like, oh wow, I can regurgitate some more stuff. They're a different person. My son, when he took his remedial math course so he could get into pre-college algebra, when he came out of that course he was a new person. He was, he wasn't the same person that he walked in. And I think what we're going to find is there's lots of courses that are like enhancing knowledge, right? And, but there are other courses that are changing who you are. That are like just kicking you up a level. That just like, you weren't, you could never do this and now you can. And at that point it's really difficult to change who you are in a basement clicking the next button. Because they're just not that moment, that moment where, and you have to kind of be with somebody, a teacher, a mentor, somebody to say, you know, I can, I'm sitting here next, next, next. Oh wow, I'm a genius all of a sudden. And it's like there's so much you can learn and so much that can be done. But, but I think these life-changing classes that might be 20, 15, I don't know. And they might be in K-12 as well as higher ed. But I think it's going to be difficult to get these things where you come out different than you went in. Not just smarter, but different. The Shoshana Zuboff writes about an idea called Activity-Centered Knowledge. And one of her examples of this is in a pulp plan that when pulp fact, paper factories were more manual, you would dip your hand in the pulp and you could sense whether the chemical mix was right. And her example resonated with me because my father was an engineer at a paper plan during the years when it went from being highly manual to more computerized. And so there's clearly something to feeling the drape of a croissant. That's, and so here in online education we're seeing a replay of a discussion that we've had for many years about, for example, can or when can video conferencing and similar replace face-to-face interaction. And one of my favorite papers on this topic came out of the Olsen's lab at Michigan about a decade ago. It's called Trust Breasts Down in Electronic Context, but can be restored by initial face-to-face context. Among other things I give that to students as an example of a paper where I've never forgotten the take home point. It's the title. And I think what you may see is that the first time you make croissants you need to go there in person so you can attune to physical variables that you need somebody shoulder to shoulder with. But maybe you could check in with somebody later online and so I think we're going to see more and more hybrid media approaches. I don't think you guys have ever watched Julia Child. She taught great cooking. So this for me is why. Exactly the same thing. And so this is why for me I still think that there is no such thing as one rock star teacher teaching, one rock star class and everyone is the student. This is why I am trying to excite other teachers because the things that change lives are a nearby teacher with no language barrier. Also just go down to fourth grade. Even the flowers in a fourth grade book have got to be local. You can't take a United States fourth grade book to the language because the flower, they're like what's that? They've never seen that. So the more you get to the... That doesn't mean that we can't create content that can influence and improve teaching around the world, but the lower it goes into the cognitive steps that have to happen, the more it's going to have to be localized and the more you're going to have to have a guy or a mentor or somebody close by. Yes. Did you expand more on how you're flushing out the certification process right now because in Coursera you get a certificate. You talked about handing out open badges, but then is there anything like say I take a course on web development or something and at the end it just says go on and make your own website and submit it to our community or something like that? What out there besides the badge and the certificate? Well, so for example in my class your final project is to make a website that you submit to a peer community and they'll assess it. There are many different strategies of certification. When I taught my course, my department chair, Jennifer Wittem who taught the database class recommended, don't worry about cheating or anything else like that. Focus on the students you want to learn. And I think there's a lot of merit in that approach. As Chuck mentioned Coursera has recently rolled out a we'll watch your typing pattern and webcam and certify that it was you that submitted the assignment. But my experience has been when teachers and students get too bent out of shape about was it really this person that distracts the focus from where I want to be. So that's all true and one of the things I think is good as I said earlier will change most radically is the notion of certification and how this happens because in my class one of the big surprises for me was that people really, really wanted the certificate and it's a PDF file. You know? You could fake it trivially. Really it's nothing more than that. And so I agree with what you're saying. You should focus on the people who want to learn because truthfully people have been scamming SAT exams since the dawn of SAT and this is not a big revelation. So I think from my perspective at Google hiring people what's going to happen in ten years is it's going to be less about which pieces of potential artificial or true you have but whether or not you can actually pass an entrance exam. And I think that fundamentally changes one of the equations and expectations about university. The university is a resource of scarcity. We can only admit X thousand people to this campus. But in my class and I can accept oh I tested up to 500,000 so there's no reason I couldn't do 500,000 students next week. So then what does certification mean at this point? There's no reason for them to not go all the way through to mastery. I don't care if they keep retaking the test. That's perfectly okay. I want them to be masterful. I want them to have changed lives. And it's not my job to fail them. It's my job to give them to be skilled. So I think that changes, right? One of the fundamental tenets of grading of the way university classes are run. Suppose everybody could graduate with an A. 100%. It's just the amount of time they take to get there. That's an interesting change. I completely agree. And I found it just like Scott. I found it personally freeing to be kind of taken away from this like must measure to three point decimal points of accuracy. Some number that's a proxy for learning, right? Just it's so wonderful. I have stickers, right? We even have one. So I give my students stickers, right? I try to take this thing and make it physical because we humans still like to pile things up, right? And I think what we're going to find in the hiring situation that when people come in and start using certificates as evidence of suitability for employment they're not going to bring in one or two. They're going to bring in a bunch and you're going to look at like, wow this person spent like a year of their life and they took two classes of time and they did pretty well. And it's kind of like a bachelor's degree. It's not so much just a giant file cabinet full of facts. It's the fact that you were under pressure for four years and you survived. And I think we'll see, we'll look at sets of certificates in the same kind of way. Do you have enough stick? Do you stick to something? Do you finish it? Do you start with it? So if you just like steal two certificates it's kind of irrelevant. But if you have a set and then a few skills that you can show that really that validates that I think we'll find that people will start seeing certificates sets over a reasonably long period of time as a proxy for a bachelor's kind of education over time. And it's going to be the aggregate of them. Much like undergraduate degrees, an aggregate of 120 credits but really it's four years of dedication. Good time for one more question. This is sort of a comment though. I read an article short time ago which made the point that one of the motivating factors why parents from the elite university want to have their children go back to that elite university is because they want their children to find their spouse at the same age. Now this is I can answer this one. Hopefully the tape is run out at this point. So interestingly at the University of Michigan I said I don't like it when people say that the online is going to be the end of whatever. It's an apocryphal story that one of our college presidents at University of Michigan was alleged to have said that in 1996 or something in a few years we will be issuing 100,000 undergraduate degrees and you will be able to get a University of Michigan degree in Camel and Arabia. You probably know more about this story than I do but that's the legend that I've heard. And that person was summarily fired as the president or resigned before the president got fired. And the reason that was given is that that's not what a University of Michigan education is. Now no one said what it was but it really does have to do with the football stadium as do with sacks, as do with alcohol. That's the last once in a while. And so I think that Coursera made it kind of the first thing that made it legitimate to think about online education at Michigan again. It was kind of like we weren't going to think about that for a really, really long time. So I'm excited to see Coursera come along and make it okay to think about teaching online. But I think there's a lot to that and that has to do with being 18 to 23. I once gave a talk somewhere and I asked the people in the audience that people have gotten online masters and how many people got online bachelors and how many... And only one person, has anyone gotten online bachelors in this room? Okay, so it's very rare to get online bachelors. And so I gave a talk and this very young woman in the back popped up her hand. She goes, I've never seen anyone say I got an online bachelors degree because it's my hypothesis that bachelors is the hard thing to do. And so I said you're weird. Like I've never had anybody say I got an online bachelors and she came up to me afterwards and I wanted to explain about the online bachelors and she said I was in the military and I'm like, of course you learn how to drink, you learn how to hang out with people, you learn how to work with teams and yeah you filled your head with some education online because she had matured the military is what matured her, right? I mean she didn't need to go to college to mature the military matured her and then she went and filled her head with education and now she's got a degree and she's successful etc. So I agree with you that some of it is just about getting from 18 to 23 in a way that you can't drive home and drop your laundry off on the weekend. I'd like to thank Dan and Scott and Chuck for the very interesting set of conversations.