 It said that I operate under the well-known Nessan Uncertainty Principle, which is that I may know what time I'm supposed to be some place, but that probably means I don't know where. So Jeff came into my life as Peter Pan, I think of him that way. He dropped in out of nowhere. And in a form that had a quality that is very Peter Pan like. He informed my internet and society class. He led us into the world of MOOCs. And then he's just lightened space in every other environment I've seen him in. I had the pleasure of listening to him talk to his colleagues at the Neiman Foundation. And it was simply enchanted. So please welcome him here. He is a wonderful man. Wow, thank you very much. Thanks, Charlie. Thanks, everybody, for being here. I want to jump right in if my clicker will work. So I wanted to start talking about somebody who is, to many, or at one point, a poster child for MOOCs. The boy genius of Ulaanbaatar in Mongolia. And Bhattashig is, the story is so good, because here we have the example of a 15-year-old at taking one of the very first MOOCs that was offered by MIT in circuits and electronics, a sophomore-level MIT course. And about 120,000-plus people signed up for it. As we know, that number doesn't mean the number of people that participated. But I think around 10,000 people actually finished the course. And Bhattashig was one of 300 people to get a perfect score. So he really kind of blew it out of the water. And it was this story that ran in the New York Times documenting, in the fall, documenting how this is sort of, this is like the MOOC vision, right? If you could have somebody so far away from MIT benefiting and mastering this complicated material, thanks to the internet. So this went in with this early vision of MOOCs. As Daphne Kuller, the founder of Coursera, said, maybe this could be sort of considered a basic human right, access to higher education. And so free education for all was the opening mantra. And actually, I'm very curious, especially in this room, how many people here have taken a MOOC or even part of one? I'm not going to, yeah, great. And how many actually finished? I'm always curious about this. As somebody who's not finished a lot of them, very cool. So I'm not going to go into much definition. But I want to just say, so you know that they basically have videos, and they have homework, and they have big, big classes. But to me, I think, so why is this any different than online education that's been around forever, right? Since, well, since forever of the online world. And even before that distance education correspondence courses, I get a lot of people that have asked, what's different? Why is this so special? And I think part of the answer is when you type in a search query in Google, you get, as you know, you get, and if you use being same thing, you get things filling in for you. So I think there are actually five kind of amazing things about this moment that we all take for granted. But five of the amazing things. One is that it knows English. Google knows English in other languages. Two is that it predicts pretty well what I was about to type, which is, when you think about it, kind of weird and creepy in some respects. But it's interesting and powerful. It also, the third thing is that it knows all of the possible things on the internet I could be looking for. Because Google knows everything that it knows anyway about cyberspace, which is darn near everything. And then it also knows what other, everyone else that's ever used Google, has actually found worth clicking on when they did a search like this. And it also knows remarkably things about me personally. Like, I'm logged into Google, apparently, because it says public speaking classes in Boston might interest me, which is, of course, I'm living in Cambridge, so that would be a logical guess that Google's made. And so this is actually an astonishing thing when you think about how many things are being pulled together here. So it's no surprise that artificial intelligence researchers are the ones that started MOOCs. And specifically, two of these people worked for Google, or still do in some respects, at least as consultants, I think. So Coursera, Udacity, both came out of Stanford and people working also at Google. edX here is a non-Agarwal came out of the computer science laboratory. And so you have people whose intuition was that maybe the Google search box, why not do that for learning? So when you're on a course, for instance, Andrew Ng, who we just saw, the Coursera founder, if you're taking his artificial intelligence class, his intuition was, why not have it as you're typing your answer and clicking Send? Why doesn't it just come back and say, I see you've got the wrong answer in an interesting way? Why don't I show you how to get to the right answer? And so literally in his first run of this course, he said he had a moment where, and he actually blames his own bad teaching on this, where thousands of people were getting the wrong answer in exactly the same way. And they weren't cheating because they were from all over the world. So their likelihood of cheating was low. But so he built a custom error message that said to the 2001st person that gave that wrong answer, hey, why don't you look at this part of your work? You might find that you did it wrong in a certain way. And so it's just natural that the MOOC premise is the premise that the AI web, or whatever you want to call it, that when we have this ability to use the wisdom of the crowd in this big data way, and that we know targeted information about the individual in front of us, that maybe education is maybe a killer app that could be the next big thing. OK, so I'm going to go back to our student in Mongolia, Bhattasek, who loves MIT, in fact, is now an undergraduate, is now a freshman in MIT down the road. So I recently got to talk with him, partly because of his perfect score in that electronics course, he wound up getting into MIT himself, and is now here in the flesh. And so one of the things, though, I wanted to go over surprising things about my talk with him that kind of complicates this narrative of the MOOC as we started off today. So one is that that MOOC that he was taking in his town in Mongolia was actually very much supplemented with human touch. In fact, his principal of his high school is an MIT graduate, brought in a recent Stanford graduate to coach every day this afterschool activity where they led labs in person every day to supplement the MOOCs. So he had the benefit of what people here at Harvard and MIT know is really good person one-on-one education. Two, he didn't do any of the reading. And if you meet, I really enjoyed meeting Bhattasek. And if you meet him, it's both charming because he's clearly brilliant. And also, he actually says that part of it is because he's busy and didn't bother. But part of it is because he was in English, and his English wasn't that great. So there's also this other thing about the form that wasn't working for him. And then the other thing was to save time, this thing. I had to have him show me because I was the slow one. He's like, oh yeah, I watched two videos at once. I don't know how to do that. So I had this on a slide because I was so obsessed with his obsession that I was like, it's too much. But he would start the next video and have that running as a preview for himself while he's watching the current video. I don't think I could do it. But he says, so he aced it. It was to argue with perfection. So he did this other thing, though, which is not in the New York Times article and I think is the most interesting is that he made his own videos. Just on his own initiative, he basically made his own teaching videos for his classmates. There were 22 of them. So he translated them into their native language by, he just went through and he propped up, he showed me how to do this, too. He propped up his iPhone on his bookshelf and then put another book on it. And then he used the camera to film the sheet of paper and he just used his pen to write on it while he translated it, the exercises that were the core of the MOOC exercises. So he learned a lot and got a perfect score by being a teacher as well. Okay, so MOOCs are not a silver bullet. I don't think anyone in this room probably thought they were, maybe, but they're not gonna single-handedly teach the world and people will be relieved to know that they're not gonna replace Harvard either. The residential college experience is so much more than classes and that goes without saying. So the other thing that's been really surprising is that MOOCs are important. I don't think we should just now say that, well, because that first vision didn't pan out exactly as advertised, there's nothing to see here. As a journalist, for one thing, I love the fact that you can follow the money and there is hundreds of millions of dollars in investment in this space. There are also so many colleges putting time in it, but there are millions of students who are at least signing up and hundreds of thousands of students taking significant amount of learning, of courseware, of doing these things and most of them are already having, they already have a college degree and many of them have a master's degree or a doctoral degree or unknown, but you just feel like, or 30 MOOCs already. So they are very educated people. They're not somebody that's never been served by education. So I didn't make this, I really wish I had, but so that narrative is maybe not there, but because MOOCs demand big narratives, I wanna propose three others today that could be a future interesting narrative. One of them is something that I've kind of seen in chatting with people that have taken MOOCs and also watching, I obsessively read the forums of MOOCs. I sign up, I'm probably part of the numbers skew because I sign up for every MOOC and then just read some of the forum postings and then don't do anything else, but it's research. So I look and many people have basically said that I'm gonna get into this discipline in some sort of tangential way, my research as a PhD, the postdoc or as a researcher or an academic and I don't know enough about whatever field it is. So I'm gonna jump forward to here. So like, it's kind of an interdisciplinary hacking. I'm gonna go to a conference on immunology. I don't know anything about it but I might work with another researcher in that area or genetics. So this person who posted on a Coursera genetics course, it's kind of like, I'm gonna learn this way and they're noting, which I've noted as well, so many people on the list are basically not who you would think would be signing up for this course. They don't wanna become geneticists or necessarily but they're gonna work in this area. And so you see that, actually you see that over and over again, the courses that are out there are very much intro courses, which it makes sense in a lot of ways that if you're gonna have, this format probably works better for introductory courses where there's a lot of kind of, you have to get your head just in the space first and you have to learn a lot of rote things before you can then go on and do more advanced things. But they're also like, if you think about it, it's almost like teaching you how to think like an X. And that's what a lot of them feel like when I've taken, I have taken a couple of them. And so you feel like you're learning a lot of vocabulary and jargon and how to decode it and what people in this field, how they see the world. And so in a way like, in the sort of tech way, I think of it almost like, it's almost like a hipstomatic lens that you can drop onto your life of like, let's see how an immunologist thinks. And so I think that I'll just be very quick on this one, but so what is a MOOC compared to things that have become before, right? Like why is this not just a textbook or whatever, or a TV show? And I do think there's a difference. And Lou Bloomfield, who is a longtime physics professor at the University of Virginia and teaches a very popular undergrad course there called How Things Work, has, is somebody who should know because he's taught his MOOC now on Coursera. He has written a book, a textbook on How Things Work, and he was a co-host briefly on a Discovery Channel show, some assembly required, which involves some physics. So he was telling me that basically he felt like TV is great at reaches a bigger audience, but the producers he dealt with were allergic to actual science in his view. And he was always fighting to get science in, he says, and it never got in in the end. Whereas in his textbook, he knows that he's reaching a limited audience just by nature of being not a popular thing flying off the shelves. So the MOOC is his hope of having the directors cut on a piece of maybe popular media or maybe somewhere in between, this new kind of space. So, so that's the kind of disciplinary part of it. And so I think that there's one thing that's kind of the big idea here that I don't know for sure. I'd love to hear you guys' comments too, but I'm toying with is that Burton Bledstein wrote this book in the 70s called The Culture of Professionalism, and he tracked kind of the rise of this kind of professionalization of very much inside the academy with the sort of silo and sort of highly specialization of every field to things, even things like professional sports and mortuary science, which was before that, you know, the funeral home. All these places, all these parts of life trying to make themselves professions and really being proud of this. And now I talked to him recently for an article and he's now thinking about writing something about how this is declining. There's this move away from that. And I think when you see this list of courses out there for free, I think you can't help but kind of think there's some sort of giving, these are not secret societies you could never ever hope to sign into, but that especially when you think about people who are not undergraduates being introduced to a field, but simply people out in the world who already have a degree taking these, that it kind of demystifies these fields potentially. So the second big narrative I wanted to sort of propose is casual learning, which this is the iPhone, I'm sorry, the Android app that was released this week by Coursera. And this is, looks like it could be kind of like Angry Birds or something, but this is the physics courses they teach and other courses they teach. This is the app for how you get Coursera courses on your Android phone. And I think that you can kind of situate a lot of the MOOC energy or at least the attempt by these companies to situate it almost in a quantified self-context of we have our Fitbit or people have all these different things that they're trying to measure themselves and improve themselves through technology. And Anand Agarwal, the head of edX has noted how the check mark that you get after every little thing you do within edX, he's been surprised by people's kind of excitement and racking these up and tallying them up and every time they do a homework or watch a video you get a new check mark and you can go back to your page and see how many check marks you have. And so there's this kind of, there's kind of this numbers game in there. And also, but I think maybe more interestingly, I think that MOOCs are, when I see the people taking MOOCs, it's kind of people that might've always taken continuing education classes, but if you think about continuing education, I kind of feel like that's not a cool area that people are not like going, oh, continue yet, I want to hear more about that topic. But why not? Like there's, I think there are a couple reasons. One is that for whatever reason they just don't have a lot of cachet. Another, like things like extension courses or things at the community center or at a museum. Or, but also they've been actually kind of not only not that great of cachet, but they actually haven't, they've actually been sort of hard to do anyway even though they're very low cost. You still have to sign up at the right time and find the right class and go to a space and all these things. And I think the interesting thing about MOOCs is that it's sort of giving you an option that doesn't require any of that and it's trying to situate it into your life differently. So obviously if it's casual learning then MOOCs colleges may not even be the most interesting players in the end or they may be starting and building an audience for something else. And already edX has partnered with all of these institutions including the World Economic Forum at Davos now has the Forum Academy that it's using the MOOC software that edX developed to deliver its own executive education courses. IMF, X, Smithsonian X, the Linux Foundation is gonna teach a course on Linux on Coursera, I mean sorry on edX. So the other thing though that I think is even more interesting is that MOOC.org, as anyone heard of this, this is the thing that edX has announced but they haven't started it yet. It's very vaporware right now. It says it's coming any minute. If we saw the whole website it's coming in early January, early 2014 so any day now maybe. It's in conjunction with Google again and what they're doing is they're gonna let anyone in the world teach a course, whoever you are you can teach a course using the edX platform. And so everyone is kind of saying oh that's a great thing but I kind of think that's a really interesting thing when you open up this AI web teaching tool to everybody. And what it could mean to professors of course is very interesting. This is one example that Udemy is a company that they don't have as sophisticated a system as edX but they do let you teach a course, they let anybody teach a course without any kind of check or balance but you can charge for it or make it free. And this one woman, Bess Ho, she made over $200,000 and they put out a press release of all the people that made a lot of money teaching these MOOCs. But when I did call her because I was curious and I was surprised by that she kind of described this first year of teaching MOOCs as kind of a fluke and difficult to repeat because she was one of the first and they were not kind of continuing both, they didn't feature her as well in the future and now there's like thousands of these courses, many on iPhone apps and her course on iPhone apps now is like for iPhone 4 and she had to totally retool it to make it to the new iPhone. So there's this high barrier to like building these courses and even the lamest MOOC on edX cost $50,000 to build and the bigger ones take a lot more money. Is it even more money than that? No. No, I just was wondering what the lamest MOOC went in there. We can debate. So I kind of feel like the early MOOCs, one of the things I was thinking was that they were like a food truck disruption to the restaurant that is Harvard or whatever because the barrier to entry to start a food truck is a lot less than starting a brick and mortar university but it's almost more like a lemonade stand scholar when MOOC.org comes into the picture when everybody can teach a course. And of course the thing that everyone is probably thinking is well do you have a lemonade stand credential? What does that matter? But I think that you can, well for one thing my physics course that I took with Lou Bloomfield is on my LinkedIn page and if you pay you can get the verified certificate but honestly like for free you can get the non-verified and still put it on your LinkedIn page and if you think of the way LinkedIn is already kind of hyping these kind of faux credentials of like your friends saying that my friends have said that I have good editing and storytelling abilities, I'm very proud. And I'm sure people that have used LinkedIn you probably have more stellar reputation than I do. So there is a way that lightweight credentials especially when it's kind of just this casual learning might be enough and might even become but at the same time you're also gonna, you might even have to compete, you might even have to start to in a tough economy do more and more of this stuff just to kind of stay in the game of this resume building. Okay, so third is the pressure on traditional colleges to change their teaching, to improve their teaching. And I really think this is a space where obviously the Amazon look inside the book, now you're really gonna peek into the universities with these courses. And I'm seeing on the forums a lot of parents of students taking these courses saying I wanna talk to my daughter who's taking this genetics course and I wanna be able to converse with her about it so I'm taking it. But there are also plenty of high school mothers and fathers deciding whether they wanna go to this school or that or pay the paycheck for that and students do the same thing. There's a lot of shopping. And then there's also a research element to and I think some people in this room are evidence of this. They're in the MOOC space, there's a lot of attention to research on how to do teaching better. And that's I think to be lauded and every time we run into people like Anant, they, you know, this was, you know, he was saying, oh, have you seen the Crake and Lockhart study on how to teach better? And I'm like, no, that's really interesting. I feel like a lot of times I talk to these MOOC pioneers and they're citing research. And there's this urge to sort of take research and inform teaching in the college setting that I hadn't seen before. And so that's interesting. Of course, you could also argue that teaching is so unique and specialized and should be just left to everyone to do it on their own. But Eric Schmidt, the CEO of Google, recently spoke in town at Tufts and I went and just asked a question from the audience about MOOCs. And I was, you know, I was struck by, he was trying very hard not to make any news and so he didn't say anything that interesting. But he did say that, you know, why is Google involved with MOOC at MOOC.org? Well, you know, we want to democratize access and we want to just, you know, let the best teachers emerge and let the marketplace figure it out. But I don't think that's always been the case, obviously, for the way teaching is gone. And so I think it's very interesting and I think to me, and maybe this isn't surprised anybody but me, but like I think that in the end, the selective colleges are gonna have to spend more on teaching as a result of MOOCs instead of saving any money. Because all of this consumer and market force is just gonna make people ask questions about are you doing all these things, either making MOOCs or just improving your teaching, which you can now get a better sense of what's really going on in the classroom. So, okay, what now? I think a couple of things that I think are really interesting to watch is in the future of this space. For one thing, I think there's kind of a full on kind of platform we're going on between Coursera and edX and Udacity about, and I think, you know, as a journalist watching this space, I'm very much looking to what their business practices are gonna be, what their privacy policies are gonna be going forward, whether they're gonna protect their users because they are these giant stores of big data. And so, you're operating in a space, and there's that great book by Jeremy Lanier, Who Owns the Future, where he talks about we're in an age of like siren servers, as he calls them, where there is only one Facebook, you know, and there's hard to compete with that because they have everybody. And there's Google, and there's not a lot of competition to these kinds of big players that end up winning. And so, who's gonna win and how do they win? And then, I think you will see a drifting off of the, MOOCs are right now very much the way college courses are, but I think you're gonna see them kind of break away from different things that have shaped the way college courses are. So I think one thing I was talking to the non-Agarwal edX, and he was saying that they've gotten a lot of complaints that they don't offer many courses in the summer because the professors aren't working in the summer, but guess what? The students who are taking the MOOCs turn out to be people in school and they're busy, and they don't have time to take MOOCs during the school year, so they want them during the off season. And then, length of the courses, there's demand to make them shorter, maybe pieces of them. And then, I do think they apparently like only partially ingest, promoted that they get, talked about getting Matt Damon for a course. And I think that, I actually think something like that is bound to happen if it hasn't already in, I think there are even some celebrity guests on various MOOCs out there, you can find them. So everything will go to pieces, but I actually mean the teaching, we're finding out that teaching is actually a lot of different activities all bundled into one. And what you're seeing is a startup to do every piece of teaching online. And so you have startups that help you find the right MOOC, you have startups that help you discuss with other students because you don't like edX's forums, or startups that do, or Google making something called Help Outs, which is announced in November, where you can get a one-on-one video chat with an expert, and a lot of them are in the education space, so get some tutoring. So if you're taking a MOOC and you're stuck, this is a way you can either for free or pay 50 cents a minute to, or different, you can set your own price. Again, it's this model of, and you get to keep 70% of the money if you're the teacher. Anybody can go on the Help Outs and try to help out. So it's this kind of, that's one little aspect that's getting pieced out and developed. And so, I talked to one of these Help Outs providers, I'm kind of obsessed with this subculture of people doing this right now. And he gave me a guitar lesson, which is great. This is Matt Gibson. And he, but it's interesting because he has like, literally a million followers on Google+, and he is a recently retired Air Force, I don't know if it's an officer or a gentleman, but he was in the Air Force recently retired, and he's trying to make it as a musician. And he wants to be a professional musician. It's not right now paying the bills. So in his basement, he offers these free guitar lessons or some of them are paying, he's trying to break into the pay. He's first trying to build his five star ratings. So his big request to me was, he'll do an interview with me, but if I give him a five star rating. So journalists in the room, I'm sure Hassid is like, did you give him one? That's an ethical, you know, violation. I did give him a five star rating. He did give me a great guitar lessons in addition to an interview. So I think this is an interesting space, but he's having trouble. Like this is a tough, just like Bass Ho in the Udemy space. It's kind of tough to make it in this world of let the marketplace decide. So I will stop and I've been encouraged to keep it short and I wanna hear everyone's questions is that, so Baddashig really is the model MOOC student, but not in this, it can teach the world automatically, but in that he is somebody that has saw, the reason he ended up taking that course, well, for one thing his principal was like begging everyone to take this course. And I'm sure he was a good student, but he said the reason that hooked him was that he watched the one minute intro video and it said if you take this circuits and electronics course, you will know, understand how the iPhone works. And so that appealed in, you want to know that. So he did as much or as little as it that he cared about to know that. And then he kind of got hooked on like making this videos and everything and got perfect score. But that was his drive and motivation. And the fact that he is this kind of person, it turns out he also in his neighborhood, a lot of people live in big apartment buildings and they have garages that they play in the kind of driveway to the garage. So he, with his knowledge from his course, invented the, I forget what it's called, the siren alarm, I think you just called it. He has a YouTube video, you can watch it, have him explain it. But he invented the siren alarm that will, it's just a sensor on the end of the driveway or when the car is coming it gives an alarm so the kids playing in the driveway can disperse and not get hurt. So this is the kind of student that is the one who's excelling in these early MOOCs. And that's the kind of early adopter for what it's worth. Now where this all goes, like I said, I think there are interesting implications for the future of the professions and what expertise means and whether a marketplace of teaching is a good idea for teaching and about whether these colleges are gonna continue to spend all this money developing them if it's not part of their core mission. Or maybe it's just they do it out of a different competitive lens than they started out. So I'm gonna leave it at that and open it up to questions and thank you guys for listening so much. Do we have the people who need a microphone? So if you could just remember to let yourself or let us know who you are before you speak, that'd be great. And if you just wanna point to folks. Awesome, well, right where you are, why not? Oh, whoever's got the mic. I've got the microphone, does it work? It's on, can you hear me if I speak? Okay, I'm Megan Richards, I'm from the European Commission and I'm here as a fellow at the Weatherhead Center this year at Harvard. So thanks, that's very interesting. One thing that I would have thought there would be a lot of potential for in the future in particular is that industry is always complaining that even though they have electrical engineers and computer scientists, et cetera, they don't have the skills and abilities that they really need to hire them. Now, whether that's a price issue or really a skills issue, that's another question. Actually, I'm really glad you said that because I blew it and didn't mention actually a project that Charlie has been very involved with in the law context. It's like professional development. Is certainly in fields or do it differently field by field, but there's a lot of demand for good professional development. And so, I mean, there is a lot of attention right now in experimentation with using MOOCs in that setting. And I actually think some of those, the slide I showed with like the IMF and the World Bank, that is really the space they're trying to, I think, enter as well. It's not necessarily somebody wanting a degree but you need to kind of, maybe you have a requirement in your field that you have to do something to update your skills. Right, but you didn't show, you didn't show industry per se because the World Bank and IMF is not industry, of course, neither is the World Economic Forum. So I'm thinking that in, did you see any signs from industry per se? Well, I mean, this is a real area where they could potentially develop the skills and requirements and they have a huge global audience potential. I think you're right. And I think the interesting thing is, so there are examples and of course, the ones that everyone knows in this room by already are the ones in computer science. And in a way I think this comes out of computer science partly because of the success of things like Code Academy and these things that were sort of like MOOCs before MOOCs even took off and so Google, for instance, has offered two MOOCs themselves on how to do search I think and one was, I forget the big data one I think they just ran, maybe somebody in this room is in it. So they've done ones and I think there is a hunger for, I guess the question though is how much industry will adopt this or call this or whether they will continue to do the kind of executive, sort of send people to universities as they have more traditionally. And I do think that the business school, Harvard Business School now announced that they're gonna dip into online education, not for free but to start offering that. So I think maybe that is a sign of potentially the start of more interest in industry. Pass it. You mentioned the business school and offering MOOCs that aren't free because you talk a bit more about that and how prevalent do you think that's gonna become? So I do think that there is this space of executive education that I think it's interesting to hear the business school just announced that this week and I don't know as much as I'd like to about it but I think they're still sort of developing it. I think their early strategy is to dip a toe in the water but the question is there's a big money actually in executive education online already. Other schools have been doing this and so the question is whether free or cheaper or whatever Harvard, sorry HPX, Harvard Business School X does, whether that will actually bring prices down and or give a lot free or some actual free options to people in a space. So the question, it really comes back to what employers think of those LinkedIn credentials so to speak. It's like, will employers buy that? And I think that the answer will differ by field just like Justin made a great point in his presentation about how there's not one narrative for all MOOCs but I think that there is a lot of interest in various fields and I think I definitely have talked to people that have said that they would see why go to business school when you can do some MOOCs and enter a business plan competition without it. That's the kind of dream but the question of whether that catches on I think it's a cultural question. What I meant was the idea of paying for them. I mean, you can see a tiered system developing where the more respectable, the more the ones with the better credentials are the ones you pay a lot of money for and the free ones get left behind. Sure. Well in the business context, it's harder to find the free stuff to be honest because they've been more savvy at charging something. So it's just a lot less than they've charged in the flesh but yes, I think that in a way isn't that replicating what we're seeing in the higher ed space before that. So then you have a tiered system where the perceived highest quality is the most expensive. So it is possible that we simply replicate the world that we have in this new space. Right here. Hi, I'm Michael Rand from Columbia University and my question is part of this also as you were alluding to at the end there, there's something where you don't necessarily have to make money but you do it to be first to market, you do it to create, to get your name out there. So as you said, before there were MOOCs, MIT had, I think it's open university or whatever, and open courseware and maybe I think the reason edX got created is because Harvard got scared and said, well, MIT and others are walking away from this so let's team up with MIT that was sort of been at this game for a while and work together. So the question being, how much? It's an accurate sensibility, some of the schools are gonna go into it sort of as a lost leader. They're gonna do it even if they don't make money and even if they don't have a large number of signups just because it gets their name out there, it's just part of marketing. No, and I think there have been some smart people that have made a similar point. I think to me I wonder if it's almost the way Ted works. You know, everyone knows the Ted videos or the lectures and Ted charges, it's like one of the most expensive conferences that you can go to and I think it's up to like over $8,000 to go to Ted in person. And yet, everyone knows them for their free and they're sort of beloved for the free stuff. And so maybe Harvard needs the free stuff as it gets really expensive continually. So maybe there is a marketing idea there. Hi, Lucas Imoni from MetaLab. You mentioned that MOOCs generally have quite high dropout rates. So in your research, could you find any best practice or strategy to counteract this dropout? Sure, I think that I'm probably not the best expert on that because I know that a lot of people are trying, the MOOCs, the MOOC providers are trying different things and the question of whether they work remains to be seen. But certainly you do see new, a lot of surveying of students, if you take a MOOC now compared to a few months ago even, you get, it asks you, what are your intentions? It's so funny, it's like, what are your intentions for coming to visit my daughter? It's like, are you really gonna stick with this course or are you just here to browse? And they wanna know that off the gate and you're kinda like, okay, all right, I was just clicking on register. So there is this sense of trying to measure which kind of person sticks and how do you do different things for them? And I think, I actually recommend, I think Justin is in this room, is a good source of what they're trying. So some people over on the other side of the room. Hello Laura. Hi, thanks Jeff. Laura Miko, former Neiman Berkman fellow from last year. I was going to ask about student retention because I think that's fascinating. My second question though was about learning styles and whether you find that MOOCs are able to cater to different types of learning styles or whether students are self-selecting based on learning in a particular way. That's really interesting. I did this article for the Chronicle last year that was really fun where I talked to, even when these things were only like six months old or a year old, there were already people who had taken 30 MOOCs and finished them. So I asked Coursera to connect me to the most MOOC-y people that they could find. So the most sort of obsessive MOOC students. And so I talked to them and one of the things that a lot of them didn't like the video format at all. They were basically like, just give me a transcript, please. I just want to just read. And so now there are in fact transcripts at most of the providers, but they weren't then. And so clearly they heard that, not probably from my article, but they heard that from students. And then the other thing was funny that they all said we don't think people are going to continue to offer these for free. So we're going to take them before they go away. So that was the biggest reason to take 30 and binge on MOOCs. Surely they are not going to continue to do this. So I'm going to get in as much education as I can. Over, let's see. On this side of the room again, I think it's easier for you to pick by one. Catalina Lacerna from HarvardX. You mentioned a piece that has to do with the MOOCs towards the teaching and learning at the universities themselves. And I just wanted to point out that, for example, for HarvardX, the second objective is not only open access, but the transformation of the residential experience. And I think the experience of professors who have MOOC-type material that then the students use to prepare for class and then come to class to discuss and have more of an interactive experience is a really fundamental piece of HarvardX. And I think a lot of the blended classroom strategies is something that the Box Center and HarvardX is looking at and is a fundamental piece of what HarvardX means for the Harvard University. And I think MIT and some others have said the same thing. Can I ask you a question back? So I mean, does that mean, I mean, I take it to mean that that means that Harvard is spending more money and making that happen, right? That's not a, just because the MOOCs are being created on the other side, it'll automatically happen that the teaching gets better, right? That's another effort, which is kind of funded. Yes, I think that very often in the proposal, when people do a proposal, there is always a question of how will this improve the college experience? In other words, HarvardX courses. So why do you think it's only if it's gonna come back to the top? Yes, because professors are used to teaching in a classroom and through the MOOCs, they can go and do videos in different environments in different settings and students can't go out there all the time. So there is something to be said about the kind of material that now professors can use in their own teaching. Case method, case studies, all kinds of things. So it is part of the proposal process actually, considering that. Thanks, thank you. Maybe on this side, no. Okay, it sounds like the barriers to entry to this are dropping and so it's not just, you know, Harvard, Stanford, MIT, that level of teacher. So once it becomes easy for anybody to teach a MOOC, how are people gonna sort out what is quality and what is junk? Five star ratings? I actually think that's a great question and I think that right now we're starting to see in those discussion forums, I've started to see if you're taking, there are a lot of the courses that, it's almost like the way Hollywood makes the same movie but like three studios make it all together. It's like there are a lot of courses on intro physics. That's one that there's like five different MOOCs on and other ones you might not even expect, there's like three different ones and someone's like, have you ever, has anyone else taken the mathematical thinking course over at Coursera? I like it better. You know, there's already this kind of like forum chatter about which one's better and I think there are a lot of like, there are a lot of sites starting that are like yelp for MOOCs where people that take the MOOCs get to say whether they're good or not or whether they had a bad experience with that MOOC provider. So I mean, I think it's gonna be a challenge. We're sort of used to you in this country, this kind of pecking order that we know what brands or what. Well plus the pecking order might not be the same, you know, what's your style of learning, you know, what I hate might be what you love. That's why the yelp might serve you best and it's probably not too late to start your own competitor to that. Yeah. Over, yeah, great. Hi, I'm Marcy Merninhan. I have a doctorate from the Ed School and for the last 30 years, I've worked in the fields of sustainability and corporate governance and ethical investing. I have a Venn diagram question, three overlapping circles and I'm actually working with a client now who is engaged in this and struggling with these issues. And that has to do with the overlapping realms of content, pedagogy and technology. To build on your question and then the woman over here's question. When looking at MOOCs or any kind of online education, how, what are best practices that you're aware of that deal not with knowledge that is being inherited from the past that's being poured in the head but rather knowledge that's being created now and is future oriented and is aimed at performance. Performances of understanding is sort of the terminology used at the Ed School. It's about displaying competence in a professional setting. And then secondly, for the learner, where are examples of what might be called individual educational portfolios that those who are seeking certification in a number of different fields. My client is going to have people who are corporate attorneys, who are securities analysts, institutional investors, you know, a lot of very, very well educated people who have to get up to speed on what it means to report on climate risk, what it means to report on human rights concerns, what it means to report on political corruption and contributions, that's forward-looking knowledge. The technology piece is, so the content piece is forward-looking knowledge, how do we do that? The pedagogy piece is what good practices have you seen where the focus is not pouring knowledge in the head but enabling the hands and the heart to do a better job. And then the technology piece is how might learners and instructors in this co-creation mode develop records, portfolios, that they can move around from setting to setting. Those are really great questions. And I wish I had more on the best practices for you on, you know, there are obviously, you know, I mentioned a lot of these intro courses where a lot of it is about vocabulary and basic how to think like a, but you are seeing more attempts at doing, I think, what you're long, what you're saying as far as practice. There are some education continuing ed MOOCs that are out there, for instance, on teaching practices of various specific types, very, you know, concrete, not just like how to be a teacher, but like how to be teaching in this particular framework or approach or theory. And as far as like the best practice of doing that kind of teaching online versus other types, I wish I did know more. But I think that there's a lot of experimentation on just that kind of thing because I think the early stage of MOOC was the intro level things, but it makes sense if you're just doing this, if you need it to be a dropdown of like, I noticed you don't know the definition of this, but the more advanced things, the portfolio question is, I think that there are a lot of project-based MOOCs and in some ways I think, and I was gonna mention this, but in some ways I think they're the more interesting, even though that wasn't what the MOOC founders were going for, is that you have a course out of MIT called Learning Creative Learning and it's about to start up again. And it's basically, you know, a course on how to do creative projects and the students have to build things in this class and you have to team up with people even if they're not physically there and you're trying to sort of create something that you can then put online and show, have something to show. And in a lot of these kind of computer science contexts like Code Academy and all those, like there are people that are showing their work in different disciplines have figured out ways of showing work. So in computer science, participating in open source community and showing that you've made a contribution is something you can document. Or, you know, I think it's gonna differ by discipline in that way. But the online portfolio question at higher ed has been one that's floated out there, I'm sure you many in this room know this, that a lot of people have tried this. So let's create a global portfolio system in a place like Harvard or anywhere where the students can then, instead of just coming out with a bunch of A's and B's that nobody cares about or they're inflated anyway, then they have like a portfolio in the portfolio system where they can say like, you know what, not only I did take this course, but I did this, here's a video of me teaching or here's the thing I made in the computer science course. So, but they never take off, they never work. And I mean, I've written about this space a little bit at the Chronicle Fire Education. And there are many efforts and you can point to them. But for whatever reason, it's hard to make I think a global one for everybody. And so I think that you're seeing kind of field by field answers. And that may be in the end where it ends up being answered. So this is like the everything going to pieces answer one of my last slides is like, I think that people are gonna build little service lets in all kinds of areas to do these kinds of important pieces. Over here, there's a mic coming. Thanks. Hi, Anthony Tech, Harvard alum. I was wondering in terms of dropout, which courses were the most popular and that was frequently dropped out of? Wow, that's a great question. You're giving me good. This is also why I'm giving this talk to give good questions that you do articles on. So I'll try to write that article as soon as I get back to the chronicle. You know, I think that the answer really has been computer science courses, but it's also one of those like the experiment is rigged because they were the first. So when you're the only game in town, you're gonna that's the course that gets the most hits. Now I think I did, we did a quick story the other day about I think the average MOOC signup is still around 40, 50,000 of just any given MOOC that's the me like the average. And so that's still a lot of people signing up to want to take it. But I didn't see in that report the answer to your question, which is a really good one. So I'll try to get, I'll try to answer that for you and put it out there in the world. In the back. Hi, I'm Anna Trandafilm from HarvardX. And do you see anything in the international scene? I know this is like extremely US centric. You know, there's Stanford, there's MIT, there's Harvard. Do you see any competitors outside and in other languages? So there are, there are some. There's in the UK's future learn was started out of the British Open University as a spin off for profit company that they started. And it's basically the same, same idea, build this platform, have technology, get partners around the, started with British universities. I think they've expanded into Europe and other areas, maybe even some US partners. So I think, and there's, I think there's one in Germany. Germany, like there have been a lot of early experiments in MOOCs as well. And so you definitely see examples. edX just partnered with Queen Noor to make the, you know, Arab wide university for people. So I think you're gonna see competitors around the world, but in a way I wonder if it'll end up being like the way, you know, China has a search engine, Baidu, or whatever that's as big as Google or bigger because they're more people. So, so like you maybe have different localities needing different, having different specialties where they develop their own over here. I know like when I've taken MOOCs, it hasn't been because I needed the degree it was because I wanted to learn about a topic and I found it easier to look into video lectures than to go on a textbook. But the way these sites worked is I was kind of forced to sign up, register for the class. And if I didn't do that, I couldn't access the lectures. And if I didn't wanna, and if the class wasn't running at the time, I might not be able to view the historical lecture. So I wonder if you're gonna see sort of a move away from kind of the online class metaphor to maybe just sort of instructional content, maybe something similar to Khan Academy originally where it was just sort of disconnected videos or maybe there'll be sort of open videos, open exercises, that type of thing. I love that question. I share your frustration. As I mentioned, I sign up for all of them so I can get access. And so, yeah, I have to now click like, what are your intentions? My intentions are never to finish this thing. I just wanna give me access. So I do think, but I think there's a pressure the other way because in a platform play, they need those numbers to get the money from the venture capitalists. So they need people to sign up. And so I think they don't have any incentive or they don't have as much as incentive to just make it totally open out on the web with no. This is also my personal grief against Facebook, right? I mean, if I wanna go, if my friend Facebook messaged me, I get an email, but then I have to log into Facebook to really respond, right? So they want me on Facebook. They don't want me just to take care of it outside of it. And I feel like there's this, in this, because unlike previous course management systems like Blackboard or whatever, where they didn't care because they weren't a platform really, but this is trying to be the platform. So they really would rather you log in and see the course. Even if it's a slight hassle for you, I think there's an incentive for them to claim you as a student. You think they'll move to a different metrics, like maybe page views, time on the site. I mean, I know all those metrics have problems and everything. I think they want your information though. Like I think they need to know who you are. They need you to go through the gate, the door. And they need to just, even if they already know you, they need to like just check again your ID to be in this platform play. So I think that's why I think it's something about the way that these popular services are moving. Like I think Facebook has written this largely. John. Tell people who you are and you can put your book. Jonathan Haber, degree of freedom. I was actually one of the people in that article that John mentioned of the crazed MOOC takers. We've talked a lot about among other things that these things are not gonna replace colleges. And while I still agree with that, I've been thinking lately that as colleges are sort of broadening out as to the type of students, older students, younger correspondents, or distance learners anyway. And then kind of very early murmurings of the notion of a three year BA. Some of the things that like Michael Roth talks about is that you'll go to Wesley and maybe you'll only go there for three years. That's a change that I can see happening because it's revenue neutral for institutions. But it does mean that MOOCs might have a place, right? They could be there alongside. But they make that happen. They make it possible to have a three year BA. Well, they would be your party your fourth year alongside gap year programs and AP programs and a variety of other things that are emerging. So I guess I'd be curious just not from your MOOC experience, but from your kind of long educational experiences is even that too much of an educational form to expect in the next 10 years? No, I actually think, and I think you make a really good point. And so people don't know there's this movement of a question about whether you can get a BA in three years instead of four years, whether there should be that option, especially in this, with college costs going up as high as they are. So, and especially if college costs do, if MOOCs end up driving up the cost of teaching on campuses. And as you've seen, my colleague, Jeff Zalingo, wrote this great book called College Unbound where he documents how colleges have had been in this arms race to build climbing walls and better dorms and all this stuff for competitive reasons. And in a way, I'm sort of saying that MOOCs are just gonna be a competitive reason to make your classes better because people are peeking into that a little more. But if they do that, you almost see something has to give eventually, right? We can't just keep raising costs in infinite. So maybe a three-year, maybe that does put pressure toward something like a three-year degree. But then the biggest, I think you proved this in your book, I think, which is that the biggest thing is, yes, MOOCs are free, but the biggest luxury item is time. Taking a MOOC and getting it, it takes really, I mean, I shouldn't even admit this, but that physics course, I almost failed it because I wasn't keeping up. Like I wasn't, and I was like writing about my experience so I would have then had to confess to everyone that I failed it. But like, it's hard to keep up to do all the work that these free courses are requiring. And so time is the big question. So how are the high school students gonna, or how do you do a gap year? Is it still a year of your life but you're just in your parents' basement or you're saving some money somewhere else? And I don't know the answer to that. I know you've talked a lot of people on that one as well. How about this side of the room? Hi, I'm Priya. I'm coming from the UK where the higher education used to be free and now is increasingly commoditized. I find it amazing that major universities are offering free courses to anyone who signs up. I mean, is there any sense of unhappiness from people who actually pay to go to Stanford or Harvard? I mean, there's a lot of money involved and now you're offering a course for free. That's a great question. And there was a piece we did about a Stanford student who was a little bit ticked off when that first course that the kid in Mongolia was acing, that was going on because he, but it wasn't so much that just that he was paying and this other kid gets it for free. It was actually more that the way the MOOC filming process worked, I think he was in one of these, it was a different course, but it was like he was in a course where they were filming it for the MOOC and it was disruptive and he's like, why are you guys disrupting my $50,000 a year education to teach these other people that aren't paying anything? And so then of course, a lot of people flay more at him and say like, how could you be against free education? And there's back and forth that ensued and it was fun to write about. But I mean, it's certainly a valid question, but I think with open course where, which was about a decade ago where MIT first gave away some lectures, people had the same ideas like, why would you, how could that work? How does that map work? But I think it does come back to what you're getting when you go is so different than just getting the, even when there's assignments and the robot grading. I think that there's a pretty high recognition, I think still by the people paying the bills that they're getting something different, that they're getting something that, I don't know if they think it's worth it or not, but that it's different. Socialization is one thing as well. The parties. The parties. Well, in contact, just like you know, teaming up with the guy who was here at Armory. Yeah, Mark Zuckerberg. So yeah, I mean that's the question of like, you know, the famous dropouts that are our idols in industry right now, right? The Steve Jobs and the Bill Gates and Mark Zuckerberg, like, they all dropped out. But they dropped out of places where they met their business partners for their thing. So they actually did not benefit from college. They just simply got a good deal and left while before they, they didn't need all of it to do what they then went on to do. That may argue for the shorter BA. Yeah. Thank you. Maxime Lomaric, University of Louvain and visiting researcher at Darkman for now. So the story about that wonderful little guy from Ulan Bator got me into thinking. So, because then you said that he got into MIT and so on. And so do you think that these MOOCs are going to be increasingly used as a way for elite universities to actually recruit genius around the world, you know, maybe a genius in Yaoundé or in Kinshasa? Yeah, and don't you think there is a risk that actually MOOCs increase the tendency towards brain drain in developing world? So what do you think about it? That's a really interesting way of putting it. I've definitely heard university folks tout as one of the reasons they're spending all this money that that might help in their recruitment or finding people around the world that are worthy and help diversify their community. The brain drain argument is a great one, which is the flip side of that exchange and what that might do. I think it's a great point and worth an article as well for me. Thank you for the ideas, guys. Over here. Hi, Deep from Cogni and at Tech Startup in Assessment Technology. So I think one of the main complaints against the MOOCs is the lack of engagement because multiple choice or peer grading does not really assist the knowledge. And without that better quality assessment, we cannot provide the certification which employers would find useful to hire people. So what's your view on when better quality assessment technology becomes available to measure the competency or the content of the knowledge of the students and is as good as human engagement? At that point, does the MOOCs or the spokes would become more disruptive? Wait, could you do some one part here which is when what becomes available to make them? The better quality assessment technology. Assessment, so whether we can then measure and prove that they're working. So is the assumption in there that you think they will catch up to being as good or that you're asking whether they will be? Right, so the question, because in real classroom learning, it's the interaction where the professor asks the question to the students. They provide, they ask them exams, they assist them. And it's that engagement which leads to better quality certification that the student has actually learned something or not. In online currently, we use multiple choice or peer grading which does not actually really measure the learning of the students. So when the technology is as good as human in terms of assessing students, at that point, does the online education general, whether it is MOOCs or spokes or small private online course would really become acceptable at that point? Yeah, I think if I understand your question correctly, I think that you've really touched on something really fundamental, which is that are these things gonna be as good and where are they gonna be as good? What disciplines, which subjects? And I think that a lot of that is gonna hinge on whether kind of group projects can be done in an online forum. And I think there are experiments right now in that. I do think that these intro courses are maybe, is MOOC only gonna work for an intro course or for a very introductory material that is really the gen ed kind of thing that people gloss over as they kind of get up to speed in a college level traditionally. Or is it something that can go beyond that? And I think these measures, whether when they can start to measure outcomes, that's gonna start to answer that question. But I do think that gets back to the, how do we measure outcomes at Harvard? How do we measure that right now? I'm here as a fellow, I think I'm learning a lot. But there's no, how do we know? And so I think that's another bigger question that I think it sort of calls back into question. How do we know that about the stuff everyone thinks is gold standard as well? You've had your hand up for a while in the middle of the room here. Hi, my name is Sam Engelstad. I wonder if you can go back to, bring the slide back and show the various percentages. So people with college degrees and graduate degrees. Just let me look for, bear with me, it's a while back. So this is just from edX, I'm sorry, it's just from MITX. Right. And so their HarvardX has similar graphs, the, it's a pretty similar trend I believe, I'm getting a head nod from HarvardX. I looked at that and I saw the ad for this particular lecture. And I mentioned a very high number of people with bachelor's degrees and I can see the very high master's degree, quite a few people with doctorates, secondary school, but very few people with secondary school education, which leads to the question. At the end, yeah, I think all they have, yeah. Whether this is actually, and I listened to the conversation here, whether this is not really sort of a first world kind of phenomenon. And to what degree is these being brought into, you know, the hundreds of millions of people around the world who live less privileged lives, who have access to smartphones and could be trained as, you know, community health workers, et cetera, et cetera, using MOOCs. So in other words, is it, I mean, is it right now serving the Fitbit crowd? That's great. And I think that that's one reason I kind of put that up there is because I feel like right now the people taking them, it is kind of serving the Fitbit crowd. And the question you raise is a really good one because the idea, the whole vision was to serve people who are not served at all right now by higher education. Or are not served by high quality higher education. And so I think one of the things that I think these numbers show though is that MOOCs work, when they work at all, they work for people who already know how to do a college course. So I think some of these numbers just reflect, it's almost like, it's like a VCR with a VCR tape as the way to, how to, your instruction manual. It's like, if you don't know how to work a VCR, how are you gonna watch the video and how to do it? So there's a little bit of that I think in the MOOCs because they just have this assumption that you're with them and they're very, some of them are quite demanding. And I know a lot of the professors that are teaching them are very proud of how demanding they are, but it almost serves as a weed out. Because if you're not keeping up, then this has got to be a hard course. So not every professor's like that. So please, but you know that there is this, that's the kind of fear is like, if they're designed with a professor teaching students that are of a certain type, that's the kind of assumptions that are going into it. So I think you raise a good question about, so does the technology, can it be tailored to a different crowd, to a more first generation crowd wherever they are? And I don't know the answer to that. I actually don't know if it is true. I don't know if you can do that. I don't know with just technology. Maybe you do need more human activity there. And that's what you see people stopping to do, right? Leah, thank you for chiming in. My wife, Leah, who knows too much about this stuff because she has to hear me talk about it. But yeah, I mean the pivot that a lot of people talked about with Udacity Sebastian Theron, who was one of these first pioneers of it. And now he's kind of, at least in this interview that he did with Fast Company Magazine, I haven't talked to him since myself to confirm this, but he was quoted as saying to something in the effect of like, that didn't really work, that whole serve the world. We're gonna focus on, he didn't say the Fitbit crowd, but the people that are already computer science that are kind of doing a lot of open to this and then I've already got a degree. So. We only got time for two more questions. Two more questions, okay. Let's see, in the back, I'm getting an enthusiastic arm raving. I'm raving, oh. Thank you. My name is Sutlana and I'm a founder of one of the startups that actually works with MOOCs. And so we're trying to connect people within the classes to each other for collaborative groups, right? And as we think more about this, and as you were mentioning, you know, little startups that are kind of like trying to come into the space and you know, like taking more, yeah, but so the question is, what aspects of maybe the residential education do you think MOOCs could like help, you know, or like substitute or like disrupt in this world? And do you even think like there is room for that? Thank you. I didn't mention just because of time that the idea of the flipped classroom, right? Which is the big, that's the big idea that people talk about in a residential setting. And there's experiments here at Harvard, there are experiments around the country, where if you don't know this one, it's like you watch videos like a MOOC style video. You have the students watch the videos for homework and then you use class time instead of lecturing which is in the can. Now you can do the discussion and group projects and act. You just sort of do different things with people that take advantage of actually being in the same room. So that's the one that gets talked about the most as far as can something change. The trick is, and I've told a number of professors have said the same thing, which is like what do I do in the class then? And with all seriousness, you know, like I've sat in on some brilliant examples of when they do it. One professor in the class I'm auditing is like he gave out lottery tickets to everyone and then he made as a democracy, you know, kind of exercise, made the class vote on how to distribute these $1 scratch off lottery tickets and then the winning proposal, you know, give everyone, give that guy every one of them or give only the women the tickets and not the men. And then, so people were proposing these crazy ideas and it was brilliant. But then it was kind of like, how do you follow that week after week of like having in class exercises that are truly unique and wonderful. And that's why I think if you did that, it would be expensive and probably people should do that. But how do we pay for that then? So I think these all come back to questions of support and money and creativity and I hope, yeah, we'll look for what your company comes up with. Charlie, you wanna ask the last question? No, I just wanna say that that answer to the last question seemed to be a fitting conclusion. Great. And I wanted to ask everyone to join in thanking Jeff for coming. Thanks guys.