 I'm David Thorburn, Director of the MIT Communications Forum and a Professor of Literature at MIT. I'm delighted to be moderating this particular forum, which has been long-contemplated and has been spearheaded in some degree by our third speaker today, Alison Byerly. Our plan, our format for the day, is for each of our three speakers to make brief formal introductory remarks lasting between 12 and 15 minutes. I will warn each speaker when they get close to 12, and I will stop them with an iron dictatorial hand at 15 minutes. There will be a brief response from other members of the panel to the presentations. And after these presentations are concluded and a few brief comments from the panel, we're going to open it to the audience. And I'm counting on the usual situation in which the audience makes the most brilliant, remarkable, and often disturbing points. So my expectation and hope is that the first hour will be remarkable and the second hour in which the audience is posing questions and arguing and suggesting things to the panel will be even more remarkable. Our three speakers today, and they'll be speaking in this order, first remotely from California, Daphne Kohler, the Rajiv Matwani Professor in Computer Science at Stanford. As many of you must know, she's the co-founder with Andrew Ng of Coursera, the famous online education platform. She is also the recipient of a MacArthur Genius Award, and in 2008 was awarded the first ever ACM Infosys Foundation Award in Computer Science. She has offered a free online course on the subject of probabilistic graphical models since February 2012. And the course is based on a textbook that she published. Our second speaker, well known to folks at MIT at the end of the table here, Anant Agarwal, is the president of edX, Coursera's famous rival or partner, depending on how you interpret it. He is a professor in the electrical engineering and computer science department at MIT, teaches edX offerings, teaches the edX offering of the MIT class circuits and electronics, and has, as many of you know, served as the director of MIT's computer science and artificial intelligence laboratory. Our final speaker, Allison Birely, to my right, holds an interdisciplinary appointment as a college professor at Middlebury College, where she previously had served as provost. This academic year, she's a visiting scholar in literature at MIT, and we're especially proud she's the first of our visiting scholars to become a college president. In July, she will become the 17th president of Lafayette College in Easton, Pennsylvania. Professor Birely is a quite well-known influential scholar on 19th century literature and culture and has written about aspects of Victorian technology, as well as recently especially on aspects of MOOCs and of digital education generally. We begin then with Daphne Kohler. You have the stage, Daphne. Thank you, and I need to hope that I can actually figure out how to share my screen. OK, sorry. Still trying to make the technology work. I need Skype premiums to share screens. I thought we figured this out. It's not letting me do this. Our tech guy is checking, just a minute. I apologize. 1-2, 1-2. No, you don't need the premium to say go ahead with free. OK. Call free. OK. Not necessarily succeeding for us here. Do you see my screen? No, not yet. Well, I think it needs the Skype premium to do this. No. During the test, remember how we selected the free option you get on the corner? And that's what I did, and it didn't let me do this. So the audience is being treated to this inside the screen. When this idea was proposed to me, I was nervous. I, let's see, did it work last time? I can do, I can show the slides for you here. I have your PowerPoint presentation. OK, I did this again, and it's not letting me do this. It is true that Anant and Daphne have been on panels together recently. And I remember we got it to work, but it didn't. You know, I'm going to do something. You sure? What's the idea that Daphne can talk to it? OK, I'm going to share a screen, and it says you need Skype premium to share screens, and I'm going to click the thing. There is no free option on that window. It says continue with a free call, and I'm clicking on that, and it's returning me back to the same place where you're not seeing my screen. Oh, it's not working then. No, it's not. It used to, but now it doesn't. I don't know. All right. Let's switch to our backup plan, which is we show you PowerPoint here locally, and we do it. You can stay on Skype, but we can switch the order and let Anant speak first while you try to fix it. I think they're almost buttoning it now. I think you already have showing the slides. Let me show the slides here. OK, it's not going to be as good. Sorry, Daphne. This is our default, and we had actually worked all this out. You were never supposed to hear any of it. And I can't hear any of you, by the way. Right from your mind. Or the slides are up now. OK, hopefully you can keep up with me without me saying next slide all the time, because otherwise it'll really be awkward. So just please try and infer. OK, because I don't even know which slide is showing, you see. OK, so good afternoon, everyone. And I'm sorry for the technical difficulties. We actually spent 45 minutes trying to debug this a few days ago and thought we had it, but obviously not. So thank you all for inviting me to this event. I'm going to tell you about the online revolution education for everyone. I'm going to start with the online revolution and end with education for everyone. Next slide, please. This revolution really started in the fall of 2011 with an experiment that was initiated by Stanford University that elected to take three graduate level courses in computer science and open them up to everyone around the world for free. We were expecting an enrollment of a few thousand people in these classes. Oh, that would be great. Thank you. Now I can actually see my slides. Thank you very much. We were expecting an enrollment of a few thousand people in these courses, but within a matter of weeks, each of those three had an enrollment of about 100,000 students or more. To clarify what the number 100,000 means, the biggest of those courses when taught at Stanford is the machine learning class. It has an enrollment of 400. It's considered large even by Stanford standards. To get that same size audience that we got to, the Andrew got to in that public class, Andrew would have to teach his class for 250 years and would reach only 250 generations of privileged Stanford students as opposed to the much more diverse population of people that he reached in this format. Next slide, please. Since this project started, after the success of these, we realized that we had a really amazing opportunity to offer education, not just from Stanford, but from a whole range of top institutions to students everywhere. What you see here are some of the institutions that are working with us. These are the US-based ones. And you can see some of this country's top private institutions, Princeton, Penn, Caltech, Duke, and others, some of this nation's top public institutions, Washington, Wisconsin, Michigan, and many others again. We also are privileged to have 24 non-US institutions that are offering courses on Coursera from four different continents. We have North America. We also have Latin America specifically, Europe, Asia, and Australia. We're currently able to offer courses in five different languages, English and French that are currently going on right now and soon to come Spanish, Chinese, and Italian. These institutions are offering close to 330 courses across the range of disciplines. People tend to think of this effort as involving mostly computer science because that's where we started. But if you look at the courses that are currently on the platform, you can see there's courses in philosophy, in medicine, mental health and illness, global challenges, business, photography, astrobiology, performance music, performance art, and many, many other disciplines, pretty much anything that you can think of. Next slide, please. Let me talk a little bit about the students. Next slide, please. The students that are taking these courses, we hit three million students two days ago. This is starting from 200,000 legacy students that we had from the Stanford courses back in April. So in less than 11 months, we have reached three million students. These are some of their stories, and I can stand here for hours telling students stories. Click, please. On the left is Raul. He's from Columbia. Doesn't have a computer science in his country to speak of, took several of our computer science courses, used that as the basis for his Fulbright application, and now is coming to study in the United States as a Fulbright scholar. Jolene is from Pakistan. She founded an NGO in Pakistan with a fellow student that she met in their sociology class from Princeton University. They're now taking other Coursera classes in business and other topics to learn how to run their NGO better. And in the middle is Achint, who grew up in a small town in India, wanted to become an entrepreneur, took the Wharton gamification class, and is now competing in an all India entrepreneurship competition. These are the kinds of access stories you would expect to find in this type of effort. Next slide, please. The next one is a little bit more surprising, I think. On the right, you see Daniel. Daniel is a 17-year-old boy who's severely autistic. He has a speaking vocabulary of about 150 words and communicates by typing on a special iPad designed by his father. Daniel, despite that limitation, was the star student in the University of Pennsylvania Modern and Contemporary American Poetry class. That's Professor Al Fillory sitting on the left. He's since taken successfully several of our classes and says, not only is this the first meaningful educational experience that he's ever had after a life of special ed, that this type of learning is actually helping to diminish the severity of his disease. This type of access is something that transcends just autism, although we have a lot of autistic students. We also have people with other kinds of access difficulties. For example, we got an email from a woman with stage four breast cancer who can't really leave the house except to go to the hospital for her chemotherapy, and this is her window to the outside world. So let me talk a little bit about what the course experience looks like for these courses. And one of the things that we tried to do in these inaugural Stanford courses back in the fall is to give students something that is very different from the traditional open courseware experience. This is not a set of static contents, it's a real course. The course begins on a given day, and every week there is material that the students are responsible for learning, and every week there's homework. The homework has to be done by the deadline, the homework gets graded, and if the students don't grade the work, if the students don't do the work and don't get a grade, they don't get a score. And you can see the effect of that on this graph, please click that you see over here where the x-axis is time and the y-axis is number of users on the site, and each one of those little blips is the day before the deadline. When the students log in at the very last minute as students everywhere do to do their homework. And at the very end of the process, please click there is a certificate that the students get and that helps them benefit from the work that they did. And I'll talk a little bit about that at the end. So let me talk about the three pillars of the student experience on this platform. The first is the video-based learning and the fact that it allows us to personalize the experience in many ways to the individual needs of students. So one of those ways is simply by moving away from the constraints of in-class teaching which force us to work with 50 minute or hour and 15 minute intervals for the purposes of room scheduling. That unit of time is not ideal for either the material or for the student's attention span. Here we break up pace. We can also now make the curriculum more flexible in other ways. So for example, if we click, we can see that some students might benefit from introductory content that not everyone needs or please click, some students might want to learn more about particular topics that not everyone is interested in. Furthermore, the students can browse this at their own pace, rewatching when they need, fast-forwarding when they want, allowing them to really study at their own pace as opposed to the pace that the instructor select is suitable for the median of their lecture class of 300. So this really allows us to break away from the one-size-fits-all model of education that we've had to do for the need to scale, teaching to 300 people in an auditorium. That's the first pillar. The second pillar is of course the fact that students don't learn by passively watching videos. They learn by actively interacting with the material. So in order to do that, we've had to build in a significant number of assessments and interactions into the material. And if you want to do that in a way that gives students meaningful feedback about whether they're learning, you need to be able to grade their work. And so we've put into mechanisms for grading students' work. The first is to use computer-based grading. And you can now do computer-based grading for all sorts of things, not just multiple choice and fill in the blank questions, but also meaningful grading of math, for example, math expressions. And then please click anything that involves a structured output, be it the output of a computer program, the output of a computer model of a physical system, or an Excel spreadsheet that's a marketing model, anything that has a predetermined format can be graded automatically by a computer. Now, one of the nice things about computer-based grading other than the scalability is also the fact that it gives the student immediate feedback about whether they're doing well or not. To contrast this with the model that happens in traditional college classes where it can be three or four weeks between the time that the student learns the material, does the work, and eventually gets their grade back. So by the fact that the student gets immediate feedback, it turns this into a computer game so that when they get their grade back and they say, what, I only got 60 out of 100, they're incentivized immediately to try and beat the system. And so what you see here is a graph where the blue bars are the grades for an initial submission, for a particular assignment, and the green bars is the same distribution for the final submission of that assignment. And you can see that almost all students achieved either a perfect or close to perfect grade by this mastery-based learning, which they incentivized themselves to do. Now, you might say, well, that's great, and who cares if the students by trial and error eventually get to a better answer, did they actually learn anything from it? And so the answer is that they did, and we can show that if we click, we looked at students that had similar performance, say on problem set one, before doing mastery. The students who engaged in mastery-based learning also did better on the initial submission of problem set two, which means that what they learned in problem set one helped them do better on other assignments as well. And why is my, okay, thank you. Good, no, back, back, thank you. The other, now, of course, computer-based grading only takes you so far. There is a, if you want to do the kind of open-ended, critical thinking-style work that is so essential in the humanities, but also in many other disciplines, you can't do that with a computer-based grading today. Technology just isn't quite good enough. So in order to do, to deal with that kind of work, we ended up putting in place what we call a peer grading pipeline, where students grade each other's work using a set of very carefully defined criteria that is provided by the instructor. The students are trained and the use of that grading rubric are assessed and whether they can apply it successfully, and when they can, they go and grade the work of five of their peers, so each student gets five pieces of feedback for their work. Now you might question how good are unqualified students at grading the work of others. And so Professor Mitch DeNier from Princeton in his introsociology class actually tried that. He had every one of the final exams that consisted of three essay questions graded by both of his TAs and compared the TA-based score, which you see on the x-axis, with a peer grade score on the y-axis. And you can see that the correlation here is very, very good. Now of course, the quality of the correlation really just depends as we've discovered on the quality of the grading rubric. This happened to be a very well-designed grading rubric, which is why we see this correlation and what we're constantly doing is trying to refine our understanding of how peer grading ought to work. And when it works, you can see that it works very well indeed. Peer grading allows us to deal with a lot of other types of assignments that then you could using computer-based tools. So for example, this is the design class from the Wharton Business School, which was where the assignment was a semester-long project that started out from a problem specification, a concept, a prototype, and finally an actual artifact. And what you see here are some of the artifacts that were created. Now, because this was a semester-long project and it was graded each week, by the time the final project was submitted, students got over 40 pieces of feedback for the project they were constructing, which gave a real richness to the work that they created. And you can see here, for example, a laptop table, a charging device that I've often wished I could find in hotel rooms when they never put a piece of furniture next to the outlet, and a space-efficient work desk. Now, only after I selected these projects, because I liked the way they looked in the pictures, did I notice that they came from three different continents. There is one from India, one from Spain, and one from the Philippines. Now, one final point about peer grading that we were not expecting is that it's not just a useful mechanism for at-scale grading. It's also something that really enriches the learning experience of the students in the class, because when a student does the grading, it is immediately after they have tackled this assignment for themselves and really have thought through some of the issues that are encountered. And so their mind is really open to seeing other solutions that people have come up with and understanding how it relates to theirs, which I think arguably is an important step towards learning creativity. The final part of the user experience on the Coursera platform is the community. And this is something we needed to put in because the one thing that does not scale to a class of 100,000 people is the instructor time for interacting with individual students. And so we have replaced that with four minutes with peer-to-peer interactions and students to ask questions and other students answer each other's questions, giving a real richness to the student's experience. So now let me talk very briefly about credentialing. This is the thing that happens at the end. This is something that we've put in place recently. It's an identity verified credential where the students of Nameless Certificate is tied to a government issued ID. And we can ensure that they are the ones that are actually submitting the work. This is something that a lot of students have already chosen to avail themselves of because they want to have a verified certificate that they can present to an employer. This already costs a modest amount of money, not for access to the class, but it's a way for us to sustain the effort. The access is still free, but the identity verification process is cost something like 30 or 40 or $50. But because we believe in open access education, for those students, for example, in Sub-Saharan Africa for whom $40 is a lot of money, we've put in place a financial aid option and all the student needs to do is explain why they need the certificate and why they can't afford it and they get it for free. The final piece of certification is an effort that we've undertaken with the American Council for Education, which is an umbrella organization for US institutions that, among other things, assesses credit equivalency of courses that are taught outside of a traditional college environment. For example, courses taught to American service personnel in the military. They have assessed a pilot set of five of our courses for credit equivalency and have granted them credit equivalency, which means that students who now successfully complete these courses have the option of transferring them into a degree program that they subsequently enroll in at the discretion of that institution that they enroll in. And this allows students to really get benefit from taking these courses. Okay, let me talk very briefly about a few higher level topics. The first is the amazing data, an opportunity that this allows us to learn about student learning. We now finally have the opportunity to improve pedagogy in the same way that we improve other things. So some of you have heard the name AB testing, even if you haven't, you've participated in it. It's what Google and Facebook and other websites use in order to refine and iteratively improve their website. At any time that you log on to Google, there is a 5% chance that you're in the B-group, which gets a slightly different user experience. And if performance metrics suggest that that B-group is better than the A-group, the entire website flips over. So that allows the website to improve on a matter on a scale of days rather than years. We've never been able to do that in pedagogy, but now we can. So for example, you can do AB testing to decide whether the instructor's face should show in the video or not. There's been papers written arguing both sides. We actually, with Dan McFarland at Stanford, ran this AB test. And you're probably expecting me to tell you what the answer is, but I can't. And the reason I can't is because the B-group complained so bitterly that they didn't see the instructor's face that they had to stop the experiment in the middle. But there are other AB tests that we've done that really have taught us a lot about online pedagogy. Two minutes. Two minutes, perfect. The next, the other high level point that I'd like to mention is the quality improvement that one can use in order to improve the student learning experience. And for that, I'd like to refer back to a paper called The Two Sigma Problem by Benjamin Bloom, where he studied three populations, the population of students that were taught in a lecture class. You can see the distribution of achievement scores. Lecture class with... I'd like to point that out. I'm not responsible for technology. Yes, go ahead, go ahead. Can you hear me? Yes. Okay. So if we take the median point of the, actually let me skip past this since we're already past time. So we get a Two Sigma improvement in performance. And the question is, how do you get that kind of improvement in performance in using mechanisms that we can afford? Because as Bloom pointed out, we cannot afford a human tutor. Hello? Okay, sorry, can you still hear me? Can you talk to, tell her we can... Okay, sorry, this guy just decided to quit completely. Sorry. So as Bloom pointed out, we cannot afford a human tutor for every student. So how do we get this kind of improvement in performance? And now we can potentially afford to get maybe a tablet for every student. And if we can do that, maybe we can use technology to get us to the red curve and ultimately to the green curve. And the nice thing is that computers are really good at mastery. They don't mind showing you the same thing five times. They're quite happy to do that and they won't even get judgmental. The next place where we can get improvement in performance is inspired by this quote by Edwin Slosson who said that college is a place where professor's lecture notes go straight to the student's lecture notes without passing through the brains of either. This I think is a motivation to all of us to change the way in which we piece in the classroom. So we move away from this of students sitting there in rows taking notes to something that looks more like this. We're outside the classroom, students learn the material and then they come into class for a much more in-depth discussion about the materials with their peers and with an instructor. So if you look at what this means, Professor Christian Turwish who's an operations research instructor who taught his Coursera class, his last segment was an analysis of Coursera, a case study of Coursera. And he talks about the Pareto optimal curve of faculty productivity versus student learning where you can invest a lot of effort in each individual student such as in office hours for great outcomes or a lot less time in each student in a traditional lecture for poor outcomes. And he says that MOOCs and Coursera really move this frontier into a different place and you can use this frontier either to keep students learning the same and decrease costs which is what MOOCs do by offering a free education to everyone or you can keep faculty efforts the same via blended learning and improve learning outcomes considerably. So finally, and actually I'm gonna skip to my last slide please which is education for everyone. I'm going to come back to the title of this talk and point out that currently only about a third of our population is in the United States. Less than 30% are in Europe and the rest are around the world. In fact, 40% of our students are currently in what the State Department defines to be the developing world. And I think this is really a priceless opportunity that we have here to really make a turn education from something that is a privilege of a few to a basic human right. And I'm going to end this with a story. In September I was invited to the launch of the Education First Initiative at the United Nations in New York and people who were launching this initiative were talking about their travels and refugee camps in Africa where people are living in little grass shacks on a cup of food a day. But when asked what they most want right now they don't say more food and they don't say better shelter. They want a better education for their children because education is the one thing that we have that can really turn people's lives around and make the world better for everyone. Thank you very much. Thank you. We'll get you back as soon as we can. Anand, you're up. So before I begin the talk let me start with a really quick poll of the audience. So one thing we do in online learning is make it turn that into active learning where you don't sit and listen to a professor for 15 minutes, instead you get engaged. So in that same spirit let's start with that. So I'd like you to think back, say, is this working? I'd like you to think back on the order of 400 years. And I'd like you to shout out the biggest innovations you've heard about or seen in transportation. The wheel, the steering wheel, automobiles. What else? Steam engine, airplanes, rocket fuel, inertial guidance, navigation, so the list goes on and on. So I'd like you to think back the same 400 years. And I'd like you to think to what have been the biggest innovations in education? So we are, after all, in this little institute of technology in the northeastern corner of the US. And this is considered a great educational place. So it's natural to ask the question, what have been the biggest innovations in education in 400 years? We're no longer teacher rhetoric. We're no longer teacher rhetoric. What else? I'm not having compulsory attendance. Not having, textbooks. Textbooks was 500 years ago. The printing press was 500 years ago. That's why I said 400. The proliferation of the paperback. So we keep going back to sort of printing and so on and some of that. Smartphones. Smartphones. But we really haven't applied smartphones to education. But you get my drift, which is that. We've made dramatic applied technology to a variety of fields. Transportation, healthcare, agriculture, and so on. But really we haven't done the same to something really as basic as education. And we constantly keep going back to a lot of things to derive from the printing press. Like books and so on. People didn't bring up the blackboard in 1872, but that was a big one. But either way, we really haven't done a whole lot. And just to give you a sense, so this is a classroom at MIT about, I think, 50 or 70 years ago. And this is a classroom today. So what's changed? This is from my friend, Eric Klopfer. So what's changed? Well, the seats are in the chairs, exactly. The seats are in color. The women, that's a big one, absolutely. Yeah, here, here. Some people did point out that there's a projector, there's PowerPoint, but it's not clear to me that PowerPoint was an advance. But, you know, I'll not go there. So here's another slide. This is from my friend, Pure, who's sitting right here. So Pure taught at Abu Fahmay University in Nigeria. And this is a picture of a real classroom there. Okay, this is a classroom. So here's a teacher teaching, and students are all sitting here. This is a real classroom. Now, we've all heard about distance education. Okay, I suspect the people here are having long distance education. Okay, so there's a real issue of access to education around the world. I mean, here, at least, they have someone teaching. They have some access to education. But really, there's large parts of the world where people do not even have good access to education. And if you come back to our own United States, in a number of high schools, students don't even have access to good first-rate AP courses. And the quality can vary quite a bit based on where people are. So this is the context in which edX was formed. edX is a non-profit venture. It was founded by Harvard and MIT with $60 million, $30 million apiece, to really transform education, to really rethink how education should be done. And a large part of our mission is to rethink how we do education on our campuses. Really reinvent education, rethink education from the ground up. A big part of our mission is to increase access to education for students around the world as well, but really combine both access to education and improving the quality of education in our own campuses. So we're based on an open-source platform where we are making available not just the courses for free, but we're giving away the platform code as well. So hopefully we can get the benefit of people around the world contributing to the platform and other people using the platform if they cannot put up a course on edX themselves. So we have a number of university partners. We have 12 university partners so far and adding quite a few more. Top first-rate institutions that appear in institutions of places like Berkeley, Harvard, MIT, Georgetown, and so on. This is our portal here. As you can see, some example courses. Here, for example, is the Berkeley course on quantum mechanics. Here's a course by the famed Walter Lewin on physics. These courses are high-quality courses. We try to work really hard to get high-quality courses up, which are the same rigor as the campus courses. You know, if the campus course has an exam, these courses have an exam, and the professors work really hard to make the certificate that students get at the end really meaningful. As part of the platform, we also do a lot of research in the pedagogy. We're gathering a lot of data. I like to call the edX platform the particle accelerator of learning. So where we keep, where we track every student interaction with the system, you know, when to pause a video, when to rewind, when to access videos. So for example, we drag our students to class early in the morning. People want to guess at what time of day are most of the, do we see most of the video access on the platform? Between midnight and 2 a.m. Okay, and then why do we still insist on dragging them to class? It would help two people if you didn't do that. It would help me as a professor, and it would help the students. So we do a lot of research on how students are learning, and we gather a huge amount of data on the platform, and I could give a whole another talk focusing on the kind of research that's been done on the platform and the kind of learning that is coming out of it. So as an example, the numbers are really staggering. So the very first course that we offer on the platform was a Circuits and Electronics course from MIT. And this is an MIT hard course. A number of students who skipped out after the first week said, man, these guys were serious. And they said this is an MIT level course. This is a hard course. This includes differential equations and complex analysis as prerequisites. But despite all of that stuff, and we were pretty honest about the prerequisites in the announcement. And despite that, we had 155,000 students take this course from around the world, from 162 countries. So let me put this number in perspective. You heard Daphne put it in perspective in one way. It's really funny. Daphne and I are good friends and colleagues. And at this time, we could give each other stocks. So let me put this number in perspective in a different way. So 155,000, this number is bigger than the total number of alumni of MIT in its 150 year history. So it's pretty interesting to see what, how big these numbers are. Of these, 26,000 students tried the first problem set. So we call these students active learners. So they were serious about the course and they were trying to engage with the material, as opposed to those who just want to check things out, check out a video, for example. So these are active learners. And then if you notice about 7,200 students passed the course. So a grade of 60% and above was passing grade and this many students passed the course. And they got a certificate from MITX. So if you pass the course from Berkeley, you get a certificate from BerkeleyX, signed by the instructors and so on. And people are using these certificates for a number of things. One interesting story, the story is a bound. And we have a number of students. For example, a student from India called Amal Bhave. So he did really well in this course, he applied to MIT. He heard two days ago that he got into MIT. I'll just talk about MIT central and MIT audience, but I could say this for a number of universities. Another student, Batushig, who took this course from Mongolia. And he was 15 when he took it. Absolute genius. He had a perfect score in a sophomore junior level MIT course. He applied to MIT. He got into MIT. He heard two days ago. There was a third student from China. He took a Berkeley AI course. And them doing well in the course, showing initiative. He got into MIT as well. So again, I'm talking about MIT. But there's an equal number of students who applied to other universities. These are high schoolers. So you can see how these things are changing people's lives and making a meaningful impact in giving them access to these courses. And then being able to show that they can master this material in a way that they could not do before. But the interesting parts which the press is going to focus on, unfortunately, is the fact that online learning can make education not just higher quality, but efficient. So for this course, we used about the same staffing resources as we would use in a typical 150% MIT class. It's pretty remarkable what is possible. Another thing we do is try to improve campus education. So one experiment we did was San Jose State University licensed this course and taught it at San Jose State. So we teach a course on their campus. edX hosts the course. And there, the professor used the videos, think of it as a next generation textbook. And the students would come to class after watching videos and exercises in the dorm rooms, then come to class and ask questions of the professor and interact with the other students in the class. And the results came out pretty staggering. Traditionally, this course was a hard course. It had a 41% retake rate. And with this blended on-campus class, the retake rate fell to 9%. And these 86 students were randomly picked, not volunteers. They were randomly picked from a class of 250 students. And these are some of the first results coming out. And these are just staggeringly good. So this is to show that we can improve the quality of education while we increase access to education as well. So just to give you, I'm just going to give you a few quick vignettes of what a course looks like. So this is what a student sees. There's a bunch of activities they can do. A professor can organize a course in a number of ways. Here, the professor has chosen to organize it by week. They can do it by chapter, like a textbook if they want to. They're done by week. Each week, these courses replace a lecture with what we call a learning sequence. A learning sequence is a set of interleaved videos, short video snippets, interleaved interactive exercises. So this is a video learning sequence, and the student kind of goes through that sequence and can do it at their own pace. This kind of learning where the student is engaged with the material, watches the video, answers the question, and really engages with the material is called active learning. And this kind of technique has been shown to improve learning outcomes as well. So here's a classic paper in the field by Craig and Lockhart written 40 years ago. This should be required reading for anybody that starts to teach. I wish I had read it 25 years ago. I read it last year. And this shows that if you engage the student, so the Socratic method applied to learning can really cause a much higher engagement level with the student and also increase the retention of the material with the student. So this kind of, so what online learning is done as a minimum is brought education and education research to the forefront of everybody. Even I who hadn't read an education paper for 25 years of teaching, finally read one last year. And so education researchers and learning scientists, instruction designers are now becoming really a hot commodity as people are looking around for, wow, we can actually improve teaching and learning. So here's an example video snippet. Play this video quickly if you don't mind. This is a Kahn style video to give you a sense of the kind of videos you can have. This amazing thing happening. And the key is that the stable point is when V plus equals V minus. Okay, it's not exactly equal. Okay, it's more or less equal to each other. Okay, V plus more or less equal to V minus. So in a number of ways you can do video. This is a Kahn style video. Kahn Sal was our student here and he's certainly done good. We have a number of ways of computer grading exercises. The computer grades the exercises here. We have, show you how we can do chemical equations. We can do a large variety of things. These chemical equations can play the video please. And so a student enters an equation and the computer checks it out and they can get multiple tries. So you'll notice here they got it wrong. They can retry it. You heard from Daphne's talk that how giving students multiple tries to correct themselves really helps them get engaged and improve learning. And back to the gamification aspect, students get a little green check mark when they get something right. And students are telling us that they go to bed at night dreaming of the green check mark. That is really, really engages them. They keep trying and trying till they get that green check mark. Really, really engages them. And one of the students that took our spring course went ahead and took a Berkeley course in the fall. And this is what he had to say on the discussion forum about the green tick mark. Okay, oh God, have I, have I missed you? Okay, so this whole gamification thing where people are learning by trying and getting positive feedback is really helping tremendously. I won't talk about discussion forums. You heard Daphne talk about it. This is a big part of learning experience where students help each other and learn by teaching. One of the interesting aspects of the edX platform is many of our courses have a virtual lab where people can design and construct things. Run that very quickly, please. So your students get a bunch of components like Legos on the right-hand side. They can do a design and they can submit the design to get that checked out and grade that automatically. So all of this can be done by a computer. It really helps the learning outcomes. We also introduced what we call AI assessment. It's a great grading system for essays and so on based on machine learning. So we have three kinds, self-assessment, peer grading and AI grading. So this is based on machine learning and it's pretty remarkable what sophisticated technology can do. So a lot of this kind of machine learning research happening around the world, a lot of it at C-Sale, computer science and AI lab here. So this just shows you a different set of essay types. This is the Hewlett data set. And here's a percentage error between an instructor grading it and another instructor grading it or a machine grading it. So blue curve here is the variation between instructor to instructor and the red curve is instructor to machine. And so what you can see is that machine scan grade essays reasonably well. Okay, there's a training process where the professor grades the first 100 and then after that the machine can grade these essays very quickly. So we can do various forms of free form content, short answer, essays and so on. So this is also a really cool piece of technology. And finally at the end of it, as teachers and educators, people come and say, why are you doing this? It's a non-profit, it's a really hard work to create a course. So people say to why are professors doing it? Why are people doing it? At the end of the day, it's comments like this where people around the world are telling us that this has changed their lives. They've gotten access to some very high quality courses and really learned something that they could not have done before. So he has a quote from a student from Pakistan talking about what an experience he had. So these statements like this and the way we can improve education on campuses like the San Jose experience really make all the effort that goes into this well worthwhile. Thank you. I'm happy to say as the third speaker that I am the least technologically enabled speaker today. I don't think anything can go wrong other than what I might say that I wish I'd said more articulately. So hopefully it'll go smoothly from here on in. For those who came in late, I'll just say a thing or two about what I'm doing on the panel and why my level of commentary might be a little bit different. I'm not a practitioner who has taught a MOOC. I'm not someone who has been as closely involved as Daphne and Anant have. I'm an educator like many people in the room. I'm someone who has an administrative background. I've been provost at Middlebury College and as David mentioned, I'm going to be assuming the presidency of Lafayette College shortly. And like many people in higher ed, I've been giving a lot of thought to what MOOCs mean for our understanding of education, for what we do as institutions, for what we do as faculty in the classrooms, for what it means in the general landscape of higher ed to think about this revolutionary new perspective on teaching. Those of you who have followed some of the commentary in places like the Chronicle of Higher Ed and certainly in places like the New York Times know that there's been a lot of discussion and a lot of commentary about MOOCs and what they mean for education as an enterprise. It's interesting that it seems impossible for people to even think about what MOOCs mean without turning them either into the savior of education or the destroyer of education as we know it. There doesn't seem to be a middle ground. I'm hoping some of our discussion today can help sketch out that middle place. What I'd like to do is just talk briefly about MOOCs from a kind of institutional educational perspective. What do they teach us about our own enterprise in the classroom, whether we use technology or not, and what we can learn from that experience as we start thinking about how to or whether to incorporate that kind of teaching into the institutional structures of colleges and universities that have not yet jumped into that space. I'm gonna organize my comments around three primary questions. The first is how do we account for the extraordinary interest that has been generated by MOOCs? Why have they come to sort of symbolize higher education and the promise of innovation in higher education today? The second question is what does it mean to talk about a digital classroom? Why do we use the classroom as a metaphor for the learning experience? What are the shapes of those kinds of experiences and to what degree do they or don't they require structures like classrooms or like colleges and universities in order to create an experience for students? And the third question is what is the relation between a MOOC and the curriculum that a college or university like MIT or like Middlebury or Lafayette or like Stanford, what is the relationship between the MOOC and the rest of the curriculum? How do these kinds of courses help us understand what is happening elsewhere in the university? What can we learn from them? But also just from a kind of concrete institutional governance perspective, who controls that level of curriculum? What input do faculty have into the question of what these sorts of experiences mean in technical terms like credit, like degrees, like teaching loads, like the things that faculty ultimately have to contend with. Those are less interesting and less inspiring questions than the ones that the other speakers have addressed. But I think at some point we run up against those questions and we have to confront them. To start with the question that I led with, how do we account for the extraordinary interest generated by MOOCs? I think anyone who follows the high red landscape would recognize that there has been no bigger story in, I don't know how long, in decades. You cannot open an issue of anything education related without articles on the subject. And it's extraordinary the interest that it's generated in the popular press. I think, in many ways, MOOCs have a kind of symbolic or metaphoric dimension that makes them a wonderful space to talk about a lot of the interesting and potentially threatening things that we see in the high red landscape now. They've become a kind of flash point for discussion of higher ed because they represent, in many ways, a kind of easily graspable, almost parodic version of what everybody wishes they knew more about, which is elite higher education. Those who have not had the privilege of attending MIT or Stanford have a wonderful opportunity either to access that material by becoming students in these courses or to comment on what those universities are doing and what they're offering by looking at the MOOC landscape and thinking of that as a kind of picture, as a kind of representation of higher ed. I think, in many ways, colleges and universities that have been quick to offer MOOCs have seen it as almost a kind of inoculation against the threat of technology getting ahead of them. It's a way of saying, we're not afraid of technology. We can do that, too. We're not going to be undermined by the ways in which education is being fundamentally transformed. And I think listening to the two presentations today makes it clear that these are enormous changes in the way in which we approach teaching and the way in which we approach our understanding of the kind of learning experience we try and offer students. And so the kinds of changes that are being enacted within that space have tremendous implications for what any faculty member does in any college or university. I think even if you are not using any technology, even if you are not planning to offer a MOOC, I don't think that the role that you play in the classroom will be untouched by the experiment that's being described here. And one of the reasons I say that is if you think about the way in which the way we talk about education has already started to shift in the wake of these discussions. The way in which terms like student and course now means something slightly different from what they meant when a student was someone who was formally enrolled in the class for credit within a larger institutional structure. I think the fact that edX talks often about learners rather than students is a way of reflecting that difference. But I think by and large in the popular press, the word student is starting to mean anyone who learns anything at any time. And it's wonderful that we are all lifelong learners who don't limit our learning to our undergraduate educations. But there is a sense in which the term student normally reflects a kind of specific relationship between a teacher and a person who has come to that university or college to learn that in some ways has the potential to be deluded or at least redefined in the wake of this kind of discussion. Similarly, thinking of a course as a set of lectures or as a body of material rather than something that is designated within a specific university curriculum for a specific goal changes our understanding of what those structures mean and how necessary they might be. And so there's a lot that's very exciting about the way in which the discussion is blurring the boundary between formal education and informal learning experiences. But I think in the wake of the excitement of this first year or two of the experiment, there will come moments when we will have to confront questions like does a faculty member get additional credit for teaching a course like this? Does the number of students who are signed up for a course like this have some effect on the way in which we understand its place within the structure? They're not exciting questions. Maybe only administrators like myself care about those questions. But in the end, they do come back to a fundamental truth about the educational experience, which is that up until now we have defined it as something that a student enrolls in or signs up for with a particular structure or particular purpose in mind. And the wonderful openness of this set of opportunities changes our understanding of the need for that structure. This is why I think even the title of this particular session, which was, I think, David's idea, MOOCs and the Emerging Digital Classroom, is interesting because we keep turning to the spatial metaphors like classroom to try and describe what that learning experience is. How do you define the space in which Anand or Daphne is interacting with students across many continents, across many different time zones, in an enormously broad environment of learning? What does it mean to say we'd like to find a way of describing that as a kind of classroom? I think that even that terminology recognizes that sense of structure, that sense of signing on for particular experience for particular purpose and defining a kind of relationship between the teacher and the student is part of how we have always understood the learning experience. And that may shift. Listening to the descriptions of the way in which, for example, automated learning, automated grading can provide data about learning experiences that it's much harder to accumulate over time in the classroom. That offers wonderful opportunities for thinking of how we might refine the kinds of pedagogies that we use. At the same time, I think it's important that we not lose sight of the way in which the kind of structural elements of an education have always been critical to our understanding of what happens within a particular class. And this leads me finally to my third point, which is, I think the time has come to really think about the relationship between MOOCs and the conventional curriculum of a college or university that is offering them. I think that they've been treated almost as performances or publicity stunts in the press. They've been taken very seriously by those who are offering them. They've been talked about in an almost theatrical way in the popular media. I think the point has come that we have to think very seriously about how we incorporate them into our structures of university and college governance, how we think about what designates a particular course as part of a curriculum or not part of a curriculum. It's important to recognize that when a student is given credit for a course, it's not because the course is valuable in the abstract because it's simply a good course. Usually, if we give a credit for a course, that suggests that that course fulfills certain requirements that are part of a certain structure, a certain degree, that are part of a particular set of goals that are appropriate to that student. The same course might not get credit in another place. It might not even get credit for another student because it doesn't count towards their major. The reason we have those structures, and I'd be the first to agree that many of them are antiquated and many of them could be more flexible than they are. But those structures do suggest that learning goals are important to the learning experience and that having some understanding of what the trajectory is, what the structure is that you're working within has in the past been seen as important, if not fundamental, to the learning experience. In places like residential colleges, the kind of academic experience that students are offered comes in tandem with a learning experience that takes place outside the classroom with a kind of 24-7 learning environment that's very different from what happens when you're simply interacting with a computer screen or even with a set of students within a particular class. And so one of the things I think we need to think about is how we distinguish between delivery systems and intellectual communities. I think it's already been demonstrated that many of these courses create intellectual communities as subsets of the course experience. So I don't see those two things as mutually exclusive, but I think they have to be understood as slightly different entities and that we have to recognize that the kind of community that we create within a university or a college is something that it might be a struggle to create in that kind of broader, more dispersed environment. I think finally that colleges that are offering courses that are largely initiated by individual faculty or that fit within particular parts of the curriculum have to be thought about in relation to the institution's mission. That for many institutions, and MIT and Stanford are great examples, these courses seem very consistent with the way in which they've defined their research mission over time. There are other colleges and universities for which those might be less consistent with their mission, whether the college has a liberal arts or a professional or a vocational orientation. I think it's important to think about how these things fit into the larger picture because fundamentally, though a well-endowed university like Harvard or MIT or Stanford can afford to put a few million dollars into developing this experiment, other schools will have to struggle to fit it within budgets that are already very constrained. As an administrator, it seems like a very short time ago that we were all recognizing in 2008, 2009 that none of us had enough money to do the things we were already doing. It's hard to imagine how you will carve out millions of dollars to invest in a project that doesn't serve the tuition-paying students and the donors who have given money to support the institutional mission as already defined. I think that means that for institutions that wanna move in this direction, you have to think concretely about how this fits into the goal of your institution. And it may be that for some institutions, it doesn't. There may be colleges and universities for which that model needs to be further developed before it's really applicable. There may be others that can move very quickly into this space and do extraordinary things that they haven't done before. But I think that the controversy surrounding how MOOCs will or will not reshape higher education, whether it's for better or whether it's for worse, all rest on the fundamental reality that they have made visible and dramatically visible and exciting, a kind of education that previously was seen only by a privileged few. And I think that makes for a very interesting conversation. Thank you. Daphne, can you hear us? Daphne, can you hear us? Yes, I can hear you. All right, we now, we have a packed audience, an overflow audience. I can see many people eager to speak. We're going to move immediately now to the audience participation segment of our discourse. And may I ask members of the audience to respect what the speakers respected, which was time limits. Let's try to get as many questions and comments in as we can in the next hour. There are microphones on either side of the space. And if you walk to the microphone, we will begin to call on you. Since no one has yet reached the microphone, I'll begin. My first question, perhaps the only question I'll ask if there is activity from the audience, is really a question that many of my colleagues and friends have mentioned to me over the week since we announced, months really, since we announced the formation of this MOOC. And it has to do with what, to put it in the simplest terms, the fate of second and third and fourth tier colleges. In your final remarks, Allison, you talked about the idea that many colleges might engage in this kind of project. But I wonder about whether or not the brand name attractions of Coursera, of Stanford, of Harvard, of MIT, of Duke will create a situation in which there are going to be these monopoly systems of the elite schools who will essentially drive the smaller schools into difficulties. How do the people who are, Daphne, why don't we start with you? I mean, how do you respond to complaints of that sort that I'm sure you've heard? Well, I think that, by the way, can you see me? I've put myself on video now. Okay, good. So, you know, okay, I can't share you though. Okay, yes, okay, now I can hear you. So let me try and answer that question. I think that regardless of whether we might think this is a good transition or a bad transition, what seems to be happening is that high quality content is rapidly becoming free and ubiquitous. And that means that any institution, no matter of which caliber and which size, is going to have to think carefully about the value proposition that it provides to its students and realize that it cannot be anymore about the conveyor of content. It has to be about something else. And I think accreditation is a very lame answer to what is the value proposition of an institution. It has to be about more than that. So I believe that what will happen is not that institutions will go out of business, at least not the ones that really grapple with this question and come up with a really meaningful value proposition, which can consist of the kind of blended learning experience that both Anant and I talked about, project-based learning, internships, peer-to-peer interaction, there's a whole range of possibilities of what an institution might do in order to provide value to its students. And I think institutions just need to engage with that and make sure that they do, in fact, do that. Sure, I think Daphne said it well. The way I would address that is that online learning, I think far from putting universities out of business, I think it's going to make them ever more popular. I think universities, whether at any part of the totem pole, are going to be turning students away because this is going to improve the quality of education dramatically. And what this is going to do is online learning is, I think, going to bridge the divide between universities that have little money and universities that may be very rich. And I think because very rich online content, this new textbook, for example, will be available to everybody. And as we improve the quality of education, students are going to say, look, it makes a lot of sense for me to go to these universities and get something out of it. So I think, I like to think of online learning as a rising tide that is going to lift all boats. And so I think everybody is going to do a lot better. In fact, universities today that may not be offering rich courses in a number of areas, I think can begin offering those courses because all of this content is going to be available to them in this new textbook form, if you will. Much as we did the San Jose experiment and teachers and professors will not be doing what they've been usually doing, which is purveyors of content. Standing on stage and spouting forth content is better done through video or through textbooks. I think as professors, we will all have to really think about what is the transformational value we are providing to students? Anybody can spout content. I think what we will have to do is begin thinking about how we enable students to think about that content, to process that information, and really teaching students how to learn, interact with students. I think that's what teaching should be all about. And I think this will take teaching and learning to a whole different plane in every university. Something to that, as someone who comes from a smaller institution, I think that a lot will depend on the structures whereby these kinds of content opportunities can be enabled in different institutions. I think that the fear that David described is that the smaller and less well endowed schools become simply outsourced camps where we offer versions of what the real faculty elsewhere are presenting for us. I think most faculty who go into College of University education see themselves as a purveyor of content as much as a kind of transmitter of content. I do think that all of us have probably experienced what I found in the last few years, which is the way we deal with content in the classroom is already quite different because it's so available in so many different ways that I don't think anybody who is paying attention teaches the same way they taught 10 or 15 years ago because so much information is already available to students outside the classroom. But I do think to go back to Daphne's point about high quality content now being ubiquitous, I think colleges don't sell the content, I think they sell the interaction and the ability to do something with that content. And so it all will rest on whether the primary site of that interaction is the you and 150,000 students and the machine grading and whatever sense of personality you're able to convey through that large format or whether it takes someone on site to work individually with students to do that. And I guess the final point that I would say is another thing that distinguishes colleges and universities is that you have a defined audience of similar enough abilities that you tailor what you're doing to the people that you're addressing. I think working with a smaller group of students in a classroom is simply different from trying to address the more diverse group of learners that you would have online. And it's hard to imagine how those things won't continue to be differentiated. We turn to the audience now. I would be grateful if you would identify yourselves when you speak for the historical record that we keep. Over here. Hiya, thank you for the opportunity. A very exciting panel. My name is Martin, but I finished at MIT in 2000 and we co-founded an entity called AITI that's currently under MISTI. And then early in the decade, I spent some time working at Harvard in the treasury function. So I've seen all these different sides. My question to all three on the panel, if you had the year of a president right now, somewhere in Africa, if you had the year of a central banker or a minister of finance, and finally, if you had the year of entrepreneurs and innovators, MOOCs have fundamentally transformed the landscape and pieces that are in the Economist or in the MIT Tech Review are discussing MOOCs. What would you tell them? What two or three things would you tell them? But most importantly, and I like to announce, you had that slide on Nigeria for the innovators and the entrepreneurs on the ground and potential private equity investors. What's the one line you'd mention to them in addition to the things you tell these three players? Thank you. One idea, one idea, which is post offices. I was at Davos at the World Economic Forum in January and I spoke to a leader from a young entrepreneur leader from Africa. And we brainstormed about how online learning can be brought to Africa and how can access be increased in a dramatic manner. And a couple of ideas came up. One is that in terms of infrastructure, one of the things that governments can do is given that this free content is not going to be available worldwide, governments can enable that by creating better communications and wireless and internet-style access in various places. Second thing governments can do or companies can do is provide rich, inexpensive devices to access the content. So as an example, India has taken a lead in that area. So India has produced a device called the Akash. How many people here have heard of Akash? It also happens to be my 18-year-old son's name, but so Akash means sky in Hindi. And they built this little tablet costing about 40 bucks and they're making it available broadly to students all over the world. So if I was a government, I would invest in devices and infrastructure to access all of these things, that's one. And the second thing I would do is for entrepreneurs, think of existing infrastructures and see how you can repurpose it. So when I was talking to this entrepreneur at WEF, they talked about all these post offices all over Africa that really don't serve a major function anymore in the whole age of email and Twitter and things like that. And so they said, maybe we can think about turning all the post offices everywhere into online community centers that people can gather and take and gather and do these small group discussion free course kind of setting. So I think there's a lot of opportunity. You just have to think creatively. I think Anand summarized it really well and I agree with everything that Anand said. I would like to present the sort of converse side to that coin, which is what governments ought not to do. So going back to the India example, India has the goal of increasing its tertiary completion rate from 13 to 20%. And they, at some point in time, I think they've fortunately come away from that. We're planning to achieve that goal by building 1,500 new universities in India. Now aside from the logistical nightmare of building 1,500 new campuses, part of the problem is that even the existing universities in India, once you move down below the top tier of IITs and IIMs are already understaffed. Many of them are at 50% capacity in terms of the instructors that they have. So where on earth are you going to find instructors to staff 1,500 new universities? So I think that what countries in the developing world ought to do aided by governments, entrepreneurs and so on, is to leapfrog brick and mortar in the same way that they leapfrog landlines and move directly to cell phones so that they can now basically build up an enormous amount of capacity using high quality online content augmented by local help, whether it in post offices or coffee shops or even US embassies in order to help bring people together to a place where there is broadband connectivity, high quality computing equipment and as well as just a safe place to learn together. And I think that's going to be a way for these countries to build up educational capacity which they desperately need in order to bring their workforce into the 21st century. And it's a much more successful way than pretty much any possible mechanism of capacity building that anyone has proposed. In fact, I would argue that for many of these countries it's the only feasible way to build capacity in a short amount of time. I would just add to that that one of the things that I think is noticeable about the way in which MOOCs have developed in the United States is that the excitement more or less parallels the decline in public funding for higher education, that the sense of wanting to privatize something that previously had been seen as a government responsibility is hard to miss. And so I really like the question addressed to, I like the idea of a question addressed to how can governments help to support whatever opportunities MOOCs can offer rather than simply assuming that now we don't even have to educate anyone in our country because they can just phone the United States for an education. I don't think that's the model we would want to see. My name is Chris Peterson. I am a graduate student in the Comparative Media Cities Program and a MIT admissions officer. So thanks, Anant, for a 6-002. I wanted to ask a question actually directed mostly at Anant about this though because you brought up the idea of MIT hard for these classes, which is something that I think is really wonderful about MITx, that it offers an MIT quality and MIT rigor to students all over the world. One of the problems with MIT hard though is that you have to be pretty smart to be able to handle MIT hard already. And there's a pipeline development problem with getting students to the level where they can jump in in 1801x and handle MIT hard. One of the things I was wondering is to what extent edX in particular, because I'm provincial, has thought about building out classes to help develop students from a secondary school level or even from other levels and a gentler on-ramp to the point where they can get to MIT hard activity? First of all, let me point out that the kind of courses a university puts up really depends on the university. And so I think for a number of universities on edX, the mission is to really provide courses of the same rigor as campus courses and that is advertised as the North Water Down. These are campus rigor courses. There might be other universities that choose not to do that, but it's really their call. Although edX, we really ask people to think of courses that are of the same rigor as campus courses. In terms of the pipeline problem, which is the funnel, how do you get people to that level? It doesn't help if you have someone take a cold shower with a hard course. If you haven't given them an on-ramp mechanism to build up to that level of competence. So just an example of 6.002X, it's a sophomore junior level course, but the number of courses now that have gone to the freshman level and so the freshman level course is being offered and I think a number of universities have talked about other courses that are much more accessible that can form what you call on-ramp courses and people are talking about on-ramp courses. And I think this will happen over time, but you're absolutely right. We really have to build a pipeline and start early. You just cannot give people a cold water shower late in the game. Add to that. David? Yes, we hear you. Okay, so I'd like to say that one of the lessons that we learned and we also encourage our institutions to provide courses that are at the level of rigor and academic standards they would like to see at their own institutions. And one of the things that we have found is that in fact, it is not the case that a Princeton or Stanford level course or a Caltech level course is necessarily appropriate for all students in the United States or outside the United States. And so for example, in order, and frankly, even if they wanted to offer a course that they thought was geared to students coming in with a lot less background, it's not clear that they would even know how to teach a course like that because all of us who teach at institutions like Stanford or MIT know how to teach Stanford and MIT students. So one of the things that we've started to do, for example, as part of the Gates Foundation that funded a bunch of courses back in the fall that were gateway developmental courses and the general education courses is to offer courses from a broader range of institutions. So one of the developmental English courses that's coming up on the platform very soon is from Mount San Jacinto Community College in California. And with the reason we elected to have that course come from Mount San Jacinto is because none of our institutions could possibly offer a well-designed, I think, developmental English class because that's not the population that they teach. And I think it's important to have a broader range of institutions represented so that we can teach a broader population of students with different levels of incoming background and ability. So we can create a funnel. Hello, my name is Eva Rios-Lalvarado. I'm attending Simmons College and I'm a library student. And I think, well, from a library student perspective, this is very interesting because either it'll scare the bejesus out of you or it's very exciting and for me it's very exciting. So I guess my question to the panel is how do librarians and considering partnerships on campus and departments, how do librarians support a MOOC's environment to faculty and students? Perhaps you would talk about your experience, how your... Sure, so I think anyone who's a purveyor of knowledge can really play a big part of this and really MOOCs in particular have served as a lightning rod, if you will, for people to re-examine what they've been doing in these areas and say, how can we do better? So one way in which libraries and some of our university partners have been helping is that in a normal course, you can use any material under fair use and you can apply that to your course. Well, when you teach a MOOC course, it's the fair use hasn't been clearly articulated yet. So one way in which libraries have been helping is to help professors collect material that and work with them to work with copyright clearance and things like that. Also, maybe for libraries to have sections with open content, open source content, if you will, available and make it very clear what that content is so people can use that in MOOCs very freely. So that's one example. And so there are other areas where the help professors find content of various types, digitize the content, maybe move into the electronic age and provide content in various media forms. And so, but that would mean going beyond what libraries do today, but become more media centric and media savvy and offer content in different forms and help professors who in the past would go grab a book or go grab a piece of paper, or here help them with media content and so on. So that's one example. I can think of many more, but that's one example. Basically, what an onset. If I could just add something. I think one of the things that is most noticeable in the role of libraries of the last decade or so is the way in which the boundary between faculty, member and librarian has become blurrier as we have become more like what you're describing as content providers. The role of organizing information and presenting it is starting to look more like a faculty role and less like a library role and librarians are starting to function more like faculty and instructors in many institutions. And so I do think that when you look at what's happening in areas like digital humanities and the key role that library staff are playing in many of those places, my guess would be that that kind of question will be an important question in the future. My name is Sam James. I'm a junior here studying material science and engineering. So my question is that I've noticed that all the MOOCs that have kind of been created so far, they sort of offer sort of the same sort of learning style where the instructor picks like the sequence of the course, similar to what goes on in the regular classroom. And my question is, what sort of ways do you see digital education, online education, being able to create new learning structures like guided learning pathways or really structures where the student has more of an ability to explore the content in the order that best fits his or her needs? So let me take a stab at that. First of all, let me explain why it is that we're doing it this way. We actually started out with models that allowed a graph of dependencies which students could explore in different trajectories. We found that that was actually cognitively more difficult for the instructor to prepare because they needed to think about all the different paths that a student could take to get to a particular point in the course. And so it was actually harder for them. And because it's already difficult to prepare an online course, removing one source of cognitive load just makes the preparation process a little bit easier for the instructor. But I think that's one of the arguments. The other argument which I think is even more fundamental is the community that gets built around the course materials. That is because we cohort the students and they're all going through the same material at the same time approximately. It means that when a student asks a question on the forum, other students are cognitively there and are prepared to answer that question. Whereas if they were in a different place in the course, they probably wouldn't bother quite nearly as much. And because what we're doing in these MOOCs is using peer-to-peer interaction to a large extent to substitute for faculty involvement which just doesn't scale to that kind of scope of number of students, then this is a way for us to ensure that there is some level of interactivity. So I think that's sort of the arguments that speak on the other side of this, which is not to argue that allowing students to explore things that their own trajectory is a bad idea, but it just to sort of explain what are some of the issues that arise when you do that. I do think that for certain topics, it's not that difficult to provide alternative paths if you could find a way of addressing some of the issues that I mentioned. We already see for example in solutions such as Newton for basic math, as well as in Khan Academy that people are allowed and even encouraged to go in one direction versus another based on what they know and what they're missing. So this is what the whole framework of adaptive learning is about. I think there is an open question of the extent to which one can reasonably apply adaptive learning in a broad range of disciplines beyond basic math and maybe basic physics, basic chemistry. So can you really extend that? And I think that's a fascinating research question. I can add some more. Just to add a little more. I was talking about Sal Khan a few weeks ago. And as Daphne said, they do provide students with sort of a concept map. And a young learner comes in, I'm a 12 year old, I come in and I see this big ball of wax, if you will. So, okay, where do I start? Where do I end? And it can be complicated for a student. And so what many students are telling us is that you are the professor. Why don't you tell us where to start and where to end? We don't know how to do this. So tell us where to begin and show us a path and give us a road that we can follow and get to an end point, which if we follow, good things will happen. Now, that said, a professor can provide a guided path, but that said, even with the guided path, what we're seeing students do is that, first of all, they by and large follow the path. We found that if you put something in the sequence, so we have the concept of learning sequences, if something is in the learning sequence and sequence from beginning to end, students are much, much more highly likely to access that content than some of the content that is not on the beaten track, so to speak. But that said, students are accessing other content as part of each video we link to a textbook. There's a discussion forum link there. There's other resources linked. So people can explore around, but by and large, you're finding that students are following the beaten track and they seem to like it. But I think the longer term, you know, some form of sequential, you know, sort of a linear path, but allowing students to explore in an adaptive manner off the path and coming back to it might be the way to go. Hi, I'm Nicole Yankelovich. I have a educational technology startup. I was curious in the diagram that you showed of 150,000 or so students starting with 7,000 or so completing. What ideas do you have for increasing those completion rates? So first of all, you know, first of all, I want to point out that I think it's important to understand what those completion rates mean. And rather than saying, oh my God, it's a disaster. How do we improve it? If you understand what it is, it's actually pretty darn good. So first of all, if you look at 155,000 students who started out and 7,200 completing, that's 5%. But then I define the active learner as a learner that clicked on a meaningful exercise or a problem. That is 26,000. So look at 7,000 as a function of 26, that's 30%. So right there, it's not great, but it's not embarrassing either. And then you realize that of these 26,000, they have not passed an admissions test. Anybody can do it. So now just imagine, this is huge. We are telling people that I'm not gonna judge you based on what you know and whether your parents could afford to send you to SAT school, but anybody can come in, the doors are open. If you can master the material, we will give you a shot at graduation. And I can also go into how, from 26,000 to 7,200, you can improve things. So for example, in the blended class of San Jose State, the availability of a professor on the ground to answer questions improved rates to 91%. Second is students are motivated, right now the courses are free. Now some people have argued that we should be charging students because once you have some skin in the game, you feel motivated to complete. Although I do pay for my healthcare membership, health club membership and I don't go to the health club. I just feel guilty and bad about it. But that said, maybe the MOOCs are free, but maybe charging students a small fee might do them good. Maybe they'll complete the course because they put some money into it. So there are a number of ways that we can think about to improve it. But I think fundamentally, 30% out of 26,000 isn't bad. I add to that. So I think our funnel is in many ways similar to Anans. For our courses, if people submitted the first meaningful exercise, peer graded or computer program or anything like that, our completion rates are actually 40 to 45%, which is really not very bad. And talking to Anans' point about skin in the game, I mentioned in my talk, the signature trap that we recently put into place. This is where students pay a very small amount, about two weeks into the course, for the possibility of earning a verified, identity verified certificate. They basically pay for the identity verification as they submit the work. Our completion rate for those courses is something like 75%. So that's actually a pretty healthy completion rate. Now you might say it's because these were committed learners to begin with or because they had some skin in the game. We don't know the reason for that, but 75% is not bad. That having been said, I'd like to sort of amplify on Anans' point about the fact that anybody can do this. And the phrase that I like to use in this context is the notion of risk-free exploration. If you're somebody who doesn't know like an incoming college freshman, you don't know what you want to do. You don't know what's interesting to you. You just know that you'd like to study maybe something in the STEM discipline, but you have no idea of what. Or you've heard about psychology, but you have no idea what it really is in a way that does not involve a huge amount of college tuition and a potential failure on your transcript. At the same time, we hear a lot, including in a New York Times article recently about the so-called under-matching problem where students, especially those from disadvantaged minority groups, feel like they are unable and don't even think about highly selective institutions like Princeton or Stanford or MIT as a possibility for them to attend. And by taking some of these courses online in a way that doesn't really cost them anything and doesn't, again, imply a failure if they drop out, I think there is the possibility that they will see that they're able to succeed in some of these courses and apply it to some of these institutions and avoid the under-matching problem that really prevents many students from this disadvantaged background in succeeding in life. Which side are we on over here? Hi, my name is Daniel Adzit. I'm a fellow in the systems design management program here at MIT. And it seems that compared to traditional classroom courses, online courses at the moment seem to have a stigma to them. And based on some of the information that we've heard tonight about some of the technologies that are available through online courses and some of the technologies being integrated into these platforms, how do you feel like there's an opportunity to enhance and transform the actual experience in the classroom by redefining them using some of these technologies for students who are in the classroom who don't have the opportunity today, you know, to do multiple problems and get feedback through them? How do you feel like, in addition to providing access to people through these platforms, we can also redefine and come to some sort of a convergence where we've got a classroom both virtually and on campus that are providing a unified kind of course work to them? I'll start with that, even though I'm less expert than either of the other two in looking at the kind of data that has been generated, I can say what we would want to learn from these kinds of experiments and how we might apply them to the classroom. I'd be interested in knowing, for example, whether it's possible to track how highly the level of interactivity that students display in being the peer commenters on each other's work, how highly that correlates with their success. You probably know that information. That would be something that would be worth knowing. If I were someone who was teaching a course that was similar in content to the courses being taught, it would be wonderful to have a lot of, to have those grading rubrics publicized and to have the metrics that arise out of that experiment publicized so that you can see what were the most successful kinds of problem sets, what were the most successful kinds of discussions, what were the most successful pieces of the course that worked most effectively for a large number of students. I think it was Al Filri's in relation to Coursera course who just said, in terms of the number of, in terms of the number of respondents to a survey, it's just more information about teaching than you get when you get 25 course evaluations. If you have more than 100,000 students responding one way or the other, that's just a larger kind of sample size. And so it does seem as if there's a lot of information to be gathered there. The question is whether we can really apply that to other kinds of classroom interactions. And a lot of that might depend on how we absorb that into the kinds of curricular structures I was talking about. Just an enhance on that, we're getting all of that information. There were even some slides that I skipped that really do this kind of very large data analytics on courses. And you can see, for example, as you teach a certain amount of content where people drop off and don't re-engage. And that tells you that maybe didn't explain that particularly well. You can see that certain exercises or assignments, people consistently get wrong. And maybe that means that you didn't convey the concept or maybe it means that the assignment was ill-designed. You can learn a tremendous amount from that that goes back into the classroom. You can learn a lot about just what works and what doesn't even in the online format itself. So for example, we have methods of assessing peer-grading rubrics to see which ones are successful and which ones aren't. And there's metrics for evaluating that. Finally, I'd like to point out that pretty much every single one of our instructors, and we now have over 150 who have completed courses, has told us that two lessons. First of all, it was a lot more work than they ever expected it to be. And second, what they learned from teaching this class has completely transformed the way they teach their on-campus classes, whether they end up using the content in their on-campus class or not, it completely revolutionized the way they teach. So one example of that is our sociology instructor, Professor Mitch DeNier, who taught his class an introduction to sociology to a global audience. And every week he had this discussion section where 10 students from around the world would talk about online in a video chat room about the sociology question of that week. And he says that he learned more from teaching that class in this global format in one week than he'd learned in his 12 years at Princeton. Because at Princeton, every year there is a great discussion, but it's a very similar discussion from year to year because the students are very similar in background. And here, the range of perspectives, the wealth of opinions that he got from students completely transformed the way he views his own discipline. Are you at MIT? So I'll give you some statistics. I think to your point about how we bring this technology back on campus, how do we improve and apply this back on campus? I believe the number is we have 6,000 undergraduate students at MIT, or five to 6,000 students. Sanjay Sarma presented a statistic that said that at this point, close to 1,500 students are using the edX platform in one form or another on campus. So for instance, many professors are teaching in a normal manner using a lecture hall and so on, but they're giving students exercises and problem sets using the edX platform so that students can get instant feedback. And the instant feedback is one of the most popular aspects of online learning that students care about. You saw the feedback on the green check mark. And so that can be brought on campus. In other examples, people have flipped their classroom. They've stopped lecturing and they have people watch the videos, but they spend their time in classrooms in much more of a blended interactive format. So that's another example. In the third example, professors are using the platform to do experiments with simulations and virtual labs and so on, so students can try out a lot more experiments than they could in a classroom. So there's a fourth example. And here's what I claim that online learning can be better than real life. So in Eric Landers biology class, for example, students are able to play with a molecule builder. So they can actually play around with that and build things that they could not do in real life. So I think the number of ways in which we can dramatically enhance what is happening on our campuses using computing technologies. Hi, I'm Scott Osterweil. I'm a research director here in comparative media studies. And I'm designing what may end up being our first online humanities class here at MIT. We're just starting that. And I want to introduce some optimism and pessimism into it. I'm very much a big fan of the flipped classroom. And I'm hoping that we'll be working much more toward a model of the course delivering, providing content which is then used in an apprenticeship or at a conversation in onsite wherever the course is offered, where the quality of the interaction among the participants is really what matters more. And so I'm optimistic that that might even be a process by which we can improve the pedagogy onsite, that the teacher who's doing the online course gradually develops new practices and becomes a better teacher over time. But my pessimism, my fear is that the distance sites will see this as an excuse to have even less well-paid, less well-trained instructors that they'll see it as a way to sort of warehouse students. And I guess my question is, how do we as providers prevent that future from occurring? Just a footnote to that. I mean, that seems to me, that's part of the pessimism that I was hoping we would talk about. It's very hard to understand, just to reinforce the question. Why if students in the country have access to Harvard, MIT, and so on, some of the smaller colleges would not be transformed essentially into colleges that were essentially teaching assistants? I actually don't believe in that at all. When textbooks first came out, just think of textbooks. Before textbooks, the teacher would spend most of the time memorizing stuff and explaining, describing that to students. And there was one professor, and this was before 1400, they would have a few three or four or five apprentices. When the textbook came out, the same thing was said about the textbook. Oh my god, now this professor has put this thing down in a textbook. We have finished. OK, that's it. You just need one chemistry professor in the whole world. What happened instead? We got 10,000 more chemistry professors in the world because they were all enabled. Previously, I just needed to have professors who could memorize a bunch of stuff, but even if they're great teachers, they didn't help them to memorize stuff. I would be a dead man. I can't remember. I can't string together a sentence from memory. So I think what will happen is that online learning is the new textbook. What this will do is I think just as the tech coming out of the textbook enabled the whole new mushrooming of the industry, whole new cadre of teachers, people who may not remember due to spout or knowledge and write down complicated equations on the board, but who are phenomenally inspirational people who can inspire students to learn. I think we'll see a new generation, a new class of teachers spout in addition to all the ones who are around. And so I think this is a disruption. I think we are in for a really exciting ride as I think we'll have just a lot more great teachers who may not have been able to do this before. In fact, sorry, go ahead. Sorry, I was just going to add that, of course, one of the things that's most exciting about MOOCs is that they do dramatize good teaching in a way that was previously invisible, that it has the capacity to allow people to see what terrific lecturing or terrific interaction looks like. It does require that everybody up their game a little bit. And I do think one of the additional sort of pluses and minuses is that if you imagine a world in which a lot of people are using those materials on site, I think they're going to be looking to the models that are developed online and applying them to the classroom in ways that will be disruptive to people who don't alter their teaching styles. The example I was thinking of was the instant feedback. As someone who, as Provost read, roughly bazillions of course evaluations, I can tell you the number one complaint of students is work not being responded to quickly. And that complaint is not going to diminish when they're starting to get green checks within a 30-second time frame online. Now I think it would actually be great if people were faster at grading their work and returning it. But I do think that there's a sense in which the model and the speeding up and the pace and all of those things will transfer back into the classroom. And it will mean that even people who are not themselves teaching MOOCs or are not themselves using those will probably absorb a lot of those techniques. I think a lot of that will be positive, but a lot of it will be negative or at the very least to use your neutral word, disruptive. Yes, ma'am. So just to add to that, because I think this is one of the fundamental risks that people perceive here, I agree with Anant that we're not going to see a death of teaching. In fact, I think that the very notion that the kind of meaningful one-to-one interaction that a student has with an instructor is something that only happens with a quote-unquote teaching assistant, I think is in some way the meaning to the whole notion of teaching. I think what we're going to see is a return to teaching the way it ought to have been, the way it used to be before we needed to scale up with one professor and 300 students. So I think all of us, if we think back to our college career and we think what it is that, what were the pivotal events that really inspired us, that moved us to go in one direction versus another in terms of our career choices, very few of us are going to say, it is when I sat in that big classroom with 275 other people and listened to this person speak, they will talk about an interaction that you had with a coach, a small freshman seminar, an advanced undergraduate class with 12 people in the room, something that somebody said to us in office hours, it was a much more personal interaction that you had with a role model, with a mentor. And I think that what we're likely to see and what we're trying to do here is in fact open up more time for that kind of interaction because there's going to be less time that's spent on preparation of content for our rating and on grading of the same papers over and over again. And I think that's what we're trying to do is to create more of that interaction rather than us. Over here. My name is Jonathan Haber. I'm in the midst of a MOOC experiment myself. I'm attempting to take the equivalent of four years of undergraduate courses in 12 months in the humanities. So I've been exposed to a lot of MOOCs. I've been trying a lot of everybody's courses. And I guess so far the most, I'm about done with my freshman year and the most sort of glaring thing I've noticed so far is in those courses, like philosophy, like the humanities where there's requirement for some degree of subjective evaluation that seems sort of the weakest in any of the MOOC classes I've seen so far. Either it's machine scored assessments, which have to boil down sort of concepts into small enough units that is not thorough enough or it's peer graded assessments, which I think people know a lot of the challenge with those. So I guess I'm sort of thinking you had a year from now, two years from now. Where are you seeing the assessment component of MOOCs going and will they be able to do more or will we have to go more in the direction of breaking these classes into smaller units so that there could be more direct human grading? I could take a crack at that. I think, now first of all, if I just look back to my own experience over the past year, there were a number of things that when people asked me a question like this a year ago, I said, I have absolutely no idea. And today we've learned so much already. It is unbelievable. I think once really talented people around the world, you know, at Coursera, Udacity, edX, Khan Academy in a bunch of places, just a bunch of concerted effort of just great talented team members applying technology to education and learning in a concerted manner in a way that is really having the world pay attention, we've come dramatically far already. Today's online course is not your grandfather's online course. I think somebody asked a question here earlier, which is online courses have got a really bad rap. They did, but they haven't seen today's online course. I think today's online course is even a year into it, even today, a year from when we began, our course today or a Coursera course today is far different from what was through a year ago and certainly dramatically different from what was through even as much as two or three years ago. The differences have to do with greater assessment, or why are they better? So it's better in a number of dimensions, pedagogical, technological, so on and so forth. So for example, I'll give you one example. I'll go back to assessment, for instance, where we said, look, you can't have machines assess free-form content, can't have machines as great assets that have been greater as before and you can string together a bunch of random words and you can get there are examples of essays where some random words got a high score, but we've come a long way. So our AI assessor is really remarkable and I think there are others that can continue improving that. And a year ago, if you'd ask me could you do that, I would have said impossible, but we've done it. And so I think there are examples. I think pure grading is another example where for a class of problems, it can work really well, but it's not particularly when the good rubric is specified. Same thing for AI assessment, you need a reasonable rubric. Will we be able to do a lot better a year from now? I think combinations of techniques, I think better research, just better machine learning techniques, I think we will do a lot better a year from now than we are today. Mr. Adams is that there is a really beautiful blog by Clay Scherke that I highly recommend reading that talks about complaints that came about when people first started doing recorded music by LPs and there were all these people that came and said recorded music will never replace live music because look at all that quality issues that it's scratchy, it bounces. I mean, it's just not never gonna be the same. And after LPs, we had CDs and after CDs, we have MP3s and you could still argue that there's some subtle differences between listening to an MP3 on your earphones and going into a live symphony. And I agree that those differences are there, but it's certainly good enough to serve the needs of many, many more people in a much more economical and accessible way than going into a concert hall and paying $150 for an orchestra seat. So I think that the quality is a non-set is something that is constantly improving. What we see today is not what we saw a year ago and it's certainly not the online courses that have been with us for the last decade. I think in two years, three years, it'll look unrecognizable. And I think it's important to remember that even if the quality that we get is not going to be quite, using a peer grading, for example, will never be quite the same as what you get if you had the world's best instructor giving you meaningful detailed feedback say on your 10,000 word essay. I think we need to balance out the other benefits of this technology in terms of the accessibility, the scale, the cost and other things that I think trade off with a difference in quality that is rapidly shrinking over time. I'm Peter Refros, I'm the Chief Science of edX and I had a question. There are a lot of very bad things that I'd like to see MOOCs disrupt, bad instructors, educational debt, but there's one very fundamental value to academia which I'd like to see not disrupted and that's openness. And when MOOCs first came out, there was a potential to really have open educational resources which you could reuse, remix and so on and so forth. Instead I've seen the industry moving the other direction where we're starting to see things like closed courses on how to teach online, closed conferences on how to teach online. We're starting to get very basic optics into how the human brain works in pedagogy but that's being controlled by a small number of organizations, some for-profit, some non-for-profit and right now those are run by very good people but what can we do structurally to make sure that we don't end up in a world that's closed where basic information about how a brain works doesn't end up closed? How can we structure the Coursera's, the edX's, the Udacity's, the Khan Academy's so that this remains open and we maintain this very important value? So I think, you know, I think we're all about openness. I think with MOOCs, the courses we've made open and made freely available, edX has been built on a platform of open source and so we began open sourcing the platform. Content, honestly, when we began edX, it had been certainly a major mission of mine to make all the content open as well but sadly, I've not succeeded and I've got scars on my back from many university and professors who don't feel comfortable putting out courses that will then be remixed, if you will, by others. It's like, the same thing happened in the textbook world. The number of open textbook efforts where people try to remix textbooks and those efforts have not succeeded and so people have tried to build open resource, open content textbooks with remixing and it just hasn't happened. So I'm not sure I believe that a group of 30 people who are working together can create a better course than one or two dedicated people working with themselves. So I think the jury is out on what we can do with open content but personally, I really would like to see open content but I think we'll have to convince all our colleagues to do that. In terms of research and how the brain works and how people think and how people learn as well, keeping that open. Definitely, if you want to add something. Sure, I mean, I don't think that we have kept any research secret. We have been aiming to publish everything that we do in the scientific literature in the same way that one publishes all research and attempt to make it accessible. As Anand said, this is not our content. It's our colleagues' content. The people who write the courses, prepare the courses, it's their decision, whether they want to open that content up to others. I think people are to some extent rightfully afraid of having their content be arbitrarily remixed. We've all seen commercials around elections of somebody taking another politician's words out of context and putting them in an ad that makes it sound completely opposite to what the original intent was and I think people are afraid, especially when it comes to controversial topics, that if they do that, that their words will be taken out of context and misinterpreted. So again, I'm not saying that education ought to be closed and obviously our content is open and free and freely available. And that's the only way in which we're going to offer it. None of our content is paid and pay well anywhere in the entryway to any of our content. So I mean, I think it's as open as one can possibly hope for, given people's legitimate concerns about having their words taken out of context. If I could just add something as well, in addition to the concern about taking words out of context, I think going back to the question of how university and college structures account for this, in a world in which getting tenure requires a demonstration of expertise where your intellectual property is what you are being tenured for, your capacity to generate new classes or to publish material is what most universities and colleges will award you tenure for. I can certainly imagine that giving that content away for free will be less attractive to people who haven't yet gotten the institutional security that puts them in a position to sort of put that out in the world freely. And so I say that not at all because I'm opposed to the idea of open educational resources, but just recognizing that it has to be part of an ecosystem in the educational world in which there is a shared desire to make those resources public because a faculty member can't do it unless their own university or college is supporting that movement. I think the first step has been taken though. I'm very proud to say that Delft University came on board as an edX partner a few weeks ago. And to my knowledge, the first MOOC university that is gonna make available all the content as under the Creative Commons license so that anybody can take it and remix it. So may the force be with Delft. I encourage you all to vote up Delft in any ratings and so on. I think they're a path breaker here and good for them. We have seven minutes left. May I ask that we ask very quick questions and ask the panelists to respond with a special concession? I'll do my best. My name's Tom Pounds. I'm an alumnus of this loan school which will probably be further revealed in my question. I'm interested in business models. Coursera, as I understand it, a venture backed, Silicon Valley venture backed for profit entity. EdX is an experiment by two institutions that have $50 billion of assets. In your boardrooms, you must have different conversations about what your goals and your metrics of success are. Maybe could you open that box a little bit for us and help us understand how you define success in each of your organizations. And then Dr. Byerly, maybe you could comment on how you see the differences between those two approaches. Do you wanna start? Should I start? I don't care. I don't actually know that, and obviously I have not been in a non-sport room so I can't speak to what's said there, but I can certainly tell you that in our boardroom, the question that keeps getting asked is not how do we bring in money, but how do we educate people? We were very careful in selecting our investors so that they're very well aligned with our educational mission. Our lead investor, for example, is John Doar who sits on the president board of economic recovery, is on the board of multiple charter schools and was the first personal investor, together with his wife, Ann Doar, in the Khan Academy, the first donor that gave them their startup money, so to speak. So these are investors that are very much aligned with what we'd like to do, which is teach people. But I think that when we went into this, first of all, nobody gave us $60 million so we did not have a non-sportion in this respect. So we had to raise money from other sources. And I think that we knew, and I think Annan shares that view, that we had to be self-sustaining. Donor fatigue quickly sets in if you count on philanthropic foundations. So we are aiming to teach people and we're looking for revenue streams that will help make this sustainable without ever charging students for access to education. So, as we started edX, I've done five for-profit companies. I'm a serial entrepreneur. But in this case, I sort of felt that the structure of the organization as Harvard and MIT, we're brainstorming about what to do and MIT was thinking about how to structure this. We decided to make it a non-profit. And one of the, and the discussion in the boardroom, in a sense, reflect kind of that vision, which is that because I'm a serial entrepreneur, I tend to think of sustainability. And as Daphne said, we have to be self-sustaining. And so I tend to really think about, okay, we can do this and we can be self-sustaining in this period of time. And what is interesting is that my board often reminds me, and I might say wraps my knuckles sometimes, which is that Annan, remember, you're 10 years from now, you will be judged not based on how financially successful edX has been, but have you, and this is the reason why the founding partners, I might in Harvard put $60 million into it, is that have you been successful in reimagining education on our campuses? Have you changed education as we know it? Have you done research that completely transforms what we know about education? That is how we will be judged. That is how you will be judged. And I think that is the discussion that happens at the boardroom. And so that is one example of, I think how structures can change the way board members judge you and their limited partners judge those board members. And some of the limited partners for Harvard and MIT are the corporation members. And so really you have to go down the value chain of investments to see what people are saying. And certainly for us it is, Anand, have you done good today? So, well. Well, it's wonderful to hear that those are the discussions that lie behind those particular projects. I think recognizing that other schools don't have the same resources means that what people fear at other schools are a different kind of conversation that would be perfectly understandable in an environment where most schools are struggling with complaints about high tuition and limited revenue. That being able to say that you're willing to forego profits is not the same as being able to sustain losses. And I know that the institutions that I work with would probably find it hard to give up hundreds of thousands of millions of dollars for something that they see is very worthwhile. Partly because the students who pay tuition and the people who've donated money to the institution are donating for a particular purpose or paying for the tuition of their children, not the tuition of the world. So it's an educational process that clearly you've been successful with your own boards, whether everyone can expect to do the same, I don't know. And I think that does return us to the question of how this model transfers out or scales out to other less elite, less well endowed institutions. That the worst case scenario is pressure to do something that isn't affordable with less worthy goals at other institutions. Another model is what we discussed earlier, which is a kind of outsourcing of what a few institutions do to different places with different levels of interactivity. And I guess it remains to be seen, what'll happen. Final question. Thank you for the question. Dr. Eiff last week said we are in 195, sorry, edX is in 195 out of 195 countries, which is amazing. I know there are partner institutions across the world now, but I'm wondering what conversations are happening about the cultural influence that means we will be having across the world. Short question. So let me take that because one of the things that we have really tried to do is to expand in having an international footprint, not just on the student base, where we've been in all countries of the world for a very long time, but also in terms of the universities. So we now have 24 international universities, some of the top universities in Asia, in Europe, in Latin America, in Australia, our Coursera partners. And one of the reasons we did that is so as to open up the ability, not just to consume education, but also to produce education that is inspired and motivated by local cultures, local information, local knowledge. For example, I think that an institution in Africa, we don't have any yet, but we hope to, will be able to provide some really unique insights, for example, in tropical diseases that you might not get from anywhere else. And so we, as well as local perspectives on culture, local perspectives on global issues like sustainability and so on, that are really unique to different cultures. So we really hope to partner with some of the world's best universities, not just the ones in the United States or even Europe, in order to allow multiple cultures to present their perspectives. I think that's well said. I just thought an example. I think if this was just a US based enterprise, and where US universities teaching only in English, for example, that would have the drawback you mentioned, which is it would be viewed as trying to homogenize the world's culture. But what is amazing is that I've just been, I'm astounded every day when Australia National University came on board as an edX partner. They said that the second course that we're gonna put on would be, believe it or not, would be in Hindi and Sanskrit. So go figure. Did I need to say more? I'd like to thank the audience and especially thank the panelists. Daphne, wonderful remote.