 Well, good afternoon, everyone. I don't see many Argentinians or Swiss in the audience today. Some of you with laptops open. I kind of wonder if you have a split screen going to be cool. So yeah, the World Cup matches on right outside. But in any case, good afternoon. Or for those of you on our webcast of viewing if you happen to be in different time zones, good morning or good evening. My name is Andrew Ho, and I am a professor at Harvard's Graduate School of Education, just over there down on Appian Way. More relevantly for these purposes, I am the co-chair of the Harvard X Research Committee. And I co-chair with my colleague Dan Levy at Harvard's Kennedy School. As many of you know, edX, and we have some representatives from edX today, edX was launched in May of 2012, jointly by Harvard and MIT with great fanfare, and a tripartite mission. First, to increase access to educational opportunities. Second, to improve learning on campus for residential learning. And third, to advance research. In the fall of 2012, Alan Garber charged the Harvard, created the Harvard X Research Committee to advance this research on Harvard's campus and beyond. But I don't think we really got that much work done until Justin Reich joined us. So Justin is the Richard L. Menschel Harvard X Research Fellow, and he has really been the nexus of Harvard X Research since he joined us this last summer. Justin has been an author, a lead author or co-author on all of the Harvard X working papers. He has almost single-handedly led the survey effort. We actually have a Harvard X survey that I think is one of the most comprehensive currently exists, and he has just been a connector for researchers and faculty and ideas here at this university and beyond. So Justin really has been essential to our Harvard X research effort. He, of course, does many other things. He is a fellow for the Berkman Center for Internet and Society, and I'd like to thank the Berkman Center for hosting this luncheon today. And he has a blog at Education Week. He has a lively Twitter feed, and he really is just in this scene, not just at this university, but beyond. So we are very fortunate to have him as a research committee. And he also teaches at Harvard and MIT. And I should mention, too, that he is a father of two great kids who I often see playing at the playground by my house. Justin and I actually live two blocks from each other. But we have this pact to not discuss research when we're pushing our kids on the swings. But in any case, it is really my pleasure and my privilege to introduce Justin today. He'll speak for about 30 minutes, I believe. And then there will be what I hope is that what I'm sure will be a vigorous discussion. If you have questions, keep in mind that we are being recorded. And so either speak very loudly or ask for this microphone that I have right here. And we'll have some runners for you. But that's the sequence. I'm looking forward to the presentation and the conversation. And without further ado, the Richard Elmenschal, Harvard X Research Fellow, and Berkman Fellow, Justin Reich. Thanks, Andrew. That's great. That was, as they say, too kind. So thank you all for making the time to be here. If you're on the interweb somewhere, thanks for eating your lunch with us or watching the soccer game or whatever else. I'm reasonably good at the Twitter machine. And we'll try to keep track of it and other people in here, I think, as well. So if you tweet things out to hash Berkman, hopefully we can bring your questions into the conversation and have your thoughts connected there as well. Actually, according to the margin orders from Kerry, I'll try to speak for about 20 to 25 minutes. And then hopefully have plenty of time for discussion and conversation. I have like six slide decks of backup slides of all kinds of things that I wish I could show you if I talk for an hour and a half, but I will constrain myself to the best of my abilities. So maybe one thing you should know about me is I love education and I love teaching and learning. So I started teaching lifeguarding when I was in college and I was a camp counselor when I was 16 and I was a high school history teacher for a bunch of years. And so for me, helping human beings become better versions of themselves is the most fun, most fascinating, most extraordinary things that humans do, that make us humans. And so MOOCs are sort of a space where we think a lot about data and platforms and networks and routers and logins and browsers and all those kinds of things, but as much as possible, I try to keep in mind that what we're really trying to do is help human beings be better versions of themselves and develop capacities that they didn't have before. And so I try to have my work be inspired by that spirit. I'm gonna try to do two things with you all today. One is to give you a bit of an overview of sort of where I see the field right now, some of the things that we've accomplished and some of the limitations of the current crop of research that we have. And then try to point the direction to what I think can be more fruitful, more exciting, more engaging things that we can pursue in the future. Online learning is not a new field of research. We've been teaching kids over the web or people over the web for 20 plus years. We've been teaching people at a distance for decades and decades, but there is a lot that I think is new about these platforms and so it's a good moment to take stock of what we've been trying to do over the year and a half. If there's a single hypothesis that sort of animates what I have to say for the next 20 minutes, it's probably this, we have terabytes of data about what students click and much less understanding about what changes inside their heads. We know lots and lots about their behavior. We know lots and lots about their clickstream data. It's much, much harder to get a sense of how they're becoming different people because of the time that they invested in us and I would actually say in some ways it's almost the ease of access to the terabytes of data about what students click that in some ways, it certainly shapes, maybe I go further and say, corrupts, perverts, moves potentially off in the wrong direction of the kind of work that we do so I'll start by sort of making a kind of critique of four studies. They're all four studies that I think are fine contributions, great for the first years of this work. I've sort of selectively pulled out unfair examples that create a caricature of these four studies to give a sense of what I think happens when we spend a lot of time thinking about what students click and have less emphasis on what changes inside their heads. So here are four studies that I think give a sense of the kinds of things that I see come across my desk as I look at journals that come out or I get asked to review papers or whatever else. So we'll look at four different organizations. Here's one about Khan Academy, a study that was done by SRI this past year which is a great third-party research shop. Khan Academy isn't really a MOOC, it's sort of kissing cousins of a MOOC kind of. Most people know Khan Academy as a series of YouTube videos that this guy, Sal Khan, makes, but actually when it's implemented in schools, the videos are only watched maybe 15% of the time or less, about 85% of kids time is spent on these randomized worksheet problems sort of functioning as an intelligent tutor. Khan Academy has vast stores of data about how people are engaging with those problems. So there are billions of problems that have been completed, they have unique user IDs for all kinds of people, they know exactly how many problems you've answered, they know when you've answered them, they know whether you got them right, how many hints you asked for and what order you answered them, all those kinds of things. So we have these huge stores of big data and sort of one of the most common analytical moves that researchers have been taking is to take that enormous fine-grained granular real-time data and shrink it down into person-level summary statistics. So you sort of have access to all this data and then you sort of shrink it down so that you can summarize people with a single number. So in this particular study, they summarized the number of minutes that people spent on the platform in the fifth grade and sixth grade and the number of problem sets that they completed. And lo and behold, people who spent more minutes on Khan Academy got higher than predicted scores and people who spent less minutes on Khan Academy. Science. So another important piece about this study too is that the authors are very, very clear that you can't draw causal conclusions from these correlations. So it's entirely possible that people who have really great instruction outside of class, outside of the Khan Academy system, sit down in front of Khan Academy and go, oh, I just got some great instruction on this material. It's really fun to do these problems. I'll spend a bunch of minutes doing this and do a bunch of problem sets. And the people who got really lousy instruction on it go and sit down in front of Khan Academy and go, oh, this is kind of boring. Like, I don't want to click on this very long. And so in fact, the causal area might really run in the other direction that people who have high competence for one reason or another are more likely to spend more minutes clicking their way into the system. Here's another study that I think has another set of similar moves. So here's a study about from Google. So Google, a lot of data scientists and data analysis and so forth, they have a class that they offered on their Google course builder platform about mapping with Google that was broken up into these 10 sections, each one a skill for using sort of mapping features within Googles. And so this study looked at, as a predictor, the people who did or didn't do activities every week and then as the outcome, whether or not they did the final project. So one of the things that you'll see here is that the people who did the activities from week to week went on to do the final project and the people who didn't do the activities week from week did not go on to do the final project. We know this because they have both a figure and a table. So again, taking this sort of enormous fine grain data, boiling things down into this person level summary statistic using a pretty coarse outcome measure which in this case actually has nothing to do with learning. It's really just whether or not they did another kind of behavior on the system. Here's a third one. Here's from Udacity's, this is research done by on the San Jose State Udacity pilot where Udacity ran a series of courses for San Jose State in remedial math and statistics. This is a little bit more complicated so it's a little bit more of a regression model but again saying sort of the same kinds of things. On the y-axis is your probability of passing on the x-axis is the number of problems that you've completed. The different bars represent different courses or students in sort of different kinds of courses and what you see is that as students do more problems in the course, the probability that they pass the course continues to go up. Here's one sentence from that paper. The primary conclusion from the model in terms of importance to passing the course is that measures of student effort doing problems eclipse all other variables examined in the study including da-da-da-da. So a shorter version would be students who do things in class pass. Here's one that I did. I did it with a bunch of other people but I won't hold them accountable for that. I will take full responsibility for this. So in this version I don't even bother with a table or with any kind of regression modeling or things like that. I just say let's break people up into two groups. Those who earned a certificate in this case in a Hero's X course and those who didn't earn a certificate in Hero's X. And let's just count the number of times they press play on a video and aggregate that together. And sure enough if you look at the distribution of people who earned a certificate they clicked play more often than the distribution of people who didn't earn a certificate. And that if you want the kind of, I think of this as another one that I did, kind of the paradigmatic, summarizing these complex activities into this really, really simple person level summary statistic and then using this extremely course outcome variable. So we're just looking at the number of times you clicked on anything comparing certificate earners and non-certificate earners. And it turns out that people who earn a certificate click on things more than people who don't. So that's my contribution to the science of learning in MOOCs. One way you could summarize this is what I've characterized as Reich's law which is that the students who do stuff do more stuff and the students who do stuff do better than the students who don't do stuff. And like in some ways there's some other things that we've learned but boy there are an awful lot of studies that come to conclusions that are dangerously close to this one. So like what's the action item you're gonna take based on this? It's something like just make the students do more stuff. And you can't even quite do that for two reasons. One of the things that I alluded to before is that these are sort of correlations and not causal relationships. So one of the things we know about MOOCs is that there's an extraordinarily diverse population of people who come to these courses. Some people come knowing absolutely nothing about the subject and some people are already experts in the subject and they come to get a refresher or test their skills or do whatever else it is that they were doing. And so evidence that people who do stuff do more stuff and do better stuff could just be that the people who are already competent sit down in front of these systems and have a good time with these systems and click away and get little green check marks and little smiley faces and little dings and dongs and whatever other sort of Pavlovian rewards we offer them and they're perfectly happy to do that. And the students who walk away are the ones who sort of were low in competence to begin with. I think the other real danger is in the four studies that I've showed you I've told you almost nothing about the ways in which learning was actually measured. I've told you almost nothing about exactly what kinds of competencies were being measured or developed or improved. And I think that's not entirely uncommon to see studies with these really course measures of learning. There are lots of paradigmatic cases where people who do really well in measures like passing classes or grades or other kinds of things like that actually aren't developing the competencies that we hope they are. So I think one of the paradigmatic cases of this comes from physics. So about 25 years ago there were a bunch of physics educators who came up with this instrument called the Force Concept Inventory. And the Force Concept Inventory was a series of multiple choice questions that asked people about their conceptual understandings of force and Newtonian physics. And it turned out that people who did really, really well in all kinds of introductory physics classes at places like Harvard and high schools, people who would do well in the kind of plug and chug formula, algebra-driven final exams that you might get inside an introductory physics class, actually many of them saw no change at all in their force concept inventory. Basically, many introductory physics classes seemed to be really good at helping people sort of solve the formulas and do the equations of physics but not actually change the sort of underlying intuitions that we have about how physical forces work in our daily lives. Basically, physics is probably particularly acute in this problem and that we come to physics with a set of intuitions that we develop throughout our life. Heavier objects fall faster than lighter objects. Look, if I take this rock in this feather and drop it, the rock will fall faster. So the heavier object must be falling faster. Those kinds of incorrect intuitions are often preserved entirely through first year physics courses because oftentimes it's really hard to measure the things that we care most about in courses. So if I were to stop here, my sort of admonition to you would be that as you're looking at MOOC research that gets reported or comes off the shelves or things like that, I'd encourage you to be really attentive to asking questions about what kind of learning is being measured here. To what extent is this sort of user behavior, user interaction research versus research that really contributes to some kind of science of learning? And we seem to be doing much, much more of the former than the latter. And I will say, I guess in critiquing these four studies, including my own, again, I don't mean to say that they're not useful contributions in the sort of first year of messing around with these systems. Andrew can tell you and other folks who have messed around with this data that it's hard to make sense of, it's hard to figure out how to use the data, it's hard to use the right questions. I think there's a bunch of sort of features in the sociology of the field that are driving some of these phenomenon. There's enormous pressure than their young, unemployed, untenured researchers to crank out papers that can get out in the field quickly. There's a huge first mover advantage in terms of citations and things like that to be the first writer about a topic. But what that means is that we may be sort of aiming for the easy things to write about and publish about rather than some of the more important questions that could really, you know, I'm not sure any of these studies give us a whole lot of guidance on how do we make courses that help people be better versions of themselves. So if I were in charge and in my own practice, these are some of the, you know, I could probably just say this as things that I'm challenging myself to do and imperfectly following through on. As I think about the kinds of research that I think would advance the science of learning, there are three characteristics that I think we could pay more attention to to try to advance that and certainly some of this is happening and I'll mention some of those studies along the way. So the three pieces would be to measure the most important competencies, the things that we care most about students developing, to measure change in those competencies over time. And I can go into that a little bit more too. And then to really think carefully about how we're gonna build chains of causal reasoning that help us give feedback to instructors or learners to be able to say these kinds of things seem to not just correlate, but really seem to be affecting and predicting what students are learning. How many of time? I'm rolling. So let's talk about assessments first. The domains in which we do a really good job building assessments are ones in which the performance of understanding that we're asking students to do is highly constrained and sort of highly constructed in some ways. So for instance, at Harvard, our introduction to computer science course CS50X, which has 300,000 people registered for it, I think has some really extraordinary assessments that are baked into it. Computer, we can write computer programs that can test whether or not other people's computer programs are working and writing good computer programs is pretty central to becoming a computer programmer and pretty central to introduction to computer programming. And so this seems to be having kind of a nice alignment. We can tell whether or not your code compiles, we can tell whether or not the program you've written meets the engineering tests that we've set out for it. We can tell how parsimonious your code is, how quickly it runs, whether or not it meets the design specifications that would allow you to collaborate with other people. CS50 is one of the domains which we have some pretty good measures of whether or not people are doing things that we care about. There are plenty of other domains which are much, much harder. Broadly speaking, the humanities and professions are a place where these things are much, much harder. Although I'd say one way that I often categorize a domain that computers aren't particularly good at assessing is any kind of reasoning from evidence. Reasoning from evidence is kind of the thing we teach in higher education. So not having really great tools right now to assess reasoning from evidence is a huge, huge limitation. This is an example of a question from Heroes X and it's trying to capture whether or not students are developing the capacity to read an ancient text bringing the perspective that would have been brought by a person in the ancient world and not bringing their own present preconceptions. I mean, that's a hard thing to teach. That's a hard thing to assess. It's a hard thing to get at a multiple choice question. One little side note that I wanna make here is that another challenge that we have is that assessments have to serve multiple functions and courses. So one of the things that we use assessments for is that we use them assessments kind of for learning, including assessments of learning. So this is a kind of question where there are three, this is an annotation question where you would read a passage of text and then there are three possible answers. One interpretation, which is kind of the way an ancient Greek person would have read it. One interpretation, which is the way a contemporary person would have read it and one interpretation that's a combination of both. Now as an assessment of for learning, kind of serving up these three potential options, I mean, I don't know, but it seems plausible that this might be a really good way to help people develop those skills. From a sort of psychometric perspective, this question leaves a lot to be desired. So for instance, I told you that one of the answers had to be kind of a combination of two things. Like, you don't even have to look at the words to guess which one is the combination of two things, like it has a conjunction in it. So if you have even a kind of modicum of test taking skills, you can cross off this one and then you're guessing between two answers and you got a 50% chance of getting it right at that point. The passing grade for the course is 50. So as an assessment of learning, this doesn't have all the sort of psychometric properties that we might want. As an assessment for learning, it may work just fine. Our colleagues down the road at the Harvest Business School have been building their online platform HBX and partnering with Pearson to develop some assessments. They were saying that for every 70 multiple choice questions they wrote, one of them would get chosen. So they had to write about 70 of them to find one that sort of had the psychometric properties to be a good assessment of learning that they were looking for. You know, all of this points to really deep challenges that require pretty diverse teams to be able to tackle. We need content experts who really understand the domain and what we're trying to do. We need instructional designers who can think about what kinds of assessments are possible in online environments. We need the platform developers to help us build new technologies. We need psychometricians to help us think about what these assessments should look like. We need educational researchers to think about what kinds of questions we should be asking and how to put these chains of causal reasoning together. This is a non-trivial problem. And some of it makes you think like, man, let's just go back and doing some user behavior research. That was pretty easy. But I think the tougher challenges are gonna be the ones that are more rewarding. We've been messing around with trying, at HarvardX is a couple of things that are going on that are trying to find some different ways, especially in the humanities, to get some other kinds of traction on what people are learning. I could talk more about this. I wanna explain it all the way. But some of you say we're doing some topic modeling. We're asking students to do some writing and discussion forums and then seeing what kinds of themes, surface in the discussion forums. You could imagine circumstances where instructors were trying in a unit to teach about sort of six different topics. And if the students then write in further discussion forums about all six of those topics, then you have some sense of as a community what people are picking up and making sense of it. If they're not writing about all those things or if they're writing about some more than others, you might be able to sort of tune your instruction along that, maybe not at the individual level, but maybe at the course level. We have some folks who are working on annotation tools. Annotation is a strategy that humanists have been using for a couple of millennia to develop and demonstrate their understanding. And so there may be some ways that we can sort of get some traction on what people are learning in the humanities around that. But these are super, super hard problems. But I think exactly the kind of problems that if we wanna be able to have a science of learning that we need to tackle. We need to measure change and competency over time. In residential circumstances, it is reasonable to expect that people come in with fairly homogenous levels of competence. So if you come to Harvard and you already know all the stuff in introduction to physics, you're pretty unlikely to wanna take introduction to physics. Like if you can only take 16 courses while you're here and it's kind of expensive to hang out here, most of the people who can get past that requirement will do so. That's not at all the case in the courses that we teach. We have a wide, wide range of people who are showing up. Some people who come into our introduction to physics classes are taking them because they know the material and they just wanna test themselves against MIT's problem set. So they just wanna refresher. In some courses, in many courses, we can do a good job of building really good sort of assessments of competence that we place at the end of the course. But if that's the only place we assess people and we don't know their skill sets beforehand, we may just be making inferences about what people's pre-existing capacity were. I think about that a little bit with the Copyright X class. Through Berkman's Law School, or Berkman with the Law School, they launched this class called Copyright X. Fabulous class offered to 500 people. They had a really rich assessment exam at the end. They're only 500 students so they had third year law students grade it. You can get really nuanced feedback and get really detailed critique. But some of the people taking the class are law professors and some of the people taking the class are high school students. And so it's reasonable to believe that the number one predictor of your success on the Copyright X final exam would be your ability to write a three hour timed final exam as opposed to what, or at least partially as opposed to what you learned in Copyright. So there are more people who are messing around with these kinds of things. We've got a couple of colleagues at the Center for Astrophysics who have been developing this computing readiness pre-test. So it's a pre-test they put at the beginning of the introduction to computer science class to try to get some kind of bandwidth on what sort of competencies people bring into the course. So we can get a sense of how their competency changes and it's just a kind of hilarious sort of logic-y manipulating variables sort of problem. If you can figure it out, you can yell it out later. And then maybe the piece to end with is that even if we do these things, there'll still be quite a bit of work to do to sort of carefully build chains of causal reasoning that help us understand exactly what kinds of things students or instructors or peers or other folks can do that directly affect learning. So this is one figure from I think one of the best studies about MOOCs to come out recently that really addresses some of these learning issues. So David Pritchard is a guy who runs a lab at MIT in physics but has been doing physics research and physics research online for a long, long time, much longer than edX. And he designed this course that used the force concept inventory as a pre-test and a post-test to be able to sort of track people's conceptual understanding. He measured people's skill on sort of the quantitative pieces of things as through a series of homework problems throughout the course. And then what he did here is he did a study where they looked at how time on task correlated to gains in behavior. So they looked at the time on task on the checkpoint questions and the discussion forums and the edX with the problems, with the total time in the class. So instead of having one person level summary of activity, they're sort of breaking it down into pieces, have a couple of different well-validated measures of learning. The circles, this is sort of a funny representation, but it's the correlation between those things. So if they're gray, it's not significant. If it's sort of light green, then it's a little bit significant at the point of five level or something and at the point of one level for dark green. So you can start making some correlations towards how people spend their time and how the time invested in courses correlates with particular kinds of skill improvements or different kinds of pre-post gain. Even if you do all that, there's still a whole bunch more work to be done to try to figure out to what extent those correlations really represent causal, are places where you can make causal inferences. So the thing that stands out to me in this particular forum figure is that the amount of time people spend in the discussion forum seems to be weakly correlated to things like skill improvement and pre-post gain. There's a whole bunch of reasons that could be the case. It could be that people who are super highly competent going into the class, spend a bunch of time in the discussion forums, helping other people out and they didn't have that much to learn to begin with. It could be the people who are really struggling, spend a bunch of time in the discussion forums and maybe the time is helpful for them, but it turns out that they just were having a tougher time developing anyway. So really the next step in this kind of research is to think about addressing causal mechanisms in two kinds of ways. One would be doing some kind of experimental intervention that either gave some people access to discussion forums and some people no access or temporary access or encouraged more people to be involved or encouraged people to be less involved in some kind of manipulation to see if you sort of add more discussion time or take away a discussion time, how that affects people's performance. And then another sort of research strategy which I think is equally important is if we want to better understand some of these kinds of causal mechanisms, I think we have to spend much more time talking to the people who are taking these classes. So it could be that everyone is having discussions about the problems in 8MREV. However, if you have a really strong expert network of people who can help you with this stuff off the platform, then you go and talk to those people and they help you out and you do better. And if you don't have that really strong expert network in your social circles, this is that you go online and in fact the online time is weaker than the people who can go into off course platforms and to talk with experts in their network. So I think really trying to understand exactly how people are going through these learning experiences, there'll be no substitute for talking with those people. I mean, we've been doing a whole bunch of work survey. It's hard, we can't talk to all of them because there are 300,000 people who registered for CS50. So we surveyed a lot of them, but we should be randomly sampling people and we should be making phone calls and talking with people in different parts of the world and trying to better understand what they tell us about their experiences. So those are sort of three challenges that I would lay out for the science of learning, moving forward, most of which really has to do with making sure that measures of learning are central to the science of learning. That no matter how, if what we've done is sort of massively magnify the amount of data that we can collect about user behavior, but we haven't made substantial gains in how we measure people's competence, then I'm not sure that the hordes of data that we have on the right side of the equation are gonna help us that much as we try to figure out what's going on on the left side of the equation. I'm not sure how much. We can find ourselves very easily in traps where studying engagement to figure out what engages people as opposed to figuring out how people are learning or developing their skills or competencies. So if you're not in this field, if you're just kind of checking out what's going on in the field, I hope that one of the things that you could take away from this talk would just be sort of a sensitivity to these issues as you're reading research that comes out and reading news reports and things like that that you could look at published research and really ask the first question, how much of this is a behavior study and how much of this is really a learning study? At the end of reading this study, how much do we know about how we help people learn? And then I think for those of us in the field, whether we're course developers, whether we're researchers, whether we're platform developers, I think the signature challenges that we face are gonna be ones where how do we get a better handle on what kinds of competencies people have, how do we get a better handle on measuring their competencies at multiple times so we can really understand not just what people are clicking in these systems, but what they're learning. Oh, one, I'll skip it. We can talk about that more later if you want to. So I've written about a whole bunch of these things at EdTech Researcher over time. So if you wanted to, I mean, don't do it now because we're talking, but later on, you can pull up some of these kinds of things and see what I have to say. A lot of this talk comes out of a blog post that I wrote, I think called something like Breakthrough MOOC Data Finding. Yeah, there's some great videos on my blog, things like that. And so I'd be more than happy to, so I'll stop talking now, which was roughly 25 minutes maybe. I'm more than happy to engage with you on any of these kinds of pieces that you want to. I can talk about the other things that we're doing at Harvard. I can talk about what you think of all this or studies that you're excited about or research that you'd like to see done or anything else that you want to talk about for the rest of our time together. I don't know, Daniel, do you want to help with the mic? So we'll run around the microphone so that the people on the interwebs can, and if people have been asking things on the interwebs, someone should shout out for them. Shane has a lot to say, but. So you said towards the end, I'm not sure. So in a way you've answered my question, or you're not sure whether asking the right questions or rather that the terabyte of data that you already have will help you answer the right questions. That was really my concern. So when you ask, for example, improved learning over time, if you have terabytes of data, surely you can go in and learn something about that. For example, if they're doing problems that they're learning to do the problems faster or something, I mean, if you think about it well, you should be able to get something out of terabytes of data, surely. Or is it really the case that the terabytes of data are useless and then you have to go off and find new measure, new, you use the word measure also all the time. So you have to get new terabytes of data. Yeah, so here's how I think about this. I think we should be suspicious of the hypothesis that because we have a lot of data, the answers we want are in it and the things that are important are in it. So that the answers are not in it, surely we have to be careful not to jump to that. Yeah, sure, I think that's absolutely right. I think we sort of have to interrogate what's there. I mean, to me, the way that I sort of think about the problem that we face is sort of something like this. Like, imagine you're Amazon and you've got all of these different programs. So you've got Amazon Prime, you've got Amazon Smile, you've got customer reviews, you've got things that people are bought together. You sort of have all these programs, all of these behaviors that people are clicking through in all the ways they're navigating through this purchasing system. And what you're really, in many ways, what you're trying to optimize is the number of dollars that people spend inside this system. What would happen if I told you you could have all the data about what people are doing but I weren't gonna tell you the number of dollars they were spending? The kinds of research that you could do in that circumstance would be a lot harder. For me, the measures of student learning are the dollars in Amazon. That's really what we're trying to get at is are we helping people develop these really difficult capacities? So for sure, I definitely don't mean to say, I mean, I'm in this field because I think the fact that we can capture tons of real-time clickstream data about people's behavior is super promising. I think there might be all kinds of things that we might be able to get out of it. I think just gathering that data itself without making some real progress in sort of how we measure people's capacity might lead us into some of these circles where we're correlating people's engagement to other forms of their engagement and making some perhaps not well-validated inferences about whether or not they're actually learning and doing that. But those are good cautionary notes. So Charlie, or anybody else? Thank you. Well, my question really relates, just to give you a bit of background, we're platform developers. So my question really relates to the fact that- Do you wanna introduce yourself? Who are you and where are you? Oh, sure. Well, I'm about, and I'm actually currently based out of MassChallenge. So we're working on gradberry.com and what gradberry is doing is that we're trying to basically enhance skill levels of students and fresh graduates for the employer. So we're looking at basically you're enhancing skill sets in terms of very practical skills and skills that are, you know, let's say maybe six months old. So for example, the Swift programming language or any source of new languages that are coming out. So in the last eight months or so, the data that we've captured from our roughly 32,000 users, what we're trying to do is we're trying to enhance the skill levels of these students and graduates. So you mentioned the prior study, which showed that there was no really direct correlation between the amount of time spent in discussion boards and an increase in skill level. So I was wondering what sorts of, what can we really do in order to enhance skill level and what were the conclusions from that study? I mean, what about like interaction? What about having live sessions? Did that help at all? Maybe the students were learning through the Socratic method where they could directly ask questions from professors and instructors. So that's really my question. Yeah, so I mean, I would definitely go read the full piece. I mean, I gave you just a little snapshot of Dave's research and I think it was that discussions predicted some conceptual gains, but not as much of the sort of skill quantitative improvement and it predicted less than some of the other kinds of things that were there. So it wasn't that it was sort of nothing per se. But I, you know, I think in any of these domains there are tons of things that we could tackle. I mean, you know, and again, I would say if we wanna understand better how people are learning from discussion forums, you know, some of the first things that I would do are try to talk to people about what, why do they go to discussion forums? What kinds of things are they doing there? I would love to do some studies where we ask people to let us sort of snoop in on their sessions, you know, do a Google hangout with us and go, you know, do your studying for an hour and just let us watch and see what kinds of things you're doing, maybe think aloud. There's a whole bunch of research that this guy, Sam Weinberg has done on developing historical thinking skills, which is this great body of work based on just having people talk aloud about the kinds of things that they're doing while they're trying to solve intellectual problems. I think thinking about what kinds of, so I think that will give us a better sense of sort of how people are using these sorts of resources and then there's all kinds of literature that's out there, you know, so for instance, one of the things that we know is that people's, you know, people's social identity is super important to them in learning and online spaces. So edX has been working on all kinds of challenges over the last year. One of the things that they haven't gotten to yet is building sort of robust social features for people. So one of the things about edX is when you sign up for it, you put in a username and your username might be, you know, kittyunicorn47 and that's just like the username that I use for all the things that I log into and I go to the discussion forums and now here I am in this, you know, introduction to physics class and like I have these questions being asked by kittyunicorn47 and I can't change it to be, you know, Justin or, you know, or someone else or something like that. So there may be certain kinds of platform things that we can test and we can evaluate that would improve that. There are a lot of folks that are out there who think for these kinds of large scale settings that discussions are just totally unwieldy and not really great platforms for a lot of kind of learning to happen. If you talk to the folks who sort of developed the earliest things that were called MOOCs, the earliest connectivist platforms, they gave up on discussion forums pretty quickly because they felt like people just, you know, sort of an awkward place for human interaction. Either nobody was there and then they were empty or a lot of people go there, you know, they fill up with things. So I mean, I think there's probably no, you know, there's no single answer to that question, but I would say if you go back to sort of those three things I would offer, I mean, it sounds like you've got some pretty good measures of what their skills are. I think it'd be really important to sort of measure their skills over time and then a good starting point is to sort of correlate what kinds of things are they doing in the discussion forum and how does that relate to skill, but then I would think about talking to people about what they're doing in discussion forums and trying to figure out what some hypotheses of what causal mechanisms might be and then think about what kinds of experiments we might be able to do that would take whatever advantages there are in those platforms and use them more efficiently. And then we should also, I mean, I think there's a lot of inertia behind the forums that we use, you know, like why do we use tons of video lecture capture? Because there are some other platforms that use tons of video lecture capture. Why do we use discussion forums? Because like you can go to places and people have discussion forums, but there are potentially other forums out there that we could be experimenting with that would look a lot different. Here's, you know, on that point of video thing, let me see if I can, maybe I'll let somebody else ask another question and then pull up this little piece. I have this sort of shocking piece of research that. I was wondering about the thing where you're not able to get some info from folks at the beginning, like where they're starting from. And I work with a group called Open Hatch and we're always trying to get folks to learn to program from zero. So some of the people we work with were able to pepper with lots of questions at the beginning when we do in-person events, but then we have other folks that use our online tools and we have no idea where they're coming from. But you have a lot more data than we do. I was wondering if you could kind of cheat on some of those because you must have people that sign up for a second course. So you must have some sense of like, well if they took the first course then we know they're at least in this range. So I don't know if you've looked at. Yeah, yeah. So that's, I mean those are great questions. Here's some interesting things about that. At Harvard, we don't have a ton of courses that are in sequence. We have some poetry courses that are in sequence. We have a China course that are in sequence. You could argue that the China course actually doesn't have assessments. So there's not much to measure there. The China course does. It's not clear how much there's sort of each piece builds on the other, but there might be some more to do there. I think some places like MIT have more of these courses that are in a sequence and I think it would be definitely a great way to get some sense of what kinds of learning experiences people are having is if they take course A, do they perform well in course B? One thing that's really interesting about the sort of X consortium and this relates to a bunch of sort of data privacy things that a bunch of folks, the kind of Facebook studies probably on a bunch of folks minds. If you're a data scientist, we would love to, right now we don't have data from all the X consortium members. We at Harvard have an agreement with MIT where we share data with them. If someone drops our introduction to computer science class CS50, it might be because they just started MIT's class and liked that better and that's not really a sort of weakness for us. It would be great if we had the courses from Berkeley and McGill and Utexas Austin and all these other places. If we merged all those data sets together, there's much more research that we could do. If we merged and shared all those data, that data would be at a much greater risk of being disclosed to other people as it's moving between institutions and trying to be de-identified and things like that. So there's this real tension that we're wrestling with that there's a whole bunch of things that we could do with data that would improve the science of learning and they tend to be the same kinds of things that increase risk of disclosure and exposure. And so there are a bunch of folks that are wrestling with that. Yeah, that's tricky, I understand. Thanks. Yeah, sure. Dan can just pick people. Just a follow up question about the data. So you keep on talking about data which comes from the online learning platforms. The question is do you also use data from outside sources? I don't know, social networking and things about the student to actually know background of students, things like that. So I mean, those are great questions. We're definitely at number one, we're totally overwhelmed with just the data that we can pull off the platform. So it really is quite a bit that the real time logs that we have of every student's interaction on all of these platforms, it sort of feels like right now enough to keep us busy for a long time. All of that kind of, another thing is that we sort of, I mean, these come back to some of these sort of ethical issues. We feel like that we've, and we probably could be doing a better job, but we feel like we're doing at least some job of communicating to people. Like when you come to our courses, one of the things that we do is we study, we study your behavior in the courses so we can offer better courses. Like ideally, all teachers are doing that. They're looking at their learners and making sense of what they're doing and trying to figure out how to teach them better. We don't tell our users that like, if you come and join our system, we're gonna start creeping you on Facebook and on Twitter and sort of pulling these other things. There are some folks that have been doing that. So for instance, there are a bunch of folks who created the original set of things that were called MOOCs, these connectivist MOOCs, who actually, I think would disagree with most of what I've said in, almost everything I've said in here sort of takes as a given that instructors should decide what's worth learning and measure whether or not students are learning that. But a whole other perspective is like instructors should just create spaces for learning and only students themselves can decide what whether or not what they're learning is valuable. And so there are people who started doing some studies to see like doing sentiment analysis on Twitter to see, you can't ask people whether or not they're happy with learning because the people who are happy answer your surveys and the people who aren't don't and so you don't really know. But they were doing some like, or at least contemplating doing sentiment analysis on Twitter to see if the people who took their classes were sort of happier after taking their classes than the people who didn't take their classes and that would be indicative of these kinds of things. We do, it's very, very clear that we're at a huge risk of, when we only look at platform data, of making bad inferences because we bind ourselves to that data. We have no idea what people are doing for their learning which is, for instance, online but off platform. So it could, oh, so actually this connects to this really nicely. This is the thing I wanted to show. So there's an undergraduate who's been doing, this guy Tommy Mulaney just graduated has been doing some fabulous research for us. One of the things that he started doing was looking at the number of people who started and finished video. This I think is actually a really provocative example of the ways in which user behavior research can be quite helpful. So the top line is the people who start the videos, the bottom line is the people who finish videos and the red line is the number of people who earned a certificate. So one of the things that you'll notice there is that by about a third of the way through the class the number of people who are watching a video is lower than the number of people who earn a certificate. You could actually see this in the first course that Anant offered of the 6002X at, when it was originally MITX. Like if you went to the, at the end of that course, if you went to the last YouTube videos in that sequence there were like 7000 people who earned a certificate and 400 people who watched the last video or something like that. And so Tommy's got tons of these from all these different courses where again the number of students who are watching videos by about a third of the way through the course is substantially below the number of people who go on to earn a certificate. So this raises a whole bunch of questions. One question is we spend a lot of resources making videos. Like it's, it's not, I mean what you should, in some ways when I look at this thing one thing that we know is that people they're, they're about as many people who sort of audit their way through most of a course as they're due who earn a certificate. So you'd actually like to see the number of people who start and finish a video be, be you know maybe half again as high as the number of certificate earners. So this raises a whole bunch of questions like what are the people who earn a certificate doing to learn? Did they know all this stuff already? Are people just banging their heads against problems until they figure it out? Are they you know looking for keywords and problem sets and then going to Wikipedia and reading articles there? Are they just reading the transcripts? I think it's entirely possible that and we don't have great instrumentation over sort of how and whether people read transcripts. So it could be that like you watch the first few videos and you're like all right I know what this guy's voice, this woman's voice sounds like you know I don't really, you know most people who watch the videos watch them maybe you all do this at 1.6 speed, at 1.8 speed, at 2.0 speed. There's this guy who is the boy genius from Ulaanbaatar, the Mongolian kid who got a perfect score in 6.002X and went to MIT. He watches two videos at the same time. He pulls them up on two different screens. He feels like it's sort of more efficient that way. But there's a whole bunch of questions that to be asked here about you know if we think videos are sort of the primary way that we're delivering content and people aren't watching them what are they doing to learn? My wife is on the faculty at MIT, she's building an online course this semester and she's spending like a gazillion hours making these videos and I sort of like started to share some of these findings. Like, well don't worry too much about the last videos because people aren't going to watch them anyway. She's like, well what are they going to do anyway? Well they're going to read the transcript. She's like, wait a minute, they're going to read text. I was like, yeah, they're going to read text. She's like, like isn't this back to where we started? Like if they're just sort of like reading textbooks and then taking problems, you know, sort of it seems a little circular. So those are some of the issues raised about off-platform. I don't know, Dan's got to. Oh yeah, I was wondering like, you know, I don't know, you kind of answered it a little bit but I was wondering about like, you know, what's your feeling is like, you know, the people who control data, do you understand what I mean? Like the control of data. My thoughts on the control data. Yeah, your thoughts on that. I mean, you've been kind of answering it as you go along. Yeah, you know, I mean I think, yeah, let's see. So if I put on my researcher hat, like I think researchers are good guys. I think Andrew and I are technocrats to be trusted and I think, you know, we should, you know, particularly with, you know, but we're not doing market research right from the most part, we're doing a little bit of market research but we're mostly trying to figure out how do we help people learn? And so we want folks to, you know, folks sign up for our courses, they're voluntary. I think it's pretty clear that we're studying the people who are coming into these environments. So it'd be great if we could share data, it'd be great if we could merge data. I absolutely think students should have, be an important stakeholder in their own data. It'd be great if their API is built into edX, so you could take your data, download it, move it somewhere else, do other things with it. I can sort of put my more Berkman privacy kind of hat on and go, whoa, this is a terrible idea. Like technocrats like me are not to be trusted. Like there's all kinds of terrible things that we will do. And you should severely constrain the kind of research that's possible. So I, you know, I guess depending upon where I, whether I spend my day on Everett Street or 125 Mount Auburn sort of determines what I think about those questions. You mentioned the possibility of doing a study by varying the conditions that students are seeing in the class. Like, you know, we might be taking the same class, but she has access to some form that I don't or I get an extra video that she doesn't. And where's the ethical boundary of doing that kind of research, especially when we might actually be able to discuss the fact that we're not seeing the same, exactly the same class, even though we think we are. So Victor who's here from edX has been this champion who helped us build this AB testing platform that we've just sort of got built in edX and we're super excited to mess around with. So let me separate two things. One is sort of an ethical question and one's a contamination question. So the second one is a tricky one. If you two are in the same learning environment and we give you access to the same learning, to different learning experiences and you can talk about those experiences, then that problem isn't an ethical problem. I don't think it's really just a problem of it. It pollutes our ability to make inferences about these two groups that we're hoping are separate. In terms of the ethical questions, so educators do experiments all the time. Every year in public schools, we take a bunch of people who've just graduated from college and we ship them out, usually to our most vulnerable students and we say, huh, I wonder which of these folks is gonna work out as teachers. If I, another way that we do experiments, which is not as good, this is just a not as good way to do an experiment. Like you come to my class last fall and I'll do a set of things. I think some of them work and some of them don't. You come to my class the next fall, I'll give you a different set of experiences. Some of them might work better or some of them might work worse. And that's an experiment too. Is it ethical to conduct that experiment? Absolutely, the only way that any individual instructor or we as a field can figure out what works better is by conducting experiments. And I would actually say the more ethical thing to do is to conduct those experiments in rigorous ways that we can make inferences that we feel confident being able to share with other folks. We have to build in the same kinds of issues that biomedical researchers do. So if we find things that we think work a whole lot better than other things, then I think it's really important to make sure we stop doing the things that aren't working and do more of the things that are working. But I think it's important that people know that we're conducting experiments. One set of experiments that people get really kind of, one thing that sort of sets people off quite a bit is if some people have access to things and other people don't. We sort of think about that as unfair. Here's the thing though. It's not totally clear to me in learning environments that having more stuff is better. If we prove that parsimonious learning environments were better than learning environments with too many options or too many pathways or other things like that, then we'd be doing a real disservice to people by not testing and examining some of those kinds of things. I will say though that the onus is definitely on educational researchers, hopefully in the not too distant, I mean one of the challenges I think of some of this is that I'm not sure how many sort of blockbuster findings we have from randomized trials and online learning experience and say boy, howdy, don't we know for sure that this, that or the other thing is true and really important. But I think general, I think there are ethical issues. I think we should be wrestling with them really hard and we do spend a lot of time with our IRB board here. They come to our meetings and things like that. But I think the alternative to not doing, taking experimental approaches to improvement is kind of idiosyncratic approaches to improvement. How do we deal, hi. Hey, how are you? Wonderful talk. I'm a sociologist, so I'm always gonna be thinking about probably two things that didn't come up quite so much, which are differences in group inputs, particularly as they relate to inequality. And this will tie into the last question, which was, I understood it as fundamentally a question about how do we get a baseline measure for inputs? I would like methodologically. So how do we get at this idea? Could it be an experiment? First of all, if we're not randomly assigning people, how do we control for the fact that there is still an extreme amount of self-selection that happens in this environment? An extreme amount of self-selection, not just at the individual level, but a level of analysis that we haven't talked about quite so much, which is at the group level. We do have some meaningful descriptive statistics about the users who tend to self-select into these courses. So I wonder if you could kind of address those two issues both methodologically and what it means about some of the inferences we're making from the data about to whom it applies. Yeah, that's great. So one of the things I think, and it's Tressie, right? We've met online. I don't think we've met in person. How are you? We, yeah, so the people who self-select into MOOCs, I could have spent more time talking about the demographics of folks. They're disproportionately highly educated. Most of the people who show up for our classes have a bachelor's degree or above. We're doing more and more research now to try to figure out sort of where they are. I mean, it's tough to ask people a lot about their socioeconomic background in the background in a survey, but we're starting to do some things like asking about their parents' level of education to see if that gives us a little bit of insight. We're doing some things like we have their address and see if we connect that to census tract to figure out sort of what kind of neighborhood they're from and things like that. I mean, certainly the people who show up to our courses are different than the people who don't select into our courses. And so any of the kinds of experiments we do shouldn't generalize to everyone. They should generalize to our populations. And then actually I think the field of sort of intelligent tutors MOOCs has an interesting debate going on in it about how much to take demographic factors or groupings into account. So actually for a long time, a lot of the folks who did intelligent tutor research said we're not gonna collect any data about people's demography. We're not gonna know anything about their backgrounds. We're just gonna represent them as a series of actions inside our platform. Now one of the great things about not collecting any data about people's backgrounds is you could go to CMU's data shop right now and there's data sets from the last decade or two from different kinds of intelligent tutoring systems which are nothing but sort of lists of actions that people have taken. Totally divorced from people's backgrounds. So there's some folks in the field who will say pretty vigorously like don't collect any demographic information because then we can share data more easily and we'll do better science. And then there are other folks who say a bunch of the questions that we have have to do with people's backgrounds. We wanna know do people who come from the UN list of least developed countries have different experiences than the people who come from Western Europe or people who come from the United States. I think one of the sort of worst-case scenarios of these systems is if you ignored demography, let's say you believe that people from different kinds of backgrounds learn differently in some way. Let's say people who are English language learners needed more kinds of academic literacy supports than people who were not English language learners. But you decided to ignore demography. You just look at people at one big clump. You do a bunch of experiments maybe on academic literacy supports and you find that in the aggregate they don't matter that much because you don't know about people's backgrounds. One could start imagining a system where you sort of tune these tutoring platforms to the majority because you're ignoring all of these demographic differences. But of course, and then there are a whole bunch of challenging, there's equally kind of horrific circumstances you can imagine of some, if a lot of people are really interested in how would we build prediction algorithms? How would we figure out sort of how people will perform really early in the course? Well, if a lot of our prediction algorithms identify that people who have less advantaged backgrounds are likely to pass a course and we sort of treat them differently or inequitably, there are other ways that we can sort of wrestle with that too. So I would say, you know, part of the, if I had done a talk on sort of like MOOCs and policy research, I think there'd be an awful lot that we'd have to say about where people with different backgrounds come from and what kinds of experiences they have and so forth. And I think those questions would be really important. I think they're really important questions in learning too. We haven't even really gotten to the point of zeroing in on sort of the whole, including subsetting. And then there are also really tough questions about how much data do we want to gather and share about people's backgrounds? A lot of the way in school settings that we know about people's backgrounds is we know what neighborhood they're in. And sometimes we know even more intrusive data about whether or not they're on financial aid and whether or not they're eligible for free and reduced price lunch. But those are huge questions to be tangled with. Let me ask a question about a group of learners that often are not included in these kinds of studies, but yet for which MOOCs and related innovations I think are very relevant. And those are practitioners or professionals who are facing major changes in their work environment and are finding that they need to be almost in a constant state of continuing education or professional development. That speaks to this category of performances of understanding where the knowledge is not so much poured into the head as it is reflected in what's in the heart and what the hands are doing. It's more applied knowledge than received knowledge from mathematical equations or things like that. So what kind of research is going on with respect to the performances of understanding question as it is revealed in professional development, professional training where the idea of a cannon or the idea of a calling are central to what people are doing. Yeah, so I've got a few things there. One is I think actually across all of, this is not specific to professionals. I think one of the things that struck me most in the last year is talking to faculty who teach these courses. How many of their sort of outcome goals, learning objectives are not sort of academic changes, but they're behavioral changes. So the people who teach the human health and global environmental change class, I think they want people to understand the health consequences of global environmental change. They also want people to be more active and better informed advocates for addressing issues of climate change. That's a sort of like social impact goal rather than a learning goal. Some folks at the Kennedy School are building an education policy course. I think they care about whether or not people learn what the recent changes in education policy are and theory about how policy works, but I think they also care to people vote in their school board elections. Are they more likely to be involved in their school board? And I think those kinds of behavioral outcomes are just as legitimate for us to aim for and study as anything else. I mean, it's really striking how many, talk to a bunch of scientists who want people to sort of understand the basic science, but also become more science literate, read more science in the news, be more critical consumers of science, have more public support for public funding of science, those kinds of things. So as I talk about measuring the learning outcomes that we care about, I think those kinds of behavioral outcomes are totally legitimate. And they're super hard to track. In civic education, we have a whole literature on how do you measure, whether people will go and vote or volunteer in their community and things like that. And they're hard to measure, but we should do it. In terms of other kinds of research in the professions, I'm really enthusiastic about efforts to adapt the case study method to online venues. That's something that folks down the river at the Harvard Business School have been working on is trying to think about, they have this really distinctive pedagogy that they think prepares people as practitioners for these settings and they're trying to figure out ways of recreating that setting in a high scale environment. And I think that as they release more stuff from their HBX platform, they've got some pilots going on right now, I think it'll be kind of fun to watch. I don't think they have any sort of startling breakthroughs to reveal, but I think they have some things they're trying to be able to work towards that. But yeah, I mean one thing that I guess I would wanna add on to that point is I really hope that we define learning really capaciously. I think great schools do three things. They prepare people for their roles in civil society, they prepare people to lead meaningful, reflective, ethical lives, and they prepare people to be of service in their community through the economy and through the labor market or workforce or whatever else you wanna call it. We are totally obsessed with number three, but number one and two are absolutely kind of at the heart of the vision of public education and we should be thinking about those things as well. What does the timing look like, Dan? What are we supposed to say? Okay, great. I don't think this has really been a talk about MOOCs. I think this has been a talk about education and how we figure out how people learn. MOOCs allow us to do different methodologies and to test them in different ways. But listening to what you said about physics and looking at the, I heard John Sturman talk about system dynamics at MIT and he was talking about testing graduate students at MIT around system dynamics and they don't understand what's going on. And then the invisible gorilla problem. The psychology that we have built into us as human beings, there are a lot of blind spots there that inhibit learning and inhibit reasoning or inhibit action. And how do you get over those? How do you even, we have political discussions and all of the time there are all of these mistakes that people make that are known mistakes and we continue to make them as a culture, as individuals, as groups. This is built in to a certain extent and how do we get over those using what you're talking about? I would say, if what you take away is that I'm really interested in learning and that MOOCs are just a tool for that, like that sounds like a fabulous takeaway for me. That's how I sort of see things as well. But I will say that online platforms have a series of characteristics that are associated with them that do make them different than other environments. So you're bringing up some issues around kind of human psychology and how human psychology connects to learning. There are a ton of, like in the next year or two there are gonna be a gazillion studies that come out from behavioral and economists and social psychologists who are trying to use a wide variety of sort of little primes, little nudges, little changes to get people to learn more and learn differently. So for, you know, Joe Williams is here. He's done a whole bunch of this work, say with Khan Academy taking messages inspired by Carol Dweck's mindset research which shows that people have a flexible view of intelligence who think intelligence can be increased as opposed to being fixed, learn better. So, you know, take these kind of randomized worksheet problems and put little mindset messages on top of them and see what kinds of insights that adds. We have two studies that are going on right now at HarvardX. One is a study supporter system which basically you nominate a non-cohabitating friend of yours. So it can't be your wife or your mom or something like that because they already have too many things to spouse. Spouse mom, father, they have too many things to nag you about. Like the course is 100 on the list of 100 things. But if you get like your uncle, we're gonna send an email to you and your uncle every week that says something like, hey, Justin just learned about Oedipus this week. Ask him why Oedipus gouged his father's eyes out. And that, you know, sort of spark a kind of conversation between these folks. We have another study. This is, you know, this is a more, so that that study is basically saying how can we leverage the weak ties in your network to provide more social support for you when you're feeling low. Here's another one that's maybe a little bit more manipulative, you could say. We know that people like folks who they feel like are similar to them. And we think that when people like their instructors, they're more likely to persist in their classes. So we're gonna run the study where we give a survey to, we give a survey to instructors and a survey to students and about with the same questions. And we tell students when they answer the same way as their instructors. So like you like jazz music, like guess what, Dr. Elmore also likes jazz music. Oh, you think this is really important? Oh, Dr. Elmore does too. And the idea with that is that actually if we ask enough questions for pretty much everyone we try it on, you're gonna have six or eight things that are in common with the instructor, which hopefully makes you feel more similar, which makes you like them more, which makes you want to persist more. So, you know, so this is definitely, you know, like that's taking insights from social psychology. The difference between doing that in a face to face setting and in the online platform is the reason why social psychologists and behavioral economists are so excited about these platforms is because they can scale high fidelity interventions really well. So the problem with like teaching Carol Dweck's mindset style, you know, there's seven million teachers in this country. You gotta go to them one at a time and say, well, this is the right way to do a mindset message. This is not the right way to do a mindset message. This is what you gotta watch out for, you know, and building human capacity at that scale is super hard. But taking some of those things and programming them into platforms that hundreds of thousands or millions of people use, you can, you know, you can program them with kind of perfect fidelity. And they raise a whole bunch of ethical questions like how much do we want, you know, educators manipulate their students, that's what they do is you try to like create conditions around people that get them to change their behaviors and minds. How much do we want to tolerate those kinds of things operating at these sort of psychological levels? Some things, you know, seem totally great. Like if you can trick people into doing spaced practice that helps you memorize your multiplication facts better, that seems great. If you're sort of trying to get people to like each other or you're doing other kinds of things, that, you know, there's huge ethical conversation to have there. But I can tell you, in terms of MOOCs and the science of learning, in the next two years, you're gonna see a gazillion studies of social psychology interventions that are baked into these online platforms. The last one that got a lot of press was, you all see David, the article about David Yeager and the University of Texas in the New York Times. They did this little mindset study as part of the University of Texas orientation, where they have people sort of write a letter to themselves about what they should do when they face adversity. And the first pass analysis seemed to be that, especially for students who come in without some of the advantages of other students, that they were much more likely to persist through 12 credits in their first year than other students. So, tons there. I have two questions. One at the sort of macro level. I'd be interested if you might give some kind of summary of the different background demographics of people in, let's say, obscure rural areas versus places like Cambridge, that when you have people logging into the courses from, let's say, rural Alaska or Brazil or Malawi, do they have a much different or China? Are the students logging in from Chinese high school students hoping to get into an American university, or are they teachers trying to pick up material to use for trading? We're starting to get some of that. Again, this is the first year that we've had a survey that would give us sort of some more information on that. For instance, one thing, maybe I'll state this as a hunch rather than something that I'm sure of yet, is that there seem to be way more people that I would call co-learners outside the United States. So this isn't really a rural urban divide. This is a U.S. non-U.S. divide. There seem to be way more people who tell us that they're enrolled in degree granting programs who are also enrolled in HarvardX outside the U.S. than there are inside the U.S. So that's sort of an interesting kind of thing for us to continue to explore. But the demographic details that we can get about people are pretty, we're still, we don't actually really, we're having a hard time figuring out, say, within the United States, like exactly where people are from. We do a pretty good job figuring out that you're in the U.S., we've sort of gotten stumped recently about figuring out what state you're from, for instance. And now we're gonna see if we can figure out what census tract you're from, you know, which again, maybe, I don't know if you were expecting us to be able to do that when you signed up for edX, but so there's this tension between finding out some of these real details about you and how much we think that might be able to support what we're doing, how much it might be able to inform policy. To me, the more interesting differences are actually sort of not between countries, but between subject domains. So for instance, the background of people who enroll in our professional school classes, health and society, introduction to clinical trials, data analysis for genomics out of the School of Public Health, seem to be different kind of people than the folks who are signing up for even our science and cooking, medical anthropology classes in the FAS. They're more likely to have higher degrees. They're more likely to have master's degrees in PhDs. They're more likely to articulate instrumental motivations for taking the course rather than learning motivations for taking the course. They're more likely to say I wanna earn a certificate, I wanna advance my education, I wanna serve my community. Everyone who signs up for our courses say they're doing it for the love of learning because they wanna be affiliated with the lead institutions and because they want access to learning experience that they wouldn't have otherwise. But folks in the professional schools, I think we'll probably find out that they finished courses at higher rates, that they intended to finish courses at higher rates. So those populations are pretty interesting to look at. The other side was sort of the micro level. Have you been able to analyze, for example, if a lot of people are getting a question wrong, especially a multiple choice question where they're getting it wrong on the first try, is that because the question is difficult or possibly misleadingly phrased so that you can give feedback to the instructors that maybe if they had phrased the question in a slightly different way, they would have gotten different results. So we have actually a doctoral student right now who's starting to do some of that kind of item level stuff. I think some of the teams themselves started to do some of that item level stuff, like how many people, he can ask a question too. How many people perform well in a class? One thing that we did that was kind of neat is we have these figures, probably won't be able to pull them up. We have these figures about persistence that look at when people drop out of the course. And when people drop out of the course is usually actually pretty consistent. So about half of a cohort will leave in the first week and then about 15 to 20% in the second week and then somewhere between five and 10% of the remaining cohort will leave every week after that. But sometimes we see these sort of spikes in particular weeks. You'd sort of anticipate that maybe 5% of the cohort should sort of try to that point, but like 9% does or 12% does. And those are some places actually that we've started having some instructional teams go back to and be like, oh, we released twice as much content that week as we did other weeks. Like it looks like we had this sort of unstated social contract with our learners. And when we doubled the amount of work that we asked them to do, maybe we violated that contract. So that's exactly the kind of feedback loop that we wanna have with developers where we say, here are some things that we were finding out and how can you make some decisions? Maybe one, I think this is a good sort of broad kind of thing to think about. I think very little of the research we do is gonna say, oh, this was the wrong thing to do, you should do this. I think much more of the research that we do will involve characterizing trade-offs. These kinds of instructional decisions benefit these sorts of folks over here and these kinds of instructional decisions benefit these folks over here. So you have to make some choices. All right, Charlie, you've got the final word. Great, Justin, thank you. I came in late, but immediately resonated to the question you were posing, which is whether the right question is to be found in the data. And I did so because just yesterday in a faculty workshop, we all, the faculty of Harvard Law School gathered together and David Wilkins kind of led a session in what is the purpose of legal education. And he has a whole analysis that essentially posits a combination of values in some ways akin to the triad that you articulated. His triad is what we're trying to train at Harvard Law School are lawyers who have technical ability, who are wise counselors and are leaders. And those were his three. And I forget what your three exactly were, but they were, they were similar and they're a Aristotelian character. They're individually transcendental and in combination, absolutely. And so when you, when we found ourselves discussing, well, what is then the purpose? How do we articulate that? How do we express that in this new medium? It didn't seem at all as if data was the place to go, except in the brilliant way that you've done it, which is to show things like, hey, this certificate that's being handed out is largely farcical. It's like a frame into which we're painting with an effort to educate people where the means that we have of feedback are these crippled, robotic, multiple choice kinds of things with no personal interactivity of a teacher with a student at all. And so I don't know, for me, the question that you put right at the beginning is profound in its way. Yes, tremendous excitement over the data that's being generated and man, there must be huge amount of work and things to be learned from it. But somehow I feel that the deep questions about at least the future of legal education, which is not, not evaluatable in the way you, again, I thought perfectly articulated, money is not the measure. The metric is not easily quantifiable. Whether the future for us as a law school lies in looking at the data or somehow digging back into the core of dialogic experience and how we propagate that in an expanding digital environment. Yeah, that's a great set of challenges. And I guess the way that I would frame that is let's be totally honest with each other about what we can do today. And if there are things that are important that you've just described that we can't assess today, let's not pretend that we can, but let's also not despair that we can't. Let's get the brightest folks that we can get together. Let's get great legal minds. Let's get great platform developers. Let's get great instructional designers. Let's get great educational researchers. Let's put them together. And if there are places that we can make progress on that, if there are ways that the computers can do things as well or better than humans can, then by golly, let's figure out what those are. And if there are places that we are pretty sure that real high quality education can't be substituted with the computational tools that we have, let's find out ways to make sure the society invests in necessary resources, particularly not just with professional education, but with our public education systems to make that possible. So those are great thoughts to end on. Well, thanks for sticking with me for a big chunk of your afternoon. Thank you.