 Turn the floor over to Eric Klauffer who's going to introduce today's speaker. Thanks Scott. So I will briefly introduce Justin who's going to be talking today about Failure to Disrupt, his new book on educational technology and its failure to disrupt educational practices. Justin is a professor here in Comparative Media Studies and Writing. Before that, he was a researcher and lecturer here at MIT so he's been associated with the community for many years and he taught my education courses previously and ran our partnership with the Woodrow Wilson Academy for a number of years. Justin's been doing work that I think I would describe as sort of healthy skepticism about educational technologies for I think most of his career including his doctoral work and a lot of his work more recently. It's sort of taking this lens on educational technology that's both sort of hopeful and critical in terms of looking at the ways that it ultimately influences the practices of educators as well as the lives of the many students involved in the educational systems. Justin runs our teaching systems lab here which looks at a lot of issues in professional development of teachers, use implementation of educational technologies, equity in education, innovative practices for training both current and next generations of teachers. So with that, I will hand it over to Justin and thanks for coming to Colloquium today, Justin. Thanks Eric. We have some great instructors, writing instructors who help us teach the media studies classes to undergraduates at MIT and they try to help media studies undergraduates think about what argumentation in media studies looks like and one of the ways they describe it is to say that there are producers of media and audiences of media and they sort of interact in some way through media in some kind of broader social context and everything can get much more complex than that. The producers can become consumers, the consumers, producers, all that kind of stuff but there's a kind of fundamental structure to a lot of common arguments and I would say a lot of my work is interested in that model as well that there are people who take and facilitate media as educators and there are people who consume that media as learners and sometimes they pass it back and forth. In some kind of broader context, you know, the thing which distinguishes me as a learning scientist from other media scholars is I tend to be keenly interested in how does that interaction change human development? How does the learner develop new capacities? How are they able to do different kinds of things than they were before on the basis of that interaction and I'm hoping that one of the things that can come out of this conversation is I'll present to you some of the ways that I see the field of education technology as a learning scientist and I hope that folks will ask questions from a media studies perspective and for the graduate students make connections to the readings and other kinds of things you're working on, faculty, the research and other practices that you see or germane and we'll see if we can make some connections that way. So one place that I start my work is that for the last 20 years people have made some really extraordinary claims about how new media might transform the education landscape. In 2009, a Harvard Business School Professor Clay Christensen predicted that by 2019, last year, half of all secondary school courses in the United States would be online or blended, that they would cost a third as much to provide and they would produce better learning outcomes. In 2012, when massive open online courses exploded in higher education, edX and Corsair in Udacity, Sebastian threw in the founder of Udacity said in 50 years there's only 10 institutions of higher education left in the world. They're going to be a concentrated set of mega universities providing the world's best learning content all around the world and Udacity might be one of them. Sal Khan in a 2011 TED Talk said let's use video to reinvent education. Let's have individual students sit down in front of computers. Let's have a personalized learning pathway around mathematics for them that's sort of optimized for their individual learning rate and we'll still have students and teachers and things like that, but they'll mostly get together to do interesting projects and to reflect on what they're learning and build community, but the sort of heart of skills development will happen through these machines. And then 2013, Sirgada Mitra won the TED prize for his proposal that we didn't even need schools or educational institutions anymore, that we could simply give kids laptops and broadband connection and without any institutional support they could learn anything by themselves and then earlier this year the world was blighted by a global pandemic and 1.6 billion learners were sent home and to some extent you might think that like this was the moment that education technology was poised for. I mean prior to the pandemic the case of education technologists had to make was that they had a set of offerings that would be better than the existing traditional educational system, but they didn't even have to make that claim anymore. They simply had to claim that they could offer better learning experiences than you know pandemic hobbled system in the middle of an emergency pivot to remote teaching. And I think during the last 20 years one of the reasons why the arguments of education technologists for the transformative potential of learning media was so powerful is that we saw these transformations happen in other sectors. Journalism has been profoundly reshaped by media. Government and civic media is in the midst of a transformation. The word friend means something different than when I was growing up dating our relationships. I mean some of our most intimate experiences are mediated by technology, by social media in particular in new ways. Doesn't it stand to reason that the same kind of thing should happen in education as well? Like why would education be a sector that's different from any other sector? But I think most of you know what has happened during the pandemic which is that you know education technology did not write in on a flaming horse to a winged horse to save us all. In fact I think most people, most families especially of younger children but I think many folks in higher education and well have experienced something ranging from like eh this is adequate to wow this is really a disaster for us and for our families. And in fact two of the technology, I think the two most prominent technologies of the pandemic are two of the very oldest technologies that we have. So I would argue that the two technologies that have dominated the pandemic are learning management systems which are basically platforms that allow people to pass documents back and forth with one another. Canvas, Schoology, Google Classroom, these were theorized in the 60s and 70s. They were commercialized in the 1990s. They were made open source in the 2000s. And then the other you know perhaps dominant technology of the pandemic has been what in the 1930s when it was introduced was called videotelephony. Now goes by the term of videoconferencing. And I think what we saw both in higher education and in K-12 education was you know a series of collective acts of conservatism small C conservatism. The likes of which we will never again see in our lifetime. Faced with dramatically changing circumstances, you know most of the professorate like walked away from their lecterns and sat down in front of their home office video cameras and kept teaching you know roughly the same way that they were teaching beforehand. Despite all the transformations happening in the world and despite the promises of people that technology would rearrange those relationships even in normal times, let alone pandemic times. So my task as a writer in this book is to explain why that is, to explain why these arguments of transformation can be so tractable at times can be so compelling at times. But also the argument is that they shouldn't be so compelling because they're routinely not true and that there are more productive stances to take towards the role of technology in schools. Last year before the pandemic, Sal Khan gave an interview to a little trade magazine called District Administration. So I'm sure millions of people have seen Sal Khan's TED Talks. I promise you that it was like me and four other people who read this interview in District Administration Magazine. It turns out that in the last decade, Sal Khan had not only built Khan Academy, this library of videos on instructional topics and an adaptive tutor and other kinds of tools, but he actually built a regular in-person school. It was a private school. I think it's in Silicon Valley area. It costs $20,000 or $25,000 a year to attend. And his observation after working on this project for a long time was now that I run a school, I see that some of the stuff is not as easy to accomplish compared to how it sounds theoretically. And his new argument was that actually a better way, it's not going to be the case that Khan Academy is going to transform relationships in schools. We are not going to have students who are spending most of their time developing skills through these individual pathways and at the end of their skill development coming together for rich intricate project-based learning. Instead, the model that he's recommending is to teach about the way you had been teaching four days a week and then use Khan Academy as practice problems one day a week. And he says that's doable, that's tractable, and it also has some benefits for students' math learning. The thing that struck me in a powerful way about that argument was how incredibly well-established it was. So you can go back to the 1990s. Here's an article from 1997 published by Ken Catinger at Carnegie Mellon University where they had done in the Pittsburgh schools the exact thing. They had built a series of cognitive tutors, adaptive tutors, that responded to students' responses and gave them progressively easier, harder or properly sequenced math challenges. And they told teachers to use it three or four days a week. I think they told them to use it three days a week or they told them to use it one to two days a week and teach in a regular way three or four days a week. And they ended up, for the most part, if they used it using it one day a week. And they found the same kinds of things that Sal Khan found 25 years later, which is that if you have people teach in a regular way four days a week and then use adaptive tutors as practice problems one day a week, it works a little better than what had come before. Another way to frame this is that Khan Academy, I think, has raised like $150 million in philanthropic support over the last eight or nine years. And what they learned from that investment, you could have discovered with it with a trip to the library. So when I contrast Sal Khan of 2011 with Sal Khan of 2019, I see two different kinds of stances towards education technology at work. And over the last two decades, one of the most powerful stances has been the charismatic stance. And I borrow this term from Morgan Ames, who did a really lovely anthropology of the one laptop per child program that was situated from the Media Lab here at MIT. And she wrote a book called The Charisma Machine and talked about charismatic technologists, people who envision technology as tools that can disrupt and transform and rearrange existing systems and who imagine futures that are brand new and different because of new technologies. And she contrasts that with actually sort of what I see in Sal Khan in 2019 with the tinkering stance that she draws from a book from two historians, David Tyak and Larry Cuban called Tinkering Towards Utopia. And in the tinkering stance, the assumption is not that new technologies will disrupt and transform educational systems, but rather that these existing conservative, complex political systems will domesticate new technologies. They will take new technologies and they will slot them into particular niches for particular students, particular context, particular subjects. And that the future in many ways can be seen as an extension of trends from history, that things change, but they don't change disruptively. They change incrementally. They change step by step. And in some ways, the book Failure to Disrupt, you know, is like a love letter to the tinkers or a passionate plea for honoring and respecting the work of tinkers. And it is to say that the charismatic stance leads us to sort of boom and bust hype cycles around education technology in which we misallocate our resources. And by contrast, there are ways that technology can help improve existing systems for learning, but they tend not to be breakthroughs. Like a lot of things in human development, they tend to be a few steps forward and a couple steps back and kind of maddeningly, slowly plotting, but perhaps ultimately, you know, leading to improvements or potential improvements in human capacity, or at least if we're going to make investments in education technology, that's the sort of stance to bring to it. So the first half of the book sort of reviews what I think is one of the most useful disciplines for the tinkerer, kind of personal discipline, which is to start from the assumption that any new technology is situated in some kind of history. And if we know something about that history, we can make some pretty good guesses about how a new technology will operate. So to illustrate that principle, I look at a set of technologies that I call learning at scale, learning environments for many, many learners and few experts to guide them. Typically, education technology evangelists have not promoted calculators as transformative tools of teaching and learning because there's not a sense that calculator, kind of on its own, provides, you know, a personalized individualized curriculum that is scalable to many millions of people. But there are other technologies we've created in which ed tech evangelists have made that promise and they tend to fall into three different categories and you can define those categories by who guides the sequence of learning activities. So there are learning environments in which an instructor selects the suggested sequence of activities and those are things like massive, open, online courses. There are algorithm-guided large-scale learning environments like adaptive tutors where a computing algorithm measures student performance on some dimension and on the basis of that measure selects some new learning activity to suggest for a student. And there are peer-driven, peer-guided network learning communities, you know, at MIT, I use the example of the Scratch programming language and community where the learning experience of Scratchers is in profound ways shaped by the community of peers that they interact with on the Scratch platform. Almost all of us participate in these kinds of networks in some way if you're interested in how to do makeup on your face in new ways or how to style your hair or how to beat a level in a video game or how to do different kinds of handicrafts. If you're watching videos, posting pictures, reading Reddit threads, you're probably participating in one of those kinds of networks. Each of those three genres has a history behind it. They tend to have similar kinds of pedagogical proclivities. They tend to use similar, you know, to borrow the terms of software developers, sort of similar technology stacks. So the first, and the other thing is that for most of them, we have a kind of track record of efficacy behind them. As the example of Ken Catinger's 1997 research and Sal Khan's 2019 proclamations point out, you know, we human beings have been using computers to try to teach other human beings for as long as we've had computers. This is not a brand new field. This is a 60 year enterprise conducted with substantial funding by super bright people at research labs all over the world. And so if you find a new piece of education technology, I argue that you can sort of ask this question, who guides the sequence of learning activities here? And you can see how it sort of slots into one of these three genres. And if you know something about these three genres, you can make a good guess about how some new technology will shape the future. I think I won't spend a ton of time with this, although we can come back to it, except to say that, you know, for instance, I think that instructor guided and algorithm guided genres of learning technology, they tend to be inspired by pedagogies that emphasize direct instruction and experts communicating information to novices, whereby contrast peer guided learning environments tend to be interested in pedagogies of apprenticeship, where you learn not so much from the received wisdom of experts, but through the process of trying things, interacting with others, sharing your experience. In the middle slot here, I indicate some technologies that I think are quite common to these genres. So for instance, most adaptive tutors, they really have two main parts. They have an auto grader, and the auto grader is what allows us to determine the level of human performance so we can select some other task. And almost all of them, no matter how fancy they purport to be, no matter how many billions of cells of data they purport to collect, no matter how fancy, you know, the algorithms or the parameterization they claim to have, at their core, they almost all use some variant of a statistical toolkit called item response theory. An item response theory was developed by the Educational Testing Services in the 1980s. And I'll describe it to you now, but I take maybe three pages in the book to describe it to readers, with the point being that this is not intractable. This is not impossibly complex, which is what education technology evangelists often try to convince us of. Like the fundamental building blocks of this are well known and they're usually old. You know, if you see a sort of EdTech Roadster coming your way, like lift up the hood and you will see like, oh, that's a pretty old engine or that's a pretty old chassis or there's some well-established pieces here. And if we understand those well-established pieces, then we can make some good guesses about how a new technology will operate in the future. Or we can identify the pieces of a new technology that are in fact distinctly new. Like it's very unlikely that the whole thing that we've created is gonna reimagine, you know, pedagogies that we've been working on for thousands of years in our society. But maybe they have some particularly interesting tweak. And if you can identify that particularly interesting tweak, then you can think of that as a place for study and exploration and other kinds of things. And I can talk in more detail in doing the book about, you know, how I think a math department head should examine this history of technology when making a decision about how to implement a new piece of math software in their school, how a vice provost for information technology might do that, how a researcher might identify what kinds of interesting problems are out there, how a designer might think about how to approach the development of new software. So in the second half of the book, what I try to tackle is making clicking work, are what I call four as yet intractable dilemmas. Four issues that come up time and again across all of these different genres of technology, problems that if people interested in technology at different levels, funders, developers, researchers, implementers, teachers, students, if we thought about these problems in creative ways, if we were diligent about trying to address them, that would be our best chance of sort of tinkering our way to success. And the problems emerge from three key features of educational systems. Three key features of these systems that I think are really central to understanding why technology doesn't lead to disruptive society shifting change in education in the way it does in other fields. And the first piece of the education landscape is that education is just immensely almost unfathomably complex. Somewhere today, there was a teacher who was a seventh grade earth science teacher who was starting a new unit on plate tectonics. And somewhere else, there was a kindergarten teacher who across a Zoom video conference was trying to explain to students how to open a new tab in a browser or how to tie their shoes. And somewhere else, there was an advanced Mandarin class being taught in a liberal arts college, somewhere else. And then somewhere else, there's a two-year degree program where people are learning how to be radiology techs. And it's not just that those subject areas and those contexts are different, but what you do, the interactions you have with your students day to day, the content that you cover, the skills that you address, they are constantly changing across the 180 days of a K-12 school year or the 28 weeks of two semesters of higher education. It's extremely difficult to think about how technologies could be built to address all of those kinds of diverse use cases. And indeed, the technologies that we have are very uneven. They work well in some circumstances, but not others for some people, but not others in some contexts, in some subjects, but not others. And I think that is one of the sort of fundamental misunderstandings that people often have of when they predict sweeping changes from education technology, they're imagining that our technologies, and I haven't found the right metaphor here, but they're like bulldozers or they're like Swiss army knives. They sort of clear everything out or they do everything, but our technologies are not that way. They solve very particular problems typically. They're like very specific pegs and the complexity of education is just this manifold, huge sweeping landscape of lots and lots of different kinds of holes. And then our education system, especially here in the United States, but certainly in lots of other places in the world is shaped by profound inequality. We provision students in their schools and students in their homes very, very different levels of resources with which to approach the challenge of education. And that shapes every part of our education system. And certainly some of the saddest thinking and reflecting to be done about the pandemic is how it is both revealed and exacerbated those inequalities. So I'll briefly talk about these four dilemmas and then maybe I'll stop for a bit and see what questions or thoughts that folks have. But here are four of the kinds of things that I think whether you're interested in network learning environments or learning games or adaptive tutors or MOOCs, I think these are problems that sort of cut across these different kinds of approaches to learning at scale. So one has to do with complexity and I call it the curse of the familiar. If you build a technology that is familiar to people, you can get that technology adopted. The most widely used technology, education technology maybe in American schools is a tool called Quizlet, which was created by an MIT dropout and a former students of Eric's and the terrific guy. And Quizlet lets you generate online flashcards. And when you glance at the operation of the Quizlet website, you will instantaneously understand what's going on. You'll go, oh, this is a flashcard. Like I'm already a question on one side and answer on the other side. I'm gonna test myself. I'm gonna share the decks with other people. These are flashcards. Because it is instantaneously recognizable, it can spread very, very widely. But if we sat down a bunch of experts in the American education system and said to ourselves like, what are the real problems that we face here in education with inequality, with the challenges of the future of the labor market? Like what do we really need to work on here? I think very few experts would come up with the answer like, man, we really have a dearth of flashcards out there in the schools. Like we really have to improve flashcard access for children across the United States. The new flashcards are neat, but they, because they don't offer any kind of substantial change in the way that teachers and students interact with one another with the content, they are very unlikely to lead to substantial improvements. They gain a certain number of inefficiencies and they help students memorize some things better, but they're not sort of unlocking new pathways of teaching and learning. By contrast, when we do build things that unlock new pathways to teaching and learning, users often find them confusing. So if we create things, new technologies that incorporate alternative pedagogies or create new routines or relationships between teachers and students, it's often the case that this confuses teachers and students and these things sometimes get passionately adopted in small niches, but very rarely spreading scale across systems. In the cases in which we do have some of our most interesting technologies sort of get into schools and start to spread and start to change the way teaching and learning happen, they tend to do two things. And this is where in the second half of the book, I try to propose some, so not sort of pat solutions, but some approaches to addressing these intractable dilemmas. They tend to be able to be used for familiar ways initially and then span out into new kinds of opportunities. So Quizlet gets in in a very simple way with flashcards but doesn't really take you anywhere beyond flashcards. I have colleagues who built this graphing calculator tool called Desmos, which at first glance just does everything a TI-84 calculator does on your computer for free. But then beyond that, there's a whole set of ways of using this graphing software that enables a whole different kind of approach to teaching and learning in mathematics. And so it meets people in a familiar place and takes them somewhere else. And then the second thing that people do when they successfully navigate the curse of the familiar, and this is sort of a theme that cuts across both the book failure to disrupt and a lot of my current critiques of how we're addressing online learning during the pandemic, is that the people who are good at taking technologies and spreading them and having new pedagogies spread with them, they don't assume that new ideas will travel with the technology. They assume that they have to engage communities of faculty members, teachers and learners in pedagogical exploration about how to do new things. So the MIT Media Scratch Lab, they built Scratch but they are also in the midst of building this giant apparatus to teach people all over the world what are the pedagogies of computational creativity that are associated with Scratch because Scratch by itself, if you sort of drop it into schools, it will be domesticated by those schools for conventional teaching and learning but it provides an opportunity for allowing people to reimagine those ideas but only if faculty and students and communities are supported in doing that learning and doing that rethinking. It's a way of thinking about scaling of new ideas not through technology distribution but through movement making, through community building, through the kinds of things that we see in other forms of social movements. A second challenge I call the EdTech Matthew effect and Matthew effects are commonly observed across sociology. They, it comes from a line in the book of Matthew which is paraphrasing like to he who has much, much more will be given and to he who has little more will be taken away. There is a persistent story in education technology that it will have the capacity that new technologies will democratize education that they will make education more free, more fair, more just. My colleague Larry Cuban who's an emeritus professor at Stanford has marked how these arguments go back to the days of radio. He's a great book called Teachers and Machines in which he has a picture of a bunch of young people sitting around a radio receiver that's the size of a small child, you know, one of the big standup units. And the caption is, you know, with radio the underprivileged school becomes a privileged one. And you all know that radio did not, you know squash the inequalities that occur between our schools in the United States and nor will other technologies that we develop on their own. New technologies disproportionately benefit the affluent because those people have the financial and social and technical capital to take advantage of new innovations. And it's only through deliberate efforts at really thinking about what would it look like to create technologies that close gaps rather than spread us further apart through things like do our technologies, you know can we measure and assess or observe how people from different backgrounds and life circumstances use technologies differently. A tragic thing that we're gonna research over the next year and learn in different ways is that white children in American schools and black children in American schools are not gonna be treated the same way on Zoom. That the behaviors of the subjectively inappropriate behaviors of black students on Zoom calls will be policed in ways that they are not for white students. And if we don't find ways of looking at what's happening during the current pandemic and looking at it through the lens of how do people from different backgrounds and life circumstances experience technology differently, then we will miss the opportunity to learn about those things and figure out how we might be able to address them. A third challenge relates to unevenness which is the trap of routine assessment which is that many of our large scale learning technologies depend upon automated assessment. And we have some domains in which we do really good automated assessment and some domains where we don't. If you ask someone a question with a well-defined right answer, a computational question, a question in a physics system where the laws of physics are well-defined, we can build good auto graders that evaluate those responses from learners. One of the fields in which we've built the most impressive auto graders is in the field of computer programming where computer science professors and teachers can assign their students computer programs to write that have to meet certain engineering challenges. And they can create computer programs that grade those assignments. And when you can do that in an automatic way, you can build systems where people at their own pace and time can participate in learning experiences, get some feedback from an automated system, be motivated and inspired by their progress or be supported by different kinds of feedback and then proceed and move forward in their learning. There are lots of domains where we can't do this very well, probably the most important one is that we really don't have good tools for evaluating writing, for evaluating people's ability to reason from evidence. This is a problem because much of what we teach in the liberal arts education is how to reason from evidence. Perhaps most problematically, the things that we're good at building computers to assess tend to be the kinds of things that computers are already good at. They're highly structured, highly routine kinds of problems that arguably we don't really need people to do that much anymore. By contrast, the things where humans have a comparative advantage over machines in the labor market or sort of equivalent concept in the civic sphere are areas in which we don't have very good automated assessment. So the problem here is that we're really good at creating assessment systems for things that we don't need people to do anymore. And this is not because there aren't smart people who are working on this problem, decades of work by very, very smart people in technology companies and universities have thrown all kinds of resources at this challenge. And instead of having some kind of Moore's law-like exponential growth and improvement, we've seen very, very little progress here in decades. And I think that should give us some humility and caution about our predictions for the future. For lots of education technology evangelists, it will make a prediction. And then when it doesn't come true, let's say, ah, we just need a little longer. Thomas Edison in 1913 said that by in 10 years, all textbooks would be replaced by the films that he was producing. And then in 1923, he gave, I think it was in front of the FCC and he made a similar argument except, ah, he said, well, actually it's gonna take 20 years, but it'll happen that all the textbooks will be replaced by film strips. And we still have not replaced 100 years later all of the textbooks with video materials because it turns out that print is a pretty good media for learning in a lot of different ways. Then the last challenge that I map out is that if you're someone who's excited about building and improving software platforms, you're probably very interested in the large amounts of data that these systems can collect and then the ability to rapidly run experiments that let you test how changes to the software platform affect people's experiences. And there are all kinds of regulatory issues with this and there's all kinds of cultural policy issues. There's some sense in society that if I go to Amazon to buy a book, that they should be able to collect some data about my experience there. And that if I find out that sometimes they're doing randomized controlled trials to see if I'm more likely to buy a book with a blue button or a red buy button, that at least I as an individual entered into that learning experience, into that software experience somewhat of my own free will and volition, not entirely. But as wicked as the problem is in retail, it's much, much worse in education because right now all across the world, young people are both compelled to go to school and they're compelled to use the software platforms that are assigned by their teachers and school systems. And so there are all kinds of reasons to be very seriously concerned, both about the immediate risks of surveillance in these systems, but also about the long-term kind of educational social risks of normalizing and socializing young kids to live in surveillance cultures. There are lots of circumstances in the social sciences where we think experiments are like generally good. We tend to not intuitively think of experimenting on young children learning as a thing which is good. I think there are ways to do these things responsibly, but I also think that there are real serious concerns that are embedded in these practices. And if we want to see the same kind of improvements in learning technologies that we've seen in other kinds of retail software platforms, we're gonna have to resolve and negotiate some of these kinds of challenges. So that's what the second half of the book tries to address, it tries to chart out, the first half is really more of a history, the second half is more of an engineering text of trying to say, here are four common problems that we run into all the time. And here are some approaches that I think might be able to solve them. And as I've reflected on what the lessons of this, in the prologue I mentioned that the final copy editor to the book were done like March 23rd. The book was written right at the end of one era and just at the beginning of this new one, and a lot of what I've done over the last eight months is both try to be helpful, but also try to reflect on what do we learn from the pandemic that's salient from history? And I think two important lessons at least come through. One constant mistake that education technologists make is to describe technology as sweeping, to describe it as something that can sort of sweep away existing systems and usher in new ones. And instead for a variety of reasons, our education systems are conservative institutions because they're extraordinarily complex and they're managing all these different competing interests. And as a result, new technologies tend to be domesticated. We tend to slot them into existing functions, which is why we perhaps shouldn't be surprised that all the faculty walked away from their lecterns and went to their home webcams. And you can decry that as a sort of pitiful conservatism in a system, or you can say, wow, this system is so well honed to meet its competing interests that it's actually kind of found a local maximum, a local optimization of all the different resources and competing constraints. And then the second problem that I think technologists often make, which is quite salient to our moment now, is that technologists often describe education technology as a switch that you can flip on and on. You sort of buy it and install it and then it works. And that's not at all how education technology works. Education technologies are only as useful as their communities of users are well supported and strong. They are powerful tools for rethinking, learning for imagining, iterative and continuous improvements to learning. But there's very, very few things that we've created that just kind of instantly in meaningful ways benefit learning, rather they become useful and we're seeing now all over the world, millions of faculty members engaged in the process of asking themselves, okay, I'm forced to use this new technology now. What am I doing? What are my colleagues in my discipline or my subject or my school doing? And how do I have to rethink my practice or rethink my approach to have more powerful, more effective, more connecting, more inspiring experiences from learning? So those two things of technology is particular and domesticated by systems rather than sweeping and the strengths of technology not being its instantaneous effects, but by its ability to be absorbed by community of learners are perhaps the two things that I'll leave you with before taking some questions. I just want to invite everybody. If you're currently a panelist, I think it's fine to just unmute yourself and call out at this moment. Just be mindful of someone that speaks up before you. If you're a guest, feel free to put a question in the Q&A. I'm gonna start with a question then because no one else is calling out. I know in the beginning of my work in educational technology, and I realized it was a fallacy, but I had this sort of, I guess it was a wish that if a technology, in my case, game sort of made, that it could somehow make people who used it aware of whole different models of pedagogy and when in fact, what we've cost learners that teachers tend to, even if they're using something as radical for them as a game, they're gonna tend to fall back on the same pedagogies they've always used. Are there exceptions that? Or is that, as Eric point, yes, mentioned, the Trojan mouse, that was the term that he always used. Are there no, is there no evidence of that? Is there some evidence of that, of the technology actually shifting the thinking of the people who adopt it? I mean, my view is that there's not good evidence of the technology, or maybe three things. There's not good evidence of the technology in and of itself shifting people's views. A second thing that there is good evidence of is that technology has an ability to catalyze thinking about those kinds of issues. So it's possible to, you know, you can go into a group of faculty members and say, the future's gonna be really different, the world is changing, our learners have a whole set of new experiences, and as a result, we think you should rethink your relationships, your curriculum, your pedagogy, all those kinds of things. And many faculty respond with like, no, thank you, we're doing fine. And then you bring a learning game into that same community experience, you know, or some other form of technology, and it, you know, operates as kind of like a symbol of the future to be able to say, oh, like we could do things really differently here. We could, you know, this gives us new opportunities and new affordances, new ways to rethink things. And so I think it can catalyze those conversations. And then another thing that we know is if people do move from more familiar practices to new practices, it's a developmental process. You know, Judith Sandholz did this project in the 1980s called the Apple Classrooms of Tomorrow, where she, with a team of folks, you know, got some K-12 classrooms and got a bunch of Apple II Plus computers and had wired networked with, you know, some of the first networked computing environments and they had robots and printers and things like that. And she described teachers going along a developmental trajectory from using things in more familiar ways to using things in slightly novel ways to doing really imaginative work. Faculty members travel along that continuum at different rates to a limited extent. They start in different places, but really a lot of them start at that first spot. You know, how do I use this to extend existing practices? And then I can think about new and different and interesting kinds of things. So, you know, and that's why I think it's so important to pair the idea of technology integration with the idea of community learning at the same time. Like, can games do that? Sure, but not when games fall from the sky. It's more likely to happen when games arrive with a community of people who are interested in teaching and sharing these ideas. I mean, Mitch Resnick has a great line that he had with the release of Scratch 3.0, which was something like, you know, we're gratified at how widely Scratch has spread in schools and we've been somewhat surprised at how challenging it's been not to spread the technology of Scratch but to spread the ideas and the pedagogy behind Scratch. And, you know, one of the things I admire about that group is that, you know, rather than saying like, oops, disruption didn't work. Like, let's go try something else. They're like, okay, great. You know, let's work in a really diligent, devoted way on that human development problem as well. I have, again, people, if you're, feel free to talk, but I've got some questions in the Q and A. Someone said, I very much enjoyed logic. Amuk, what do you see regarding the future of this platform? So I have a chapter about Mook's instructor-guided learning and I claim that they had three big bets that they would provide new pathways for different kinds of people into higher education, that they would unbundle and rearrange higher education systems, and that they would usher in a new era of data-driven learning science. And I think none of those things basically has happened. I was one of the people who was working pretty hard on trying to usher in a new era of data-driven learning science. And I don't think that we've been particularly successful. And I think we may make incremental progress, but I don't see breakthroughs on the horizon. What ended up happening with Mook's is that they were domesticated into the existing higher education system for the most part. Many, I have a whole section of it. Many Mook providers have started describing themselves as online program managers. These are people who largely create professional master's degrees in a set of topics that are amenable to being taught online. And they have a familiar business model of paying upfront for the costs of course and curriculum development and then taking an ongoing fraction of student tuition revenue after that. There are some respects involved outsourcing the core competencies of universities, which generally across businesses thought of as a bad idea. You outsource your janitorial services. You outsource your accounting. You outsource things that are peripheral to the core operations. You don't outsource teaching and learning. You know, I think Mook's are just a terrific illustration of the incredible power of education systems to domesticate, to incorporate new ideas. And there are a few places where I think there's some really neat things happening. You know, the Georgia Tech has an online masters of computer science, pot through a series of MOOCs, which is like 7,000 people at any given time enrolled in it. You know, it'll probably end up graduating on an annual basis, 16% of the computer science master's degrees. But it was not a harbinger of a whole new set of higher education. There was like one particular subject area at one particular university, which has captured a modestly large niche. And there'll probably be some other, you know, accounting degrees or data science degrees that like kind of work the same way, probably not as well. And, you know, and it's better to understand, you know, I think the saddest part of the MOOC story is that there are lots of universities that took limited funding and invested enormous resources and saw very few benefits of them because they weren't, you know, one of the lucky few early adopters or people who got things just right or institutions at other places that have sort of bottomless funds for these kinds of things. And I still, you know, I make MOOCs, we launched one last week called Sorting Truth from Fiction about civic online reasoning. They're really good at teaching already educated, already affluent people. Some of the already educated, already affluent-ish people in society are teachers and teachers lead busy, complex lives. And MOOCs are a great way to reach lots of them at relatively low marginal cost. But that's, again, sort of slotting into a particular niche into systems in which we're hoping to incrementally over time improve our practice at them rather than arguing that, you know, we're on the cusp of a transformation in teacher education. And I see we have a question from Anbar. Hi, Justin. Thank you for your talk. I was wondering a question about related to students and what about advice do you have for students like in the current educational climate, to make the most out of this experience right now? You know, what advice do I have for students? I had a whole set of slides that I was sort of clicking through as I was talking, I just realized that I forgot to put them up. But I also think that sometimes it's nice just to hear people chatting, but I at least wanted to have a link to the book up there. Yeah, my advice for students is, first of all, I think students at all age levels and around the world should celebrate all of the resilience that they've shown and all of the unconventional learning that they're doing during this period. So I think the most common narrative around learners right now is some kind of deficit framing around learning loss. Like, oh man, these kids are just not learning all the stuff that they were supposed to learn in the curriculum, they're gonna fall behind. And there are very serious issues of that and they're very serious inequality issues of that. But that's not all that happened. I think there are lots of people around the world who have spent this time as part-time or full-time learners who've learned all kinds of awesome stuff about negotiating technology as learners about showing more independence and more self-directed learning. And so I think, you know, I'm a big fan of sort of asset approaches to thinking about these things. There are not, there's not a set of good sort of simple advice along the line of study tips that make people better online learners. If there's one basic one, it would be something like, people often do, this is true for everything, but particularly online, people do a better job learning things they really care about and they're interested in than people than what they were, sort of what you're being forced to take. You know, if you were a freshman at MIT right now, I would encourage you to save some of your GIRs, some of your required courses that you're not that excited about for when you come back and do the things that you're just most intuitively interested in now because motivation is really a central part of learning. And then a piece that I say mostly to faculty, but I think applies to students too, is I hope we approach this period of pandemic learning as something that we're all in together. I'd really try to encourage faculty to say like, how can you partner with students in designing your response to this? Because there's exactly one group of Americans, one group of people around the world that have been learners during a pandemic and it's the students who are in our classrooms this spring and this fall. And they know a lot that we don't know about what good teaching and learning in a pandemic looks like and we should really listen to them. So maybe that's a plea for you all as students to try to share that with your faculty and to be in those partnerships with students. And we do have some questions. I'm gonna take them in the order they share. Will, you have a? Yes, thank you. I was wondering if the kind of like shifting views about timescale that we've been considering with the pandemic have like changed people's stances from either like a charismatic stance towards like interventions or just like a bandaid kind of, we're just gonna deal with it for a couple of months and then we'll go back to normal. To more of like a tinkering stance of, maybe we're gonna be in this situation for a year or more so we can think about like slower kinds of experiments and other sort of solutions that might take a longer time to figure out if that's something that you've noticed and if so like what does that look like if it's coming from educators or technologists? Well, I'm always amazed at the technologists like, there's some people who are just really devoted to the charismatic stance and like nothing will stop them. So there's a guy, Michael Moe, who helps run Arizona State University's partnership with GSV and he had this, they ran a series of events called like the dawn of online learning. It's like, didn't we have the dawn 10 years ago or 20 years ago? Like, maybe you don't think dawn is like a watershed moment in history. Dawn is just a thing that happens every day early in the morning that we all kind of sleep through sort of these repeated cyclical pieces. So the charismatic are still out there but the thing which is like really taking the wind out of the charismatic sales, I think, is that like this should have been their moment. And I just see like no hue and cry for the kinds of large scale learning technologies to an extent that has even surprised me. I mean, at the beginning of the pandemic I told a lot of my colleagues at MIT, the odds that you can in the midst of a pandemic like go home and sort of whip together a decent online learning experience for your students is pretty low. And you've built a bunch of stuff already on OpenCourseWare and on MITX. Like just point people to that and kind of help them out. I actually thought it would have been more a moment for MOOCs and some other kinds of things. But it turns out that at MIT, and as far as I can tell in lots of other places around the world as well, it's not what people wanted. There has been nowhere that I can tack some groundswell of students saying like, my introductory microeconomics professor is doing a crappy job teaching us intro micro online. And I just want to be able to take a MOOC and learn it myself. Instead, I think what we overwhelmingly see is like people really do want the connection to their individual professor who's like doing lousy job, managing kids in the background and putting together their first online course. Because I think that human connections enormously important. So I mean, the whole purpose of the book is to try to inoculate educators from future hype cycles to try to convince, the next time someone comes around and says like, oh, it's gonna be AR or VR. It's gonna be data science, artificial intelligence is gonna change everything. It's to say, well, like that's very unlikely to happen. And there are gonna be a lot of places that don't spend resources wisely chasing those kinds of pursuits. So I mean, one of my answers to your question is like, of course the Tinkers are gonna win now, but you should be cautious because that's who I'm rooting for anyway. Mike, you wanna go? Yeah, sure. Apologies, Justin. This is a little too heavy of a question, but it's something that I'm really curious about throughout your talk. So maybe in like a charitable way to assess why the charismatic technology and charismatic technologists are so successful is because they are appealing to these values which are like pretty noble, right? Democratizing and like fixing inequities. And like, I think what that Morgan names book is so great at showing and what your presentation is great at showing is like, you know, technologists of technology, you know, if you wanna fix that stuff, you have to fix inequity and you have to fix democracy and all that type of stuff. And this is a moment where we're kind of having this wrecking of like, okay, there are a lot of issues that we have with our system set up. There are a lot of issues with what we expect technology to do. And there are a lot of issues with the people who made a lot of money from technology. So I guess the question is, you know, in a system that has previously assigned to the value of certain technologies and education trends to monetary value that they can get from investors and nonprofits and all of that, what might creating a new set of values are going to a more fundamental set of values while dealing with this technological space look like, right? Like maybe we can't fix, we can't democratize technology with Zoom or with Khan Academy, but maybe we still need to democratize and we have technology. What might that look like? I mean, I think the stance that the skeptics that I find most compelling take is that the project of education is a fundamental part of civil society. It should be fundamentally thought of as a public good. It should be sort of fundamentally democratic and therefore accruing enormous power to technology firms to be able to influence these environments. No matter how well-intentioned they are, no matter how useful their products are, it's the wrong people doing it. I think it matters a lot that Zoom is a piece of consumer software sold in a publicly traded firm and Scratch was developed in a research laboratory at a university and then transferred into a nonprofit entity, which is actually funded in some ways by hedge funds and other things. There's all kinds of problems there, but yeah, I mean, I think that's what you're getting at is absolutely sort of a crucial theme of the book, which is that all of the technology doesn't in and of itself solve all of the hard problems. There are ways in which technology reveals these problems. The fact that we can't get broadband access and we can't get computers to literally millions of the 57 million school children in America is revealing, exacerbating the same time, but revealing new kinds of inequalities as well. So I'm hopeful that the moment leads to, more than anything else, sort of social movements that demand that we do more for children and their families in our society. And I hope that we look at them through these lens of technology, but the problem that skeptics have right now is that like Zoom and Canvas are the only games in town, Zoom and Google Classroom are the only game in town. You can't actually critique and resist them, perhaps if you were to launch a full-throated critique resistance, sort of Luddite smashing of them, you'd be like, well, now we're just like mailing paper packets to kids and that's pretty terrible too. I mean, to go back to this idea, it's actually, in other writing, I've sort of contrasted skeptics and tinkers and Charismatics and position tinkers as a middle way. And there's a lot of really good education technology skepticism that's out there. I think it is challenged in this particular moment by the fact that if we want schools to keep operating in a reasonably functional way, online tools, including ones created by monopolistic corporations are gonna be pretty central to that. But I think it's part of a broader movement in society of saying we should not tolerate these monopolies operating however they want to because they're monopolies, we should regulate them. And in some cases, we may really should be looking at publicly funded alternatives to them, especially for projects that are as close to civil society as schools. Does that get at some of your issues? Yeah, absolutely, that's great, thank you. Great, Kelly. Hi, this is maybe also kind of a big question, but one thing that I wonder about is how you teach topics related to privilege and things like racism to people that have already left formal education. And it seems like one way to reach people is through technology. But I also kind of dislike the like Instagram size. Here's how you learn about racism. And I'm wondering if you have any thoughts on that or if the answer is just you have to have individual conversations with people in real life? No, what a great question to ask because there's been such a flourishing of that. I think there are a whole bunch of white privilege syllabi and Google Docs and reading clubs and other kinds of things that have been generated in various kinds of ways. And so in media that's not explicitly educational, The New York Times put out a fabulous podcast through serial called The Nice White Parents, which is a terrific five episode investigation of those topics. And one of my colleagues at MIT is a professor named Peter Sanghi at the MIT Sloan School of Management. I mean, he wrote this book called The Fifth Discipline and he defines really successful organizations as learning organizations, as organizations in which the process of getting better at things, in doing your day-to-day jobs in performing the function of your firm, you should also be creating opportunities for all of the actors in that firm to learn. These issues of anti-racism bias white supremacy, they're of keen interest to corporate America who are generating all kinds of learning experiences. They're probably millions of Americans who do in their workplace, are privileged to participate in or subjected to, depending upon their point of view and the quality of these things, sort of learning experiences. So yeah, I'm quite interested in those kinds of questions. And part of the reason why I stay in education technology is like I just don't know, it seems to me that all kinds of media are gonna be central to addressing that challenge. Another challenge that I'm personally really interested in right now is people don't know how to search effectively online and to sort truth from fiction. My colleague Sam Weinberg at Stanford has done research on lots of groups, including middle school students and Stanford freshmen and tenured historians who by and large are terrible at identifying misinformation in falsehoods online. And he's also studied this one group of people, fact checkers at prestigious news organizations who are not only extremely good at sorting truth from fiction online but are like quite efficient and use a series of fairly simple techniques. So I'm interested in this question like, there are about 3 billion people connected to the internet and we need to teach them all how to do this. We need to teach them in schools, we need to teach them in libraries, we need to teach them in their corporations, we need to teach them in senior centers, we need to teach them in civic organizations. Like how are we gonna, and when you think about the question from that scope, you're like, well, there's gonna be some ed tech in there somewhere because that is a huge learning goal that we should have for people in our society. I mean, that to me is that's the thing for me that keeps pulling me sort of away from skepticism is just this yawning need for learning felt by billions of people around the world that it seems like we should be able to figure out how to use these tools to address. Great, I have a couple of questions in text. I did see someone with a hand up who then put it down. If you do have a question as a panelist, if you put your hand up, use or use the little tool to put your hand, the virtual hand up, I'll be able to see you. But in the meantime, let me get with these questions from the Q and A or from the chat. One was about, you mentioned the technology cannot be expected to work as a switch say that people need support from a community. How do you envision this kind of community to take place in an online environment with MOOCs? Have you seen any platforms today that do a relatively good job? Yeah, the simplest answer to that question is that MOOCs don't provide that support. MOOCs are good for self-paced learning and most people are not good at self-paced learning and the people who are good at self-paced learning tend to have had a formal apprenticeship in the educational system. And so if we want to sort of support community learning, we should turn to other approaches of technology. I think there are neat things that have been done, incorporating MOOCs into community-based systems of learning to have people take them together in libraries, to have them built into school systems in different kinds of ways. But I think also to like, it's important when we see limits to recognize them and say, don't try to solve a problem with a thing that's not gonna work. I mean, maybe keep doing research and experimenting in different kinds of ways. In the MOOCs that we create, we try to support people in creating those community supports and social learning environments by encouraging folks to take our courses in groups, to take them in learning circles, but we're not because our learners are teachers, we know that they are embedded in these social institutions that have mechanisms for supporting community learning anyway, which is not necessarily the case for people who wanna become computer scientists or wanna become accountants or data scientists or other kinds of things like that. So it's sort of like having healthy respect for the limits of technology and saying, oh, that looks more like a social problem than a technology problem. Another question is, do you see technical innovation in means of awarding learning credit, grade, certificate, diplomas? Do you see that as having any significant effect? You know, the place where I think there's been the most discussion around this is around some like badging and micro-credentials and things like that. That has been an interesting phenomenon to me because it's a place where people have been able to generate supply and demand has not followed. Like people have built it and others have not come. And it strikes me as a place where people, especially for folks who are trying to use some form of micro-credential in a labor market, just like fundamentally misunderstood human behavior in the labor market. They're saying like, what we wanna do is like give people who are hiring or people who are admitting people to graduate programs like really granular data about people's abilities. And most people hiring folks do not want really granular data about people's abilities. They want like very simple summaries that allow them to go through hundreds of resumes at the same time. And by the time you get to the sort of two or three candidates that you really wanna go into, you don't wanna see a micro-credential or a badge. You wanna see like actual evidence of that person's performance, which some micro-credentialing systems have. But I think it was an example of people saying like, well, here's something that's possible. And if we could change human nature, then it would allow all these really kind of interesting things to happen. You could start sort of mixing, matching, micro-credentials from different places or things like that. And it's not to say that also that not all that innovation, there are still spaces for innovation, but also to recognize like, there's lots of ways the system has already done this work. I mean, MOOC people have been sort of crowing for a while about like, oh, we can create these sort of like new micro-credentials that are assembled in different ways. This is like a brand new innovation in higher education. But in fact, like in Britain, in I think the end of the 19th century, they developed junior colleges and community colleges and they invented a degree called the associates degree. And that was a kind of micro-credential or a stackable credential that built into another kind of degree program. And in fact, like the heavy lifting of inventing that happened 110 years ago, and there's the spaces for innovation that are left are sort of fewer and narrower. Most of what MOOC based degrees have done, micro-masters and things like that, they've created associates degrees, but for people who already have degrees, they've created like easier pathways into master's degrees, but people who have master's degrees are for the most part already affluent and already educated. So again, I mean, in the courses that I teach at MIT, especially in undergraduates, the main thing that I try to communicate is that developing effective technologies requires a rich understanding of the social technical systems in which they'll operate. And so many of the false starts in education technology are because you have people who are really good at programming and react or whatever else it is, and they don't understand complex social technical systems very well, and they don't understand actors and educational systems very well. And so they build things that don't work very well. And I mean, that's one of the joys of sort of teaching in comparative media studies is just having people who take these things very seriously. One more moment for, I have one last question from the chat, but I wanted to see what anyone else here among panelists had a question. And the question in the chat is, how could teachers and students use technology without being overwhelmed in learning and trying new things? I mean, that's a great question. And I like just such a wonderful question of the moment because so many people have been thrust in against their will to these challenges. And certainly it has been some of the sources of deepest frustration, I think particularly for students and families, who's feeling overwhelmed by all of the different things they need to learn to participate in school. And in normal times, you would say, pace yourself. I mean, we have all kinds of heuristics that we use to help people address that challenge. We say things like, if you're a teacher and you're thinking about incorporating technology, identify a target of difficulty. Identify an area in which you're teaching something that's really important to you. That's really hard to teach and where you think technology might have some leverage. And through that kind of three-part rubric, most faculty can identify some part in their curriculum or syllabus. We're like, oh yeah, I'm not happy at how people learn these things. And I bet there's a way of doing that better. And that's one pathway for some people that gets them excited about working on that in a way that feels manageable. The challenge in pandemic times is you sort of have to do that for everything all at once. And I mean, my main piece of advice there in this moment is just to show yourself and the other people that you work with a great deal of grace. There are lots of conflicts that are happening between teachers and families, between students and teachers, between teachers and school systems. And they're because we as a country have failed to manage the pandemic. And one sad thing about that is you have like lots of local actors getting into conflicts over problems that are created by broader social systems. So there's no magic trick to not getting overwhelmed. Pace yourself to the degree that you can. Show yourself and the others around you some grace. But then also recognize, and this is the hopeful piece I came back to, if you can find one of these holes where technology peg fits really nicely, it's wonderfully satisfying. And builds human capacity in a way that if you're patient and willing to take a sort of tinkerers mind frame and be comfortable with continuous incremental progress, then my colleague, Ken Catinger, says that the step change is just what 20 years of incremental progress looks like from a distance. Good, if folks are interested in continuing these conversations, there's a book club that we're doing, a free virtual book club at 3 p.m. on Mondays at failuredisrupt.com slash virtual book club and would welcome any of you to drop in for one session, a lot of sessions. If you go to the webpage, there's a whole list of guests and other kinds of things that are there. If people have other questions, you can find me on Twitter at BJFR or MIT and other folks know how to find me. Thanks for some great questions and thanks to Scott and Eric and Andrew for hosting. Thanks very much, Justin. And thanks everyone for coming. We had a great turnout and we look forward to seeing as many of you as possible next week when Jim Warren will be presenting. So thanks everyone again.