 So we are happy today to have Justin Reich with us. He is a fellow here at the Workman Center. He's a former high school teacher, teacher of world history, as his time might give away. And he recently completed his dissertation at the Graduate School of Education here at Harvard, where he looked at the use of social media tools in K through 12 classrooms in the US. In addition to being a fellow here, he's the co-founder and co-director. I'm looking at this because I want to get it right. Which is a professional learning consultancy that helps schools and districts leverage technology to create student-centered, inquiry-based learning environments. Take it away, Justin. Well, thank you, everyone, for coming from far and wide to hang out with me for the afternoon. That's a pretty good introduction. So I won't say anything else about myself. The one other thing I will say is that if there are people who are watching this on the live stream, hello. Thank you for doing so. And I've got my iPad here. And when we stop and break for questions, if people out in the interverse want to tweet a question, either at Bejuffer or to the hashtag hashberkman, then we can try to take some questions or have some conversation that's coming from the rest of the interverse as well. I think it is an incredibly exciting time to be in the field of education. And we're beset with all kinds of challenges and difficulties. But the blossoming and flourishing of collaborative media production tools, of online resources, of online tools and content that can make education richer, makes it an incredibly exciting time to be a teacher. When I think about this, I often think of my seventh grade US history class. In my seventh grade US history class, we were lucky because we got our own books and we got two books. We got a textbook and we got a primary source reader. And the primary source reader had 20 documents in it, from the Mayflower Compact to the letter from a Birmingham jail. And those 20 documents were the only primary source documents that we were going to read for the entirety of the year. And I was trying to calculate, but there must be, if I were to teach that same seventh grade US history class now, there must be five or six or seven orders of magnitude, more primary source documents available to students now. From the National Archives, from the Smithsonian, from the Library of Congress, the explosion of resources that are available to young people is, to me, incredibly exciting. And it's incredibly excited to be part of the movement of people who are trying to think about how these tools can productively support better educational outcomes for kids. But I also think it's a time as the movement around education technology, particularly around open educational resources and free resources, as this grows and becomes more mature. I think it's a useful moment to also pause and to think about how education technology innovations will have an impact across people from different kinds of backgrounds and different kinds of neighborhoods. One of the things that we know about education in the United States is that it's profoundly inequitable. There are 14,000 school districts across the United States. Half of the dropouts in our system come from 2,000 high schools. The concentrations of poverty, the concentrations of schools that aren't meeting students' needs is really extraordinary. And so to me, it's really important to think about, as we put free and open resources out into an ecology of schooling that we know is profoundly inequitable, how are those inequities going to interact with the work that we're trying to do? And I should say that I'm primarily, probably my first love is as a classroom teacher and then as a researcher, I'm really interested in what things look like in classroom settings. And so this to me is sort of a new foray into thinking about policy, but trying to think about policy informed by what we know about what goes on in classrooms. And I think this will, a lot of what I say will probably reflect quite a bit on things that happen outside of classrooms too in informal learning environments and after school. And I'll talk about that a little bit, but my sort of first love is what happens in schools. So let me start by telling a couple of stories. And all of these stories are to a certain extent characters and they're to a certain extent oversimplifications. But let me tell these stories and kind of map out the landscape of education technology and free and open resources and think about how they might differentially impact different populations and then we can sort of make these stories more complex and more nuanced. Let's start with some gross oversimplified characterizations. So there are a bunch of people out there who make stuff, who make social media platforms, who make media production tools, who make peer production tools, who create videos, who create online problem sets who create aggregators that take lots of different kinds of content and pull those different sources of content together. And tons of these folks make this available entirely for free by distributing it on the internet. And the hopes of, and I've talked to all of them, my hopes is the people who are distributing this stuff is that it will go out from the internet and it will find its way into classrooms across the country in a very kind of organic typically, you could either say unorganized or you could say organically organized or individually driven kind of way. But all of the folks who are making these things hope that at the end of the day that students' brains change in pro-social ways. That's kind of the end of all education is we wanna see sort of how many neurons we can re-engineer in kids' brains to produce pro-social outcomes. If you want it to be a little bit snarky about this, if you wanna be maybe a little unfair, you might call this something like a dump in hope scenario. The idea that we're gonna make a bunch of stuff, we're gonna dump it in the ecology of schooling and we're gonna hope that it gets taken up in ways that lead to pro-social learning outcomes. It leads to kids getting smarter, passing tests or feeling more creative or all kinds of things. One way that we can complicate this is to think about, and again, only slightly complicating it because we're still using gross oversimplifications, is to think about how these kinds, how this dump in hope strategy might interact with profoundly inequitable national school systems like the one that we have in the US. So I'm gonna use a number of these figures throughout that are kinda stylized models and on the y-axis is learning, broadly conceived, so the number of neurons that we can rearrange in kids' heads in pro-social ways. And then education technology, innovation, which is sort of a lump of all the new stuff that we create, every new resource that we put out there, every new problem set, every new video, every new thing. So one story that you could tell about the distribution of free and openly available resources is that they actually disproportionately benefit low-income students, and probably the better way to phrase it would be that low-income students is fine. I'll characterize that differently maybe a little later. So if you were to tell a story like this, and probably you all can start kind of conjuring in your head how you would narrate the story of what this model would look like, but you say something like, well, okay, so we know that starting right now that there are wide disparities in educational outcomes between low-income students and more affluent students. But with all these new education technology innovations we're providing a bunch of freely available resources that weren't previously available to low-income students. So affluent students have always had access to networked collaborative peer production since they had Apple II eLabs with hypercard in them. That they've always had libraries which are stocked with tons of media that they've always had plenty of access to all different kinds of resources. And so all the new things that we're making will have some perhaps marginal benefits to affluent students, but they already kind of have a lot of this stuff. It's really students who don't have these kinds of resources who all of a sudden are getting access to them who are gonna really benefit from this. So we could call that the closing gap scenario. And I think there are a number of people who sort of explicitly argue that this is gonna be the outcome of free and open educational resources. So like Curtis Bonk in his book, The World of Open I think gets towards this. I think probably much more common than sort of openly making this argument is having this argument be tacitly made or kind of the subtext of a lot of discussions about the impact of that open education resources and freely available resources should have on student learning outcomes. Another story that you could tell is that again, there's gaps to begin with between affluent students and low-income students. But in fact, all of the stuff that we put online is much more likely to be taken up in school serving affluent students and in homes and families serving affluent students that essentially even free stuff takes quite a bit of social capital, of human capital, of free time, of time to be creative and we could talk about all these kinds of things. But essentially that sort of places that have more resources are more likely to be able to take up the resources, take up even free resources. And so affluent students are likely to disproportionately benefit. We could call this the rising tide scenario. So one thing that's important about scenarios and I'm sort of hypothesizing that all of this stuff coming online isn't harmful to anyone, is that everyone kind of benefits in one form or another. And some people would argue from a public policy perspective that this as an intervention is totally fine, that it happens to be that affluent students benefit more than low-income students, but everyone is benefiting from this, that a rising tide will lift all boats, that it sort of lifts the mega yachts more than it lifts the dinghies, but it lifts all boats. There's a complicated set of locks that makes that possible. But good. So what I want to say about these two stories is that I think we're at a moment in looking at sort of free and open educational resources where it's worth considering sort of which of these stories seems to be more likely the case. Which of these stories is kind of more desirable? Is there one of these stories that has better public policy outcomes that we should be working towards? And if one story is more true than another, sort of one of the ramifications of that. So I'll try to kind of chat with you for another 15 or 20 minutes or so, about three questions. Well, actually, mostly I'll talk about the first question and we'll see if we get to the next two. So one is kind of which of these stories seems like it has better empirical data behind it? Will technology innovation ameliorate or exacerbate an educational equality? If technology innovation exacerbates inequality, should we care? So is that a problem? Is that something that's a non-desirable public policy outcome that we should think about? How we fund OER, how we design OER to wrestle with? And then if we do think that the closing gaps scenario is the optimal scenario, how would we design and deliver technology innovations so that they do improve equality? Now this is a good moment to sort of pause and kind of think, all right, I sort of laid out these two stories. You know, which of these two stories do you think is kind of most compelling? Which of them do you think is happening? This is the kind of thing that I do with breakfast, with my daughter over breakfast. So we sit down in the morning and we give her some Cheerios and we say, okay, you know, you're gonna grow up in just the most extraordinary period of time. You know, but I think that, you know, we have this extraordinary inequitable system. What are the ways in which education technology is gonna interact with these inequitable systems? And the point I wanna make here is I think for the areas in which we haven't done a lot of research, I'm not sure that Adela's opinions are all that much better or worse than my own opinions. So I think she might be, you know, is capable of guessing, you know, what she thinks is happening. If we don't do research, if we don't have some empirical evidence to kind of look at these things. So what we need to do is kind of run the numbers and see if we can figure out which of these stories seems to be more likely to be plausible. And what's so exciting about, you know, being an education technology researcher is that the capacity to look at empirical data about practices around new media and social media is extraordinary. You know, if I were the Gilder-Lariman Institute, and I released a new primary source collection of print documents and made it freely available and kind of sent it out across the world, once those things got out there, you know, I might be able to track kind of who I sent it to, but I'd never be able to track sort of who is using it and how. But for all of the online resources that leave some kind of public data print, we can start digging in, we can start answering questions about, okay, so how are social media being broadly used in different kinds of environments? You know, so part of, so if there's one thing that I think I can do, you know, I think all the public policy stuff is harder for me to speak about really confidently, but I think there are a few things where research can really help us move past speculation. So here's a question, will technology, innovation, ameliorate or exacerbate educational equality? As a case study to dig into this, I wanna look at the case of classroom wikis. So classroom wikis are different than mass media wikis, so I'm not talking about the classroom use of Wikipedia or the classroom use of the world of Warcraft wikis, but there are plenty of services out there that make it free and simple for teachers to go out and to create their own wikis or for an individual student to create a wiki or for a group of students to create a wiki or for a class to create a wiki, these are multiple classes. And I think that these kind of peer production platforms are a great example of the kinds of freely available tools, you know, which nothing, there have been no equivalent to these kinds of things that were widely available and freely available a decade ago. And we'll ask two kinds of questions about them, but first let me show you sort of an exemplar of what I'm talking about. So this is a wiki called the Flat Classroom Project. The Flat Classroom Project has been running for about seven years now. It's a global collaboration that happens every semester between students from schools from across the world. So all these schools kind of sign up to participate and then kids are putting groups and each group is built out of kids from around the world. So a kid in Jordan and a kid in Saudi Arabia, a kid in Australia and a kid in Vienna, a kid in Georgia are put on the same team. And they're responsible for creating wiki pages that explicate the 10 world flatteners that are discussed in Thomas Friedman's The World is Flat. So you've got a group of students from around the world using network technologies to develop explications of the impact of network technologies and globalization on their world. It's an amazing project that I've been in some of these classrooms and I've had students talk to me about their experiences. But you know, this is one I pretty much grabbed at random, this is a wiki page about the impact of workflow software. And the students have worked collaboratively to create an overview, to put in images, to cite their sources, to have links, to have other videos. As you get towards the end, they also create a series of their own videos and every video has to have footage which is shot in one country and then outsourced for editing in another country. They're organized into teams. So each of those sections is not really sort of cut and pasted together, but it's woven together by a group of students. They've got a discussion page at the top where there've been 123 comments about the development, about the planning and evaluation of how things are happening. So if you're looking for learning opportunities for students to develop and deepen their understanding of a topic, to present that understanding using new media, to communicate and collaborate and develop skills of working across culture and within groups, I mean there's some pretty extraordinary evidence of opportunities for learning here. So wikis have the potential, in my opinion, to be totally awesome. So then we've gotta ask two questions. One question that we've asked for a long time around issues of the digital divide has to do with access. So do students in schools serving different populations have equitable access to wikis? This is, access is sort of the number one way we frame the digital divide, especially in the United States, of who has the computers? Who has the fastest internet connections? Sort of who has the stuff? What we can do much better now is ask questions about usage. So the question there is, do wikis created in schools serving different populations provide equitable opportunities for learning? So if I can pull up a wiki like the flat classroom project and bring it up on my screen, and as it turns out, there are sort of hundreds of thousands of wikis that I can pull up and look at, then I can not just sort of survey teachers and say, hey, are you doing this? But I can actually look at evidence of what they're doing and see if there are differential opportunities for student learning. So for assessing wiki access, there was a great survey which did not get a lot of press but is really valuable that was done by the National Center for Education Statistics. It's called the Fast Response System Survey and for the last 20 years or so, this has been kind of the go-to survey of tracking the digital divide in the United States. And they asked a couple of questions that were pretty useful for the work that I'm doing. One is, do you use a blogger wiki in any form in your teaching? And this is to a nationally representative sample of teachers, so they draw a sample, but it should represent the whole population of U.S. public school teachers. And then do you have students contribute to a blogger wiki? And then kind of a fun footnote to that question, it was originally crafted differently and then I asked them to write it that way. They had sort of posted, like on the federal register, that they were gonna do this survey and I said, actually, it'd be helpful if question seven looked like this. And then they did. So if you want the federal government to do useful research for you, you should tell them what you think they should do. But to me, the numbers here are actually really striking. So in the columns, you can see the proportion of teachers who responded with rarely or sometimes. And they're organized in four levels of socioeconomic status. So a really common measure of socioeconomic status is the percentage of students in a school who are eligible for free or reduced price lunch. That's what FRPL stands for. So you've got four categories there where less than 35% of students are eligible for free and reduced price lunch, 35 to 49, 50 to 74, and then 75 to 100%, which are sort of most hyper segregated schools. Two things I think are really surprising here. The first is the relatively high proportion of teachers who report using these things at all. So about 40% of teachers say that they use a blog or Wiki, about 20% of teachers say that they have their students contribute to a blog or Wiki. So this was in 2009. And you can decide for yourself, what was the date that sort of web blogs and wikis became widely popular. But for a fifth of the nation's teacher force to report that they're having students actively engaged in contributing to blogs or wikis by 2009 to me is really striking. But then the thing that totally blows my mind is that there's no differences across these four strata. They didn't do T testing, but I played and stayed a little bit. And there's no difference between the wealthiest and the least wealthy schools. So if this is the only piece of data that you were to have, sort of surveys of what teachers tell you they're doing and we were to sort of use this on our little chart here. Oops, I need my pen. This was the only piece of data that you'd have. Maybe you'd hypothesize something like this, that there's some kind of inequity that we know that they're starting inequities. But because students have equal access to these opportunities that probably those inequities are not necessarily changing. Now another thing that you could do is not just ask teachers what they're doing, but instead go and actually look at the wikis they're creating. And this is sort of what I've been doing for the last three years. So this is some work that I've done with Richard Mernain and John Willett that's coming out an educational researcher this month or next in a paper called the state of wiki usage in US K-12 schools. So we took all 179,851 publicly viewable education-related wikis that have been created on the PB Works platform from the founding of the company in 2005 through 2008. So we had every single one of their publicly viewable education-related wikis. We did some analysis on the whole set, but which is kind of cool. But mostly what I'll talk about is we took a 1% random sample of them. So our random sample should be representative of this whole population. And PB Works is sort of a good example of other kinds of wiki farms. So I don't think it's particularly different from wiki spaces or wet paint or other things. So we had about 1800 wikis and we threw away all the ones that were used in foreign countries and ones that were used in higher ed, ones that were used in independent schools and ones we couldn't identify where they're from. So we ended up with 255 wikis that were used in US K-12 public schools. And because they're used in US K-12 public schools, we have all of the data on the wiki which is extraordinary. We have its entire historical record. We have every page revision to every page that was ever created, every discussion, every comment. But then we can also associate it with socioeconomic data from the common core of data kept by the National Center for Education Statistics which is in the Department of Education. So those 255 wikis came from 41 states. So in addition to being representative of the population of wikis, they seem to be broadly representative of the nation or at least drawn from the nation. And what we did with them is I had a team of research assistants with help from the Hewitt Foundation's Open Education Resources Initiative examine every revision to every page and to evaluate it using a tool we developed called the wiki quality instrument which measures opportunities for students to develop deeper learning competencies like expert thinking, complex communication and new media literacy. And these are categories we picked both because we thought they were important and because when we interviewed teachers about why they had their students use wikis these are the kinds of things that come up. So the questions that we asked are actually five categories. We're things like, do students use wikis to get information? Do they have an opportunity to contribute to the wiki? Do they own any of the pages? Do they sort of really control them? Do students use academic content knowledge in wiki activities or do they just write about their hobbies or their pets or something? Do students reflect on the process or product of their work? When students collaborate, if they collaborate at all, do students merely kind of concatenate text to a page? So you would write a paragraph and you would write a paragraph and you would write a paragraph and you'd post it all on the same page and say that you'd collaboratively built that page. Or do they actually substantively copy at each other's work, substantively edit each other's paragraphs in that example? Do students use formatting? Do they use hyperlinking? Do they embed their work in multimedia? So really kind of a detailed analysis of what we thought were 24 measures of behaviors that at least provide opportunities for students to develop 21st century skills. So all of that then is going to be summarized, the findings from that are going to be summarized in a single slide. And I don't know if you all have had this experience of taking three years of work in a dissertation and putting it into a slide with 10 numbers on it. There's something both rewarding and deeply depressing about it. But what I'm trying to convince you is this is simplicity on the far side of complexity. So after doing all those analyses, we lumped wikis into four categories. What we would call failed or trial wikis. So these are wikis that someone had created and then typed like hello world at the top and then never changed again. Or wikis that looking at them, there's really no possibility that students could have done any learning from the experience of interacting with them. The second category were teacher centered content delivery wikis. So teachers often do things like hand out syllabi. And a high proportion of wikis are used for teachers to post their syllabi online. And then some of them post content or post links or things like that. A third category of wikis are what we called individual student owned wikis. So wikis where one student, the most common use scenario is essentially turning in an assignment by wiki. So I ask you to write a paper about Henry the eighth and instead of typing it up and handed to me, you put it on a wiki and put some links and a few images in it. And then the fourth category were collaborative student owned wikis, which is actually the category that we think of what, when you think of wikis, this is what you're thinking of spaces where different people can come together and collaboratively plan and develop and produce some kind of performance of understanding. So a few things that should strike you here about these categories. So obviously one row is low income schools for those of you who are in the education world. These are schools that are eligible for title one funding. So 40% of their kids are eligible for free and reduced price lunch. And then what I call mid to high income schools. And the thing that strikes you here is that there's, well you can decide what strikes you, but what strikes me is that obviously there's a tiny percentage of any wikis which are collaborative and sort of used in wiki-like ways. But wikis are much more likely to fail in low income schools and high income schools and they're much more likely to have some kind of student involvement or student contribution in mid to high income schools. So, and there's other ways that I can prevent this, but we know for sure that wikis that are produced in schools that serve wealthier students are more likely to provide more opportunities for students to develop deeper learning competencies, 21st century skills, that kind of thing. The other thing which is much harder to tease out, but I'll just mention is that title one schools, low income schools make up about 60% of the schools in this country. So if low income schools and middle income schools were sort of making wikis at the same rate, you would expect there to be about 60% of the wikis would come from low income schools, but actually more wikis come from mid to high income schools. And it's actually difficult to sort of quantify this exactly, but to me it gives a hunch that at least sort of more wikis are being produced in mid to high income schools. Now one thing that's incredibly important about looking at this is that the only comparison I'm giving you is between different kinds of schools, talking about sort of potential inequities between different kinds of schools. One thing that I, again I put in the strong hunch category is that I think there's widespread inequalities of opportunity within schools. So we did a bunch of qualitative research, we did a bunch of interviews, we did a bunch of visiting schools, and I work with teachers and talk to them all the time. And one of the things you hear regularly is that teachers are much more likely to use technology in all different kinds of forums with their AP classes, with their honors classes, with their higher attract classes, which we know are disproportionately filled with students who are more affluent. So I would say as much as I think we can see quite a bit of inequality of opportunity by looking between schools, we're not even accounting for the equality of opportunity which happens within schools, which I expect is quite significant. Another thing, which maybe I won't dwell on for too long, is you can also measure how long each wiki persists, so how long it continues to change. So this is a little bit of survival analysis that I won't spend a ton of time on. But the x-axis is time in days, so the number of days that a wiki continues to change, and the y-axis is the proportion of wikis that survive past a given day. So some things that strike you are about 20% of wikis in middle and high-income schools fail within 24 hours, so they get created and then never changed again compared to about 40% of wikis in low-income schools. Both have failure rates that drop pretty quickly, but actually mid- and high-income schools have a much higher proportion of wikis that persists through 25, 50, 75, 100 days for things that you would imagine sort of project length time, portfolio length time. If you want to characterize it, this right here is kind of the digital divide. It was one other thing that I wanted to say about this, but I forget. So if you were to look just at those few pieces of data and just around the case example of wikis, it seemed like wikis, while teachers report that access to wikis is equitable, if you're just used survey data, you'd say, well, teachers are telling us they're making them at the same rate. There seem to be more wikis that are being produced in higher-income schools. They seem to be providing more opportunities for students to develop 21st century skills, and they seem to be persisting longer, so if they have any kind of positive impact, presumably the positive impact is lasting longer. So given those three things, it's pretty hard for me to imagine how the closing gaps scenario would be true, at least in the case of classroom wikis. That sort of, if you think of as an intervention, places like PB Works and MediaWiki and the MoodleWiki and wiki spaces sort of putting this opportunity out there freely available online, dropping it into the ecology of schools, the impact of that seems to be, I don't think it's harming anyone. It doesn't harm low-income students that their teachers post their syllabi online, but it seems to be disproportionately benefiting more affluent students. This, I would say, should probably, all of, I mean, conceivably I could show you this first, but then you'd be like, oh, you're just telling us what we already know. You know, if you were to look at, when I look at education technology research from the last 25 years, what I'm finding is really consonant with what we've been finding for a long time. But first of all, when provided with a new technology, teachers are much more likely to use it to extend existing practices than to develop new practices. So the fact that wikis are mostly used for teachers and students to sort of push content out rather than as a real collaborative platform, I don't think should surprise us. And then folks like Harold Woodlensky and Paul Atwell and Mark Workhauer have argued for a long time that in affluent homes and schools, technology is more likely to be used for higher order skill development with more adult supervision than lower incomes homes and schools. What's cool about this research is that we were able to do it both by looking at actual examples of classroom practices and at scale, but I think it's pretty consonant with what we've known. So if you were to kind of modify the story that we started with, that there's a bunch of people out there who are creating these kinds of tools and resources, they're putting it into an ecology of schooling which has profoundly different kinds of learning environments. So there's places like high tech high and there's places like the strawberry schoolhouse which I don't think is still in, this is an example of using images in an unfair way to make an argument. But you put these kinds of tools into ecologies of learning and what we should expect is that we have differential outcomes for different kinds of students. So I think I'll pause there and collect some questions. Some of the things we could talk about is sort of why that's the case. So what are the difficulties that school serving low income students have? Some of the things we could talk about would be what are the public policy consequences of that? Some of the things we could talk about if we wanted to design a system, if we wanted to design interventions that did end up closing gaps, what would some of those gaps look like? But that was me chatting for a little while, so maybe if people have any questions at this point, let's sort of collect some and I might collect a few and then respond. If it seems to me unfair to compare the use of technology, any educational innovation in affluent communities, affluent schools to low income poorer schools at the same time, because there's a lag. And it makes more sense if you wanna investigate the effect if you take account of that lag. And so look at what happens when a relatively new technology like equities is used by affluent schools today and then used by less affluent schools five or 10 years down the road. I mean, let me, the example of the use of graphing calculators in math classes, I think should provide an example of that. That's great, that's great. So I have some ideas of that, let's hold on to it. But that's a really important point that conceivably I'm looking at the wrong time frame that it would be important to be fair to say, okay, well, of course, new innovations are gonna be taken up more quickly in affluent places, but let's give this five or 10 years and then hopefully we would see more quality develop. That's a great potential sort of counter argument to this. I think that I have a comment that may be related. Do you get a sense in those results? You're striking the show that you have a difference between low income school and high income schools in terms of the survival rates of the wiki, whether the students are empowered by the wiki and whether it fails. Do you get a sense in your data of what part of this effect actually comes from the social capital of the students themselves and what part comes from what the teacher, which may be systematically different in low income and high income schools, try to do with those tools, how much do they push those tools? So this is important in terms of police implications for your story, so. Yeah, good, so all I'm showing you is sort of descriptive outcomes from, is outcomes from descriptive analysis, which doesn't, which particularly just sort of slices the wiki and says let's just look at what's happening there and see if there are any kind of differences and then what kind of research would you want to do to be able to say whether it's teacher inputs or student inputs that are responsible for that because obviously if it's teacher school inputs then you think about educational interventions, if it's entirely driven by school inputs, but you'd still probably think about some educational interventions, but you'd also want to, you know. I would say a number of people sort of that I've talked to online have brought up this idea of, to what extent do we want to resolve issues of inequality within school systems and to what extent do these sort of larger society issues is another kind of way of framing it. So we can talk about that too, let's sort of gather fillers. I was just wondering whether there was any difference of whether you looked at within the low income population between like say in urban low income areas versus rural low income areas where like there might be in rural areas this additional structural issue of like whether it just isn't broadband and sort of like that. I went to a school in rural South Georgia where an insult was that's so dial up. That's like when something was totally laid like, oh that's so dial up. And so the more you sort of slice the data more thinly, some of that we start running at the sort of limitations of the data. So we did a little bit of that. It doesn't seem, rural urban divides didn't seem to be real relevant. I think there's some indication that things in really hyper segregated schools so where you see like 90, 95% of students who are, you know, who identify themselves or identify in the system as African-American or Latino have even sort of disproportionately worse outcomes than what you'd expect. But yeah. I went to a school that was half, like half and half. I went to a school with an international baccalaureate program. And basically it had like white and Asian and like, and then black and Hispanic. It was a very sharp divide that we did not interact. They're completely different curriculums. I'm wondering how you dealt with such data. Like, do you look at those kind of schools at all? I'm sure there were innocent, you know. So we didn't do a whole lot of research about what the schools were beyond what the, for this reporting, what the proportion of students were on free and reduced price lunch. But you're absolutely right that within, that I haven't captured at all, the quantitative data doesn't at all capture within school inequities, which are probably as significant or more significant than between school inequities. But the story that you're describing makes total sense to me that we have tons of tracking systems in this country that the tracking systems correlate really highly with socioeconomic status. And I don't think there's really good research on this, but I think it would be worth doing, trying to figure out to what extent to those kind of within school differences map onto technology usage. And my hunch is what you're describing sounds a lot like what I see when I go visit schools in somewhat systematic ways. I have a question about perhaps success measures or units of analysis. And you seem to have taken a very broad systemic level of bird's-eye view, but how granular does the analysis get or can it get? And I'm wondering about measures of success, not across time. So looking at perhaps technologies that may be a benefit now in one sphere that weren't there a few years ago, but margins. And so is there a way for you to track and to be able to measure success for a particular tier of the low income students? If 10% of low income students rise and their levels of engagement and access to and benefit from educationally, the Wiki technology, is that a measure of success rather than showing the differentials between the two with the low income students? Yeah. Yeah, like, and I'm particularly interested in this in another context. I'm, you know, it's always hard to measure success when your goal is not systemic change, but perhaps educating a few with the margins or trying to improve not with an idea of total change, but some portion thereof. Yeah. Well, one way I briefly responded to that is that, you know, one huge limit of this is that we decided to focus, in order to get at scale, in order to look at sort of hundreds of these, we decided to just look at Wikis. And so we didn't actually capture changes in learning that happened in student brains. If we wanted to actually, so we measure opportunities for learning rather than sort of particular learnings. One thing we do find is that there are examples of even in low income schools of these tools being used in ways that seem to provide, you know, so if you go back to, you know, this slide, one thing that's important is in low income schools, there are, you know, collaborative student-produced Wikis. There are individual student-produced Wikis. And then certainly in some of the qualitative research we've done, you know, we've gone to places where teachers have gone to incredible lengths to make really rich technology mediated learning opportunities available to their students in schools in unbelievably difficult settings, where they're using MacBooks that are six years old and all the batteries have failed and they're in a library room half the size of 30 kids and the cords are falling all over each other and things like that. So I think, you know, there are lots of proofs of concept that are out there that these kinds of tools can be really powerful. And I can talk about some other examples where I think people have done a really great job of designing interventions that particularly support students in low income environments. But I think, you know, one, you know, maybe what a lot of this really comes back to is if your hope, you know, I think in this I'm not speaking as much to people who are designing a particular intervention, targeting a particular group of students hoping to get, you know, particular gains, whatever they would be. But people thinking about a strategy which says, okay, we're gonna build something and we're gonna put it out there. And because it's free, we want it to be really good for people and, you know, in our hearts or for public policy reasons or for whatever else, we're really hoping that lots of different kinds of kids will benefit. Good. What back to the side, because, so it seems like the kind of wiki that you're saying is the most desirable is this fourth type of wiki. So that's pretty amazing then, right, that the low income schools, there's a higher percent. So what- So I wouldn't call it a higher percent because it's probably not actually statistically significantly different between those two things. You know, 1% and 2%, it's about the same. And, you know, so the thing which is exciting about that is that in all kinds of schools, we found really exciting uses of wikis to enable all kinds of, you know, development of expert thinking, development of new meta literacy skills, development of collaboration skills. I do really worry, you know, one thing that we can't do is sort of get into the wikis and sort of figure out exactly where in the school is they're from. A lot of teachers don't sort of label their work as like, this is for the honors class. Like, all of my students are really wealthy. And so, but my hunch is that, I mean, I know that there are great cases and proofs of concept that leveraging technologies for really empowering, exciting 21st century learning can happen in all kinds of schools. But I think, you know, not surprise, you know, this is continuing evidence from a lot of previous evidence that suggests that it disproportionately happens in higher income schools. Well, let me use my minutes to talk about a few other sort of things. Maybe some of this will address some of those questions. So what I think really important question is, if technology innovation exacerbates inequality, should we care? And here are potentially sort of three reasons why you wouldn't care. So I've been kind of telling a story that looks something like this, where again the top line is more affluent students and the bottom line is more low income students. Wait, it can't go down. It's gotta go up a little bit. So I don't think there's evidence, well in the story that I'm telling and from what I looked at, I don't think there's evidence that education technology innovation is actively harming students. Like I don't think the introduction of Wikis into the ecology of learning damages student learning in some way. I mean, maybe you could argue the opportunity cost. Let's see, it's not evidence that I would draw from, well let me give you, let me say that there are certainly people who would draw the same figure, you know, Larry Cuban. Demonstrate the technological innovation has resulted in decreased learning. I don't think there's any say which says that it's taken a student backwards. I think I'm sure there are examples. In fact, I know that there are examples where trying new innovations have not been as effective as what was happening beforehand. Well that's negative, that's decreased learning. Yeah, well increasing learning at an increasing rate, but decreased from what the value added might be. Sure, and I think that kind of argument would, I don't know that Larry Cuban would go so far as to say, Larry Cuban wrote a big book called Oversold and Underused, where he basically said that of the millions of dollars that we've spent on technology, there's very little evidence that there's been any kind of positive learning gains in the broadly across schools. To some extent, we see like in a few pockets of places people doing really interesting things. I don't know that you would say that students were harmed by. You would, you would, and you say you do. I asked, I know Cuban wouldn't say that, but you just said you were. Yeah, I think you could find design, I think you could find design research experiments where teachers have been doing things a certain way, tried something and found the outcomes of that. Like you would, you know, which is not, yeah, which you would expect with any kind of, you know, the innovator's dip might be something that you describe it as. If you sort of try something new and it's not as good as it was before. You know, I'm sure there'd been probably systematic rollouts of things. You know, I don't know if there are examples of things like an elementary school instituted a computer class and their reading and math scores went down afterwards and we can demonstrate that or something. You know, I can't, you know, maybe that's possible, but one thing. So I think one important sort of caveat to this story is that conceivably, you know, what we're, you know, what we're talking about is that if education technology has very few or sort of negative, you know, mildly negative or strongly negative ramifications and all this isn't really all that relevant, that you can only, inequalities are only interesting to talk about if you're talking about the inequality of something which has value. So if we think that education technology has no value then this isn't a particularly relevant conversation but it's also not particularly helpful to do things like build education technology innovations. Another way that you could characterize this and I think this is sort of along the same lines of saying that you're sort of looking at the wrong things is the thing that I've been pointing out here is this gap here that sort of at the end after a series of innovation that there's more inequalities that there's greater inequality of opportunity, potentially greater inequality of learning between these populations but it's possible that that's not really what you should be paying attention to. So this is, Tom Van Der Arc makes this argument in his book, Getting Smart and he sort of postulates that there's a line that you might call college and career ready, oops, and well, college and career ready. This is in sort of NCLB terms what in theory is supposed to be called proficient. So the idea here is that there's some kind of basic level of learning that we want folks to, that we want young students to achieve and that it doesn't really matter. The important thing is not that there's a gap here. The important thing is that over time, education technology innovation will sort of lift more students above the floor of proficiency. So if each of these sort of points on the line represents a mean, then at the beginning, kind of most of the kids in affluent schools, one standard deviation on either side were above the proficiency level. And starting off, most of the students down in the low income group were below the proficiency level, but conceivably what education technology can do, even if it sort of can't help low income students sort of keep up with high income students, it can at least get more students to be past the level of proficiency. So Tom Van Der Arc calls that the preparation gap. That would be another thing that I don't know that there's any kind of evidence to support that technology is doing a particularly good job of helping students kind of reach that minimum level of preparation. But if you're a policymaker, you might say, you know, it's not at all worth focusing on this difference or those numbers. As many students as we get to achieve as highly as possible in our society is absolutely fabulous. What we really wanna focus on is how many students we can get to that kind of baseline level of proficiency. And if technology can be of service in reducing that preparation gap so that kids can enter college wherever they are and not have to take remedial classes, and that's a big success. Again, I don't know that there's any evidence that that's necessarily what's happening, but I think it's a good argument for why you might say, oh, Justin, you're focusing on the wrong things. The expectation of what that proficiency level is and that a lot of people will continue to fall below that. Yeah, so education technology innovation not only sort of potentially raises our capacity, but it certainly raises kind of the demands in a society. So as technology is capable of taking over more routine tasks then more students need to be able to do more cognitively difficult work. Yeah, no, I definitely think that's right. Let me do a third kind of example. So here's rising tide again. One argument which I think you were making before is that I'm sort of looking at the wrong timeframe here. So all of this is actually just taking place right here. So this is sort of now and then there's the future. And now we see something that looks like this, but in the future it's gonna end up looking something more like this. That over time what we expect is that the sort of the benefits of any kind of education technology innovation will initially be more beneficial to affluent students, but later those benefits will sort of after a lag kind of approach lower income students. And I think there are actually some public policy folks who would say that this is essentially the desirable scenario. That if you want public policy to interventions, innovations to take hold in society, it's really important that fairly early on in the life of those policy interventions that middle class and upper class voters see the benefits of those policies. So I was talking with someone who worked in an education ministry in an OACD country and he said, when we were doing our work, we got a lot of heat from the left for having a lot of work that really supported middle class and affluent students. But if we couldn't rope in those middle class and affluent voters to be supportive of our policies, that it's not gonna move forward. An argument which I think is particularly compelling sort of in the politics of the United States that when we look at social programs which have succeeded for a long time, things like social security, they benefit all people of society and then ideally they have kind of a carve out and a target for reducing inequality but that's not the only job that they're doing. So I think people who would sort of see the world kind of from this perspective would say, actually these kind of inequities are conceivably kind of beneficial from a public policy standpoint that making sure that the first job of OER and other kind of free educational technology resources to make sure that people who are likely to vote for support for them are seeing the benefits of them. If you wanted the, I think the most compelling sort of flip side of that argument to me is that one of the things that we know about really successful educational systems is that they're highly equitable. So the school systems that have the nations or school systems that have the highest scores on PISA tests and TIM tests and other things like that also tend to have the least inequality. So as a correlation, we know for sure that that's the case and I think a number of policymakers would say that it's not just a correlation that in fact a deliberate focus on improving equity of outcomes is what leads to excellence. That equity is not merely a byproduct of trying to be really good but if you want to be really good you actually have to focus on developing systems that serve all students. Here's another thing that I've been thinking about a lot related to this actually was a guy named Ryan Blitstein at the Susan Crown Exchange who helped me think about this. But one of the things that's kind of exciting about all the tools and resources that we're creating is that they leave behind for people like me tons of data to look at to analyze to think about how we would iterate and how we would redesign what we're creating. So let's say folks create all kinds of awesome stuff. They put this awesome stuff into the ecology of schooling but then it gets used at differential rates. So it's much more likely to be adopted and taken up in wealthy schools and wealthy homes and wealthy families that it is in low income environments. And then we're collecting data about that usage and the data we're collecting about that usage doesn't necessarily disaggregate between those kinds of users. So for instance, Sal Khan talks a lot about all the data they collect about how their videos get watched. And so they do a very good job of saying okay, in this video I just made 37% of people stop watching at minute four, second 13. So maybe there's something wrong there. If most of that data is being generated by predominantly affluent students and families then that's the data that's gonna be used when it gets sort of pulled back in. So there's a whole kind of terrific group of researchers in the field of OER who are developing a set of standards around what they call paradata. So if metadata is kind of the data that gets slapped on a learning object when it's created, then paradata is the data that gets collected as it gets used. So who uses it, how do they use it, what do they think about it, what are the results, that sort of thing. If the paradata around learning objects around these kind of education technology interventions is primarily created by affluent students then we'll conceivably sort of the design, we'll, if we have data-driven design will those data-driven design decisions be driven more by the needs or interests or reactions or experiences of more affluent folks. That to me is another kind of interesting consideration to think about given that one of the great advantages of these tools is that we have all kinds of data that we can gather about exactly how they're being used. Let me in my last couple of minutes as promised talk about three examples of interventions that I think are doing a really good job of being deliberately designed so that they particularly try to replicate the closing gaps scenario. So I'll give you one example from inside schools, one example is from outside schools, sort of informal and one example of after schools. So if I start thinking about what is it that would not produce kind of a rising tide scenario, these are some of the things that come to mind. So one is the example of the leadership public schools, which is a network of charter schools in Northern California and here's their problem. They want their kids to be able to go to college, they want them to pass all of the classes that are required to get into the UCAL system. A lot of their students are second language learners and so they come in with difficulties with academic language. So if you put them in a math and science class, they don't pass the class, not because they're not smart enough to do the math and science, but because they don't have the academic literacy skills to take the class. If you pull them out of the math and science classes to teach them English language, then they don't get enough credits they need to be able to get into college. So what these folks have done to solve that problem is they partnered with an organization called CK12, which creates open access textbooks. And they said, we're gonna take your algebra textbook and your, I think they started with algebra and are gonna move on into biology and we're gonna design them so that built into the algebra and biology content are specific academic literacy supports for English as foreign language learners. Is this the kind of thing that could benefit students in all kinds of learning environments across the country? Sure. We have all kinds of students from every walk of life who are English language learners. We have all kinds of students who struggle with develop academic literacy. And so learning the words that you need in academia to understand math and to understand science potentially is a huge benefit to everyone. I think it will disproportionately benefit schools that have high proportions of second language learners. And so to me, it's a great example of thinking about, okay, well, here's an intervention which is particularly designed for the neediest students in our society who we really wanna focus on making sure that our school system is one that provides an equitable foundation and equitable access to college and secondary learning, tertiary learning. But also doing it in such a way that can kind of broadly benefit all kinds of folks in society. Another program is one called Tech Goes Home which is right here in Boston. And Tech Goes Home partners with both schools in Boston and families in Boston. So families that can't afford technology, hardware or access have it provided. So it's a grant program that gives away netbooks and internet connections. But it doesn't just give away netbooks and internet connections. The catch is to be able to get this, your family has to come into school and take classes about how to use the technology, how to use it for learning, how to use it for playing, how to use it for getting work done, how to use it for looking for work. And those classes are taught by teachers from the schools. So you basically have technology that's going home to support learning both for all kinds of members of the families that education is being provided in a school environment. So it's drawing families who we want to be more involved in education into the educational system. When Deb Socia talks about this, she talks about the schools where it's happened. I mean, not only do you get a bunch of technology and training out to people who really need it, but on your school open house day, they saw double digit increases in the percentage of parents who started coming out to school open houses that basically by using technology as a lever to bring parents into schools, we got families more involved in the education system. And one of the things that we know really well from research is that anytime we can get parents and families more involved in students' educations in virtually any way you can think of that students' educational outcomes improve. There are other, again, there are other places that sort of use models like this of, this is a model that can be reproduced by schools of all kinds that are trying to sort of use technology training as a way of bringing families into schools and to strengthening those partnerships. But it's a model that's sort of particularly designed to think about closing those gaps. And then the last one is a project by a woman named Betsy DeSalvo called the Glitch Game Testers, which was an after school program, so the sort of third category. And this was a program that was essentially co-constructed with a group of African American men with an aim of thinking about how could we help African American men get excited about attending schools with strong science and engineering programs and then actually enrolling in computer science and moving from, you know, with all kinds of learners, moving from creators of consumers of computational content to computational creators themselves. And so they, working together, sort of developed a game testing lab, which they then hire themselves out to companies like Electronic Arts and other places to get paid to work after school, to beta test games. And then while they're doing that, the students actually began insisting that they also get an opportunity to learn the AP Computer Science Curriculum. And so again, sort of thinking about, you know, if you're designing a program, which is not merely sort of putting opportunities out there, but really thinking about, you know, if we wanna make sure that opportunities to use technology target students that we most care about, how would you sort of design those opportunities? How would you design those interventions? And certainly having people participate in the design of those kinds of interventions is a great example from there. Good, so I think we're supposed to end at 1245? True? Lies? 1230? Okay. So those are a few more examples of if you were to think about, you know, and to me, what those three examples have in common is that if the very first story I gave you sort of, you know, if were to think about the kind of cathedral to bazaar spectrum, so if you were to think about, you know, designs of interventions which are more highly structured and designs of interventions which are more organic like the bazaar, more kind of let people come together, it seems to my hunch, and we, you know, people can talk about this and tell me what you think, is that the kinds of interventions that that involve sort of putting things out there and hoping that they'll have impacts, particularly if you're, you know, hoping that not just they'll have impacts on student learning, but hoping that they'll differentially and disproportionately impact the students who's learning we most care about, or we most wanna support and become more equitable. It seems pretty unlikely. It's hard for me to imagine how those kind of organic supports are not gonna be taken up in ways that are more advantageous to affluent students, and it seems to me that if we want technology to support the students we most care about, then we're probably more likely to think about interventions that have more structure, that have more design and are more targeted. But what do you think? Hi, that was great, thanks. I'm Sam Klein, one laptop per child. You were talking about aggregating different kinds of different sources of data, and it seems that some of this research aggregates changes in sharing policy, changes in technology, and changes in classroom implementation. And I wondered if you did any desegregation yourself when you were studying these things, working with different schools. Well, can you give me an example of what? So you're talking about OER at one level, and the simplest form of OER is just a change in sharing policy. It's not technology like most things that involves costs and implementing different interventions, right? Yeah. Then you're talking about changes in technology in terms of access to different materials and classrooms in ways that people could collaborate. And as an aside, you mentioned that a lot of the differences that you noticed are related to changes in implementation, because some teachers just don't know how to implement. You showed that in the fifth note, twice as many teachers in the lower income schools just didn't know how to implement. Or weren't able to be successful for some other reason, even if they knew how, had some other kind of obstacle. Right. But those are two very different things. There's a question of whether a changing technology is useful when implemented, and whether a change in implementation is possible for different audiences. Yeah, well, I would say that almost nothing I do has anything to do with licensing. And that, in fact, every time I go and talk to teachers, I'm struck at how little licensing has any bearing on their day-to-day practices that teachers will take whatever they can find online and use it however they want to. And if you introduce any of them, licensing is like, whoa, that's a new thing for me to learn about. Even people who I think are pretty technology savvy, or at least new for me to actually think about how that would be relevant. I mean, I think the ways in which I do, in terms of the statistical work I do, that that's mostly looking at issues of access. So what happens when things appear in systems? Because we actually don't know a ton. The research that I do, which is just looking at how wikis are put in place, all we can do is look at the wiki. All we can do is look at what they've done there. We don't get to go in the classroom beforehand and see what kinds of practices are developed beforehand, what kinds of relationships teachers have with students. We're trying to do some more of that stuff right now by surveying teachers at the moment they create a wiki, finding out some more about their attitudes and then seeing how that predicts outcomes. And I think that'll be kind of fun to look at. But so I think in terms of the statistical work that I do, I probably can't disaggregate too much about implementation. I guess maybe there's some things that we can start to tease out. So one of the things that I didn't really get into was that we actually measure wiki quality at six points across the wiki life cycle. So we measure it at day seven, day 14, day 30, day 60, day 100, and day 400. So we measure all changes up to that. And the figures I showed you were actually a summary of what it looked like by day 14. And part of the reason that we can do that is in fact almost all quality changes that we measure happen within the first week or two. So basically whatever kinds of things you see happening on a wiki within the first week or two are likely to be the only kinds of behaviors that happen on that moving forward. So it's very unusual for a wiki to start as sort of a teacher-centered content delivery device and then 50 days later, 75 days later to see students kind of co-constructing some really interesting piece of work there, which suggests to us that the initial norms that are developed by communities as they start implementing some kind of wiki project are incredibly important. That essentially the norms that are shaped in a community about what kinds of things are supposed to happen in this online space are really powerful in shaping what happens going forward. But that's a kind of inference about implementation that, you know, I mean, I think the kinds of research is much better at digging into those implementation questions. Our design research, case research, qualitative research sort of gets into places and looks really closely at, okay, in these kinds of school environments, these kinds of, you know, I mean, I think we have a lot of really rich historical depth about each wiki, but we don't have a lot of contextual depth about sort of exactly what are the key factors that make things tick. Does that answer your question? Okay. State the economic argument, which is you're making, you're drawing conclusions in the end about whether making certain new kinds of technology available can be equitably useful to people across the spectrum or even more useful to people who don't have access to excellent teaching resources now. And you don't seem to be differentiating in your research between technology changes that do not require changes in investment by students themselves with their families or by students with their teachers and ones that do require investment. And I would suggest that things which have a very, very low marginal cost actually have the opposite impact of the things of the sort of the aggregate data that you were showing. And if you were able to separate out the different kinds of implementations and whether in a given school system implementing a wiki, which at some level sounds like it's easy, is actually really expensive and places where it is actually free because someone nearby is already handling all the maintenance and upkeep and overhead, you would end up with very different data. And it hurts me a little bit to hear people drawing conclusions that making things freely available and reducing marginal costs might not incrementally benefit really poor students more than they benefit wealthy students because I think there is a regime in which that's obviously not true. And there's probably a regime where it's obviously true just because there's no technical infrastructure to make something that could be free happen. I think if what you're saying is that with in addition to making the technology accessible, that there are other kinds of sort of aspects to the implementation so that there's other kinds, you're talking about other kinds of professional learning, other kinds of support that if education technology is paired with say like a clear set of learning goals for what those technologies should be able to support, a clear set of professional development for teachers in those contexts, additional administrative support to sort of solving technical solutions behind the scenes, like could those kinds of solutions disproportionately benefit low income students? To me that seems extremely likely. I actually like the leadership public schools I think is kind of an example of that perhaps. My sense is that what you're describing a sort of more structured intervention than sort of let's put some stuff out there and kind of see who takes it up there's not a lot of that happening in schools in the United States. Like I think if you were to look at say like what Idaho is doing right now is sort of requiring various kinds of technology things. It's not clear to me that Idaho's efforts to require students to take online classes and to spend a lot more money on laptops and those kinds of things is closely tied to a set of clearly defined learning goals and a set of professional learning opportunities that are gonna help different kinds of people take advantage of those kinds of things. I think to me it looks like an effort to sort of drop a bunch of stuff in schools and then kind of see what happens. And that kind of effort is a lot further away from classroom wiki so I don't know how much you can generalize. My hunch is that when Idaho does all that stuff what they'll find is that most teachers adopt new technologies to extend existing practices and where they're not doing that it's the most exciting kind of interesting innovative uses of technologies are gonna be disproportionately happening in schools serving affluent students. And if they want something other than that to occur then they'll need to think about sort of how they design their intervention to be able to make that happen. And if I hurt you I'm sorry. You gave the example of the time lag effect and the kind of convergence as the optimal sort of policy outcome. As what some people might argue. Yeah. But do you think it happens? Is that the fact or is that just what we're aiming for? I don't know. I don't think I know that there's I don't know that there's sort of recent we certainly like is there a research about education technology that would demonstrate that there are particular kinds of education technology interventions which in their early days produced sort of accelerated the widening of the digital divide and then over time sort of that converged. I can't think of an example that looks like that. But I also think part of the issues is that they haven't had is that education technology over the last 25 or 30 years in the aggregate it's hard to see how much impact it's had in producing positive learning outcomes that we can get really excited about. But I do think, but I think there are other I mean, maybe people would use examples like libraries that the development of a library was in the Athenaeum was around for a long time and wasn't broadly of service to people in the city of Boston. And then the free public library system was developed and those have been much more broadly accessible to people across the city of Boston and part of what made libraries have the public policy support they did is that they're a bunch of upper class people who are like, well, the Athenaeum is awesome. Let's make sure that everybody can get one of these. But that's far beyond my field. Time. Well, thank you all very much for coming and thanks for your good questions and all that kind of stuff. I'm easily found online in various places so please, and I'm around here on most Tuesdays. So if you have other questions or other things please track me down and we'll chat some more and thanks a lot.