 Thank you all for coming everybody. Does everybody know who Monterey's tutor is? Not a prophet? Who doesn't know? Does everybody know about the National repository of online course project? Right. A little bit? Yeah? The INROC project has been around for about seven or eight years now and the intention is to bring together high quality, multi-media based course content for high school and especially high school and community college students available as an open resource. The project has been going on a while and it came to a point where we needed more content. You know how these things are. We've continued to have content. We've been getting content from sources outside academic institutions around the country for a number of years. Ultimately it became time to build content ourselves to fill the holes. What we chose to build first is math and probably, no surprise anybody, math is a major issue in the United States. About half the students in the U.S. entering higher education have to take some kind of developmental course, rather than frequently it's developmental math. About half the students in the U.S. on average don't pass Algebra 1 in 8th grade or 7th grade when they take it the first time. So it's a major issue. It made a lot of sense to go there. The other reason it made sense was that INROC, the natural repository project and the membership organization kind of manages everything for us. They guide where we go. It's academic institutions from around the country that are members of it. Here's a listing of those so small you can't possibly read it, but I can tell you that it includes about 28 or so of the state VOEs or their virtual schools, a lot of the larger school districts in the U.S. and a number of the community college systems, state community college systems. As a group, they told us it's developmental math. The number one need they have, that's the number one pull of the library, that's what you should do. And so we listened to it and we did. We got funding from both the Bill and Linda Gates Foundation and one from our Hewlett Foundation to build first Algebra 1 aimed at first-time Algebra takers, which in the U.S. is for the most part 15 and 16-year-olds taking it in 8th grade, and then developmental math for everybody else, folks that didn't pass Algebra 1 have to see it again either in high school or not impregnately in college. The objectives were of course to get students through these courses. Our approach was to really consider multimedia as an approach to this, really look at it as a means of teaching. And we attacked this project, like you attacked this project. We started with research. We saw what had gone before. We saw what products were out there, what research told us was working, what didn't seem to work. Additionally, we ran focus groups around the country. We asked groups of instructors, administrators, even students, what did they think? What did they want, what was working, what wasn't? And ultimately, those things informed us about not only the engineering we used and the fact that professional development for instructors had to be part of this. It had to be part of this to be successful. But the curriculum itself, since the curriculum had to work with all 50 states in the nation, had to work in an alignment with the community colleges and other higher education institutions that were using it, and the design itself. Now one thing came out of this early on was the focus groups had some real serious ideas about what they wanted, and so ultimately became an iterative process. The focus groups were actually influencing the product development, first design and then development, and it was iterative over a three-year period where we were running focus groups almost every month for the last three years to get, to get lodged in what we built. And ultimately, we're taking things off the assembly line as we built them, putting them in front of focus groups to make sure we got it right. And I will tell you, I've been building products like this for a long time, 20 or 30 years, and we got it wrong more than we got it right. And we all thought we knew what we were doing, but when we actually put it in front of the users, we weren't getting it right, especially the students. So the goals of the focus groups were the ones you'd expected to be to find out what people needed, what worked, what didn't. So the students especially, students are, you know, they walk on campus kind of walking back in time. They're not using the same technologies on campus as using other places, especially in K-12 institutions. They communicate in different ways. What do they really want? We've been conducting focus groups now for over three years, a lot of them with lots of students in lots of states and lots of instructors and administrators. In sum, what we've learned is as follows. Not surprisingly, what administrators and instructors told us was Fish and K-12 in the U.S. was that the existing digital products for math were too expensive. They cost a lot of money, not infrequently as much as $80 of students sometimes a lot more, which is too much for most school districts, too much for most states. They're inflexible. The curriculum that they support wasn't easily changeable, so it didn't necessarily line up with their state framework. It didn't line up with what's going to be the common core for most states. And almost most importantly for a lot of them, it didn't work with their existing learning management system. They have almost every school district in this country, all the larger ones, have learning management systems while high-rich patient institutions have them as well. So whatever you build, better fit in there if it's going to be useful for the way in which they do business now. And not infrequently, almost without exception, the existing products have their own proprietary systems. What we heard from students is even more interesting to me, having built stuff for a long time. We're really aiming at students that are struggling, so we're not infrequently used to students that have other things going on in their lives. And what we heard from most of the students, as the product rolled off, what we heard right away was this is way too complicated. The language is way too complex. Make it simple. Make it simple enough that I can hear it and understand it right away, especially for students for which English is the second language, major issue. Some examples were a major issue, too. You're going to teach somebody algebra. So if you're bringing together your understanding of the written word with a quantitative spin on that word, that's all going to come together. If it comes together in an example they recognize, like downloading music versus how a bakery works, or how the trains work, where they build more trains, it made a lot more sense. So making it relevant and putting it in context was a major step forward for the students. And then the other was humor. Not infrequently, as producers of this, you may try to inject a little bit of levity here and there. Man, it was an absolute disaster. I wrote the first scripts. I will tell you right now that a 60-year-old man cannot write scripts for a 50-year-old student. He just does not work. And I thought I was absolutely hilarious, not so much, you can say. All those things added together, if you look at the products in the market now, a lot of those things are there. And as part of the reason they're not working, it's part of the reason. So that all came together over the number of years. We brought it together as a white paper. So if you choose to and you'd like to learn more about these data, what we've learned, I invite you to go to the website where these projects are being tracked. It's called inrockmath.org. You can go there and at the project news section, you can download this white paper. It covers the data for about the first two and a half years of the focus groups. And in the last six months, we'll be updating it through time table. So all this informed us, and here's what we decided to build when it came out. We have six semesters of material. The first two semesters are algebra, one first and second semester, then the next four semesters are data. These are the typical courses that are taught in the U.S. primarily and actually in Latin America as well. In saying that, they're not really courses. They are brought together as a collection that you recognize as something as a course. This is algebra, I know what that means. But it's actually a collection of learning objects in all cases. So this six semesters is actually more like 185 learning objects. What's the number? 184? 184 learning objects. And each of the learning objects, each of these chunks, which covers each of the chunks of content to make up these six semesters, has each of those multimedia elements from warm-up to video presentation to work examples, et cetera. At the same time, it can be organized by units as typically happens, especially it can be organized around a typical classroom in the U.S. There's about 29 units at the unit level, which is more of a summative level where now there's certain things you're expected to have learned in the context you've sort of applied these things. There's other multimedia approaches, including a virtual tutor, which ended up being a big deal for the students we're working with. These simple puzzle games which reinforce concepts and relationships would end up being although not important to most of the students using these, for the 5 or 10 percent of the students who were struggling the most mightily, this was the biggest piece for them, so we decided to leave it at it. In total, believe it or not, you probably don't care about the U.S. math curriculum, but there's 516 explicit learning objectives in the U.S. math curriculum across the state frameworks, so this embraces all of them in a grain size so you can actually mix and match this to fit all the states, Common Core, and all the alignments with all the community colleges in the U.S. Looking at them side by side, there are two selections of courses looked like this. Included with all this multimedia content is also a formative assessment system baked into this, so as the students go through, especially doing the warm-up, they're just appraising their performance, putting questions in front of them to understand what learning objectives they haven't reached yet, and then feeding that data back in the system and then changing the content they see based on that analysis. And those data are actually, we're building a new system, those are going to be passed back to the LMSs we're working with. So, what does it look like? As I said, this is a typical landing page for one of the learning objectives, but there's all those six components I mentioned were available there from a text-based warm-up to video presentation. Now, the video is interesting. I've been in video for a long time, a lot of television work, and I learned early on that video is really good at a couple of things. It's really good at concepts and it's really good at examples. It's not so good at protocols, procedures, which is a lot of what happens. And so the pound away at process so we wrote these videos carefully around exemplars that would make the concept relevant to the students, making sure the concept was upfront. Why are we doing this? What is the concept here? For each of these 184 learning objectives, the videos are short, they're YouTube length, like three to six minutes. They're highly produced. They have a presenter on screen. A lot of animation and graphics, they're quite beautiful actually. And also the other thing is presenters. There was a trial. We went through probably 100 and 125 presenters, fighting the one or two or three, actually the end of the three, that really played well across the US with the students were serving. And I didn't see it coming. How bad that was. Work examples. Each and every object, each and every of the learning objects has two, three, four work examples of work through the materials we were going through protocol. It's a typical whiteboard voiceover. Who do we choose for that? Anybody know? Who would you choose? We chose Saul Kahn. Saul, we went through a bunch of different presenters. They got shot down one after another with our student focus groups. Saul is beloved by the student population in the United States. So Saul Kahn did 800 work examples for us. We wrote him. He performed it. So they're in line with the curriculum. They followed the curriculum explicitly. Interactive problems. A review section. And even an online textbook. You can access it from each of the learning objects. It's an interactive textbook in the sense that it's typically laid out for a computer screen, but it has interactive elements to it. You can also, you can dump it and print it if you choose. At the unit level, there's something on the virtual tube. The intention is how many struggle with applying algebra concepts to unique problems like I did. Yeah. And the thing is, you understand the fundamentals of algebra and then they give you some problem where you can't see where you don't plug in what where or approve. It improves. And so, the strategy of problem solving is something my teacher never taught me. The virtual tutor not only walks you through protocols and procedures of a problem, but the strategies of that problem solving because they're not in frequently multiple routes to solving the same problem, right? You can do it more than one way. So, the virtual tutor is all about that. And this ended up being a very powerful tool for not only students, but also instructors. Instructors are using this in front of classrooms. Project-based learning so students can break down into groups is part of this. And finally, these puzzle games. Really simple stuff. No rules. You light it up and you're going to know what to do with it. And for the most part, it's all about reinforcing relationships, concepts. Nothing more than that. And again, teachers using this in front of classrooms. At this point, I was going to go out and show it to you. But we're not going to do that. Because I don't think we're going to have time. So I invite you to take a look at this. It's at our... Anybody know about HIPAA Campus, our open site? HIPAAcampus.org is our open site for students and instructors, any individual to use. It's not at the old site. If you go to the old site, you won't find it. You'll see the old algebra course. But you'll see somewhere in HIPAA Campus it'll say, go to the new site and go to the new site. It's a new site that's going to be launched and take over the old site by the end of this year. It has all this new material and it has other collections. It has silicon full collection. It has fat full collection. It also has a wonderful feature where you can actually build playlists. You can drag and drop all these pieces in your own playlists. Anyway, this new math algebra course is there. I invite you to go take a look at it. What I wanted to show you in the last remaining 10 minutes we have is the results that are starting to come in with this material. Because why build it if it's not actually working? Algebra first and second semester was published earlier this year in the spring. Dev Math the first arithmetic the first semester which is arithmetic we released a beta version in the summertime. Right now we have 34 ongoing algebra pilots in the US and we have about 50 more queued up and actually I will tell you I probably have a hundred or more queued up just keep coming in coming to queue up. For Dev Math we have pilots queued up in 15 states. We don't have the dating in on this much yet. We're just starting to get it. I was talking to people in the hallway that are running this and sort of getting it anecdotally. By the end of this year I'm going to publish a report on how this is doing but I understand that we don't know how to use this material. We think we knew. We're putting it out with the hands of doctors and districts and they're telling us and about five or six use cases are rising to the surface as being good ways to use this material. They seem to have positive results. As I said HIPAA campus is I'm sorry algebra one is a HIPAA campus now take a look at it. We're going to have an interlock math app we're going to start to publish this spring. So all this material all this math material and then later our history physics all that will be playable on any mobile device and my smart TV which I bought last weekend and it's going to play on my smart TV I'll tell you right now. So we're going to have these math apps published soon and then there's a documentary film about one experience with this algebra program as we were finishing this program it was coming off the assembly line sort of data form we had one district one of our members that was eager to get this they've been really having trouble with this Sierra Bista alternative school in Whittier, California so we were giving it to them as it was coming off and they actually ran an experiment for us a class using this material as a hybrid and a class taught traditional 20 kids in each class 21 to 19 million others just to give you a sense of what Sierra Bista is all about it's an alternative school it's kids that have struggled and had other problems and they end up at the school and the scores sort of the performance of the school is not very good it's one of the worst in the state of California actually and the red bars tell you that you don't have to look at this to close the red bars are bad it's literally only a tiny fraction because this school are even performing at competency in that background on who was in this experiment as juniors and seniors more than half had credits deficiencies that's why they were there about 80% were teen parents so they didn't have regular schedules they were working 20% were homeless all of them had repeated algebra twice before a lot of low income kids we use as we always do whatever the benchmark is for the district or the state to find out how things are going so we use the state benchmark for algebra it's been modified slightly by the district but it's the same benchmark traditional classroom textbook four times a week instructor giving lectures and then this product together in a computer lab excuse me with a teacher coach that comes in three times a week for an hour to sort of meet and through and here are the data this is from the first state benchmark this is a traditional class the pass is at 60% time and about half 45% got through it and that's typical for this school that's what they've been looking at tracking for the last several years here's a hybrid class all but one got passed yeah we said the same thing what and so it's working there and I've since talked to them and it's still working just like this so it's working for them now one thing's really important here is obviously that I taught the same way just have nothing to do with our product our product just allowed this to happen but the teachers do this the other thing is that these kids don't take the test until they're ready to take the test that's a huge deal so don't think that we're doing anything magical but the point is when they were ready to take it this is what it looked like and they're all ready to take it within the time frame that was the summer semester so it actually works here's the sort of overview results and if you have any questions about this I invite you to go to dragnath.org oh I have ten minutes do I have ten minutes? okay I have two things I have a seven minute video about this Sierra Vista experiences incredibly inspiring make me cry and I'll show it to you but alternatively or additionally I can answer questions too for the next couple of minutes so any questions? yes I had a question about it you said that the were the kids that didn't have this program in the school did they also get to take the test when they were ready? no okay so they were paced they had a normal you know you show up you show up you take the test on Tuesday I'm sure it has a lot to do with this but they've actually tried it the other way because they took it at their own leisure and it didn't help that didn't make any difference and it turns out that the kids in the hybrid course actually all took it on Tuesday anyway they just had a choice when they take it but the point is that the threat wasn't there right right would you just elaborate one more time on what the support was in the classroom? yeah it was I got to watch it actually it's a computer lab it was open it was open 12 hours a day so students can come and go if they choose to and then three times a week a teacher coach was there to help the kids for an hour or two I could show you the video you can tear her or say it for herself what made the three presenters that you chose successful? what's that? the three presenters that you chose it? it was and you know I'll tell you it's really weird what I heard back from the students we had presenters that I thought sounded good and looked at and the students would say something like they don't really believe it and I go what? and then I'd hear it from the same I'd hear it from students in different states so there's something about body language or something subtle that I wasn't picking up and our director wasn't picking up but the students were picking up and what happened whereas the person you'll see when you look at the product you're going to go you're going to think they're good but it turns out they're more pertinent they're believable and they engage with the 15 year olds that are watching it and how long did it take you to develop the algebra course? two and a half years so awesome well because the first nine months were blown away by getting a problem so I'd be through way nine months of work before we actually cut it these guys make I'm looking at the video online but do these video presenters they make all that and make sure like at least I'll make it or did the presenters the presenters? oh no the presenters are just presenters or actors so we just cured the actors yeah we went through teachers went through we went and ended up going with actors and we had scriptors were professional script writers who happened to have masters in math and then it all went through a vetting process with dozens of instructors that made sure it was running so you essentially broke up you broke up the traditional like teachers everything presenter, curriculum designer we tried it the way you'd expect and it didn't work and we just ultimately went back to professionals in every step along the way including the curriculum we had curriculum experts make sure the curriculum was right every part of this was done by someone who knew what they were doing they were the best at their job and not surprisingly it worked out okay the presenters that were successful were they successful in the judgment of the students liking them or was their attachment to the performance of the students I don't know so it's quite possible but somebody who's really dry and boring and a better presenter that if they're dry and boring and the students are it's an engagement issue it's getting the math right but it's also getting the students involved I'm just thinking in terms of the humor idea off the top of your head to think yeah humor is a great idea would be inspiring I thought so liking the presenter may not be the successful I don't know I'm not sure it's like it's connecting I mean what I heard from students that would talk about it was more that the presenter reminded them of their uncle or their brother and they were a good guy they believed them and the way they talked they seemed to understand the neighbor they came from and the example they used was one that made sense to them it was that thing it was not only the presenter as demeanor it made probably their appearance and the way their reflection but it was also the examples that they were using which stated right these are professional actors but the writers were getting feedback from the focus groups on what was working on I mean we reuse a great example how much flowers could make cookies no one cared about that but when we talk about skateboarding or downloading music or and so it was it was a matter of connecting around things that mattered and then also culturally connected with these cases what about the two demographics the ages same presenters worked for both no no actually they do in some cases but not always we had some of the connect presenters would work for the older demographic too but not always and so we had to start over do you want me to start this video two minutes? actually it's at the end of four actually okay we can be done I can play the video and they can leave that they won't how's that here we go there's another thing what 15 minutes so if the video is playing it won't earn anybody so I choose to if you like to leave take it off if you want to see this I won't be upset Sierra Vista is an alternative and a school these are what we call at risk students they've already had a situation what happened did you extend your desktop you're not going to see this video it was already on my desktop yeah but I'm not trying to what did I do you've got two screens connected drag it over I gotta see it first gather the screens it should be there now okay okay come do it yourself I think you've got so where is it coming do it yeah sit up here oh there it is here we go start slow it off Sierra Vista is an alternative and school these are what we call at risk students they've already had a situation in which they were not successful we have a fair amount of team parents we have students who are behind in credits for medical reasons we have students who are behind in credits for discipline reasons we have kids who just commit focus in traditional classrooms what we're really trying to do is provide an alternative here where they can be successful we're trying different models we're trying different methods of delivery we're trying different types of curriculum algebra was one of the things that we had really been struggling with we had tried several different delivery methods and none of them felt really what we were looking for we knew we needed to change things somehow but we weren't really sure of where what direction to go we didn't have a lot of resources as far as funding so I looked for as many open educational resources as I could and those are resources that are available freely that are adaptable one of the things I came across was Hippocampus which is a project of the national repository of online courses Enron this is the commercial part your brother takes arithmetic and makes it stronger faster if you look across the country the largest single college of students not getting to a high school diploma as algebra it stops them if you don't get through algebra you're not getting the resources you're relegated to a low paying job or a fitting job at all these days you may not always see it but you can find algebra just about anywhere we developed a course that took advantage of a lot of the research that is about on how different types of technology address different types of learning styles we have games that are based learning that allows students to draw upon some higher level thinking skills we had tried other online programs that they were very rigid and the students had to go through them kind of lockstep and they would get caught up on one concept that they just couldn't push past so we needed more flexibility our courses are built to teach it in the same order that NROC has created it the learning architecture allows them to break it apart move it around they can add their own resources we can add things we can take things out and we can make it work for our teachers and for our students that's what really made the difference we spent a lot of talking to students finding out what was and wasn't working for them we would build something out of that and surprisingly we were wrong a lot things that involved ranges of distances let all the places you could go and take a gas the language is actually critically important it's got to be clear it's got to be words that students hear and understand easily idioms aren't going to work because English may not be the first language humor doesn't work especially for a 15 year old 8th grader who just not can work it takes a lot of people who have teachers teachers who've taught teachers developmental editors who know how to write good problems so this is where you're going to all your reading looking at how do you get those bad tasks I'm in here with them I'm working with them on anything they're struggling with sometimes we have to go back and redo assignments in order to get them out to car where they need to be really whatever they need to help them be successful maybe they're helping on the way but this is where you're going to go first students to go through the class at their own pace it's been really important because the students that are really comfortable with the curriculum are able to work ahead and not be constrained by our timeline of getting everybody through at the same pace students who are struggling a bit more have that flexibility of taking X moves side to side Y moves up and down I know you do I really like this program because it is so very different from the direct instruction classes it's more exciting for me I'm not doing the same thing every day every week over and over again