 So just to give you an overview of what we're going to spend the next 25 minutes talking about. Our big question is thinking about how do you actually evaluate deeper learning? Forget about evaluating the things that are really easy to measure on a bubble test, but if you're really interested in deeper learning, how do you start to measure that in a meaningful way in a rich online learning environment, not in a curriculum unit, but actually an experiential learning system? So we're going to use vital signs as a case study for this. I'm going to start by talking about what our learning goals are for vital signs. What are the actions and the evidence that students take and students lead through their participation that we could use as raw material to mine to look at deeper learning? And then what we really want to do is have a discussion with you all, looking at how would you approach evaluating deeper learning in vital signs or in another similar learning environment? And how do we address this challenge? Because it's a common challenge that, from what I know, nobody's really cracked successfully. So vital signs, and I should say I'm from the Gulf of Maine Research Institute, and in Maine we have an extraordinary asset. There's a statewide Maine Learning Technology Initiative that's given a laptop computer to every seventh and eighth grade student and teacher, and about 60% of our high school students have them as well, as well as all the teachers. So we're leveraging that infrastructure and creating a statewide citizen science and science education program. So we have, it's an open system, anybody can participate who wants to. We have special resources and put a special focus on educators, both in formal education and outside of formal education. And then we're also working with scientists. We're working with about four dozen scientists in the state who are interested in the question that we're researching, which is invasive species, where are they and where aren't they? What kind of impacts are they having in their habitats? And that's a question that scientists really need help answering, because they can't be everywhere at once. But it turns out you have more invasive species problems, you have more people, and where you have more people, you tend to have more students. So it kind of matches up quite nicely. So what are our learning goals? Well we're really interested in seeing our participants gain scientific knowledge, scientific thinking, and developing skills with using evidence and using data to back up claims. We're interested in collaboration and communication skills. Those are things that show up on the Hewlett Deeper Learning Rubric, as well as in the framework for the next generation science learning standards. And they're not things that you can easily measure in a bubble test, but through participating in vital science we have lots of teamwork, there's lots of collaboration both between peers, but also among the peers, scientists, and the community at large. So there's an opportunity to start looking at communication and collaboration, so we want to do that. And then of course to look at content knowledge, and not just do you know some more facts and figures about invasive species, but are you learning principles about ecology, biology, and biodiversity, and even something about the science practices, which are giving new importance in the framework for next generation science standards, along with engineering practices. So again, the question is how do we measure these things in a rich, experiential, online learning environment? So to give you a sense of kind of what the vital science experience is like and what kind of resources we have that we could start using to measure, I'll just run through kind of a standard in quotes experience that a student or learner might have. Vital science is designed to be open and different educators, different learners, different people use it very differently. So there's no real one way that people use it, which is very much by design, but it is a particular challenge for starting to measure impact on learning. So the typical way that it happens is that participants start out with a question. And it might be a question that they have about their local community and invasive species. More likely they're going to turn to one of the field missions that we've developed with some of our participating scientists. And we've developed these for two reasons, one, because the scientists really wanted useful data, and two, because educators were having trouble coming up with investable questions. So they start over the field mission. They learn something about the species that are involved. They learn some background information about why is it important. And then they go outside and they actually look for that species. And they look to see if it's present or if it's not present. And they collect evidence to back up that claim. They collect pictures and they write evidence statements. They also make general observations and they get a GPS location, all of our data is geo-referenced. So they make a claim. They did find it or they didn't find it. And then they upload their observation and all that data onto our website. So this is kind of a zoom in of what one of these species observation looks like. There's a field note not required until we often see them. There's a sketch, again, not required. But we often see them and they're often really wonderful. And then they are required to have at least two pieces of supporting evidence but they can have up to three. And that's a pair of picture and a written statement. And a couple things happen as soon as an observation is posted. One, the species expert who's the designated person who's most interested in that species gets an email. And that's one of the ways that we've incentivized their participation is they get an automatic notification when there's a new observation of the species that they're interested in. And then the other thing that happens is that it's immediately public and anybody signed into the system, including that species expert, can make a comment. So we're starting to see these great conversations happening around some of the data and collaborations as well in those conversations. So just to dig in a little bit deeper, can you all read that or should I read it out loud? Let's just, you can read it. So here's a field note. It's fairly typical and there's a sketch as well. And what's fun is that this field note was posted and then the scientists who's the species expert actually picked up on the fact that they included a sketch and kind of validated that they had used it and the importance of using sketches and mentioned why he thinks it's important and then also pushes them with another question, like did you see any evidence of seeds or fruits? So this is kind of typical of the conversations that happened between student teams and our scientists. We also see conversations between student teams. So in this instance, there was a team, I think it was the invasive destroyer, I don't know which team it was. Anyway, they'd gone looking for Oriental Bitter-Sweet. They hadn't found it. And another team, Babby from another town, no connection in real life, saw this activation. They'd looked at the same species and found it. So they chimed in with the way that they've been able to recognize it. So we're starting to see students exercising their new knowledge and sharing it with each other, which is really exciting. We all know if you want to learn something, you teach it, right? So those are some of the ways that students are communicating and beginning to collaborate with each other. We also have a place on the website where they can post final projects. So some of the things that we've seen on the laptops that all the computers have is iComic and they've made some really incredible comics that bring the species to life and their interactions with their environment, interactions with other species into life in really fun ways. And even posting videos. So in this picture, you see a young woman demonstrating the use of a Lake Aquatic Species sampling tool in a video, so she's sharing what she learned. So the other thing that we're interested in is content knowledge and thinking about how do we actually look at this across the whole system and thinking maybe there are some words that we can start looking for inside the field notes or inside those evidence statements that we could use as an indication of shifting vocabulary or more scientific vocabulary that would start us thinking or start us being able to document some of the learning that's happening. You. Let's stay here for a second. So again, what we want to actually do with the bulk of this session is use vital signs as a case to get at this question of how if you were a member of the vital signs team and the Hewlett Foundation came up to you and said, wow, this looks totally awesome. You've created this incredible open learning environment where students and scientists are co-creating data and it seems like, you know, our gut instinct is that students are doing really interesting stuff and they're developing this sort of important understanding of scientific thinking and developing a connection with the main coast. But how can we say whether or not they're actually doing any learning in what ways? And not even, I want to say measure, but I want to say quantify because not all the data we might gather to do this would be quantitative. But how would you go about doing this? So I've been, I'm not going to tell you almost anything at all about my research, but I've been looking at deeper learning in wikis. So students who use things like PB works and wiki spaces as collaborative learning platforms and trying to assess whether or not you can evaluate the degree to which students develop skills like collaboration or expert thinking in those kinds of learning environments. And so what I want to do with my couple of minutes is just give you a sense of sort of a way to think about this data as an entry point to think about, so how are you going to do this measuring together? Well, we're going to do it together for a while and then Sarah has to actually do it. With your help. With your help. So I've started to think about this kind of data as scalable, real-time, individual behavior and learning data. So vital science is tracking continuously to the second. Every interaction the teachers and students and scientists are happening in this learning environment. And it's capturing it in such a way that we can look very, very closely at an individual case or we can aggregate that data. We can scale up that data and we can look at hundreds or thousands of cases simultaneously. So we can do really rich detailed qualitative work and we can also do broader statistical work. So we're calling this data scribble data. With the idea that actually what we're capturing is that all the little scribbles that students and teachers make in the process of their daily activities of learning. And to me, I think there's incredible promise for essentially replacing summative assessments with the process of sort of constantly, formatively evaluating the things that folks are doing. So to kind of prime your thinking, here are some of the ways that I think about potentially in a more generic sense sort of exploring this kind of data. So this is a state space diagram where the number of cases is on the Y axis and the duration of kind of data collection is captured is on the X axis. So if you're sort of coming at this from the field of education, technology, learning sciences, most of the work that you're doing is probably down around here. So you're doing design research experiments where you're tweaking what's happening in a couple of classes. You're tweaking what's happening either in the delivery of the content, you're tweaking the actual learning environment, you're doing observational research where you're following a group of learners for a while and seeing what they're up to. There are lots of folks in this particular group who are interested in looking at hundreds or thousands or millions of cases over years and saying, okay, let's see what kind of statistical data, maybe without even actually looking directly at the content, can we get some sense of the learning activity that's happening? I was at a DML conference where they were doing things similar to this and they were like hooking kids up to wires and seeing how their skin temperature was changing and their heart rate was changing second to second and what their eyes were tracking on the screen. So really looking like in the case of one kid, data over microseconds to see how they're thinking. That's not quite, you have to do it a little bit more to do that. But this map is meant to sort of prime your thinking that the data here is incredibly rich because there's thousands of cases but we also have this incredible depth to it. We have this incredible real-time history attached to it which makes us think that we ought to be able to measure learning in ways that we haven't imagined thus far. So again, the, well, I'm actually gonna, let me run back to one slide because I think it'll be helpful to put this up again. You know, what these folks are trying to figure out is to see, are these students making gains in scientific and evidence-based reasoning? Are they developing collaboration and communication skills? Are they developing content knowledge? And how do they go about measuring things? So here's kind of where we need your help and how we wanna spend the next, the sort of second half of this presentation, the next 15 minutes in conversation. You know, if Sarah tapped you to be one of the advisors for this committee that's gotta put together this research program to evaluate deeper learning, you know, what advice would you have for her? What approaches would you take to evaluate learning? What relevant examples do you know about? What other people are you aware of that are doing similar kinds of things? You know, who should they be talking to? What kinds of, you know, are you working on a program that has similar needs and has similar challenges and how are you thinking about addressing them? So I think what we'll do for the next 15 minutes or so is open it up to you all. Let me, and I'll stand up here and keep some notes and you can probably call hands and talk with people. But yeah, it's folks idea, you know, get them out. Sure. Are you attempting or have you been trying to capture activity-stricted data and using that tentative specification that's out there right now to, you know, face the sub-eurology that you did this, simply more elaborate. And then once you do that, then you can do analytics on that based upon assessing the models that you've already made. You have to kind of look at how it's currently pulled that data in and see what that means in terms of what's going on through social media, through the interaction between What do you mean by an activity stream? We just thought it was all the same. Well, this versus back out right now it's been you're working here and I was saying ADL is working with it right now in terms of capturing basically online social behaviors and this has to be the actor. What you have done, you might have clicked this button or you may have, you think that's very, very basic. But then you can elaborate on that and start gathering and you launch an API of the tracks. You gather that data, tracks a little bit and it reports it back to something to be interpreted for being labeled. So you're capturing the system. And if I could build on that actually. So the activity stream spec was originally built as an open spec for social networking platforms. So it's used Google plus and but the work we're doing with the learning registry and that's where the ADL is part of that. We do have a new spec out that people can use taking that sort of syntax based way of assembling the actor and the action and the object of the action and everything for learning data and for learning kinds of activities and pedagogical contexts. And so I'll post that wherever that makes sense to post that and we would love to have people test it and contribute those data from your platforms to the learning registry open data set of these kinds of learning resource peri-data social mitigation. And besides learning registry ADL also has other projects we have a project in which is actually capturing that to report back to a learning record store. So that you can and it's another way of interacting, capturing data which is not necessarily peri-data. It's a little different but it's very similar and you can use such things in the activity theory different things but we'll look at that in the practice and see what it means. But one of the things that I'd love to see and then I'll let it up is where you're coming at from that research perspective and you have vocabularies maybe that you're using to code text where you're saying okay we're going to look for these terms and these terms equate to something about a level of learning or a level of communication or even how closely people are adopting the resource a level of commitment to the resource that they're using that kind of thing. Where you have those vocabularies already I would love to see some of that come into this discussion about how do we share these data openly so other people can see where that activity is happening across multiple platforms. So if I were to summarize what I'm getting from that and so what good learning activities would look like and then kind of develop a syntax for capturing activity stream data that would let you test whether or not students seem to be doing following desirable pathways. And it might go beyond the syntax in terms of actually APIs that plug into the various services that you're using whether it be a virtual environment or social media or what have you you want to try to track things. So you're moving all the way around So once the data is established and it goes into this whole discussion about open data skills Maybe you have something totally different and we have ideas that you do not resonate with that at all It's totally different It could be the exact same thing American Public University has done some interesting things with IBM's text analytics that we used to praise where they're looking for language Are students making arguments or presenting hypotheses you might search for the word because it's a really simple way to talk about you're telling a program to analyze lots and lots of words That goes towards machine grading The best is an expert human grader but if that's not scalable enough you can make human grading easier with rubrics so it's established in standards and allows less qualified people to create faster by using rubrics even peer-to-peer grading can be really effective looking for what other students think about this and thought about this stuff That is imperfect Keep going And thinking about would you a priori hypothesize what's some good text to look for what we'd be or would you grab a sample of 1% of posts start reading them and classify them and take all the really good ones and say within this set of really good ones is there common language that we're starting to see and then use that it's interesting to think about what should be data driven and theory driven so you might do some really grounded stuff to say what words count and then use that to drive your measures What other things would folks be interested in trying if you were a if Sarah hired you what would you sell some nature back to you This is kind of different than what I'm interested in using computational linguistics an algorithm that's one of the most machine learning intervention to look at and analyze again you're doing post-hoc analysis of data but you're doing it in a different way and it would prove you have to emergency there's work going on right now and it's open and dark it's things like that that are doing that for other reasons but it's very easy to be turned towards educational space as well towards educational things who are some of the people who are doing that how would I follow up on that okay don't find terrorists find smart kids identify the future heroes in Europe I actually work with a group a senior older who is using a program that they've written algorithm that computational linguistics would find students misconceptions in science against experts that's basically what he was just talking about so it's something that people are looking at and really wanting to get better because it gives the students immediate feedback on their ideas that they're writing about and then if you wanted to do more things for other people you could really be back to the students yeah I think that's a great point that almost anything we might come up with to assess learning for our own purposes like is this working in theory it would be great to think about the next step of feeding that back to educators feeding that back to students in real time and saying based on what we've developed which ways to figure out what good scientific thinking looks like and you're hitting it or you're not this is kind of coaching a group around it or whatever one thing I think is somewhat underutilized in evaluations portfolios an example of that a really good one I was sitting on the web radio where the students had been producing a bunch of just kind of sounded like almost random content interactions just broadcasting via this radio but the teachers had them organize it such that they had actually on a blog like a blog post of all of the time so that it was really easy to go back in and look through what individuals had done and so you know sometimes when you have a bunch of kind of random information sometimes it's helpful to create some kind of portfolio assessment that students actually help you organize what they can do and then it's really easy to take a rubric and have someone go through that material that's great yeah I think all those are fabulous ideas so you know we had some of them going in in terms of using some kind of language crawler I think that came up I think we have both students and educators developing rubrics for educators to use to evaluate student work and to see how their assessments of quality change over time to think about giving the community some kind of responsibility for doing that assessment so developing some kind of rating or starting system doing a peer review which I think is connected to that Sarah's got Bridget Barron in Stanford who does a lot of ethnographic research who's just got a grant to study vital signs and the first thing that they're gonna do are case studies they're gonna spend a ton of time with a small group of students and teachers and follow them really closely and do a bunch of ethnographic work and actually when you hear that earlier conversation about kind of all the analytic stuff that you want to do and the sort of sense that you need to have some kind of theory moving into it as to what quality it looks like to begin with you know it makes a lot of sense to have someone really spend a lot of time sort of looking at the learning process looks like getting a sense of the richer content and using that as a way to think about you know how you could extrapolate from that to do some of that yeah so I mean to me you know to me what's incredibly exciting about this is the opportunity to use lots of different approaches I mean I think in education technology broadly when we wanted to learn something in scale we've done surveys and we wanted to learn something in depth we've done case studies or design research and those have basically been the two things that we've done and this kind of data I think what's so exciting about it is the opportunity to mess around with those trade-offs between depth and breadth and get some you know having data that lets you do things that both are generalizable and have really high internal validity and get into a couple different angles Are there parting comments? Thanks for your time