 So, when I submitted this thing, I didn't necessarily realize that it was the length of the talk was only 15 minutes. So, I put together a really nice deck of slides, assuming it was going to be like a normal 30, 45, 40-minute talk or whatever. And then I did see that, and I spent some time taking slides out and consolidating things and constantly and so on. And then I saw that I was the first guy out of Brady, and the third guy is one of my folks there, so I just put all the stuff back in. So, before I dive in, we did one of these talks at the Moot in Universal City last year, an update on where we were a year ago. So, was anyone at our talk? One nice. Thank you for coming. There's going to be a little bit of a repeat because I figured there wouldn't be a lot of people. So, I'm going to go over some of the stuff that got us to where we are now, and I hope we'll do that relatively quickly. And then talk a bit more about what we've done in the past year since we met last and then a little bit about where we think we're headed. So, the thing we're using for our learning analytics is called Learning Analytics Process, where it came out of a project called OAAI. It was funded by the Gates Foundation and Carnegie and so on. And there was a whole bunch of information about that, and that was three or four years ago. And we've been working on it for about three years. We partnered with Marist College, who did the original research, and somebody called Unicon that I chose to bring LEP to our campus. If anyone's interested in the research that led to this modeling software, you can take a picture of that or find it on the website later. But that is the paper that led to the learning analytics processor. So, the first year we did anything at all with a proof of concept. We just took a whole bunch of data. Let me say a little bit of historical data. We didn't have all that much. Standing up to the people at Marist College, they did some stuff to show whether a prediction was worth considering, whether we had data that was worth considering. And it turns out it was. It looked pretty good. But there were some things we needed to do to get there. So, the proof of concept was done with all relational database tools, small-centered data, like I said. So, the second year we spent making it bigger and bigger and more awesome, more like the 629 man or whatever. And so, we moved from relational database tools to large data tools. But we had to migrate all of the infrastructure that we could do to handle larger amounts of data, because it turns out we generate lots and lots of data. And again, we had similar results. And up to this point, we'd only been working with one model. So, that's a very quick way to get us to this year. So, the past year we've been working on more robustness. So, up until this point, we'd taken data from our student information system and from Moodle, but that was it. We spent a lot of time getting those things to talk to the model to get the stuff in the right format for the model. Jeff had to map things to get into the way the model worked from Moodle. All the while, we're thinking we would like to be able to incorporate information from other tools. We have other tools on campus. We'd like to collaborate with media sites, things like videos. There's a lot of blog files about, you know, when they start their video, when they stop the video, did they watch the whole thing? Did they watch it at regular speed? Did they watch it at 1.5X or 2X or shipments through it at 3.5X or whatever? All of that stuff seems like it would be valuable for predicting success, right? And so we'd love to be able to incorporate those into the model as well. So it turns out it's a lot easier to do that if we have a learning records warehouse stuff. We are in the process. This past year we started the process of standing up learning records warehouse. The idea is that you funnel all of your data that you want to collect into the LRW, and then you only have like one channel that gets from the LRW to the modeler, but everything sort of coalesces into the warehouse. The other cool thing about the warehouse is that if you've got data that you think sometime might be valuable, you can go ahead and put it in the LRW because one of the things we've learned from statistical modeling such as this is the more old stuff you have, the better your predictive quality is, right? So the old stuff is used to train the model. So if I'm like, you know, our campus does card wipes for you to get into the gym, is your gym activity a predictor of whether you're going to do well at the university? I don't honestly know, but if we think it might be, let's go ahead and collect your data now so that in three or five years when we run out of all the obvious stuff, we can say, hey, I wonder if that card swipe thing is worthwhile, and we've got some data, right? So that's like the first side benefit is if you've got a learning record warehouse, you're still collecting whatever you have. It's a very nice thing. The other thing we've done is while we've been doing this modeling, while we've been trying to come up with these predictive numbers to see if students are at risk, we haven't really given a lot of energy towards what to do with it, okay? So we get, you know, a number of students in a class on a given day, we have a predictive number. What do we do with that number? There are other people on campus who are thinking about what to do with that number, right? There are a bunch of academic advisors that all work for student affairs, and we would be great if they do what the number was. But we've also been thinking it would be nice if we had some way to expose it at least to the faculty, so a faculty member teaching a class could see, you know, which of these students are at risk. And let me just say that a lot of the time if a faculty member looks at their grade book, they can probably tell, like, who's at risk. The interesting thing is that they don't necessarily go there and look, and it's sort of an effort to think about it that way, right, because they're not necessarily. Our thought is if I have a thing that's sort of in their face that says, hey, faculty member, these are the, whatever, 15% of your students that really, really could use something extra, maybe, of course, to see that that's a thing, right? And we're also considering a way to expose something to the student. Now, you don't say, you know, you have a 95% risk of failing this class to a student, right? And that's the wrong approach. Clearly, you need a softer touch than that. But there may be ways to engage the students. And again, I mean, this is, so from my personal experience when I was undergrad, I was really good at looking at the, I mean, you can calculate the math, you have the syllabus, and you know, I did this well on test one, I did this well on test two, and you do the math, and it's like, I need like 173 on the final path. But I think I can pull it out. I think I can pull it out because your students are really good at believing themselves, right, and thinking that they've got it, that they can do more than the possible. And this is more of a shake on the shoulder to say, get some help. And the goal is to do it fairly, right? So in the first couple of weeks, and so I don't know if you guys heard John Whitmer talk yesterday and say that they were seeing sort of a sweet spot on the predicted quality somewhere in the two to three week range, which I found very important, because that's kind of where I'm aiming. I would like to be able to, within those first two to three, four weeks, know which normal chunk of the students need somebody to tap them on the shoulder and say, you know, we have some resources here. You're struggling. It looks like you might be struggling in this class. We want to help you. Not midterm or whatever when it's too late or two weeks before the end of the semester when it's way, way too late. So anyway, to that end, we wanted to open, we were implementing this open dashboard. I don't know if any of you guys have seen open dashboard, but it's gone through a lot of revision in the past year. While we were spending a year working on beating up the infrastructure for big data tools to use the learning analytics processor, there's a group in the UK, just JISC, the group of institutions in the UK, also working with Unicons to try and make open dashboard a nice, better thing. So they're working with Unicons and Maris. We're working with Unicons and Maris. It's not, we're not all working together, but because they put all this effort into OpenDash, I get to, we get to install OpenDash and a better OpenDash without having to pay for all of that work that they did. And they get a better learning analytics processor engine because we invested a bunch of time and effort into figuring out how to build all of those big data tools. It's kind of a win-win. So another big thing that we played with this year was Calhort. So originally, the first couple of years, we had a single model for the entire campus. We're a big school with 40,000 students, 11 different colleges from design to ag and life sciences to engineering to vet school and so on. Very, very different, very different classes. And we thought maybe, you know, if we split them up into smaller groups, we would have better predictive quality, right? Because if you lump them all together, stuff gets lost in the averages. So we did just that. We did it a bunch of different ways. We did it by enrollment size, small classes, medium classes, large classes. We did it by doing a lot of different software junior senior grads and so on. And we did it by LMS usage, right? So we looked at the log for the classes and said, you know, if there's no logs, they're not using the LMS. If there's a small amount of logs, they're using it lightly. We use that as a proxy for depth of use. And we learned some stuff. One is that splitting by people didn't work out so well. So if I split by freshman, sophomore, junior, senior, I'd say, there might be a course that has some sophomores and some juniors in the same course, which means that some of the students are in one model and some of the students are in a different model. And so it's better when all of the people are in the same class in the same model because they get compared who they're peers. And if you split them in ways that divide them, that did not help. It did not make things better. The other thing that we were happy about was that splitting by LMS usage showed some real problems. This is probably too small, but in the middle there's a graph. I put all the numbers because the other thing we got this year is a number of guys. We have a quant who works with us, and then it's Chris. They've learned how to do all this stuff. It's not part of your sort of bailing week, so he's having to learn all this stuff as well. And then we spend a lot of time doing it. But here's a couple of graphs. The recall is the most interesting statistic in this whole thing. And unfortunately, the order that Chris put them in is not the best order for visualizing. But the black bar is back in the old days, where we used a single model for every course, right? So you know that's old. But then if you're in your mind, you take that light gray bar that's on the right-hand side and slide it over to the left, that one's no LMS usage. Then the next, the orangey red is a little bit. Then the dark gray is a medium amount of LMS usage. And then the darker red is a heavy LMS usage. And you can see, right, it goes up like this, which means that if you use the LMS more, your prediction about student success is better. And I gotta say, we did a dog and pony show for my Vice Provost. And for some reason, I'm not sure why. He really liked that fact. It says that our LMSs have value. When we had everybody in the single model, we were a little bit concerned because the difference between just predicting with demo data and predicting with demo data and LMS data was not very big. And that's a bad thing from our perspective. What that means is the stuff people do in the LMS maybe doesn't matter as much. If all you need to predict student success is the demographic data, which is stylish, you know, you can have a guy with a clipboard at the door the first day say, yep, yep, nope, yep. You know, it's a thing to complete. Thankfully, it's not the case, right? So as we segregate out the courses and use the LMS data, it turns out it does affect the model pretty heavily. All right, we're about out of time, but real quick. Next year, we're going to integrate the dashboard into our Wolfware, which is our sort of portal as Marty said yesterday. We're going to start pushing data from our other tools, the media site, like we'll collaborate, whatever else, into the LRW. We won't necessarily be using it yet, but we'll start putting it in there so we'll have it. Maybe add some of their data and we'll start playing with it in the predictive model. It depends on how much time we have, it's tough to tell. And then we want to start. Next year, we want to start running that model regularly. So right now, we do a handwork. If someone has to say, I'm going to run the model, we've got a bunch of steps. We're going to have all our colons. It runs day one, day three, they say, well, what happens to me? I have to be in the semester and pushes the data into the dashboard or potentially to other people on campus that can use it for good. So with that, I'm done with a little bit of a pitch. If anyone's interested in joining, we benefited from the work that just did on the OpenDash. They benefited from our work on building the infrastructure. There's a lot more work and certainly there's room for other people to get involved. They're interested in this model. It's open. It's free. You have to either roll your own code or work with Unifont, pay them to build code. But I think your dollars go a lot further this way. So I paid for some stuff, but I got a bigger set of things back and you would do the same. So that's basically, if anyone has questions. So our proxy for the usage was the amount of log files that were generated. So given a certain class, if we counted the number of lines in the logs for a class and it was zero, that was a course with no usage. If it was within a certain range, it was low within another range, it was medium within another range, it was high. And I don't know, just pick where you draw the lines and it was relatively arbitrary looking at the spread. I mean, I think you looked at the spread and saw some natural places where there were gaps. Right, so right now we're working with the benefit of the entire semester for the data so we can look at all of the logs. So when we're at the beginning, there's a lot fewer logs, so it's harder to divvy them up. So we're either going to just use what there is and hope for the best, right, and hope that the courses at the high end start strong and generate a lot of log files, or some way map them to previous semesters. And no, it's tough because, you know, people don't always teach the same course in the same section. If there were a way to for sure say, this is the same guy, he can teach them the same thing, and he was a pretty heavy user last time, he's probably going to be one this time, but that's a tough mapping to do. So we haven't exactly figured that out, but I suspect that in the second week we'll see a difference. Anyone else? Yeah, if there's time after we can. Thank you guys.