 Hello. I'm Jeff Longland. I know some of you, a few of you. I'm the keen lead on learning analytics project here at UBC. Just because I'm sort of the team lead on the project, that doesn't mean that you're the only people that are doing learning analytics here at UBC. There is a community of learning analytics and scholarly research and learning analytics that far predates our institutional project. And I encourage you to join us after the keynote. We have a little showcase. We'll be talking about some of the other work that's been going on here at UBC. Before we get going, I'd like to acknowledge that we're gathered here today on the ancestral, traditional, and unceded lands of the Musqueam people. And we have Tim here with us today from University of Michigan. The last couple of days, the Society of Learning Analytics Research has been holding an event here. There's Summer Institute and three days of workshops and tutorials. I see a number of folks here that were present at that. I'm really happy you guys came today as well. Tim's an amazing three days. God bless, thank you. Tim's a data scientist and professor of physics and surrounding education at University of Michigan. He's also the chair of the undergraduate program. And after many years of working with data to map the universe, Tim decided to turn his attention to something a little more focused and a smaller data set perhaps, learning analytics. His recent work in Learning Analytics explores how the collection and analysis of educational data can lead to more personalized and inclusive teaching in large classrooms. One of the things that Tim is working on is a large Sloan grant to work on improving inclusivity and equity in STEM introductory courses. And that's a foundational change across multiple research universities in the US. And R11, you got 11? 10? 10, I will tell you more about it. And Tim will go into more of this, so I won't really go too much into that. It's my immense pleasure to welcome Tim to come and speak with us today. Thank you. I believe so many people came into a room on a day like this. I mean, why are you not all outside? We should all be outside. So anyway, thank you for being here. It is a pleasure to talk about this topic. It's one I've been thinking about and working on a lot and trying to work with other institutions to understand how we can all kind of together advance the goals that we have around education, taking advantage of the new opportunities that data provide. So what I'm gonna do is I'm gonna talk about the kind of gradual growth of the use of data to support teaching and learning at the University of Michigan. That story will be partly a personal story. How did I get involved in this? Because I didn't start out doing this kind of work. But it's also an institutional story. How did some of the pieces that our institution come together to enable us to do this? How have we created some structures that invite in people who play very different roles on our campus into this process so that they can work together? I would be happy to take questions at any point if you have them, so feel free to interrupt her. We'll always have a chance to talk at the end as well, okay? So I like to start with a little bit of background. I am a scientist who is trained in physics, did much of my classical research in astronomy, and now I have all of my attention turned toward education. So those titles you heard are actually just a historical list of the things that I've worked on through my career. I have always been drawn to projects that have very rich data sets. Because data is a wonderful thing. It lets you ask and answer questions, and when you have really rich data, you can ask and answer a lot of questions. When you have really large data sets, you can ask and answer those questions precisely. So all those things come together and are attractive to me. The real reason that I like big data sets so much is that my PhD thesis project ended up being a null result. So we went out to measure something really important and found out it was wrong, and it was nothing there, and measuring nothing a while very important is no fun. So I started to say I just want to measure a lot, something a lot. So I've worked on projects like the Sloan Digital Sky Survey, which set out to expand the map we have of the universe by about a factor of 100, the robotic optical transient search experiment, looking for very rare things in the sky like gamma ray bursts and the dark energy survey. All of these projects are characterized by being very large projects, much bigger than a small group of people could do on their own, a single lab, right? So we were required to bring together teams of hundreds of people at multiple institutions and figure out how to do that, because if we didn't, we would never be able to do those projects. So I have a lot of experience with those kind of giant science projects too. Multidisciplinary teams doing whatever they have to to get the data they need to answer their questions that they have. So why have I turned to education research now? There are a couple reasons for this. One of them is that data is much more available than it was before. It is possible to see what's happening in the world of education in ways and in forms of detail that were simply not accessible when I started teaching at Michigan in 1995. I mean, I have the file folders that have all the papers from the classes I taught then, right? It's not accessible data. And that's changed in a really dramatic kind of way. So there's an opportunity here. I also have learned through my experience of teaching at Michigan that our big introductory STEM courses have what I would consider a number of serious problems. They've been present since I was a student. We haven't fixed them yet. And I think now there is an opportunity to take advantage of data in ways that might allow us to make real progress on those courses in particular, not because they're the only problem, but because they're my problem, all right? My primary focus has been on large foundational courses, both in practice. I teach these courses. I just finished two straight years of teaching an introductory mechanics course for 700 students a semester. I want to design those courses and deliver them so that they are equitable and inclusive and also very effective. I want students to learn a lot. I want them to all learn it. And I want them to all feel like this is something they belong to. These courses are also special in a way that has to do with big data or substantial data, right? When you teach a course with 700 students, you have the opportunity to learn a lot about what students go through when they're learning physics. And to do a pretty precise measures of a lot of those things. So if we don't, every single time we teach, treat our classroom kind of as a laboratory. We're throwing away a lot of opportunity to learn. And I think big universities have a special role to play here, right? We should think about all those classes as labs, not because we like to treat our students like lab rats or something, but because they are actually experiments in education and we can learn a lot from them. So we think a lot about how to make that happen. Okay, so our work began, my work on this really began thinking about my own courses. So this is the classroom that I teach in. 240 students seated in a movie theater, right? Effectively. And we do our best to try and teach well in that kind of environment. We use lots of demonstrations. You can see there's clickers on the table, a whole deal, right? So we have done all the kinds of things as many as we can in this kind of environment and wanted to understand better what was going on. So around 2008 or 2009, I got together with a couple of my colleagues, Gus Everard and Dave Gertis in physics. And we did a first data project. We wanted to try and understand the performance of our students in our classes in a way that was better than just looking at their final grades. If you look at just final grades, you know whether students did well or poorly, but you don't know whether they did better or worse than expected. Students know that. They come in with expectations about what they're gonna do and come out with outcomes, which are either better or worse than they expected. They think about that very strongly, okay? So we went into the data sets that we had and looked at all the different things that might predict student performance in our classes. What correlates with student performance? Just to sort of say, we wanted to identify the ones who did better than expected or worse than expected, right? And what we learned is something everyone in the education world already knew, that grades in other classes is the best way to predict grades. So if you wanna predict a student's grade in any class, look at their grades in their other classes, it's the best predictor. It's better than any standardized test. It's better than anything else, okay? So to first order, grades predict grades. And I think that's because they reflect the whole toolkit of student things skills that students have. It's not just a particular thing about it. You know, the way they go at school is wrapped up in all of that. Students also use this to predict their performance. Students have an experience of having grades in other classes and expect to receive grades that are comparable to what they've received in other classes. When it's much higher, they start to feel like, wow, I'm really good at that. Or sometimes they think, that was a really easy class. I mean, to be fair, when they get grades that are lower than their grades in their other classes, they feel like, wow, that class really beat me up. Or I'm not good at that. So they're doing this kind of revenue. So I'm gonna define the very simple thing, the grade anomaly that a student receives in a course. It's the grade in this course minus their GPA across all their other classes. And we do this to keep it simple. We look at all the classes of students completed up to the time they got this grade. So even in first semester, you can do something. How did you do in this class compared to the other courses you took even in your first semester, okay? That's called a grade anomaly. So the first lesson we learned from this is that the same performance for two different students can feel either better or worse than expected. This figure I'm gonna show you a couple examples like this. It shows on the x-axis the student's grade in their other courses. That's their average performance, their typical performance. And on the y-axis it shows the student's grade in physics 140, that's the course that I teach, the mechanics course, all right? Each of these points here represents the mean and the uncertainty on the mean for students who came in with a certain GPA, right? And the dashed lines represent the one-sigma dispersion. So there's a relationship between performance in other courses and performance in this course. You can see that relationship, the mean relationship is reflected by these points. There's a lot of scatter for individual students, okay? Let's make sure we understand it because we're gonna look at a bunch of these. Now, this student who came in with a B average and received a B plus, especially in physics did better than expected, right? They're actually outside the one-sigma dispersion there. They're kind of in the tail of the distribution. They're really, they did pretty well here. But the same grade of B plus for a student who came in with a 4.0 is really worse than expected. And you all know this if you've taught students because you hear about it from them. I'm an A student, I can't get a B plus, whatever. The relative performance thing is really big deal. So we learned that and it made us think about what leads to better than expected performance and what leads to worse than expected performance. Is it chance or is something going on? Because if there's something going on, we might be able to help students find their way to better than expected performance, all right? So we did a qualitative research study based on that quantitative study. We realized that we could now identify students who did better than expected and worse than expected and we could like go talk to them. So that's what we did. We asked some students from a previous semester who we put in the better than expected group to come in and talk to us and some in the worse than expected group. We didn't tell them that's what we were talking to them for. But we just asked them, how did you go at this class? How did it feel, talk us through this semester? How did you do your homework, whatever. And what we found is that there were patterns of behavior change for both groups. Both groups talked about changing their approach to the class during the class, all right? The people who did better than expected had narratives that sounded like this. I remember one woman who said, I took the first test and really didn't do very well and I was disappointed in that. But I had three friends up and down my hall who did do well. So I went to each of them and I asked them, what did you do to study? What did you do to study? What did you do to study? And I realized they were doing different things from me. It wasn't really just more, it was different. And she changed what she did and her performance went up. People who don't change what they do perform as expected. People who do change what they do might not. Students on the worst than expected side all spoke about effort, trying really hard in the beginning, maybe trying a second time, doubling down on their effort, not doing very well. And then at some point, acknowledged that they stopped working very hard. They walked away from it. They decided they're not a physics person, so they would just throw up their hands and take a bad outcome. So both talked about behavior change and that was really important for us to learn that if you want unusual outcomes, you have to kind of help people find their way to being a different kind of student than the one they were when they first arrived. You have to help them be a good student in addition to telling them about physics. Especially in introductory courses. To me, it really forced me to recognize something I should have known all along, that my students are all individuals with unique backgrounds and interests and goals. And if I don't respond at all to that difference, then you miss out on a great opportunity. I was trained in a way that led me to believe that the right thing to do in your class is to do exactly the same thing for everyone, that that's fair and it's optimized for the median student or something. I don't think that anymore. And it was studies like this that helped me to realize that. So that was one of the first things that we did. Very eye-opening for us and it caused us to start to think about other traits of our students and how we might study them. So the second lesson we learned from this is that introductory STEM courses, especially the lecture courses, all cause students to receive something we've called grade penalties. You might notice that all these points for the average performance fall below a one-to-one line. If students receive the same grades in physics that they receive in their other classes, all those points would lie right on it if we were normed to other grades, but we're not. So when students sign up for my course, they should expect their GPA to go down. It's pretty substantial. It's about a half a letter grade, that average difference. It's not small. So when students talk of these courses as difficult, they're at least speaking about how it's difficult to get a high grade in these courses, whether it's difficult to do it or not. Grade anomalies. And all of our introductory STEM lecture courses apply these kind of grade anomalies. That was the second lesson we learned. The third lesson we learned from this came when we started to examine the data in a disaggregated way. And the first question that we asked ourselves was, is there a difference in performance between male and female students in the class? We looked at this question first because this is what a lot of people in the physics community were talking about at the time. It is only a first question that you might ask. So we did a very traditional equity analysis, comparing group performance. So for every student, we calculate the grade anomaly. For any group of students, you can calculate the average grade anomaly, just average that over that group. And then you can compare group differences by comparing, for example, the average grade anomaly for female students to the average grade anomaly for male students. Simple equity analysis. The same kind of thing you would do if you wanted to compare section one to section two. Same kind of thing you would do. When we did that, we found substantial gender performance differences. So what you're seeing here now is the same kind of figure. It has two sets of points. The smaller, lighter colored ones represent male students in this class. The larger error bars with the darker color are female students in this class. You can see that at almost all points, there is a substantial gap between these students. So they come in with the same GPA. They come out with different average performance in this class. Okay? If you calculate the grade anomaly for those two groups, the grade anomaly for male students is about minus 0.3 letter grades, about minus 0.6 for female students. So what we call the gender performance difference, the difference between these two is about a third of a letter grade. About a third of a letter grade. Very clear signal. I should say, you know, why are the error bars so small in all these things? It's because there's 28,000 students in this figure. We teach a lot of people physics. Again, you can look at things very precisely. So I first wondered whether there's any way we could explain this. But I have come to conclude that this is a sign of real equity problems in our classes, and I'll talk more about that. So data can provide insights into questions of equity. But it's also true that analyses like these can create problems and confusion. So when we first started talking about the gender performance differences we saw in our introductory physics courses, some of my colleagues in chemistry started telling their students not to take physics from us because we were clearly biased against women. And I think they had good intentions in that. But both on their side and our side, there was something unwise going on here. Because clearly we should have looked at other courses. And I'm sorry to say it took us a few years to do that. Not because we were unable to, but because we just kind of didn't think of it, but we should have, right? We should have. Okay. So at this time, we started talking a lot about what we were doing with our data and I was on a committee with our provost and academic affairs advisory committee that gave me a chance to tell the provost about this kind of stuff. And if you complain enough to leadership, they make you run a task force, right? So that's what happened. They made me run a task force. So we launched this in May of 2012 and it was a committee of a dozen faculty members that I got to pick. And our charge was to do these three things. This is basically what it said in the one page document. Optimize the University of Michigan information environment for learning analytics. The provost said, we should be doing this kind of work. Let's set up a good system for doing it, not like a wide open crazy system, but a good system, like let's take care of it properly. He wanted us to explore what was possible and gave us some money to fund a series of learning analytics projects. Often it doesn't take a lot of money for someone to do an exploration like this. They need to work with a student for, you know, 40 hours to do some kind of analysis, not too hard. And he also asked us to review the metrics we use to assess teaching and learning. So during the three year period that this task force was running, we did the kinds of things that were requested. First of all, we spent a lot of time talking about learning analytics. We ran something called SLAM, the Symposium on Learning Analytics at Michigan. And it was just like this, right? We just invited people into rooms and said, let's talk about all the things we're doing. Let's bring a few outside speakers in. Academics, if you wanna get something going. Run a seminar series, right? That's what you do. Get coffee and donuts, right? We did give out grants for exploring learning analytics projects. We ran something called the Learning Analytics Fellows Program. It kind of grew out of the conversations. There were a lot of people who said, I'm really interested in this, but I don't know how to do this, or I don't know how to get started. So we put together something that was a little like a class for two hours every Friday. A group of people would meet for the whole semester. A learning community, you could call it. We asked faculty members to come in and to bring a graduate student with them. So they'd have a little team to work on their project. And I think we had 15 or 20 of those teams each of the two semesters that we ran this Learning Analytics Fellows Program. That was a big help because we went from having just a few people doing it to having 20 or 30 people doing it. And all of a sudden we did all these new projects and we learned all these new things. And I'll show you one in just a minute. Coming out of the Learning Analytics Fellows Program, we launched a set of new things. One of them was a new unit on campus initially called the Digital Innovation Greenhouse that builds tools that act on data. And some of the tools that are being worked on here that you'll see in the showcase are very similar kinds of tools. Things that would let you communicate with students like on-task does, based on information about them, reporting tools that would help you see what's going on so on. We took the Learning Analytics Fellows Program and I made a mooka out of it because other institutions wanted to use it too called Practical Learning Analytics. One of the most important things we did was to build something called the Learning Analytics Data Architecture. And what Learning Analytics Data Architecture was about was about wrangling our data and making it available to researchers. So let me just put up the slide about that. What we did was to create a research-focused data set that contains information about students at the University of Michigan and it goes back to about 1996. The purpose is to allow people to answer typical Learning Analytics questions without needing to go into the swamp of databases that are actually used to run the university. All that data is out there. It has to be in a complicated organic system in order to run the university, but for researchers to get into is very hard. So we had a team of the experts who understood that swamp pull the data together into a very simple format and deposit it so that researchers could get at it. Now there was a box around this data. This is the data that's available for Learning Analytics Research. They didn't create it as a product, they created it as a process, right? So they built a loading system that allows them every semester to push a button and a new copy of this data is created, right? What they really do is they update that pull every semester so it gets richer and better as time goes on. But they can do it very, you know, as much as they have time to do. They don't have to make big changes. How do you get access to it? There's a very straightforward system that was created as part of this for getting access. A researcher who has a project they wanna do in this data writes a description of the project. They sign an MOU with the university saying this is what I'm gonna do with the data and I'll get rid of it at the end. They have to get IRB approval before they do their project and then they get the data. Almost every researcher gets anonymized data because that's all they need. Now I will say anonymized data is not fully privacy protecting at all. It can all be re-identified if somebody wants to do that but for most researchers they don't wanna do that and they feel much more comfortable working with anonymized data. This was a huge help for us because it enabled us to have many more people work on this data much more freely and so this kind of architecture was an important product of that. I will say it was inspired by the work that we did in large science projects. So when I worked on something like the Sloan Digital Sky Survey, a project that has very important open data goals. We really wanted to give, we were actually required to give away the data to the public. We also had a very complicated database to run the project and no one could use it except the experts. So if we just gave people access to that no one would ever done any science with it. So we created public data releases which are very like this, public data releases of our data. So it's the same kind of model, it's not a public release of course because it's human subject data but it's very similar. Okay, let me tell you about two results from the dozens that came out of the Learning Analytics Fellows Program. First is a kind of small example, a nice local one and the second is one that has bigger consequences for us. So the first one was an analysis done by a colleague named Ginger Schultz with Amy Gottfried and Grace Winchell. We have a chemistry placement system on our campus. You probably do too, right? People take some kind of test at the beginning of entering college which helps them decide where to start in chemistry. At Michigan it's fairly strong because a lot of our students go straight to organic chemistry and skip over Gen Chem, right? About two thirds of students do that but the remainder of them will be sent into a general chemistry class for a semester and then they go on to organic chemistry, okay? So the notion here is clear. There are some students who really will benefit a lot from doing Gen Chem first before they do Orgo. That's why we have them do it because it's a big deal, right? They spend a whole semester in general chemistry, a tough class with low grades, right? Right when they're starting college. I mean it's a big thing that we give them this advice and the students we don't give that advice to skip over it. The idea is it's gonna lead to a big difference in outcomes but no one had ever actually looked to see how big the difference in outcomes is, okay? Now it turns out that this question is, you can address this question with a pretty nice form of analysis called regression discontinuity analysis, okay? The idea of regression discontinuity is let's say that performance in this course depends on that placement exam score. People who had low exam scores if they went on they'd get a low score. People who had high exam scores if they went on they'd get a high score, right? So there's some kind of correlation. There's a point in that where we tell them take Gen Chem and on the other side we tell them take Orgo. Right around that point, the students are the same, right? There's some noise in that measure. So they're the same. So if you look at outcomes in this class as a function of this for people just below the line, they got a treatment. They were sent to Gen Chem, right? And the ones just above the line were not. So it's almost like you've done a random controlled trial right around that regression discontinuity, okay? So let's look at that and figure out how much gain do they get by going and spending a semester taking organic chemistry, right? So this is that figure. What you see here is the chemistry placement rating and what you see here is a grade in the later class. And you can see there's a correlation between the chemistry placement test and the grade in the later class, right? And that there's a discontinuity at this point. That's the point where we send them to Gen Chem. So it is true that there is an improvement in the later class score for people who take Gen Chem compared to the ones who don't take Gen Chem. How large is that improvement? Well, if you put everything together, what you find is it's about 0.15 letter grades. It's real, statistically significant, well measured, but not enormous. I mean, the sense I think that people had was you must take Gen Chem because otherwise it's gonna be a disaster for you in this later class, but not so much, right? So now I think advisors are much more nuanced in the way they might talk to a student about it. For example, if a student is really excited about getting ahead in chemistry and wants to move on, maybe they should just go ahead and take Orgo. It'll be a little harder for them, but it's not like the bottom's gonna fall out because they do that and they'll be doing something new and exciting that they didn't see in high school and lots of good things could come of it. So I think it's very interesting. The other thing that Ginger and her colleagues learned in doing this analysis is that the nature of this advice, this advice is not going equally to all students on campus. So it's also having kind of equity implications, right? So this is the distribution of people who placed into Orgo or didn't place into Orgo. And as a function of ethnic categories that are reported by our students, two or more Asian, African American, Hispanic, not indicated and white. So the fraction of African American students, for instance, that make it into organic chemistry right away is very small, vastly smaller than the fraction of Asian students who do. This wasn't intended, but it's a fact that it's going on and makes one want to think about these kinds of systems. So something we didn't know about before. So that's a first example of something that came out of the fellows program. They wrote this up as a very nice paper. You can go read all the details if you want to. It's a nice example. If we have things like that going on, someone should look, right? Big consequential advice is being given. Students are spending tons of time, effort, energy. Maybe it's right, but maybe it's not right and someone should check. Okay. The second has to do with the gender performance differences we saw in physics. So remember, we saw it in physics. We talked about it. My chemistry colleagues said, oh, don't go to physics, right? We should have looked across the disciplines and asked, is this a problem with physics or not? The fellows program is when we did that. You know, and the reason we did it is because we brought all these people in the room and we needed to have like an exercise for them to do. So we said, we got this gender analysis code. Why don't you run it on the class you're interested in? Right, and they did. And all of a sudden we plotted it all together. We were like, oh, I guess we should have looked at that before. Okay. So what did we find? We found that these gender performance differences exist in all of our large introductory science and math lecture courses, including economics. So this is a microeconomics course, very similar kind of structure to what we saw up to this. Okay. We calculated this then across all the courses on our campus, especially all the large courses on our campus. And we have a paper out that you can go read all the details in. This is kind of the pattern that we found. There are two things plotted here. On the x-axis is the overall grade anomaly of the course. So a course that gives students higher mean grades than they usually get is on the right-hand side. A course that gives students lower mean grades than they usually get is on the left-hand side. Then on the y-axis we have this gendered performance difference, the average grade anomaly for females compared to the average grade anomaly for males. Courses which favor females are above this dashed line here, that's the zero line. Courses which favor male students are below that dashed line. And on this figure I've plotted all the introductory science and math lecture and lab courses on our campus. And what you see is all the lecture courses sit in a big group down here, very low average grades, and large gender performance differences. And all the lab courses live up here. Grades that are more or less normed against other grades on campus, they're not a lot lower than normal, they're a little bit higher. And very little gender performance difference in those classes. We didn't know this till we built a community and looked at it all. That's what enabled us to discover this pattern of gender performance difference. You cannot look at this and not read this list and say, why are they down here? What is similar about these? Well, one of them is Bio 171. That's Ecology and Evolutionary Biology. It's really not a very mathematical class. So it's not just like, ah, what is it that makes them all the same? So I've come to believe that one of the traits these courses all share is that they are evaluated mostly using high stakes timed examinations. And that is actually quite unusual on college campuses for the bulk of evaluation to be that. We're used to it in the STEM intro course. That's what we do, right? But that's not because it's normal, it's just what we do. And I think that plays a role in this. So we discovered this pattern on the Michigan campus. Is it really an equity problem? I mean, of course, as soon as we show people this and their course gets into the circle, they're like, oh no, I must be another explanation. Well, maybe it's that the female students take other classes that are much easier than the male students. Or maybe it's that they're not as good at math or whatever. There must be an explanation. So what we did is we took all the information we had about students, the other courses that they had taken, the college they're enrolled in, the credits, high school GPA, all the standardized test scores. We ran a variety of analyses to try and explore which predictors were the most powerful ones and use those to see if we could make the gender difference go away, right? Is there an explanation? Some of those explanations might still leave equity problems on the table. I'll just say that. But we did try to see, is there anything that would make this go away? We also, instead of just doing linear modeling, we did a series of matching analyses. There's a bunch of nice techniques that are model-independent that basically say, I have two students, male and female, on all of the measured parameters, they're very similar, except for gender. And you match those two and look at different outcomes. And you do that for everybody, right? So we did all these different things. And we couldn't explain it away. So I have come to believe that these gendered performance differences represent strong evidence for what lawyers call disparate impact. So in the United States, there are protected classes of people that you can't discriminate based on, right? And if you see that these two classes of people go into a certain system like renting apartments and their outcomes are completely different, you can sue for disparate impact. You don't have to find someone who admits that they're prejudiced and are discriminating. You can just look at the outcome and say, that's a cause of action. So I think that's what's happening here, basically. Now I have not encouraged our students to sue us. I don't think that's the right solution, but I think it's the same kind of thing and that we should feel as troubled about it as a owner of an apartment building might at least. Okay, so what's going on? Is it a problem just at Michigan, right? That was the next question we had. Obviously, I hope you're thinking, is it a problem at UBC? And you should go and ask, right? Everyone gets that question, but we really wanted to know. And so we set out on a process to try and find out. And I'll tell you that in 2014, 2015, when we started this, this was really hard. Very few institutions were ready to access their own data. Very few institutions, even fewer, were ready to talk about what they might find. So they were both technical problems and kind of institutional fear problems associated with it. We got together with other universities in the Big Ten Academic Alliance, you know, the Big Ten Athletic Conference. Well, it also has an academic alliance associated with it. All big public universities a lot like us, good comparison group. And we got a little funding from the Sloan Foundation to bring them together and examine this. It was called the Sloan Learning and Research Analytics Project. And we brought together people from 10 Big Ten campuses and we regularized our data on each campus. We couldn't share it, it was too hard. But we said, put your data in this format and we'll put our data in this format and you can put your data in that format. Then we'll write code that runs on that format and share the code around so we get, you know, it's a harder way to do that kind of analysis, but it works. We had 52 people involved from 10 institutions in the end. Five of those institutions left the process and said we can't talk about this. But five stayed. So the main example analysis was, we look at these patterns of grade, penalty and gender performance difference on five institutions, 172 courses, almost 700,000 students in the analysis. I'm gonna show you some kind of early raw results because it's kind of neat to compare them in a visceral way. What you're looking at here and for this little set of slides will be like this, the same kind of plot, little different colors, same kind of plot, male and female students in different bins, right, a one-to-one line. This is the introductory physics for engineers course at Michigan and at three other institutions, I can't tell you which ones they are. I'm allowed to tell you it's Michigan because we do that now. I've also put the number on for what the gender performance difference is averaged over those classes. So 0.23 letter grades, 0.24, 0.22 and 0.15 letter grades. Modest but significant differences in physics. Now I'm gonna show you some more classes. Same structure, okay? So this is econ one, microeconomics on all four of these campuses. 0.32, 0.31, 0.30, 0.36. Interesting things here. Notice that this one, the average grades are about normed against other grades on this campus, but they're still a gender difference. No grade penalty, just a gender difference, right? Oh, no overall grade penalty, right? Some of the others, grades are much lower than the rest of the campus, right? So you see different things when you look at different places. I think those grading norms are, you know, they're local norms. This is organic chemistry. Smaller but still significant differences about 0.2 letter grades on most of the campuses. All right. This is a chemistry lab. 0.01, minus 0.01, minus 0.08, 0.09. Interesting. So we put all that together into a paper. The references here. Becky Matts at Michigan State University led this paper. It came out at the end of 2017. Patterns of gender performance difference in large introductory courses at five research universities. We broke it out by discipline to sort of see whether there were disciplinary differences. Each of these plots shows you individual courses with symbol sizes that are course size and colors that have to do with whether it is a lecture course, a lab course, or a mixed course. Some of the places, you know, lump them together. So we tried to show all that. And I'll just call attention to the chemistry one, in part because general chemistry is very often the first science course that people take when they come to a college campus. And what we see there is that, first of all, most of these courses have really big grade penalties. Like if you read down to this axis, it's like three quarters of a letter grade. Really, people get hit pretty hard by these classes. And that's, remember, in their first term, their grade in chemistry is way different from their other grades. And they have large gendered performance differences, a third of a letter grade. So a double whammy for female students, okay. So at this point, we started to ask ourselves this question, how could the same environment affect different students differently? And I actually used to think this was a valid question. Now I think it's like the most obvious, the answer is the most obvious thing you can imagine, right? People are not the same. So the same environment does not affect them equally, but really, we were like bemused by this. So now it seems obvious to me that there are many ways this can happen. In fact, it's inevitable that it will happen. Every environment will affect people differently. If you're not thinking about that, you're making a big design mistake. In fact, I think very often in universities, we have made a design decision. We've designed for a certain kind of person who is often very like the faculty member. Okay, I wanna take some time here now to talk about getting beyond gender. As I said, we started doing this analysis with just gender, this is what everyone was doing in physics and it's the only thing we thought of at the time. But as time has gone on, we've thought and recognized that this is not about gender, it's about identities and identities are incredibly more complex than binaries like male and female and gender. Even if you just wanna talk about gender, it's more complicated. But people are much more complex than that. There's a very strong emerging literature about the ways in which data systems datify people, collapse them to a formation which is not real, but begins to seem real because there it is in the data set, right? A colleague of ours at Michigan, John Cheney Lippold, wrote a book called We Are Data that's specifically about this and he writes about the way data systems collapse complicated social identities to simple measurable types. He advocates for ideas of keeping clear when you've measured it and what it really is, right? The measured thing could be useful and everything else but it isn't the thing. So for example, if you have a data system that records something about gender in it, you might wanna label that column with gender at least in quotes, like it's not really gender. It's the gender data thing, right? Gender is a real complicated identity that's out here in the world. So he talks about that. It's also true that in addition to collapsing people in this kind of way, most of our analyses just do unit dimensional things compared to groups, right? Forget about the rest of the identity. So your male or female doesn't matter what the rest of these things are about you but it does matter what the rest of those things are. So how could we explore the complicated identities that people have in a better way than we were doing? This is something we've been working on for the last few years and thinking about and I would love to think about it with all of you, try to figure out better ways to do this. So let me show you a few things about it. First of all, we're not the first people to think about this by a long shot. There's 30 years of really great feminist scholarship writing about intersectionality and social identities. They're just not separable. You can't pull these things apart and say you're this and this and then in another context you're that and that, right? It doesn't work that way. So people like Angela Davis and Kimberly Kredshaw invented this concept of intersectionality in the 1980s and people have been working on it ever since. I'm gonna talk about some of the ideas that come from a paper from Leslie McCall in 2005. She called it the complexity of intersectionality and it's kind of a methodology paper. It kind of says, all right, this is the problem. How might you analyze data in ways which are sensitive to this? All right. So McCall talks about three big different ways to go at this. One she calls intercategorical complexity and the idea here is, all right, we've got these labels in our data system. We know they're not perfect but most of those labels are in there because our systems use them, right? They're socially powerful. They're bureaucratically powerful. So we should at least examine the way people's identities intersect across all of those categories even though we know they're not perfect. They are, we can document then relationships of inequality. A second approach is to recognize that everybody who's in the same box is not the same. Intracategorical complexity. Take the people who are in one box and pull them apart. See how different they are because they actually are gonna be a lot different. And finally, the ultimate goal is to get rid of categories. You know, a category says you two are the same. You're in the category, you're just the same, right? But it's not true, right? People actually live in a complete continuum of all of these kinds of parameters and we need to think about how we might get here. We're not there, I don't know how to get there but this is the kind of goal. So first let me show you an analysis related to what we just talked about that is the probes intercategorical complexity. What's on this figure now is a set of points for male and female in introductory science lecture courses. A whole bunch of courses aggregated together just to do a first analysis. And then what you see up here is for the same group of male and female students lab courses that they're in. So there's a small gender difference in lab courses, a bigger gender difference in lecture courses. And then we've included the points for four racial and ethnic categories that are recorded in our data systems. And what it allows you to see is that there are intersections between these identities so that if we went forward and we talked to all students in our classes as if being male means this and being female means that we would be wrong. It isn't that simple. African-American male students in this class pay grade penalties that are larger than white female students do. And we would have erased that if we didn't look. So beginning to do this kind of work is a starting point for seeing what's going on at least. Intracategorical complexity is another thing that you might want to do in order to understand what's going on. So what's shown in this figure is the actual individual grade anomaly. Each student's grade compared to their GPA. Each student has one of those numbers, right? And we show you the distribution of those grade anomalies for female and male students in three different semesters of introductory physics. The average differences are here. So you can see there's an average difference of close to 0.3 letter grades in each of these semesters, just like we saw overall. But this gives you a very different impression of that average difference than that average number did, right? You can see there are female students who do quite a bit better than expected, and male students that do a lot worse than expected. And maybe what's going on here is there's something very loosely parallel to gender. Loosely parallel, that's a terrible term. Loosely aligned with gender. You can't be loosely parallel. And so you see it when you look at gender, but maybe it's really not gender. Maybe it's other things that are just, you know, sort of aligned with that. So it forces us to think that we want to focus on the underperformance of individual students. So for example, if we really want to understand what's going on with what pulls people to the lower end, we shouldn't talk to all the women. We should talk to all the people that are on the lower end, right? It sends you in a different direction. Okay. And then, you know, this anti-categorical complexity is a wonderful kind of goal. Imagine if we didn't have to reduce individuals to categories like this. And we had some mechanisms of doing this kind of study and analysis that really personalized the study. So we could really recognize that every one of you here in this room is actually different. There is no one category that includes even two of you. So we wrote a little thing for the learning analytics conference last year about this, categorization, intersectionality and learning analytics. You can see those results there. Other people are writing about these things too. It's obviously very important in the data world, maybe especially in the educational data world to be thinking about it. Okay, so what lessons do we draw from all this? I think there are a couple of big lessons and I wanna tell you about how we have reacted to those lessons. First, we must attend to difference. And it seems like an incredibly obvious lesson, but it was not one that I used to act on. It's essential that we do this and that we use data to monitor what's happening with equity and we try to explore inclusion. This is especially important in large foundational courses, both because they're good places to study it, but also because students are at their most diverse when they arrive in these courses, right? They're coming from wildly different backgrounds and as they come through our system, we actually regularize them to some extent. So by the time they get to your junior level quantum mechanics, you know, yeah, they're a little more the same than they were in their first semester. So it's especially important at that point. Not just in our own classes, it's not just my problem for my class, but across the landscape. I mean, this is happening everywhere. So I wanna work on it everywhere. I want everybody to go to work on this. And it's not just about gender, it's about all individuals. Shouldn't have been a surprise. In fact, the idea that you need to use data to do this, think about other places where we're pressing for equity in our society. Would we be talking about pay gaps if there was no data? I don't think we would. They'd be there. Each individual would feel that they'd look left and look right and think, wow, I'm really not getting paid very much. But they wouldn't know, right? They wouldn't know that this was happening. Things like proposal biases. Lots of great work has been done on biases in review processes, recently only possible because of data. We would never know these things without it. Same is true of sexual harassment. We need to have data in order to address these questions. So attending to difference is the first thing. The second thing is that we need to think about how to engage in really systematic reform. This is not a problem with physics courses at Michigan. It's not a problem with STEM courses at Michigan. It's a problem with STEM courses everywhere with the way we approach introductory science teaching. I think there's strong evidence that suggests our students can learn these introductory science topics, but we've created environments in which some people learn them and other people don't, and that those differences are not solely on the student. They're based on us. Not a problem for my class or yours or for physics or chemistry, it's for all of us. So we really need to work together to counter it. And so that's what we're trying to do now. So let me tell you a little bit about that. To create generational change in this, I think we need a new kind of model. I think back to the kind of model we needed when we wanted to map the universe. I couldn't do it by myself. We needed to get the whole bunch of people together and figure out how to do it. So I think we need to build those kinds of collaborations. We need to build the teams we need to do it. And I think that means we should aim for new norms for these classes, new ways of thinking about how foundational STEM courses should be taught. If we could change them in a coordinated way across 10 or 20 universities, the rest of the world would see that. It would be different from where we are now when one university does something kind of crazy in their intro courses and everyone else says, nah, I don't wanna do that, right? Somehow it has to get to that point. I think we also need a new form of motivation. I talked with a few of you here about this. For a long time, the science education community has been pushing improved learning gains as the motivator for change, right? And it is a good motivator and people should change darn it because of that. But in 20 years and they haven't. So I think we've done the experiment, it's not enough. We need something else. And I think equity can be that. When we started really openly speaking about equity and working on it on our campus, people started to show up to work. I would give a talk and three students would show up my office during the next week saying, I wanna work on that. Because it's a real problem, they know it's a problem, it excites their energy. So I think equity and inclusion should be at the center of this kind of movement. They should provide the standard for this. I would like to say that a course can't be excellent unless it not only is equitable and inclusive, but that it can show that it is, that it's attending to that. Not just discerning that it's true. So I would like to see that become a standard. Now, when I've talked about this in the past, sometimes people say, but no, it has to be rigorous too, right? I mean, that's the first thing. And I agree with that. But I think hidden in that statement is the assumption that the only way to make it equitable and inclusive is to dumb it down. And I disagree with that. So I wanna start by just saying, let's make it equitable and inclusive. I think we can make a great course like that. We can disagree about it if you want. We have launched this collaboration with support from the Sloan Foundation, 10 big public universities in the United States, five of them are Big Ten University, some of our former partners in this. So Michigan, Michigan State, Indiana, Purdue, and Minnesota, three from the University of California system, Irvine, Davis and Santa Barbara, also Arizona State University and the University of Pittsburgh. Together we enroll well over 350,000 students. It's a big gang of people. We all teach these giant introductory STEM courses together, we face the same kinds of challenges. So all of these places have agreed to give access to local data to their local researchers and to allow them to share the results. Again, not sharing data, maybe someday we will, but we didn't want the lawyers to be involved at the start. So what are we gonna do? For the next three years, we will engage in a set of parallel analyses and comparison of the results focused on equity and inclusion in STEM. We have many people on those 10 campuses who have already conducted experiments on their own campus with interventions, with measurements, with changes to course structures. They would love to expand those to other campuses. Now we have a collaboration of 10 institutions ready to go where everybody's agreed to talk about their data already. Imagine writing the NSF proposal to say, I'm gonna do this on seven campuses. Oh, and they've all already agreed to do this. It's a lot nicer proposal to write than the one where you say, I think I can talk them into it, right? Different thing. We will engage in continuous collaboration just like giant science collaborations I've been involved in before. And that will involve constant exchange of speakers. We'll have them coming and going between institutions, the use of graduate students and postdocs. And we'll meet once a year for a big annual meeting, including faculty, staff, students, discipline-based education researchers. How do you run a thing like this? Well, like I said, I've been in some of these before. So it has an organizational structure that only the people who have to organize it need to know about. The main action for most people takes place in this lower layer. Each institution has a big team of people, very diverse, people interested in different things. So there might be some people who are really interested in measuring equity and inclusion. A few here and five people there and two people there and three people there. When you bring them all together, there's suddenly 30 or 40 people. Imagine how many things they can do now that the two or three on each campus couldn't do. They can team up and come up with new ways of doing what they wanna do. The same is true for coordinated experiments, reforms and so on. So that's the idea of how this works. A few simple kinds of analyses. We will design some consensus measures of equity and inclusion and do them across all of the campuses. There's already a bunch of work that's happened. We're also thinking about new ways to instrument classes and classrooms. What we know very little about is inclusion. Doesn't show up in our data sets very well. How do students experience the classroom environment? And some of our colleagues have developed instruments for measuring that and they've done it in some context, but we wanna try to spread those across broader contexts. Expanding experiments piloted by one partner, I sort of mentioned this. The only way this really works is if there are people in the collaboration who wanna do all these things and if they will go out and find the funding to do it. So we expect this to lead to a bunch of new funded projects by creating an enabling environment that is this collaboration. We held our first collaboration meeting last week. It's not even a week ago. All right, we were there at Michigan. A bunch of people from the kind of administration and staff support side, people from discipline-based education researchers, science practitioners who are teaching these courses, people who run support units for them, all engaged in a conversation about what this collaboration is gonna do. The goal is that three years from now we would have an array of unprecedented multi-institutional studies of equity and inclusion. So at least we'll have that. I think that for sure. We'll have a bunch of measurements we haven't had before. I hope that we will also have enhanced the research and reform efforts in every one of these campuses and that we will have promoted a new collective standard for thinking about equity as the right kind of thing to be examining when you're talking about excellence in STEM foundational courses. Okay. And I'm gonna stop there. I'll just put up this slide to note that together with Simon Buckingham-Shum we wrote an article for Educause Review recently that was about how you structure an institution to do this kind of work. And I know that some of you here might be interested in reading that. I would be happy to take questions about anything that we've talked about. Excellent talk. Thank you so much for sharing your insight with us. I'm a big proponent of data collection and data analysis. And so with all the conclusions that you draw from this data, like this gender gap that exists, so I was just curious to know what is your vision on how to act on it and what would you do to close this gap. So we have acted on it in a wide variety of ways and we're still working on it. But the first thing we did was to do a bunch more analysis, right? So to try and find more, maybe the data contains more hints about what's going on. So one example of that is if you look inside the class at the elements that make up the grade and ask where does this emerge? Is it equally across all the elements of the class or not? So we've done this now for many of those classes and it is essentially only in the exams that these performance gaps emerge. Which is okay, that's another insight that gives you points your finger in a certain direction. So we started to ask ourselves questions about what might be causing it. You know there's a social psychological mechanism called stereotype threat, which is very real I think. It is operative in those kind of performative environments, so we started to think maybe we could act on stereotype threat using interventions like values affirmation interventions that might reduce the performance gap. So we tried that without much success. There have been some successes in some places, I think those are real, didn't work very well in our context, so that was the next thing we did. Then we're thinking about test anxiety. What could we do about test anxiety? We started studying whether people had done studies in the past. Turns out at Michigan back in the 1960s before IRBs existed, one of our colleagues in psychology actually gave each student entering the exam uphill and half of them were anti-anxiety drugs and the other half were placebo's. It was a different time, we didn't do that. It didn't work by the way in his case. I mean he reported it, you know. We though thought maybe more exam time would help. So we arranged for a class of 700 that they could get 50% more time on their exam, which was not easy, they're in seven different rooms when they're taking an exam, but we did it. And we then recorded when every student left the exam, so that we could see all this kind of stuff, didn't solve the problem. But it's in action, we tried to change things, so we are doing things, we're not done. What we're doing now is trying to restructure the introductory physics course so that it doesn't rely very much on exams. I think that probably will be part of the solution. I think it will solve a lot of other problems in the class. Student learning can be very thin if the goal is multiple choice tests. They know the real goal is to fill in the right bubble. It isn't to really understand how to solve this problem, it's really to fill in the right bubble. So getting past that with other forms of evaluation. Things we're working on. Yeah? But primarily you see the grade penalty and the gender gaps in the lecture-based ones, but surely there are some lecture-based courses where that's smaller. Is there anything identifiable for why some lecture-based courses don't see as big a penalty or a gap? Yeah, I mean, an interesting example is statistics. So big statistics courses, their students are quite diverse. They come from lots of different places when they take these courses. And when we first saw that at Michigan, the statistics course didn't show this, I jumped to a conclusion, which you always do when you see something in data and you try to resist it. We have a wonderful instructor who leads this giant class and it's a great class. And I thought, Brenda, solve this problem, she's great. But I was wrong. Other institutions show the same thing. Now, another difference with statistics, again here I am jumping to conclusions, but it is often taken by students much later in their career for what it's worth. You know, a lot of people take it as juniors, seniors even, it's different. So we don't know the answer, but there are courses where that's true, which still rely on exams relatively heavily. Sorry, I just sort of meant even like the same chemistry course, but like in one group or one area where the gap for the difference was smaller. So we pretty much haven't seen that. Which really points to me towards structural elements, right? You know, we looked for instructor effects right at the beginning. We thought maybe certain instructors, you know, either the way they teach or their identity or anything might affect this. But it really is, boom, boom, boom. And it's a place where having really good statistics, you know, is a benefit, because you can see the effect on a smaller scale. But instructor effects or possible instructor effects? I mean, just mentioned it. So I'm just wondering if you can speak a little bit more on like what kind of categories of instructor did you look at and what was shown and what was done. Yeah, so I would say we haven't done every analysis you could think of. But we have done some first order looks at it. In my discipline in physics, we can't do a whole lot about instructor effects, because almost all the instructors fall into the same general social categories. It's even worse than it might be, because a lot of the women on our faculty, among other things, are younger faculty members. We have often protected younger faculty members, protected them from teaching these courses, because they take a toll on the instructor. At any rate, we have looked for instructor effects on gender in some of these sequences and not seen anything obvious. It's not that surprising to me if simple elements of instructor gender were really operative. That would be almost very troubling to me. So I don't think that that's the case, but I encourage you to look. Maybe there are places and circumstances where it's really important. The only experiments that show really clear instructor gender effects that I know of are we're done at the US Air Force Academy, where all students are randomly assigned to courses, and instructors are randomly assigned to courses, and all students are required to take the same sequence of courses. So it's kind of a pretty good experimental environment. And they found small, but significant instructor effects so that female students were more likely to continue on towards STEM majors if they, in their first two years, took courses from female instructors. I think that was a great experiment, but it's a pretty controlled environment. We don't live in that environment. Hard to see in other places. Can you talk about the examiners who were thinking and describing behind the scripting, and their interactions examin' time STEM. So I wonder whether an exam is a social sense. It shows similar effects. The way do we ever actually reverse bias in social sense? So economics falls in that same category. You see the same things. Of course, it stands out. It's right at the extreme of the social sciences in a bunch of ways. Even the National Science Foundation now sometimes talks about STEAM with two E's, not with an E and an A. There's another STEAM. So I don't know. I would like to look at it more. I think that it is the case that the nature of exams, when they're used in a big way in those classes, which not that often, they're a pretty different in structure. So I guess it's an open question. I wish I'd be more. Calculus courses, the University of Michigan has kind of a special program to kind of smaller classes in an extent like training and teaching and this sort of thing, which would be probably unusual compared to those other programs. Do you have any comments? So I haven't made a comparison really across math. It is true that I glossed over this a little bit. But the way we did the analysis for the comparative analysis was to throw every predictor that we had into the comparison. We wanted to, when we made these measurements of performance difference, we wanted to make sure we were including other things and not just GPAO. So what's shown in this figure actually is a model that uses all the information we have to try and predict performance. And what you see is that in math, these gender differences are not as big as they are in chemistry. And that's because, I mean, the main reason is because standardized math test scores like ACTs are more predictive of introductory math classes than they are of anything else. So they kind of account for a difference more effectively. I'm not sure that's fully solving the problem because those standardized test scores also show gender performance gaps. So you see it here, and you see it here, and you say this one explains this one. And that might be fair. I'm not sure. But I worry a little bit about that. It is true that our calculus is not taught in a giant section. It's taught in many sections of 18, hundreds of sections of 18, which means it's a nice solution in a way, but it means you have to hire hundreds of people and train hundreds of people to teach. And that's another kind of challenge of teaching in scale. So I think as we get into this comparison, it would be wonderful to have individuals in each discipline look deeply at each of these comparisons, understand the structure of the classes in all the places, and think about whether structural elements make a difference. That's what we need to do. Yes? My question is, this is a great analysis of all part of the inside of how public institutions can solve such problems. I feel like that's a long shot, right? I think a lot of the research is able to get involved. But thinking from a micro level, individual data, which is a student across spectrum, is there any recommended measurement or recommended actions for those students to take actions on their own? Yes. You are falling into this type of scenarios, not categories. This type of scenarios, you should take this type of measurements to make sure itself is involved in such a process in the school. Sure. Yeah, so I think that's a great question. I didn't talk much about this kind of work. But another thing that data opens up is the possibility of understanding each individual in the context of everyone else who is or has ever taken this class. So we have built tools that enable us to watch in a way every individual student what they're doing in certain ways, what they're doing inside a class, and how that might relate to the outcomes people have had in the past when they do those things, so that we can provide advice to students that might help them move toward that better than expected outcome. And we're doing research on how best to deliver that advice. A lot of people think hard about these questions of people in the commercial world are trying to nudge you to do everything. This is related. We want to help people behave in ways that will benefit them. Can't make them, but we want to help them. So for example, if you know that a student always starts their homework or their exam studying the day before the thing is due, and you can show with good data that people who start earlier do better, you could share that data with them and maybe say, you know, here's what you're doing. Other people are doing this, and they're having, you know, that's one argument you might make to them. So there are lots of ways that this data can be used to the individual level. The on-task tool that you all are using here is one mechanism for delivering some of those messages. We've built one at Michigan called E-Coach. We work with a bunch of behavioral scientists who help us try to get past being professors. We don't actually know how to change students' behavior very much. They're much better at it than we are. OK. So we can't thank you enough for coming today. No, it's a pleasure. But I do get a little bag. You get a little bag? All right. That's a whole token of our appreciation. OK. Can you get a round of applause? Thank you. So I hope that serves as some inspiration. I think that's highly relevant to some of the initiatives that are going on here at UBC right now. So Tim's going to be around for a little bit, I think. Yeah. We're having a little reception around the corner here. We have a little showcase of some of the projects that we're working on here at UBC. And that's both in the institutional learning analytics project. But that's also with folks who've been doing learning analytics here for several years. And we're not nearly quite at this level. We do have a ways to go. But there's still some interesting stuff that you can come and check out. And I hope everyone will persist and stick around here. We'll meet back here in about half an hour. And we have a couple of UBC faculty members who are going to share some of their experiences working with David here. We have Kyle Fragman from Faculty of Arts. And we have St. Chievon Bergman from Dentistry who's going to talk about some of her work on the progress surveys and competencies of those professional programs. So thanks again, Tim. Thank you.