 My name is Kayleen Sampson. I'm the Program Director of Learner's Success at the University of Canterbury, Christchurch, New Zealand. Yeah, it's a long way to come but we've enjoyed the meeting so far. Look, what I want to talk about today with you is it's about analytics, but it's actually about engagement and it's about the human side of what we do when we engage in analytics. So the tertiary sector I think we would all recognise as experiencing an increase in student disengagement which has really been brought about rapidly by COVID and our quick shift to online. In fact, the Times Higher Education recently did a survey only early this year and they discovered that three-quarters of their academics report that post-COVID lockdown students are not returning to class so there's a higher level of absenteeism occurring. Yet, we all understand that the whole issue of engagement is a very critical one to watch within our sector. We know it leads to better learning outcomes for students engagement with peers, materials and academics, but also we know that engagement is a critical factor for those of our students who arrive with a lower than average ability at entry. But despite that and while we all know engagement matters, it's a problem we're all grappling with and our solution at the University of Canterbury is something I'd like to share with you today. So, give me a moment. Right, so our analytics for course engagement or ACE is really made up of three parts. We've got a student-facing dashboard which and all of these parts reside inside Moodle. We've got a student-facing dashboard. We've got a bunch of teaching-facing resources and then we've got our re-engagement program which is that human piece that I was talking about with you before. ACE itself is derived by Moodle engagement data that uses both cognitive depth and social breadth measures on each activity that sits inside of Moodle. But we also use other data points, so we're combining our Moodle data points with things like library information, Zoom, Echo 360, as well as a bunch of what we understand to be predictors of risk. This is then pushed from is put into Moodle and then pushed to our machine learning to produce risk scores while ACE itself also produces engagement data and we use this for our re-engagement program. I'm going to take you through each of those pieces very quickly. I only have 15 minutes and I'm going to talk super fast. So, there you go. Great. So, this is the key piece of technology that our student sees and this also lives inside Moodle. What we have here is a student's dashboard, the vertical axis is their relative engagement, the horizontal axis is engagement over time and what we're giving our students is a sense of self-direction by being able to understand how their engagement relates to that of their peers. The green band is a half a standard deviation on the average engagement for every peer related to this student in that particular course. You can see we've got EMF 109 up the top, that's an engineering course, that student could click on the next tab and see their next course or they could click on the final tab and have an overall view. On the end there we have an ability for a student to select against dates and they can also switch off the green band because some of our students did tell us they didn't like to be compared. In fact, when we had our student experience, major student experience survey, we asked, oh apologies, we asked students what they felt about the dashboards and we got comments like this. It's been phenomenal and very easy to adjust time on certain courses depending on how it fluctuates. So this is the piece that the student faced but where we've put our real major investment in the last 12 months has been into the space where our teachers are working. So here is an example of a teacher dashboard. The teacher can see in this dashboard the same graph with the same manipulatability as what the student has around date picking etc but what they can see here at a course level is the overall engagement in their course. Again, this engagement is not just what's happening in Moodle but it's a combination of the student's overall engagement. Down the bottom here you can see we have users and activities on the tab. So the names have been removed here for sort of identity or privacy purposes but what you can see here is we have all students in classes and in this particular case I've used that far filter function to select students that are distant students with low engagement and from here as a teacher I can email directly to that cohort of students and message them. Another part of what our teachers can see is actually engagement with particular individual activities. So here is another same graph with the same ability to be manipulated and this tells the student on one particular activity the rate of engagement or how their engagement is is faring across their course. Now I've just grabbed this as a really up close screen grab to show you that these are a set of exercises that are related to this particular course and what you can see it's very early on in the course but the student can see sorry the teacher can see the percentage of students who are engaging in that particular activity. Now what's interesting here is what we've found over the last 12 months is our teaching staff are starting to embrace the way their courses are structured based on the data they're seeing. So they're able to say oh some of those students are spending more time on that and less time on that and so we're starting to see a bit of a behavioral shift. The third part of our ACE system is actually our proactive re-engagement program. Now along the bottom I've got the four parts to our program. Firstly there is an ACE text or an email that's automated out of our system. The second is our advisory piece the green box and the last two are related to our student care function and in a worst case scenario we will reach out to an emergency contact if our students are presenting at high risk. What does the data look like? Well over semester one this year I'm showing two bands the light blue band is first year the the dark blue band is all the other years. Our threshold I have to say is higher for our first year students and so you can see we've got a lot of numbers there with students that are being detected as disengaging and at risk. The important thing with this graph to note and why we've managed to get organisational buy-in is that that first automated text and email does not require a lot of human time and energy and 60 percent of our students are booted back into engagement from that direct outreach. So if you're thinking about an investment the investment of this type of system has really started to pay its returns quite nicely. And then finally I want to talk about our randomised control trial that we've just run. So we took a control group and a test group of students that are included in our ACE engagement program and what we discovered over a one week period is that our test group those that were actually brought into the program of free engagement were their engagement was increasing by 33 percent just in Moodle clicks. So a lot of our academics said to us well that's great but clicks don't mean engagement they just mean clicks and so what we did is tested the data again and we said okay let's focus let's focus on time spent in Moodle and over the same week period what we found is our students in our trial group were spending 35 percent more time but what was the real deal for us in this research our data scientist undertook for us is that we also looked at how long students how much time students were spending inside their ACE dashboard itself. So not in Moodle but inside the ACE dashboard and we found over a week long period that we had 46 percent increased engagement with the dashboard alone. This tells us that a dashboard is a brilliant place it's a great piece of real estate for our key messaging around student support and so where are we going with ACE? Well we've got further refinements to our student dashboards we will be sharpening the we're sharpening the student dashboard more and adding greater predictive functionality to the analytics in the back end but also we are going to put a pretty big investment in the coming year into the teaching resources as well and as I say with our return on investment in terms of the early initial data we're getting back from our research our senior leadership are pretty stoked and happy to give us more investment to develop more. Finally I just want to say thank you to the work of Jerry Shan who has been in this project all the way with me he's done an outstanding job if you'd like to understand the architecture I'm an educator not a a technician type person but Jerry's your man and he's got a poster out here along with Paul Stevens from Catalyst who have other company that's carried us through this journey. Thank you. Thank you very much. Any questions? Yeah. Thanks ever so much that looks great really something to follow up. Just think about what sort of communication you have in terms of getting students on board and how they feel about it and do they have an option to opt out is it mandatory and then a follow-up question is once you've got the data how do you then speak to students to tutors follow the messages up for example. Right so I we actually talked with an organisation yesterday and we had the exact same question around how do you do the people piece so ACE is really two components it's a technical piece and there's a massive human piece around a change leadership across the organisation we built a workflow for ACE and that workflow enabled us to move down the organisation the automated piece is handled by one staff member for 17,000 students so it's been you know refined quite nicely there beyond there where we move into advisory that branches out into academic advisors and then as we move further up the tiers we get to things like academic deans and course coordinators who do that direct reach out. There is a job in getting the organisation on board but I think when when your staff see the benefits of students particularly students at risk we looked at the 35,000 decisions we make in a day and the increase since the 1980s and no wonder we have high levels of anxiety in our first years they struggle with the choice paradox and we need to change the way we support them and that's really one of the things that's underpinned what we've done. Thank you any other question? One more. Hi I'm Bruno Poulumber from University of Montreal we're working I'm here we're working on a student and faculty dashboard project we're just starting you're much more advanced than us really interesting so my question is quite simple how much shareable or replicable is your ace infrastructure I understand that you are probably using some cloud-based machine learning so I would like you to develop a little right so firstly at the University of Canterbury we are a public university and we are committed to students first ahead of profits or competition so our code is open sourced so that's the first thing if you want to reproduce this obviously you're going to need your own metrics you're going to have to understand your own risk factors in order to build your own model but beyond that the actual code of the dashboards is sitting there you'll need a partner like Catalyst or someone like that who can do that that sort of integrating work for you sorry what was the second question around machine learning we are using TensorFlow so the data comes into Moodle and then it is pushed to TensorFlow it then comes back to Moodle and from there into ace is that right is that right cherry yeah okay yep