 And it's really rare for me to be allowed to introduce these two wonderful speakers in person. We usually just meet online. So it's a nice opportunity for us to be here all together today at the annual conference. And this session is called Developing Data Analytics for ETECH Support Planning. And Helen, Karen are going to be introducing themselves properly in just a moment. There's also a Q&A element, so this is a very interactive session as well. But for now, please do just put your hands together to give a warm welcome, and then we'll get underway. Thank you. Thank you for that welcome, and thank you for choosing our session to come to you this afternoon. So Karen and I are senior members of Imperial College Faculty of Natural Sciences ETECH Lab. I'm the data analyst within the lab. So I'm Helen Walkie. I'm Karen Cobley. I work as a learning design manager. We were just having a little chat about what we were doing 30 years ago, and I was studying here at Warwick and wearing things like green dot martens and men's M&S cardigans, neither of which I have any more to wear to the garlic dinner tomorrow. And Karen was talking about experience of ETECH from 30 years ago. What? That was much harder than it was. It was probably not much, but yeah, I was just thinking about gaming in Nintendo and things like that, so yeah. So at Imperial College, there are strategic initiatives to develop business analytics across the university for different functional areas, and education is just one of them, and also to embed a data culture within the university. So we're gonna talk a bit today about how we've translated that to our own ETECH function. So we are part of a faculty-based team. We have 11 or 11 and a half people. I'm strictly speaking at the moment. We support five academic departments who have 4,400 or so students, undergraduate and postgraduate taught across about 100 programs. And in the sphere of learning, teaching and assessment, there's been a lot of rapid change over recent years, and for us, that's been down to the reasons that are shown on the right-hand side of this slide. So some of it's pandemic, some of it is due to other things. And we're finding that there is a growing support still for ETECH, growing need for ETECH support. And we want to use business analytics, data analytics to help inform how we respond to that change in need. So when we kicked off this project, we, the first phase of this project, as we call it now, we thought we'll better off having some aims and knowing what our output's going to be. Some of it is obviously trying to address the needs from faculty and monitoring the support requests that were coming in to help us to respond to those queries or those requests. Also giving our team a better chance to prepare for those requests and adapting to the changing needs and the supports that were being asked for. And in that sense, providing the leadership that you should probably be giving in uncertain times to the ETECH team, having a bit of a transparent discussion around what's on kind of workload level. And in terms of college-wide or more strategic decision-making, financial decision-making, with our data analytics would really support every argument that we kind of have, which is, yeah. Yeah, you do that, because it doesn't want to work when I cross it. So Imperial College uses Microsoft tools like lots of other institutions in higher and further education. There are other similar tools out there. We, as Kieran said, this is our first phase of development of analytics and reporting within our ETECH lab. And we've chosen to work with Power BI as our business analytics tool. Outlook we use for our communication with departments. We use Excel for more static data sets so they could be con-separated value files, but we just describe them as Excel spreadsheets within the team. And SharePoint as a repository for storing those static data sources, so the stuff that's in spreadsheets. And we don't use a purpose-built ticketing system for our service requests, so we thought a good place to start would be developing reporting around those service requests. So what we use instead is a shared Outlook inbox. So we encourage staff in the academic departments to contact us that way and we respond to their requests that way. So with that shared inbox, we can make a connection, a data connection, from Power BI to an Outlook exchange server, pull in lots of data about those email communications, add in some other static data sets. So one of the things we have in Excel, for example, is a lookup table that matches staff email addresses to the academic department they come from. So we're not interested so much in the individual members of staff for our reporting. We want to know their department, so that's a way of getting rid of some identifiers. And then we organize all those data into tables and then again into a data model that allows us to build efficient reporting from those. So all that modeling is done within Power BI within Power Query that sits in Power BI. And then we're in a place where we can start building reports, report pages with visuals and things that are meaningful to us and going to be meaningful to some of our end users. And we're then able to share those reports some of the time just as static pages as PDFs and some of the time we will publish the Power BI apps so that end users can access them in the Power BI service which is the web service for Power BI and then they can really use them as they're meant to be used so interactively. And we've used this method because we had all this historical data sitting in our shared mailbox and it's allowed us to look back over a number of years and it's a method that we can use in the future as well. That's the great benefit of doing something in-house. You've got much more control over the data, how you clean the data and actually what parameters you use to evaluate the data as well. So I think compared to maybe something like a ticketing system that gives or piton holes you into certain parameters, you got kind of like the full picture. And yeah, that's one of the great benefits, the flexibility and the control of the data that you've got. Thanks, Kieran. So this slide shows an example of one of those pages from one of our reports that we've actually shared when it was used as a static page as a PDF. So this was used in some discussions with our faculty operating officer about resourcing. So it was giving a bit of evidence of the change in our service requests over time. So on the top left, it's just showing by academic year how the number of emails that we've received and responded to have increased, generally speaking, and then fallen off a little bit in academic year 21-22. On the top right, it shows that split by our academic departments so we can see for each department how it's changed over time. And on the bottom right, it's looking at proportions of service requests from each department. And this is just a sample of what can be produced. But the graph in the bottom right really highlights how understanding your context is important. So you can produce all the graphs you like, but you have to understand your context as well. So we know that the departments that have got a small share of that email traffic aren't necessarily the ones who don't use technology-enhanced learning very much or the ones that might be more self-sufficient in using technology-enhanced learning. It's all to do with the numbers of programs that they have within the departments and the numbers of staff who are responsible for communicating with us about their ed tech needs. So sometimes that's a smaller, dedicated group of staff within a department and sometimes it can be any of the teaching staff. Other aspects that we've investigated include looking at the text in the emails and doing text filters to pull out mentions of specific tools. So we can see whether there's a change over time in how much support requests relate to particular ed tech tools. We can also look at peak service requests, like peak volume of service requests throughout the academic year and look at things like emails in terms of the conversations that they were in. So we can see if an issue was a quick fix within a day, a couple of emails, or whether it took place over a longer period and with a bit of to and fro between us and the academic department. So we can categorize things that way as well. And we've done a bit of work looking at trends. So this is just an example of a couple of ways that trends could be looked at. On the left, it shows the percentage difference to the previous year. So it's another way of just seeing the differences year on year and where you've got increases and decreases. And on the right, there are two curves on that graph. So it's partly showing that you can put two pieces of information on the same graph with different axes and there are cases when that might be relevant. So this was looking at resources versus volume of email traffic. So don't worry too much about what the resources one says, but the email traffic, we then wanted to make some predictions into the future. So there are ways of easily doing that and using scatter plots in Excel and getting it to fit a trend line and it offers you different methods to do that. So there are ways of highlighting trends and making predictions that are quite easy to do. So we've spoken about the data. We also need to talk about the human aspects and very much skills. In terms of data management, we needed some upskilling. I mean, I haven't worked with data management particularly. I've run reports from different vendors and just done Power BI reports but didn't have to be responsible for building those kind of reports and didn't really understand the structures and the needs of that. And likewise, several people in the team, everyone thinks they use Excel, they're comfortable with Excel and they have their own ways of using Excel. So there was some kind of transition I think is still happening. People like color coding, for instance, that becomes completely useless information if you're going to import that into a Power BI situation. So there are things that practices ways of working that we had to look at and are looking at to support better data management. And then when it comes to the main tool we're using, Power BI, it was not a tool that I'd used before and not an old key ring, very much. So we've both developed skills in that area and we are now starting to do that with other team members as well. And we couldn't do that without a dedicated team who are with more expertise than us in our IT department. So we do have a business analytics group within IT and there is also a community of practice where we can ask each other questions and get answers. But IT need to do the bit when we want to publish apps and share them with users in an interactive fashion. We need pro Power BI licenses to do that and they have a whole sort of checklist to make sure that we're adhering to GDPR and data security and all the rest of it before we can actually publish an app. So we rely on them for that. We couldn't do it without them. And whatever we produce, we realize that actually our end users necessarily within our team or the education office don't necessarily understand our outputs. So we had to give some training or guidance around that as well because Power BI being interactive. Things like drilling down on data, understanding what it really stands for was something that we need to provide as well. And we foresee having to do that when we increase our engagement with our different stakeholders, whether it's academics or senior management. So end user training is going to be part of that cycle. Thank you. Moving on. Sorry. So this is the time for a little interactive session. We've talked a lot about data. So we wanted to hear from you guys really about what your data collection methods are. And I think it should come up here now. We've got a couple of Vbox questions there. So this is the first one. So we'd like to know what tools you use to monitor your own support requirements if you're supporting, if you have that supporting function. Shared inboxes. Yeah. Comments. So email and our service now. Yeah. Well, the ticketing systems, spreadsheet magic. I like that. It can be magical spreadsheets, depends how you set them up. That's great. Thank you. Yeah, so a lot of commonalities with the emails and ticketing systems of various sorts. Powerware I mentioned in there. And various other planning, or tools to aid planning. I think you send X somewhere, somewhere. Yeah, we moved away from that. So next question is how do you use the insights from data? If you want to answer that, we're going to keep this as an open question and you can come back to this during our talk as well. Yeah, so we'll come back and look at some of the responses you've put in towards the end. So. Trends. Nice. We'll leave you adding. Okay, I'll move on. So what are you doing? Sorry, can you help? Yeah, you will. Just looks different here, sorry. So the importance of data. So we talked about the importance of data, but there are elements that we also want to mention which kind of makes, gives us the full picture and also how data analytics can help us lead. We were thinking about it in three different levels really. So at team level, we're thinking that, you know, it creates that culture of visibility and also support in terms of capacity making and going beyond, you know, coping your day to day and actually have a chance to plan and think about your workload. So that's kind of like a team level. For academic departments, we believe that using data analytics, we provide insights to identify the areas they need more support with or could, you know, actually ask themselves why they don't need more support in that area and actually just mostly use as a conversation striker where you can talk about different level of tech support. In terms of college, we think that college-wide we support the wider evidence-based kind of culture that we have and research that is needed around data and yeah, supporting good decision-making which is kind of an excellent idea. So yeah, with leadership, decision-making is kind of similar things but operating at that level, we think, again, team will be well-supported in making those decisions. So it just gives them that empowerment to do that. Go ahead, backed up by data and for academic departments, again, it gives them a clear indication of where budgets are going, where resources are going and for college-wide, we think we're supporting, again, the influence of the data structures so they know how faculty statuses are with data and we can kind of inform each other what college data is required to support faculty and vice versa. So next one is, oh, there it is. This is another visual. This is another visual we've created for a slightly different purpose so Imperial has a, it's just to give you another example. Imperial has a focus at the moment on assessment load so one of the things that we've been doing to help departments in their decision-making is to visualize or provide them with a way of visualizing where their assessments fall. So this shows all the modules for a particular program in the spring term and where the numbers of assessments fall in time for those modules and because it's interactive, it will, oh, I didn't show you that bit. They could hover over if they had the interactive access and see some details of those three assessments that fall for a particular module in the month of March, for example. So those were some examples and as I said earlier, data in itself is seldom enough and there are limitations and therefore we rely heavily on other avenues and there'll be conversations, feedback from surveys, consultations, observations going into actual lecture halls and seeing how a techie's being used and workshop outputs. So I think hard data is only valid if it's placed in our context, as we said. There's so much that we know because we do it day in and day out and it's that translation we're trying to work on and developing in terms of communicating that outwards in combination with the feedback and everything else does kind of give us a richer picture, a better picture of what's actually going on and working forward, we're hoping to drill down more on tools. So we've worked on our service request, now we want to kind of know, yes, you know, 80% of the academics or actually 100% of the academics use Turnitin but are they using rubrics? Are they using peer marking? Are they, you know, really comfortable with the quick marks? We're hoping to drill down and what data does Turnitin already give us? How do we complement this with our own data? So that's kind of where we see our work going forward. Yeah, so definitely to develop that, bringing in other data sets with complementary information and build up a better picture. And then continue some of the work that we've been doing to like the heat maps of assessments to offer that as a service to our departments. And it also has the benefit of being very good when they're assessments that require support from us, from us clearly knowing what's happening when and that's, we've tended to be quite reactive, I think, within our team rather than proactive when we're thinking about when things fall. Understanding there's peaks in events and letting our team understand there's peaks in events as well just to have that more forward planning. But we feel we've scratched the surface and that's made a start on this so much more than we could do. So it's exciting for us to have worked on it and continue to do so. We'd like to thank a couple of our colleagues who've been heavily involved in this work along with us. So Ellis Taylor and Moira Sarsfield, who's the director of our tech lab. We've got a list of some relevant reading which will be useful when you get access to slides. But we'd like to thank you for listening and hand over to you to see your responses to our second question and any other questions that you have for us. Thank you very much. So I think we can still see the responses coming in. Do you want to pick these up first and then we'll switch over for general Q&A? Yeah, what stands out to you, Kiran? Produce support material and current topic questions, yeah. I think that's something we want to aim for as well. If there is something that's constantly coming up as a support need, maybe we need to look at the help guides that we already have in place. Are we lacking something? So yeah, that's a really good one, I think. Yeah, I think all of these are familiar to us in some way or another. So like we've just said, we've been fairly reactive to service requests in the past and we do want to map our demand a bit better and be more proactive in the future and developing those support materials and interventions as they're needed is also really important. Prediction improvement. Yeah. Yeah. Good improvement, all sorts of others. Thank you. That's really useful to see what you're all doing. Well, we have a few more minutes for final questions. So maybe if we could switch over to general Q&A and give you all an opportunity to raise any additional points so that we can have any questions or comments are also welcome. So we'll give you a couple of moments. I'm gonna just jump in with a first question. You mentioned at the very beginning, this was phase one or the current phase. So I'm very curious to hear where you're hoping to take this next. We were hoping we'd sort of said a bit about that at the end. So it's, there are lots of different avenues that we could go in. So within EdTech we're very interested in how we can use reports and data that we can get out of our tools. So we're discovering more and more that there are admin reports that we can't access that our IT department can access that tell us, give us more of an overview of how many modules are using the discussion tool, discussion board tool that we have at the university. And it's sort of, we want to work out the best way of setting up data pipelines so we can incorporate that information to complement what we already have through our service requests. But also this work around assessment load and making better visual reports for our team so that they're better prepared to think about what's coming up particularly in the turn ahead. Yeah, things like that. Okay, we've got quite a few questions coming up. I think we're gonna, and there is a couple of more technical ones and then there is a couple more sort of about culture change. So maybe if we start for one about the culture change about senior management, because I think that's something that I'm sure many people in the room are interested in. So the question is, what engagement do you get in the data from senior management or is it primarily used to inform your training and guidance approaches? That's a good question. Shall I start? I don't know if I can chip in, Karen. So we did, another driver for looking at reporting in the first place was so that we could have more informed discussions with our faculty management and heads of department because we have quite a devolved structure at Imperial. So a lot of the money sits in faculties and departments. It's not from the center. So we have to win them over and encourage them that they're receiving, ask them whether they're happy with the service that they're receiving from EdTech and whether we were asking them whether they wanted more of the same or they've got further developments to give or less or they're all, oh, yes, we're really happy. We want at least more of the same. But being able to show some graphs and give them some figures to go with that was really helpful. And having an evidence base and data-informed discussions is really good and we found our senior management have really appreciated that. I think no one was questioning in the sense if we're needed, but as you said, the funding structures are first. Some posts are linked to certain projects or certain departments and things like that. So I think it really gives everyone a bit of a transparent picture of how they are being supported. And I think that's one of, as you said, one of the drivers. So it was one of the reasons we kicked off the project to kind of back up what we were saying. It's not kind of enough for senior management to say, oh, the volume's increased three-fold. We actually needed to show that the volume has increased three-fold. Well, thank you. We have quite a lot of questions coming in. There is one, it's a little bit further down that we had around embedding the process so that this happens every year and making sure that you can then analyze kind of the trends. I think we're just getting that up on the screen. And hopefully, yes, this is the one. Thank you. So what has been the main challenges to embed this culture of data analysis within team to ensure the same collection happens year on year allowing trend analysis? Yeah, well, the beauty of looking at our service requests is that most of the data have just come from Outlook. So it's not required the staff to do anything differently. We just have these odd lookup tables that Kieran and I can look after to ensure that we get the information that we want. So from that point of view, looking at service requests will continue. We just, we have a refreshing pipeline of data. So that's easier. That's the flexibility we had. And we could go, so even if we didn't start collecting this data pre-COVID, we could go back. And that was kind of like the beauty of it all that we could actually go back and see pre-COVID levels during COVID and after. Was because we were kind of basing it on data that already existed in Outlook. But when it comes to, so one of the things we talked within the team about doing is having more progress trackers for things. And I guess it's in those circumstances where people are maybe using their color coding in spreadsheets, which works really well, as Kieran said, when it's just that one person or a couple of people who need to look at it. But if we're pulling data from different sources and wanting to produce tracking reports from different places and show, give them the same look, then they need to all be, what's the word I'm looking for? Formatted in a certain way. And that's the work in progress. So we've been able to show to our team how it's easier if something's set up a certain way and get them to then use those spreadsheets to make their own updates and refresh the data and see how that changes the reports. So it's gradually. Well, we had a recent away day where we actually are getting input from our staff members to see what they prefer as well. So thinking about creating maybe templates where we're also very conscious about people don't want to be tracking the work they're doing. They just want to do their work and we're thinking about how can we automate these processes as much as possible so it doesn't feel like an additional thing that someone's doing. But it's just part of your flow, of your workflow. If that's already there and that's what you do and it works for both purposes. So I think we're trying to be super clever. What was it? Magic spreadsheets. Yeah, we're trying to create those magic spreadsheets. That doesn't create more work for anyone but it helps us to create reports. Well, I'm afraid we're coming to the end of time. There are a lot of questions and comments and I hope that you continue to be engaging with Helen and with Kieran via Discord and online. What a fascinating piece of research. Thank you so much for sharing that with us today and please do give our authors a round of applause. Thank you.