 Hello everyone and welcome to the session in the Altswinter Conference. I am really pleased to introduce Matthew Street and Laurie Phipps from Keen University and JISC on their talk on a digital levelling up. So I'm not going to waste any more time with my introduction. I'm going to hand over to Laurie and Matt. Thank you very much. Good morning. It's great to be with everyone today and we're just going to talk through the story so far with the digital levelling up. Evening from Sydney. This talk will be quite informal. It will be a discussion between Laurie and I and hopefully you'll take away some interesting points and a bit of a story of where we're up to. Okay. I'm going to hand over to Matt Keel now for over 15 years and he's the head of digital projects and I think what that means in Matt's context is he picks up everything that's interesting that's going on in the institution and is told to turn it into some kind of product or project that we then disseminate across the rest of the university. Yeah, that's pretty, that's an accurate summary there. I've got the pleasure of introducing Laurie. Obviously Laurie is with JISC for over 20 years and is currently Senior Research Lead and in the Keel context we've got the pleasure of having Laurie as a professor of digital practice and leadership which has been awesome over the past few years. So a little bit of context for Keel just to set the frame and for those of you that we often get asked at Keel, so where's Keel? It's off the junction of the M6 between Staffordshire and Stoke-on-Trent but it's an institution made up of three faculties, Social Sciences, Faculty of Social Sciences, Humanities of Social Sciences, Faculty of Medicine and Health Sciences, Natural Sciences. There's about 15 schools altogether, a large campus space university with circa 10,000 FTE, just to give you a little bit of context. And in the relation to the area we're talking about today, the Access to Participation Plan is really prominent to Keel. There's lots of proactive work with schools that access education and that's it's become embedded in schools and there's a key feature of annual programme review which really gives it some prominence for us. That sits within a clear governance structure and a clear planning process with key local links just to give you an idea to where Access and Participation sits within the Keel concept. So Matt, during the pandemic, like everywhere else in the sector and possibly the world, something happened during the pandemic emergency and Keel being a huge campus university you had some specific challenges. Just as some context, do you just tell us what happened at Keel during the pandemic? Yeah, as everywhere else, we moved to emergency remote teaching in the first stages of that pandemic. The focus there was really about pragmatism. It's about being practical solutions for staff that can implement quickly with familiar technologies. We didn't use the word pivot, for us it was an emergency move to and it was contingency teaching. At that time, we're a relatively small team. There's about five of us that work centrally and those five serve the institution. So there was a lot of work to get people up to speed to move quickly and just kind of calm people down. It was a really stressful time for people. We then moved as we moved through lockdown into more face-to-face teaching, we'd got a flexible digital education framework that had been actually in development with senior teams prior to the pandemic to provide a more structured approach to education delivery and to provide some consistency and coherence for staff. So we were hopefully moving to a more designed approach at that point and our role there was really to kind of build a bridge between the emergency remote practice into something that was a little bit more structured and more designed. We were also really fortunate in infrastructure terms. We'd been experimenting with teams for about two years and we moved that out-hole sale alongside the VLE. That wasn't the immediate focus, that the teams didn't come into place significantly until we moved through second lockdown really. We got really good integrations between those platforms. So it was effectively operating on a kind of module rollover basis where we could create modules from our student record system and manage the remote. So that was, that's like in a nutshell really, a rough quick tour through our emergency, our move through the pandemic. Obviously, something else happened in relation to assessments. We took care. Yeah, so the assessment thing was interesting. So you had all of the issues of students and things like accessibility, digital divide. There were issues coming up like that. But we also had this thing where you started switching out the assessments and exams and assessments for everybody was disrupted. But you noticed something during the finals. So what happened with assessment and exams when you moved into the emergency? Yeah, so we took again a practical approach from pragmatic one to that, recognising the things that you've talked about there a lot in terms of the student behaviours and student practices. Accessibility was an issue. Access to access to tools and spaces to do work and environments that were conducive to study were all coming through our various different pull surveys at that time. So the approach to assessments was a rather pragmatic one. We asked staff to avoid platforms that were unfamiliar and avoid, this was in the first stages of lockdown, and to avoid assessments that weren't accessible. And to start thinking about how they could introduce more open book exams to replace closed examinations. Anything that could be electronically via our existing systems and processes, and to think about the time that they could, they could allocate to students to do those. And we drew from a lot of openness and sharing in the sector, which was fantastic. We drew from a lot of work that was going on other institutions and put that into a package for schools to take on board and take control of. Reiterating that the assessment had to be accessible, valid, relevant to discipline and fair. And what we saw from that was a lot of work with schools that schools put forward to maintain really clear assessment guidelines and assessment criteria to make things a lot more transparent for students and a lot more and a lot easier to understand given the shift that we were seeing in the, in the way we had to approach a final examinations. Okay, so the assessments and the examinations, what happened with with that with regards to results? Yeah, so we saw a fairly significant shift actually in in results. Yeah, Luke. So there was a significant improvement across the board really with assessments. And we can see from this awarding gap data that there was a significant movement. You should just say that this is the attainment gap. And we just don't like the word attainment. Because that sort of puts all the emphasis on the student, doesn't it? So, so we're sort of making the point that this is the awarding gap. This is the award that the students get given by the institution rather than what they've attained. So sorry, Matt, carry it gone. No, no, absolutely. That's, yeah, I'd, I'd altered that again, because I didn't like the word attainment in the top. So back awarding gap. And for us, there was when the when we looked at the data, there was a significant change across the groups of students represented that. And this was really significant. We hadn't seen this kind of shift before at all. So it threw us a little bit at first, if I'm honest, I wanted to understand why. Okay, so, and also, this, this is a leveling up, right? This is not a coming together where the, the mean move down. This is actually an actual leveling up, not a coming together. Is that right? Yeah, that's my understanding of it. We've, as I said, we've not seen accounting for no detriment policies as well across the assessments, we still see this shift. And it's across the boards. So it's a leveling up, whether it's a digital leveling up. I mean, that that's the question that we'll, I think we need to unpack a little bit further. Yeah. But yeah, it's definitely a leveling up. So what one of the things that we're interested in, given that this is what we saw at Keele, is what are people in the in the call, what are people seeing with their attainment gap, or awarding gap, or whatever language that you're using? What are people seeing in their institutions with regards to the gap? So, you know, we're just going to take a minute or two. What we're looking at here. Yeah, Peter, I accept what you're saying there. I guess, for us, the awarding gap is the actual degree given. So it's a measurement, it's a metric. But that metric did actually close. So that reportable metric. But did it, I mean, just in terms of people in the UK, you look back to a report on this and have your attainment data. And I'm just wondering what other institutions saw. I can see there's people from quite a range of institutions in there. Does anybody want to sort of talk about their awarding gap? Tell us what they got? I'm not sure. Okay, there we go. I mean, it's really important for us to capture this narrative. And we'll talk about how we've been capturing it in a minute. But yeah, we'll come back to the chat towards the end. And look, if anybody has posted there, there are awarding figures in there. Yeah, we, so we really wanted to investigate a little bit further. You've seen the, the narrowing that we, well, the narrowing that we've identified above, and we really wanted to understand why that was statistical data was was was one thing, but actually to understand the context of that data and understand the experiences that were, that were there were important. And we were fortunate enough to have bodywork in the work and appeal to build on the work that he'd done with, with Donald Landcliffe's to put together a research project. Yeah, so what we did was is we started to put together a research project to look at what we're doing. So we used a method and Matt's just gonna go forward for me. Yeah, so we used a method called contextual inquiry to do this. Contextual inquiry is it's a process where individuals are interviewed about their practices in an open ended format, and within a particular frame designed to elicit information. And it's not just about the content of what they do, but what their motivations are, what personal history contributes to their practices, how they're impacted by the current situation, whether that's macro or micro context. And the priorities not to arrive at generalizations about populations or cohorts of students, but to help recognize and interpret those patterns of behavior. And that gives us insight. And I should say that this is not deductive insights. You know, this is actually inductive. So we don't look to sort of bring it down into a small gap. We try and induce and sort of make quite a broad statement about this work. And so that's the method that we used. And I know, if people are interested in that method, it is actually written up in a journal that will post into the chat. So you can you can look at it, especially if people are interested in replicating and doing this research in their own institution as well. And I know Emma's got the link for that. So Matt. Yeah, so Matt actually is, he's not put himself very clearly on this, but Matt put together the team to do this. So Matt, do you just want to sort of talk about the project? Yeah, so the group you can see before we are there is the project group. There's quite a broad research group there. There's always a sponsor, which is our, the director of Kite, which is where I'm located, the Institute for Teaching, Innovation and Excellence at Keel. Laurie and I, and then Katie, Terry, Nicola and Lou, all and Leroy, all with an interest in assessments, interest in APP. Terry is the institutional lead for for the APP and making sure that and making sure that ties together across the institution. Katie is an assessment lead, and Nicola is looking in more detail at students with disabilities. And Lou comes from an experiential learning background and Leroy is a student representative from the student the students union. And drawing that team together was essential, really. Here was a cross-draft presentation from the institution, and also a significant buy-in from from senior leadership. And so far there's been about 30 hours of staff and student interviews with more to be scheduled. And then this phase of the project, the next steps are to look at that, the data that we've drawn out, which I'll hear more about in the second, and understand how that fits in the context of the institution. As I mentioned at the start, the award, the APP access participation work is tied to, in one part, to program annual reviews. So we'll unpick those annual reviews and see how this information can fit in, alongside the key themes that we've been spotting that have come through from the interviews. So I've been tasked with my research is to actually do those interviews and interviewing staff and students. I should say from the start that, you know, we haven't named any of the participants, but we're grateful for their participation in this. And of course, we did go through a full ethical review before we started the project as well. None of this is just on the roof. It took us a couple of months to get all of that ethical approval through. And as you can imagine, it's a really sensitive subject when we're working with these groups of students, especially. So we went through that and that helped to surface some of the more direct questioning that we wanted to do and really helped us focus on what we wanted to get out of this. If we look at the early findings, I think there's two key success factors impacting during the students. The first thing I would say is that asynchronous access to resources definitely had a major impact on the experience of Black, Asian, minority ethnic students, disabled students, and international students. It was really obvious that they valued the asynchronous. But and this will not be a surprise to anybody involved in any kind of quality assessment, quality assurance in an institution. The biggest impact was the flexibility of assessment. And we had students telling us how much they valued the variety of assessment that was there. In terms of academic rigour, it was really interesting in interviews with students when students were telling me that well, they would say things like, well, we had three hours in 24 hours or two hours in 48 hours to do this. But it was still hard. You know, they'd come out and say, this was not easier. This was still hard. And they would say things like, I just can't copy this from the internet. I've got to do the work, you know, almost in a surprise, which is quite nice, really. So the rigour was maintained that, you know, from the interviews, it was clear that students were taking this in a very serious way. And they identified that rigour was part of it. And it was really helpful for us to sort of unpick what those other issues were. And we've got a bigger report coming out around this. But most of it was around flexibility. Just one quick comment from a disabled student with ADHD. And she told me that the flexibility of assessment was so much better because she found that in exams, it might be one of those days where she would look up from the paper, look out the window and then realize when she looked back down at the paper, it was 25 minutes later. So having the flexibility that you could do this exam within that 48 hour period to suit her needs and to suit when she's feeling good about doing it was really important. So that accessibility and that thinking about the needs of the student was really important and actually had an awful lot of benefit to the students that we're working with. As well as the assessment, other things that we identified that were impacting on their experience and impacting on their attainment was the idea about being back on campus. And that back on back on campus discourse was really focused around working spaces, especially around the library and meeting up with colleagues. And there was also some confusion for some of the students around whether or not online materials in online reading lists were accessible by the library and things like that. There was a lot of discourse about eBooks and if there's anybody from the library community in here, the eBook debate continues with students as much as it does with staff. And again, it's hard to justify to students when they say, but it's an eBook. Why can't we all take it out at the same time? It's a really hard one to sort of turn around and say, well, you can't because that's not the way a library works. Because students just want to access the material that they need. Other things that were coming up during these interviews were things like the idea of connection and communication and socialization with students. That was especially true with international students. And as many of you know, we had international students coming over still at the beginning of the pandemic. And they were feeling very isolated. So there were issues around that mental health was obviously an issue. And as we go further into this research now and we start interviewing people that were starting their degrees during the pandemic and they're in their second year, it becomes a really interesting situation where the difference between what they experienced in the pandemic, what they experienced as we were starting to leave lockdown. And of course now we're, well, I don't know actually where we are when it comes to lockdowns or pandemics, but we're still interviewing students as we go. So yeah, the other thing that I want to sort of finish on the early findings is this is especially true with black male students and with Asian female students. We interviewed those students and we said things like so what's it like in a lecture? How do you behave? And there was an awful lot of rhetoric around, well, I like to be in the lecture. I don't like to be seen. I don't like to be noticed. So I take lots of notes, but I try and keep out of the way and not be seen, which is an interesting observation from their perspective of they actually there to learn they go through that. I interviewed staff and I asked them, well, what does a success look like in a student? What does it look like when students do well? And they were saying that, well, it's they're engaged. Now engaged is a really loaded term. And so we have to unpick what engaged mean. And when I spoke to academics and said, well, what does engaged look like in students? They said, students sit at the front, students get involved, students raise their hands, they ask questions. So we've got this dissonance between a group of students that perhaps are black or Asian. And it might be true for disabled students as well, who are deliberately trying not to be seen, but still engage in the learning and a group of staff that perhaps are thinking, well, those are really engaged students who are there putting their hand up and engaging and asking lots of questions. So there's a dissonance that we need to resolve there. So yeah, so that's our early findings. And let's say we are going to write them up. Matt? Yeah, so we are going to write them up in more detail. I think the other the project next steps beyond this initial phase will be it'll become a longer tune will be so worthless, I think, in order to explore, to give it to explore and in as much detail as we can. We'll we'll start doing now is we've managed to get a data scientist involved that can look across the assessment data in more detail, noting that assessment theme was quite broad, and we'll start to try and understand that in a bit more context, understanding where the assessment changes happen, what they look like, and then approaching interviewing students from from those areas. So we're going to try and unpack that assessment theme a little bit more, I think through the next next phases of the work. We have seen not so much Achille, but looking around the sector, we have seen that the gap has started to widen again in some spaces as we start to revert to some of the non-emergency assessment processes. And I think we need to unpack what that means as well. Yeah. So just, you know, what what do we do? This is a bit of a loaded slide, because I'm not sure what post pandemic actually looks like anymore. And I'm not sure that we're actually in post pandemic. We're still mid, aren't we? Yeah. And I think that first bullet point is something I just want to talk a little bit about. It's I think as learning technologists, as academic developers or anybody in this field, I think we have to really focus on the fact that higher education, the way we deliver it now has been developed and delivered in a way that really privileges people that are white, male and able. And I think that during the pandemic, we started to design or at least take account more of people that are more vulnerable. And I think that one of the things that we have to do is start thinking about, well, we need to stop designing for people that look like me and start designing for the most vulnerable groups in the university. And I think that that will probably elevate everybody and help everybody. And that's the key. Matt put the actually listening down and you actually put that down in the notes originally. But I just wanted to say that actually listening is an act of care. So the actual project itself is an act of care because what we found is that students were saying things like, I'm really glad you're asking me about this. I'm really glad you're running this project. I didn't realize that, you know, you'd pay attention. So actually listening as an act of care is something that we should do. So, yeah. And with that, there's a I would say with that, there's a responsibility that actually listening and care means to do something with this. And that's why it's important to continue the work in that magnitude in a way and make sure there's clear there's clear points to report on and share back with the participants. I guess we better answer questions. We better add anything. Thank you so much. That was fascinating. And just just to let you know, Laurie and Matt, we actually got lots of comments on your activity. But I think there might be a bit of a delay between us, a bit of a satellite delay in the broadcast to YouTube. But if we have time, we'll go back to those comments, but we've had loads of questions. So the first one I'm going to show is from Dave White and it is how do you think the Academy has changed in the minds of the students over the pandemic and what should we do to respond to this new image of the institution? Matt, shall I take that? Yeah, go for it. We've done that one. I don't think that the students think in terms of the Academy. I think that we'd like to sit there and think that we'll, you know, we're part of this Academy, but students don't think about that. Students go into university to focus on an output that they want to get. I think that what we really need to focus on is the image of students coming to university thinking about, well, how do I get value for money for what I'm paying for? That's the reality. It's a brutal reality. So I think that there's some work that we need to do about, especially around induction about why students are coming to university and what they can get out of it beyond the transactional piece that they are, you know, with the rhetoric of the press, with the rhetoric of government coming into. I don't, but I don't think they think about the Academy. I think they, you know, we need to bring them in and actually educate them about being educated. But that's always been true. I think that's always been a thing, right? Yeah, Matt, do you want to add to anything there or? No, I mean, I was just reflecting on that. I think the. I'm not sure that when we've, the students we've spoke to, I'm not sure the sense of an Academy came out, really. I think it was. And in my through my interpretation of it, anyway, so. I don't think there's an image yet in that sense. OK, so we have a question from well, it's more of a comment, but maybe you would like to comment on it from Peter about the complexities of delivering online exams, the systems and his comment is that it was more difficult to make reasonable adjustments than it would have been for, say, pen and paper, more traditional setups for assessments. Was that your finding as well or? Matt, I don't think we did find it harder, actually. Thank you. No, I don't I don't think we did. I think so. There were certain types of assessment that will be found harder in the sense. So our Oskies, et cetera, were and those more complex do you want to expand on Oskies? Sorry, Oskies are the practical assessments that medical students take often delivered in situ and the station based time. So those are more complex, but the reports we had back from the ones that when they were moved to a different mode of labor actually sometimes were improved and a little bit more flexible. So I think it depends on the nature of the assessment what we didn't really find too many issues around reasonable adjustments in terms of time. I think it was probably because of the time frame that we gave for online assessments and the fact that the nature of the most of them were around open group type approaches. I think from my findings talking to staff, this is going to sound awful, but it came through in the research is that there was a gender bias as well in staff. There was definitely a gender bias towards female staff taking more pastoral care of students and thinking about their needs more so than the male staff. You know, and it came through very clearly in some of the interviews that I did, not just in Keel, but actually there's we've used six field sites across the sector. And we noticed male staff saying things like, well, if I don't hear from students, they're going to be fine. You know, they'll they'll get in touch if there's a problem. Whereas a lot of female academics say things like I've not heard from my students. I think over there, you know, I'm going to just touch base with them and see if they're OK. And I think that that's just the tip of a very interesting iceberg around pastoral care of students that then translates into how do we assess students? How do we work with students on their assessment when they come to us and say, I need a reasonable adjustment? What does that look like? I think there's there's two ways of approaching reasonable adjustment, a very mechanistic is the rules. This is what we do versus a, you know what, let's sit down and think about the needs of that student and what that might look like. So I'm going to ask one more question, because we're slightly over time now. And I know there being loads of other questions, but hopefully maybe we can come back to them and discord. If Matt and Laurie are on discord later, maybe come back to some of the comments made. But the one I wanted to add on to add at the end was from me McCormick. Was the assessment flexibility purely in terms of time location or could you test the same content in different ways for different students? So I guess different formats of assessment, I suppose, or? Yeah, yeah, yeah. So, yes, we did, in a sense. There was a shift away. So what would the guidance to staff was to problematic, keep things problematic, simple and to make sure the assessment still alive was still reliable and robust so they could take that anyway they wanted really. So we did have some staff that moved not just to open book, but to different assessment of different types of content or different form and in different formats. So there was flexibility bounded around with simple guidance and instruction to keep things straightforward. So that was that could definitely be an option. It wasn't just necessarily time, time, time and location. And I think the nice thing about us interviewing the staff after the part of this research is they actually reported back that the emergency gave them the opportunity to reflect on the changes that they wanted to make anyway. It's like, you know what? I'd like to assess this in a different way. I don't need to do a three hour exam. And alongside that, I think one of the things that came through a little bit, too, was there was a bit more of a transparency from the staff around assessment criteria. So thinking about assessment literacy and feedback literacy, that articulation of what the expectation was, both to engage online, what resources to use, how to use them, how to engage with the assessment, what the requirements of the assessment were because of the shift was a lot clearer and people thought about it a lot more. So that kind of that might be a factor rolled into that assessment question. OK, well, look, thank you so much. That was a really interesting, especially at the minute with the current news are really interesting and timely session. We've had loads of questions. Hopefully you'll get a chance to look at those comments. And hopefully we can all continue the conversation on Discord. I just want to say thank you so much to Matt and Laurie for the presentation. And yes, thank you very much. Thanks and thank you for all the questions as well, especially the one that Peter Bryant keep throwing at me because they're hard. I was just thinking the same I was reading those. Just ignore Peter.