 Rhaid, ffantasti! Rwy'n ffrillio'r ffliwch o'r sef, ac rwyf wedi bod yn ymweli'r cerddau phoblau, ac yn ymddych chi'n ddod i'r ymddangos ar y bwysig. Rwy'n ffwysig, ymddangos i chi ffwyd i'r cyfnodd ar wneud. Yn ddweud o'r teimlo, ymddangos i'r gweithio'r bywydau, ymddangos i'r ddigital llwythu llwythu llyfrwyr ar gyfer y criclwyr. Ymddi Richard y Blachyr Clartsen. I'm an academic development consultant and research fellow at the University of Leeds, and this is based on funding sponsored by the Leeds Institute of Teaching Excellence, and they'll be unhappy if I don't say that. So a little bit of the context. So it's quite a large rush group university over 35,000 students. We're having a bit of a push on digital literacy at the moment, amongst other things. So we've recently created what's called a lead skills matrix, which brings out a number of different skills and literacies and tries to relate them together. Digital literacy is one of the sort of three key areas of this. We've also recently updated our institutional framework to bring it in line with the high level categories from JISC's framework, but it's slightly different in that we've used the three E structure as a way to make it progressive. So there's a sense of ongoing development across each category. We also have an ambitious project called curriculum redefined. The intention is being that we're enhancing, redesigning, redeveloping every single programme across the entire university. It's a small, easy project that has no problems or difficulties whatsoever, but it is really interesting. Because the question is, well, what do our students actually want in terms of developing their digital literacy? And when I started the research, I thought I had an idea that I realised reasonably quickly that I didn't have a really robust evidence base and that I needed to create one. So over the last two years, I've surveyed 6,220 students at the university. And what I'll talk about today is some of the patterns I've found emerging from this data, as well as some connections to what's in the research literature. And the light interaction through VBOX that we've been doing is to give you an opportunity to sort of comment a little bit on your context and also which aspects of this are most interesting to you as the audience. Okay, so the first question I asked them were just about how important digital skills are to you. And not surprisingly, the students on average came out and said, yeah, they're pretty important. Now, word on these diagrams. So I followed the guidance for accessibility to use the Microsofts one so they're screen readable. But I could not for life me figure out how to get this text at a sensible size. So what you're seeing here is from not a priority at the top down to essential at the bottom. And obviously the trend is that there's a huge swing towards about two thirds of the students assets, 4,300. So digital skills are either a high priority or essential. It's a succeeding on their course. And this pattern was pretty consistent across the whole university. It's not the case that computing scientists said yes. Historians said no. Similarly, career aspirations, you're seeing much the same pattern, a bit over 4,000 on those two like sort of high level responses. And life beyond study pretty much the same as well. Not a lot of variation across all of these. So this establishes reasonably convincingly that our students across the board say yes, this is higher priority for us. This is a significant question. I'm going to come back to the next set of questions in a moment. But I'd like to contextualise a little bit with what some of the research literature said. And I'm going to start off by saying I'm going to attempt to be critical about the literature without being negative about the people who are writing it, which is a slightly tricky balance. But there's a lot of limitations in the literature out there, but they're clearly linked to the challenges in carrying out pedagogical research, not least that generally people are doing it on top of their existing workload. It's an extra. And I'm in the very fortunate position that I have as a comment. I'm paid to do this so my workload is reduced. And that's really key to some of the where I think I've done well here. That's really, really important aspects of it. It would be nice to find research aspects with articles, which do all of three things. They articulate a well-forced rationale. They document its implementation thoroughly and they evaluate its impact well. Now, if these studies do exist, I haven't found them yet. I have been looking a little while. Quite a lot of them do two out of three to be fair. And there's loads of stuff out there that's really interesting and helpful. But I'm yet to find the things which you just go, yeah, yeah, that's everything all in one place. Everything's flawed. Everything's informative. And trying to break all this down and think about how well, how can you represent some of the key concepts when it comes to embedding digital literature in the curriculum, which is the thing I'm really interested in. And incidentally, I'm just skipping over the idea that embedding is important. I'm assuming that this audience is reasoning on board with that idea. I started with a long list of concepts and narrowed it down finally to avoidability and regularity with visibility linked to avoidability. And then you can end up saying, well, I like food metaphors. I'm a simple man in many respects. Break it down into different ways of thinking about it, almost like a cake. So do you have a side salad or a sprig of parsley? There is an opportunity, a developmental thing there, whatever it might be. But it's quite self-contained. It's one specific thing or two specific things. And it's relatively easy. If somebody doesn't like it, they can just take it off and put it on the side. If you like the cherry, you eat the cherry. If you like the parsley, fine. But actually it's very easy to avoid it if your mind is to do so. So A and B are quite similar. The question is, the cherry is presented as part of the cake, whereas your side salad is presented separately. Icing, it's all over it. So in terms of regularity, it's spread out everywhere. But actually it's relatively thin and you can scrape most of it off without too much effort. If that's something you'd like to do. Most people probably do eat the icing, to be honest. Most people probably find it. Your raisins are baked in, literally. And dotted throughout. Now, if you don't like raisins, you're going to have to put raisins left in to break in this thing up and pulling them out. But you can. You can do it. They're also quite discreet. When you find a raisin, you probably know you've got a raisin. And if you like raisins, that's great. And if you don't, well, you might eat it anyway, depending on how you feel. And lastly, the sugar. The sugar is essentially invisible baked into the cake, but it's there and it makes a fundamental difference. Sugar, as no doubt it's fairly obvious, is the most embedded on this kind of scale. It doesn't automatically mean it's the best approach in every context. At the same time, probably shouldn't expect a sprig of parsley or cherry to have really dramatic impacts on everybody. Maybe a few, but probably not that much across the board. Now, this is what I'd really be interested in getting a little bit of input from you. So I'm going to just switch the question on VVOT, which is about your experiences linked to this metaphor, which hopefully has a little bit of meaning. And if I've done this right, you should get a multiple choice question that says, what approaches do you see the most in practice? For simplicity, I've combined cherry and side salad. So you've got four options. That's a good question. For anyone who missed it before, one, five, three, six, six, one, five, nine, four. Interesting. We're seeing a bit of a pattern coming out. Can anyone not access this? I'm going to move things on. Apologies if you've not had a chance to answer yet, but I think we've got enough results to see a bit of a pattern and this should be pushed out to your devices. So what we're seeing is cherry side salad icing, both hitting about 35%. Raisins on 17 should go on 10.7. So a clear trend towards the less-invaded end of things. And when I've talked about these kinds of issues at some other conferences, that's been the pattern as well. In fact, often more stark than here and there might be a reflection on the competition of the room. When I've talked about this before, people have very rarely picked up raisins of sugar as something they've encountered when it's academics. So that might be relevant when it's purely academics, I should say. All right, following question. What do you think are the best ways? So not what have you seen in practice, what do you think actually is more effective as a method? All right, very good. Now, the second, if anyone's not with my jam, okay. And again, apologies if anyone didn't get a chance, but just moving things on. So we're seeing the results are almost flipped. It's not quite, but almost. Cherry has come in at 3%. Icing at 9, raisins at 53 and sugar at 34. So we're seeing that opposite flip. And what that appears to be is that collectively, including whoever's watching online and following, our experience seems to broadly speaking that what we see is not what we think is the most effective practice. The things are happening in our context, but we do think that they could be better and that there are better ways of embedding digital literacy development. Obviously, there's a number of people for whom this isn't true, but that seems to be a broad pattern across this group. And hopefully, given that response, then this metaphor has got legs, a slightly mixed metaphor that you're given. It's a cake, but this cake has legs and there's somewhere we can go. Now, the literature, not surprisingly, is comprised almost entirely of side salads and cherries. There's not a lot else going on. There is, but the majority of side salads and cherries. The most common approach is create a few videos. There might be additional activities like quizzes as well, and either put them revealing or some kind of dedicated online space. There's often a sense that online delivery is preferable. It's not always articulated very thoroughly, but it's very frequently there. Sometimes this comes out can well at our own Darcy, but sometimes it's acknowledged explicitly and quite pragmatically. So, for example, Colin Walsfeld, I just say, look, given the scale of this piece of work, there was no alternative. That's the only way you can deliver to thousands and thousands of students, which is quite reasonable. There is a lack of comparative studies for online versus face-to-face, which of course would be difficult to do, but very informative and interesting. And when you do find people going raisins and sugar, it's only ever in the planning stage. So people are publishing articles, talking about how they're going to do this type of embedding, but they're not publishing articles afterwards. So there's a really interesting question, well, what happened in the middle? Did they do it and it fell apart? Were they unable to do it because it was too ambitious and they weren't resourced appropriately to do so? The emerging themes, there's a general prevalence for online over face-to-face opportunities, not surprisingly. Digital communication is what gets talked about the most. Interestingly second, it's reading and writing skills. I didn't anticipate this particularly. There may be some sort of biases in the literature that I haven't really been able to pick out yet, but foundational reading, writing comes up a lot. And then additional identity. Video content is the most common material, but when you look at the evaluation practices, what tends to have more impact is something that's more interactive. Again, no great surprises. Living labs, virtual exchanges, where students are doing a little bit more, whereas with a video, I mean obviously it can be interactive, depending on how it's used, but it's not intrinsically so. There is some evidence in favour of front-loading digital skills development at the start of the year and then returning to it through the year. It's not robust, but there's some suggestions in the literature. And there's some evidence that academic-led instruction has more impact than non-disciplinary experts as well. Suggestive, not robust. And I think probably a lot more research would need to be done on this because I'm sure it needs to be broken down and looked at more thoroughly. Okay, but there's a lack of what's called what I would call a complete recipe. I can't find anything where I can look at that and go, okay, so that works. That worked for you. I could take this and I could try and do my version of it. So there's lots of diagrammatic frameworks and overviews and all this other stuff, but there's not a lot of how is it related to other teaching materials? What exactly did this person do and have they broken it down? So Baker and Al, they create four videos and explicitly link two to assessments and they talk a good, they talk to talk about embedding in a really good way. They're a really good theoretical articulation, but at the end of the day, they're like, well, we made two videos that were quite long and they were linked to assessments and they weren't able to do a great evaluation. So you walk away from that going, well, it sounds kind of interesting, but I don't know what I would do with this. I mean, if fairness, you can ask the author. It's not like literally impossible. Same thing, go right out. Three big online module required to access a summative assessment. So in a sense, it's a bit of a raisin. Maybe it's more than a raisin because if you don't do the online module, you get blocked from doing the summative assessment. You're compelled to do it. It's scenario-based. It's scaffolded. All sounds really, really good. And students report in pre-confidence in the evaluation. So all this sounds great, but you come away from going, that was so broad. I don't know where. I'll be like, okay, yeah, sounds fine, but I've got no concept of how to do all this stuff. Similarly, Alma Mood et al. They collaborate with academics and commercial partners. Goose and iron learning model. All sounds nice. And they get positive. They survey the students, positive feedback, all lovely, but it's the one-hour module. It's a cherry or a side salad. And that's not to say it's not a really, really good salad, but we seem to have articulated that most of us think it's not really what we should be going for here. Okay. And Hawke, I'm just going to mention this is Ben. This actually does a really good piece of evaluation. So I am saying the literature is in a bit of a state, but that's not to say there isn't good stuff out there. Okay. Time for the next question. Will it change your own adventure? There's loads of interesting stuff when I think it's interesting that you can talk about this. And I'm reasonably sure I know what is going to be the most popular of these options, but I would like you to give the audience the chance to take a vote. So we could look at some specific studies in this area and the limitations because I've been a bit critical, but in very broad brush strokes kind of way. So you could spell that out a bit more precisely what I'm talking about. We can look at the evaluation strategies that are generally used in the literature to evaluate these kind of projects. Other recent literature reviews on digital literacy or get back to student voice to the rest of those questions that the 6,220 students were talking about. I'll give it a second, but unless there's a strange deluge of answers, we've got a clear winner. Not surprisingly, what are the students saying rather than what am I saying? And I'll freely admit what the students saying is more interesting. Okay. So I'm going to scare her. Okay. Here we go. Right. So this was, as I mentioned before, a longitudinal study. It's ongoing. Two years so far, 6,220 responses all inside one institution. The responses is across every faculty, every school, every teaching unit, all years of taught study, undergraduates, postgraduates, home and international. So I've sliced the data down according to all those categories. I'd hoped I'd come here today having looked at race, gender, disability. Unfortunately, I was a bit optimistic and some things have delayed that. So that's forthcoming. And this is a survey and a set of questions of a methodology that was co-created with academic students and professional services as well. The second set of questions was this. How important are each of the following to develop your digital skills? And you'll notice these don't map on exactly onto those different aspects of the cake. These are more specific. Part of your timetable teaching, bookable workshops, one-to-one support, online tutorials and healthy videos, self-directed learning outside your course. Okay. Now, the results I found really interesting. And one of the things I'm really interested by is whether this might hold true in other institutions. Is this a lead specific pattern? Because the level of consistency was absolutely incredible. So I don't want to be patronising for anyone, but if anyone's not familiar with the Box and Whisker Diagram, this represents the range within these different answers. The box is relatively narrow range. These whiskers, the lines are a broader range. And if you do see a dot, that's an outlier, a very unusual result. So we've got one outlier here. So the pattern here is that a very narrow response for online tutorials and healthy videos, same problem with the time text so that's why I'm saying it out loud. Averaging about 85, 86%. And this is the clear top answer. But it's not just the top answer on average. It's the top answer for every single school across the university regardless of discipline. For every single year group it's incredibly consistent. You can find, there is one school actually one year where that wasn't true but the response rate was 16 for that, it was very small for that one school so it's kind of an anomaly. But you have to break it down into painful levels of detail to break the pattern. If you start saying, well yeah but what about year three international students in this one school and this one year, then yes you can break that pattern but you have to really drill down. These three are all in the mix. As you can see they're broadly equivalent on the average those boxes overlap a lot. So you do see quite a lot of spread between them. Wherever you go these three are kind of somewhere in the mix together and relatively similar. And again with almost rock-solid consistency bookable workshops are the least popular option. So the fact that the top of this is below the bottom of any of these means that while there is overlap and that's a reasonably large whisker, this is absolutely categorically the least popular option amongst our students. Now this is responses by school so for those of you with an IFA statistics you will have spotted already that because I haven't stated these are weighted by individual responses and school size this is slightly different to what you'd see as if you just piled up every individual student's response. But the reason why is because it's an anonymous survey being repeated twice. If you do that some students will get two opinions and some students will get one opinion. So this probably gives a less biased version. And in fact even though that articulates a lot of consistency the consistency is even stronger than comes out in that diagram. Same questions again the average change from 2021 to 2022-23 was like this. So time table teaching went up a touch bookable workshops down a bit ones on support down a bit online tutorials almost exactly the same self directed learning up quite a bit. And for those who are not familiar with variance measures basically one is reasonably high half is lower point two is really quite low. So what we're seeing is online tutorials and how to videos were the most popular option in every school except one in both years very consistent across both years and with this very low variance the actual variation is not the case that the averages were going out so okay well in one it was super popular in the other one it was quite popular and they kind of averaged out together so it's a really consistent pattern and trend. Right so interpretation thank you and I'm going to leave this really short and I'm going to leave it out to you. So in the last five minutes a chance for questions and discussion I've got this chunk of data I do have a kind of I've got my own thoughts about where to go with it but what I'm really interested is what's your thoughts about working with this kind of you know what could you where could you go with it and perhaps more relevant for you what kind of work in this space would be appropriate in your context would doing something like this actually be of value would it be useful hint hint if anyone wants to collaborate that's great would something else actually be more suitable as well and because one of the things I mentioned that's really of interest to me is to whether this is perhaps is quite nice for what I'm doing personally in my one university but it's a very niche thing that just fits there right okay any questions right so your student data seems to suggest maybe that the cherry of the icing is what students prefer rather than the sugar of the raisins now that's a really interesting point because yeah you're right the two intersect together and there's quite a lot of anecdotal evidence that a lot of students have quite a profound dislike of online videos and a certain and a substantial minority simply dislike online learning in general so one of the following things has to be a qualitative research focus groups to discuss this the anecdotal evidence I have from our students which is not robust and is not reliable but just based on conversations is that they're very picky about the two things I mentioned earlier which is the nature of the resources and how they're integrated into the rest of the program and the other teaching materials so it's not I like online videos full stop it's I like specific types of videos addressed to specific types of content but what I don't have yet is the robust research to say I can state this confidently go ahead this is the joy of co-creating survey questions is that you don't always necessarily get the questions that you would have picked had you done it on your own so I'm working on the assumption that this is a better set of questions because it was developed with a wide range of academics professional services students personally I wouldn't have chosen that because they do overlap and there isn't a clear distinction between the two and if there had if and different questions which demarcated them might produce slightly different results so yeah good point are we able to check it and if there's any questions on the line actually oh actually hang on I said BROX up to do that so I'm just going to have a quick no that's fine nothing online she was saying I just thought I had a question were you wanting to do some maybe some focus groups yes so so in terms of sort of next steps I think the focus group is absolutely key because what you know what these quantities studies always do is they point you at your attention and direction but they don't give you any great insight into the reasons behind it for example well is it the case that the students just love videos because that seems counterintuitive presumably not as I was saying but you then need to actually talk to them and have an in-depth conversation to understand what they're really looking for is it that their issue with videos when they do complain is that it's people just droning for an hour face to camera and it's length is it content that they like factual videos but for other sorts of things they want something different on a separate piece of research to this I found during the pandemic we asked students what you want the overwhelming answer was we want interaction in our teaching that's the fundamental thing we want which didn't mean no static resources it meant that's got to be in the mix somewhere and it's possible there'll be similar findings from focus groups here they're like well videos have their place and they have a purpose and we value them if they're used in the ways that they're looking for coming back to that idea I guess of is it that the students want just in time information I want to know how to do this now because I need for this particular thing in this point I don't want to have maybe a session beginning of the year that I may not need that skill until six months down the line yeah I mean quite very plausibly yes I think and another interpretation which I've got from some students I've spoken to you but again this is anecdotal this is not reliable research is that what they want is that these are students in a very applied discipline they want online asynchronous teaching content including videos for facture information and facture information only so they want the equivalent of labs hands and stuff they want they're in-person interactive they don't want it to be didactic but they want to be content heavy what they want is they want all of that to be primarily online and whenever they come in interaction is what they're doing so my hypothesis is that that probably holds true across a lot a lot of disciplines but again we've got to do do the research and find out go ahead yeah this is interesting and this part of the problem is that people don't go into enough detail about exactly how they've distributed this so very often they'll say I created resources or something and then I did an evaluation at the end of module and I'm okay but where were they in the module and sometimes people do but often they don't no I think one and done you're right isn't a recipe for success but there are but often when the evidence of impact seems to correlate broadly speaking with a push at the beginning which is then revisited longitudinally so rather than distributed equally over time it's front loaded but it absolutely isn't where you did this in week one so sorted go ahead so the reason my research didn't look into this is we use the GISC insight service at the University of Leeds and so they ask those sorts of questions so we do have quite a big base of evidence around that at Leeds we have if anything the opposite problem we have really quite a rich suite of digital tools and sometimes the students tell us it's a bit overwhelming and it will be nice if we could curate a smaller set of key tools to use so yes I no I think it's really the question the answer is I'm glad to not to do it because somebody else is because you had some not that last question with the other people because I'm too late and do you know where your research will go? Well I mean I think I know I know it's going to go locally because this is an ongoing project so I'm going to build up several years of data I'm hoping to get up to at least sort of five figures to have a really robust amount and I'm going to be doing like a regression analysis to try and identify what whether there are predictors so the year of the students are studying in doesn't make a big difference what discipline they're studying in surprisingly I mean computing students are telling us similar things to artists and to engineers and everything else that surprised me the question then is but when you start looking at things like ethnicity and bearing in mind decolonising the EDIs such a huge an important topic does this tell us something and so as as professionals who are working in this sphere if that data comes out and says all right look when you break it down by ethnicity for example it could be disability it could be gender it could be anything there's a stark pattern we then need to take that into account and if my research is about university leads that means that I need to account we at my university but then if that pattern were replicated more broadly there would be a message for the wider sector as well so I guess my my sort of invitation is if anyone's like oh that sounds interesting please do drop me a line and love a chat about it next is that