 Will a Ben, o fewn i'n gweithio'n gweithio i'w cyllidau yn ei ddau'r cyflawnedd, a'r cyllidau cyflawnedd yn gwahodd reilio'n gweithio. Felly, rydyn ni'n oed i'n ei ddweud yn gweithio'n gweithio. Felly, ddweud i'n gweithio'n gweithio, Ben? Dwi'n credu i'n rhoi. Felly, rydyn ni'n cymryd â'n gweithio. Rydyn ni'n roi. Rydyn ni'n roi ddweud. Ieitha hwn yn y cynlunion. Ben Woein. Mae'r leidio Roedd Arfynu LleidREF wedi cyfnodol gyd-doedd, i gyflym y Llyfrgell Llyfrgell Llyfrgell. Llyfrgellau hyn o hyfforddiant am hyfforddiant mae hyfforddiant o cynfoddiant valley yw cymryd yn y ddechrau, ac ymdweithio ac ymdweithio sy'n gweithio i gyflym fod yn gyflym gyffuriaid ar y cyffirlau dechrau gyhoedd gweithio yn y cyfryd, ac mae gennym ni eto cynerydd gyd, i gyflym o'r strffyn ar hyfforddiant. So, we're going to just give you a bit of a background to what we did. It might be of interest to other people considering this, but we'll kind of start off from there. So, just to begin with, just a bit of a kind of size up the audience a little bit, and find out if we're on the same experience. For the people in the room, would you like to put your hand up if you've been involved in a VLE review before? OK, so for people online it looks probably about 30-40%. Felly mae'n rhaid o'ch ben sydd. Ondích yn 10% ac mae'n gwynhau y byd iawn, mae'r gweld i'r llunio y byd iawn yn ei bod yn ymgyrchu'r ysgolwyr? Felly... Fe pob ddweud o'i rhoi. Fe ddweud o'i rhoi, mae'n ardal i'n ddweud yng nghyrch. Educationon a bobl gweld i dda hi ei ddweud yn ddweud sydd. Ond mae'n gwybod yn wneud ei ddweud yn gysylltu. Aeth yn fwy o'ch ei ddweud, fydd yn ddweud, gyda'r stwyff o'r cwrwch. …a hyn o fewn i chi efallai cael cael cynallu gan gynllunio'r dyfodol Cadwyr-noedau yma i'ch gweithio, teithio eich gweithio, i mi fod yn ei gŵr. Dwi angen gael. Rydym wedi'i gwneud hyn oherwydd iddod i gael y dyfodol. Rydyn ni wedi'r gwneud i gael ddod. Ond ydych chi'n gwneud i gael gyda'r ddod. we had, which was Blackboard Learn at the time. We decided to show this between Blackboard and Canvas, so what we did is went through a sort of staged approach in terms of adoption during when COVID hit in 2019 certain people opted in and then we moved all of the first year courses into Canvas and then we went for full implementation by September 2021. We're going to focus on the actual review. So, just the standout features, what came out of our implementation, as well as having a new VLE and all the features that came on it, we also were able to enable some of these extra features such as tier one support, which Canvas offers, Canvas Studio, which is like a video platform, as well as Penocto, Ally to help with accessibility, now that people could do a review of that. And all those other tools as well, I won't go into more detail, but the nice thing was that novel approaches bit of that slide where we had an advisory group that was set up, there was a project team that worked on moving over from Blackboard Learning to Canvas, but the advisory group made up of different people around the university, so people like yourselves who may be based in different departments or faculties, we had senior leaders in that group, and we were able to meet every so many weeks and actually approve some of these technologies over on trials, etc. We also had Canvas coaches, I'll go into more detail about them, that was how we were able to engage with students, get students to work as partners. And Canvas Connect, which will was quite a big part off, was an online community of practice, and most of you established something like that during Covid where you got people to share practice and ideas, which was really useful for that peer-to-peer feedback of using Canvas. So, the Canvas coach role, we just want to give you a bit of context on this, we aren't going to focus too much on it, but these were students in terms who were hired on grade 2 contracts, they were given 10 hours a week to basically it was to do the sort of day-to-day, making sure everyone was moved over to the new system, help people to transfer materials over, the assessment was moved over to the new system, cold call and any staff who maybe haven't even touched the new VLE yet. But it did sort of expand a little bit more as it went on, we were sort of surprised in terms of they became more of like a sort of student support that we could rely on, which we'll go into when we talk about our survey. Just like you know, we grew into 20 coaches during like the full implementation, and now we have six coaches now working on a grade 3 contract just due to the nature of the work, it's gone a bit more into that grade as well. So, they were pivotal in our actual evaluation as well, which we'll come on to a little bit later on, but back to our story, why evaluate now? And that was the response that we received when we contacted our friends and other institutions. They're like, why would you bother doing an evaluation when you've only just moved to Canvas? Why would you evaluate it? But the powers that we decided we needed to evaluate it to decide, you know, were we making effective use of it and also to direct where we were going to spend the next couple of years of bedding in the technology a bit more. So, the original aims of the new VLE or the VLE review was to find a VLE to meet our teaching and learning needs and to drive and support development of excellent pedagogy. So, obviously our first question is, what do they mean by excellent pedagogy? And I think everybody would say that's kind of in the eyes of the bold, really, isn't it? So, our step one really was to engage with the stakeholders, but yeah, should we move on to the next slide? But obviously, 2020, the challenges it presented us when we actually started this evaluation was how do we actually engage with people? People's needs are really drastically different from what they were in 2019 or 2018 when we carried out the VLE review and also how to actually engage with people when they're distributed, not on campus anymore, et cetera. You know, that was a challenge for us when we're used to engaging with people in face-to-face. So, we're going to go through a number of approaches that we use that we'd like to highlight to other people. Step one, logic mappings. This was an exercise we did with people, with stakeholder groups, different groups that we'd come together and meet via Microsoft Teams and go for a mural, sorry, document together like an online space and map out the, basically what each stakeholder group thought of the evaluation, what they felt should be the short, medium and long-term aims of the evaluation or how they would like to see our use of canvas span out over those time periods. That was really nice exercise to do with students, we also did with lecturers and our SLT. The people who gave us the work, they didn't know, but we were going to get them to do some work as well by going through this process. And that really helped us to develop the metrics they were looking for and the types of questions that we would then put within the research instruments we were going to use. But that was really nice. Fantastic for us as learning technologies to hear people articulate how they see a video as well. Step two. So, yeah, we did surveys, obviously, with both staff and students. The staff survey, we were able to control using Bristol online survey, so we both got accounts, we were able to build the questions based on that exercise we were talking about and come up with questions. Basically it was a mixture of like and style questions, we went between one and five and then smoke and end the question so we could do like a thematic analysis at the end. Obviously we could monitor the data, which is quite nice, so if we knew based on some of the questions about demographic, if there were certain staff who maybe weren't responding from like a certain faculty, we could just look at the data and go, okay, we need to like speak to someone who's at the head of that faculty or someone who's involved in that school and get some more feedback from there. The student survey was done through central comms. Now this had a challenge in that, we had to adhere to the university sort of comm schedule. So we had all our questions ready and we're designed with our campus coaches by, I'd say about January or February at the start of the year, but we couldn't launch it until after Easter, just because of like NSS surveys and term stuff. It was a bit of a challenge trying to get out there, but the key thing was that we could go through their central marketing comms, so like the guilds, channels, you know, on the campus courses, whatever. As well as doing the survey, we launched it in the guilds, we had like a prize draw for people to complete it, so we had vouchers. I think they were £50 Amazon vouchers after like five. We've done it like as a random draw. And I think as well, when they came to the launch event, which we ran in the guild, we bought a load of cacti and succulents. That seems to get people to the table, probably like the vendors are doing outside, giving us goodies. We got people to come to that. And it was quite nice as well as we could ask them to follow through and join us for a focus group, which again, we'll go into more detail in a bit. I'll just talk about the surveys, just in more detail. These are the sort of comments and themes that came out of the sort of staff survey exercise. So we didn't obviously want to compile all the questions for you, but there weren't any surprises when we were looking at this, obviously around the effectiveness of the VLE, the course design, you know, how things were laid out, stuff we wanted to know about. The assessment side of things, you know, what engine to use, we had a tenet in on campus' zone at the same time. And people wanted to know about things like that, Len analytics, the use of ally, are all fed into our senior leadership team where we could sort of come up with like goals of how we could use the VLE long term. I should have said on that last slide as well, the responses there in the top bottom right hand corner. So we had 311 staff who completed that one. With the student one, it was more engaging with 1,447. So we were quite happy with that number. These were the sort of questions and themes that came from the student survey. Again, these were built in sort of partnership with the Canvas coaches. So we were quite interested in things like online to face-to-face because of COVID. Accessibility came out quite strong from students as well. Anything about templating because we didn't enforce a baseline when we moved to the VLEs. So that was quite interesting as well. And if people were using some of those fair party apps like tier one, the mobile app, et cetera. So just to talk about the focus group and that's where the diamond ranking comes in. I'll start off with this one. The focus groups, like I said, were students picked after they completed the survey. There was like a section where they could opt in and then we contacted people. Because again, it was sort of people on campus and off campus who ran these offline as well as face-to-face. What we did six was around May time of 2021. And we recruited students, like I said, through the survey. It was using the structure that we're going to go into more detail called the diamond ranking. We were asking around, how could we get students to actually give their opinion? What can make it more interesting? And that's how we did it. And we used the cameras, coaches to come up with the design, the statements, which we'll talk about now in the activity. Yeah, like Ben said, we got an idea from one of our law lecturers who were using this approach, diamond ranking, to explore with students their perceptions of group work on the core. So it really seemed really good. I've got a kind of a screenshot there, but the idea is that you use this as an activity within your focus group. It's very amenable to being delivered online. You maybe have one facilitator to controlling a PowerPoint online slide to move statements around the screen within a diamond pattern for most significantly significant at the bottom. So our statements were actually kind of pump prime from the survey, things that students had said in the free text comments, things like, the VLE will allow me to engage with my peers or has a good mobile app. It's easy to navigate, things like that. And we kind of got the students to agree amongst themselves the ordering of them. And it was really interesting to see, you know, the students were able to qualify their thinking talking to other students maybe from a different discipline area. It was very, very interesting. It was all recorded, lots of data. And then two statements were left blank where students could actually create their own statement, something that they felt hadn't been included. So really lovely activity, generates lots of discussion. And I really highly recommend that to anyone considering doing evaluation with students. That was the power, I think, was the discussion around this exercise. You weren't really too concerned about where students placed those statements. So there's a question at the top, I think, that says what is important to your learning resources or something on that line. And like it says on the slide, we could edit those as we were going along, depending on the background of the students. Yeah, and we did a pilot run with the Canvas coach as the students were helping us with this. And in fact, you know, they helped on the design of it. It was really interesting to be able to sort of classify different participants based upon their role. So the Canvas coaches kind of could view behind the curtain a little bit, and they had slightly different responses from other students. So again, it was very interesting. We've got a little bit of a summary of our findings coming up now. The findings fed into, well, we had to do a project closure report. We haven't fully communicated the findings to staff and students yet. There's a half a lot we can silly to do with it, but we are planning to. And also when we engage of lecturers in our training for Canvas, obviously we can highlight approaches that students fed back positively about, you know, particularly around structuring content, for example. Yeah, the perceptions, I won't go too much into this because we kind of wanted to talk about how we viewed things for other people's eyes and got them to talk about that. But yeah, high reliability, which is fantastic for us, feeling supported, et cetera, et cetera. It's really good. Next slide. And I guess another surprising thing that really fed back into how we support staff is that 80, well, we found 83% of the students surveyed use the mobile app for Canvas. I don't think anybody used the Blackboard app because there just wasn't anything in it particularly before. It's so good resources. Yeah, but it's high, high take up here. So that's really has an implication about how we design courses. Also a conversation started here about whether they prefer individuality or consistency. So students seemed to prefer a mixture of both. So they're going to debate now, aren't they? We really, as I said, in schools and departments took on a template that they cascade to other modules on that sort of programme. But like you see there, the students like the fact that there's consistency, but we've got staff who like to have their individuality to do something a bit more funky or a bit more interesting with pages. So students' opinion then, these are sort of general comments, general stuff that came back, which was that they did like the fact that you could offer more personalisation within Canvas, that there were study management tools like to-do list in their calendar. And they wanted to, they've seen a lot of apps obviously during the lockdown as well as in further education where they wanted to have everything all in one place like timetabling, library hours, keeping track on this sort of work. So they were very sort of complementary on that with Canvas. What they like to see in terms of what they saw in effective use of the VLA were those involved there. So they like to see a variation of approaching assessments and not just uploading documents into like a tenant and repository or something like that. They want to see more peer assessment or more presentation style or video. They want to see more mobile friendly in the design of a course. So obviously a lot of staff will work obviously off a PC or laptop. Do you want to make sure things are in the right places? People will complain or students that they were going on one course and the sort of module information was in one area and then it was in a different area if in another course. And students were interested if staff were actually being trained mandatory. Now we did offer training but it was never mandatory even when we moved over the option than to come. So that's why we still have that discussion on that last bullet point of whether we need to have a standardised approach or whether we encourage individual approaches to Canvas still. I mean our own reflections as learning technologists coming into it. I mean I guess we had kind of higher hopes or expectations that Canvas was a place of rich peer-to-peer dialogue and collaborative learning, et cetera, et cetera. But in actual fact the students we sampled maybe we sampled the wrong students then possibly. But they fed back actually they just wanted it to work. You know this was mid pandemic so that's kind of when we fed back to our mentors they kind of said well we you know bear in mind we're halfway through a pandemic all the stress of that and you know and we also had Blackboard before as well which was quite out of date and it was quite bloated with all the integrations that our own university put into it. So they were just thankful that things were reliable cloud based and reliable so that was interesting. Reliability came up an awful lot and you know I suppose in a way that underpins the innovation you know once you've got something reliable then the innovation might not be through the VLE it might be something you can do on glass because it frees up your time to be able to do the more exciting stuff face to face in your valuable time when you know that all the VLE is taking care of all the admin for example. But I think it's also it just shows that we really have to work with lecturers to see the value of that because we entered our job roles when discussion boards were probably a big thing you know and things have moved on a long way. We use Microsoft Teams we're a very strong institution using Microsoft Teams for collaborative project work so that's also something that takes a lot of the strain in those areas but that's something we're still thinking about a little bit. Some students aren't aware like that peer-to-peer bullet point that's on there there's something we're not aware of some of the tools that we integrated like there's a tool called Buddy Check I don't know if any of you've used that but it's a nice tool to allow students to give each other in a group anonymous feedback either through like a like a scale type question or a survey and when students heard about that they were like oh mind my lecturer trying that you know during the presentation or something like that so maybe we need to do better in our role and sort of showcase in these case studies and making it highlight that so many people I mean a bit of work we've just lately students have been feeding about they generally find group work to be quite a risky exercise particularly towards the final years where they think it might count against them so I'm putting that list like a TV I don't know what you're saying so Recommendations then so we thought this might be a nice slide if any of you are about to undergo the LA or anything that we got out of it so first point there just working with target group stakeholders on the research design so that exercise we'll talk about with the mural board was really beneficial and we started with blank mural boards then we have each group as well so we could target all the different people who make use of the value of support students so we went to meet the librarians one week we'd go meet the head of the science and engineering and talk to them and we'd see the patterns where things would be higher or low and it really sort of got those people to invest in the evaluation as well because the lecturers engaged with that saw their own questions actually became part of the survey they were much more likely to then promote the survey to their colleagues as well if they were kind of invested in it we could also pilot our focus groups or survey with them as well There's that saying was it too many cooks in the kitchen in terms of like when you design and a survey or design and some kind of feedback form so many people can have this saying push certain questions the agenda but like Will said if you get there buying for this it makes it easier to get it signed off by senior leadership That's true Research ethics that was a pain wasn't it trying to get that done It's useful to get it under our belts one thing it got mine and Ben's name attached to the work as well but having that was a great safety thing for students filling the survey as well that we could give them the participant information sheets et cetera and also then you can present it externally like this the data more easily as well Did it take us to do it? About two or three months Well it didn't take long to do the application but to get a response back and actually pull off and grab to do it initially probably about two or three months yet So yeah established that student partnership in the design process that goes back to the coaches again we used Muro boards got them to work for teams designed some of the questions pass it on to them get them to try it out it was just really handy so if you do have students who can work as interns or like that that'll be really useful and consider structure activity like the diamond ranking and we can happily share how we went about it how we designed it because originally like Will said we spoke to a lecturer in marketing wasn't it? I can't remember but if she was doing it to find out feedback on how people were using I think it was just mobile apps just for the work so we had to adapt it obviously for the VLE so we found it useful for moving it from a marketing perspective to something that was like a VLE review But that was definitely having an activity to lead on the focus groups because I've been in quite a lot of focus groups where you have people who are quite vocal and then tend to talk over the other people whereas this actually gives us something to focus on a little bit and bring everybody into the conversation I think that's probably our so please feel free to get in contact with if you'd like to follow it up or use any of these things that we showed you that'll be very welcome to do that we'd be interested to hear if anyone else uses these approaches in the future Yeah, we're around till Thursday so yeah, happily answer any questions on this today and anytime after Great, thank you Thank you very much It sounds like you've been incredibly busy I think that's probably an understatement a lot of work gone on there and it's really interesting to hear it and we have had quite a lot of things come through actually and I think as you can see on the screen we've got quite a few likes for that first question there What were the biggest differences conflicts between staff and student and staff priorities? I don't think they are quite unified really I think they both wanted the easy-to-use system it was reliable It goes back to the reliability doesn't it? That was the common theme as others stood out The difference I suppose maybe staff maybe had higher expectations on what they wanted to see from the VLE in terms of course design the third party apps like Ally Accessibility we didn't really get too many students interested in that to begin with but yeah, it seems to be they weren't too dissimilar it was quite surprising that they were quite similar when we asked That's quite interesting isn't it because you do sort of think oh it might be that they are very different but really we're all human underneath everything aren't we so there's a lot of things that tie us together and looking I know that there's there's a couple of different things on here and one that sort of stood out to me did you consider areas of social justice and themes of inclusivity decolonisation and how this is addressed through design of the VLE and good luck answering the question oh yeah, I don't know we did get into the decol didn't we let's begin with, yeah how do we adjust that one do we have a question about that specifically we might have had a question to say very not specifically around decolonisation but inclusivity we did especially around accessibility but not really around race or gender I can't remember, I thought we might have gone we need to look for our data set again it is something we're picking up now cos now what we've done is we've got the report we've submitted that there is a list of recommendations based on sort of thematic analysis of the open end stuff and that came up as something that students would like to see improved with the use of VLE so it's something we're working on cos the advisory group still exists in Liverpool so it's something we'll pick up but yeah, that's fortunate we can't really answer that one yeah, I think it's an area that we're all striving to and I think you'll probably find at the conference there's a lot of people who will be having these conversations so we can all learn to about it and to talk more about it I think we're just coming back to it now actually I think we had a question around like equipment and what sort of equipment students are bringing and I think that's feed into the social just this where students are well equipped to use a VLE before coming to the university but sorry that's just come to my mind it might have addressed yeah, so the next sort of one that we've got the most likes on did you use any change management methodologies? We don't know much about that do we? We were in the project team but we were in project managers we had a strategic change department you actually were seconded for a time there Ben and I was for the evaluation yeah, so for the next few years I broke away from the centre for innovation education and joined that team but I was just working, they called it faculty education co-ordinator so it was just to go out speak to staff, make sure that people were engaged in the VLE review process and move over but yeah, we had two project managers at the time, Trisha and Michael Dandy but they were probably able to answer that question better than I can they came up with the methodology It wasn't really owned from from our team it was more of a strategic change project which it did have the APVD for Education overseeing it but it was kind of an IT project Lots of different stakeholders I'm sure we're all familiar with there's quite a lot of questions coming in as we've had before so I can see as we're looking the second one down about this I mean anyone who's done anything like qualitative data, you end up with too much data don't you and we're probably not really equipped to be able to handle all that data in a quick time frame but it's something we could probably look towards in the future we actually hired a student to do a thematic analysis of just the survey data but we haven't really looked at this focus grip data too much yet which I think will be another rich data source What's happened at Liverpool is that the advice groups still exist but they've created free work and groups has won around accessibility innovation and student staff training and they're working off the recommendations of the canvas report to improve those free areas so that's a fair step but there's a lot more like Will said and a lot more data that we need to work through and actually react to don't we there's plenty more that we need to do but it is certainly driving things around staff training where we direct our focus, things like that There's a question at the top there which was something I was considering as well so you started the review in 2018 a migration in 21 how much did the pandemic affect the length of this project i.e. could it have been done quicker without the pandemic in the middle and it's something just to sort of caveat onto that that I was thinking about was I wondered how sort of focus might have shifted as you were having these calls within teams rather than maybe something that would have been in person would that have affected how things were done I think strangely enough even though it seems a lot a few years there it actually accelerated the fact that people moved over to canvas and we got it done I think if we didn't have the pandemic people probably would have kicked up more of a force of moving systems saying that it was too integrated in the old system and now it's not a right time because we've got certain start dates etc because we had the pandemic and our VLE was limited people wanted to go over to canvas because it had a few extra features around stuff you could do like in a seminar online and it was integrating with teams and zooming better so the fact that we could do that staggered approach and get like sort of early adopters first then move all the first year courses in 2020 then go for a full implementation in 2021 it worked quite well I think in the project team we were under pressure to deliver by September 2020 but again we had kicked back from certain departments who didn't want to completely move everything out of the VLE so we extended that for another year which meant we were paying for blackboard and canvas at the same time which was a bit of a pain but yeah even though it doesn't sound like it's quick for those that sort of span of years I think we probably might or not even migrate until now if it was there so yeah a bit of a strange culture at Liverpool So we might have time for one more question here I think so I can I'm sort of looking they've all got two likes trying to decide which one you should go for I think if looking at that top one did you shortlist canvas and blackboard before engaging with the students and if so how did you come to the shortlist that's a quite interesting question So there was a process the word has completely gone out my mind of what you call it but anyway everyone got together in terms of again heads of schools and departments senior leaders and we had all the different suppliers come in and deliver a presentation over a couple of days and then there was a list of questions that someone had designed might have been strategic change so it just came in actually first of all and did like a sort of VLE review in terms of what makes a VLE good et cetera with staff then the designer sort of question it and we went through I can't remember the name what you call it now but anyway we all went through we scored things based on we didn't get hands on with canvas or blackboard or any of the other suppliers then we shortlisted both canvas and blackboard and had the review from there So yeah we weren't really engaged with that too much at that stage though we were talking I mean I only came on really around the VLE evaluation stage at this point in time I think that wasn't really part of the short version Yeah there's a few of us I think I did that survey Simon Thompson who some of you might know who's in Leeds now he was our head he did it as well but for the life of me I can't remember the name I've got a bit of a brain melt down so I'll remember the name what you call it That to me sounds like an indication that it's lunchtime Yeah definitely But thank you thank you very much for your presentation thank you for your questions I'm sure if people would like to get in touch with you then they can but yes if we want to give you another round of applause and for our previous presenters as well