 So hi, so my name is Elliot, I work for University College London, I'm a senior learning technologist and product owner of UCL's new theme which is what we're going to be talking about today. I'm joined by my colleague Stuart, do you want to say introduce yourself maybe? Hi, I'm Stuart, the more I do UX stuff. Thank you, we rehearse this many times, don't worry. Cool, so we're going to be talking about our theme, here's a bit of a run through of what we're going to be talking about. So we'll give you a bit of the UCL context, a bit about the UCL design system, we'll talk about more, and then we'll get into some of the UX approach that we took, so top task analysis and benchmarking, the initial designs we tested with users, with learners, and then the redesign that we did, and then a bit about evaluation, how we're looking at this going forward, and then there will hopefully be time for Q&A. So to start it off, so UCL context, so UCL, like a lot of universities and institutions in the UK and the globe, I guess, we're needing to update to Moodle 4 to get a lot of the benefits, the security and all these things, so we updated to Moodle 4.2 in July 2023 from Moodle 311, and then part of this update was a good process or a good chance to update the theme, and we also were, we had some general ideas that we wanted to target, obviously the landing page makes sense to improve, and the dashboard and the my courses, that split that was on Moodle 4, you know, what could we do with the dashboard to make it more about personalized learning and maybe build functionality within there. So we also had some business requirements, a big one was the UCL design system, and we put beta because it was a developing design system, and that was one of the challenges. We also wanted, you know, marketing spots to advertise events at our campus at university, we want to advertise them on Moodle, until a lot of users when things occasionally are broken, so that's a pretty key business requirement. The other thing is UCL is pretty big, we're in the northern hemisphere, just to make that point, and the active users are about, you know, close to 40,000 a day, 70,000 browser sessions, plus 5,000 courses for the academic year each year, and about 150 plugins. So, you know, it's important Moodle to our users, and this update was definitely a team effort, bringing it together, and we wanted to make improvements for our user base and make sure things kept working. So number two is the design system. Do maybe raise your hands, if people know what a design system is, and do they have one, maybe keep your hands off if you have one in your workplace or institution. Okay, sweet. I know Moodle's kind of got a design system which looks excellent. This is kind of like UCL's design system, gives you a sense of some of the components. So this helps us, you know, across UCL when we're building different systems, including Moodle, to create components with, you know, similar UI, keep to brand, keep a strong identity. So this was kind of what the UCL's design system was looking like. It was developing, so that was something we had to keep on top of. But that, this is kind of what we wanted, you know, we had it in the back of our head that we wanted the new Moodle, the new theme to look like. This was kind of interesting because on the left is kind of, you can't really see the image, but maybe you can see the kind of colors, key colors in the field. This is Moodle 311 at the time. You compare that to the design systems, you're like, whoa, there's quite a big difference here. You know, so we actually found through our UX research, which I was going to talk about in a sec, but when students were kind of, you know, they're going through the apps in the different Web of State of UCL, and then they come to Moodle. They feel like they're stepping outside the university and going somewhere else, which is a really bad experience. And once you didn't even say it, it kind of felt like they were going to this dodgy environment for their learning. So that's something in the back of the head we wanted to improve on. But at the same time, you know, what from the old Moodle could we keep? What was the key functionality that we wanted to keep? And what areas did we want to improve on? And so that's where we started with top task analysis and benchmarking. Cool. Thank you very much, Elliot. So top task analysis and benchmarking is about finding out what the hell your users want to use your site for. You choose what to measure, which is what people use your website for. You measure it based on its ease of use, how easy are those tasks to perform, and decide based on that evidence what to improve. If people say they can't do something and it's a real problem, you should probably fix it. So if lots of users found it perhaps hard to find where their feedback and grades are in Moodle, might be a real thing. We'll tell you later. Try and build something that improves that, test it with the users, the prototypes and things, and then iterate. This is a very important cycle. So we did an online survey. We just built a block in Moodle, HTML block with a link saying, hello, tell us what you think. We asked some questions. How often do you visit UCL Moodle? Because we wanted to know. What's the main task you visit UCL Moodle to do? How easy do you find it? What's your favorite thing about UCL Moodle? And what's the worst thing? I'm not going to show you full results, but Elliot can if you're nice and buy him a drink later. We intend to repeat this survey over three months to measure as part of our baseline and we keep measuring it and then we know if we're doing better each time. Hopefully we never do worse. So what did we find out? Students actually loved UCL Moodle. Elliot said some nasty things about the design, but some of them referred to it as affectionately retro. And I hold that in my heart. I'm like, cool. Learners have UCL Moodle open all day in the tab next to them. So they have like the tab. When we're interviewing them, they're showing us and going, oh, in this tab, I've got UCL Moodle open. I just have it open all day. The top tasks we talked about, course content, activities, feedback. What did they like about it? Ability to locate all the modules. It has all the information I need. How easy was it to use? So we got a 3.96, five scale. I mean, that's damn good, man. And then the things I didn't like about it are exactly the same things I guess most people think about Moodle. It looks clunky, old, inconsistent, which now Moodle's got a design UX team. We're going to fix Moodle, aren't we? So yeah, we repeat that every three months. The initial design, we needed to test the initial design with learners. Whoops. Okay, user testing. How do we do user testing? So we do, we did once one moderated distance user testing with learners. Each session is 30 to 45 minutes over Microsoft Teams this time. Sessions conducted remotely. They're all recorded and we capture qualitative and quantitative data. I show this map, this diagram all the time after you test with five users, the kind of value of usability problems kind of starts to level off. Once you've got five, it starts to repeat itself. You kind of do three interviews and you're like, yeah, these people are saying the same thing as everyone else, even if you're testing from different demographics. And when you get to five, you're just like, yeah, they're just telling us the same information. Now, can we do something else? So yeah, here's what it looked like. That's me at the top. This is Roy, he was testing as earlier, being very nice and taking excellent notes. Really good. It gives you feedback that workshops and focus groups don't because users can speak honestly and openly without being biased by other people in the room. It's really powerful seeing real people try and use the system. I like seeing them go like, have wrinkles in their forehead or get a bit like, what is that? Why am I doing this? I'm watching their eyes because you're real empathy. You don't get unless you do real user testing. We had a facilitation guide. We asked them three common tasks from that top task survey. The designs, the tasks were designed to go through the main page elements and each page participants were asked to stop, tell us what they could see and think out loud. So we asked them whereabouts would you go without clicking? What would you expect to see there without clicking? Then we asked them to click and tell us what can you see? Was it what you expected to see? Just follow that. We also did a debrief survey afterwards in Microsoft Forms and asked them to fill in ranking the elements in order to importance. How useful were they and what else would they like to see? So some scores on the doors. Elliot, do you want to do this one? Yeah, landing page. We're testing the initial design. So this is the landing page where the user goes to log on. It contained a login link, a carousel, course library link, free courses link, new courses, site announcements. So let's see how it did. So users were primarily looking for their courses. Kind of makes sense. And they focused on elements that they thought would lead them there. But they actually often went to the wrong place. So an example is the course library. We didn't expect users to click on that. And they were looking for their courses, you know, the courses they were rolled on. Makes sense. But that was actually like a library of all the courses at UCL. So those 5000 courses, I guess they could browse through them, find their 10 courses. It's going to take them a long time. So not ideal. So we had to rethink this landing page for our demographic. But it's clear, okay, login link, get to the courses. That's the key things. The other things, you know, less important. Then the dashboard. Yeah, we tested this initial design on the left. Site announcements, I guess for the business, yeah, up there. Items accessed by my peers. Sounds interesting. Last access course, timeline, core Moodle, my courses update, search course catalogue. So these are some of the designs we put them in front of students in the interview. What came back. So again, the ranking of kind of what they found useful. So last access courses, my course updates. So again, you know, give me my courses. Where are they? And then some of the other elements down the page. Timeline was highly rated, which we liked because obviously, you know, that's just Moodle functionality that we can use. And so the dashboard, again, so participants just wanted to see their courses as the primary requirement. Anything that hinted at this was rated highly. And then the second most useful element was the timeline. Participants were unable to find their feedback in grades. So that was obviously a gap that we had. And some things such as, you know, items accessed by peers, that type of thing, that was kind of a distraction, as an example. People didn't really know what that meant. So we knew some good ideas of what to keep and what to change from these results. This is quite so okay. So I don't think this is for me. Maybe it's for someone else. There were lots of good intentions in the elements we are trying to put on the front page and the dashboard to give information that we thought was useful for students. But by the time you've read through kind of three blocks of information, and you're going, that one's not for me, that one's not for me. And you get to the bit that is for you. They were kind of often like in this state where it's like, oh, this mustn't be for me either. This must be for someone else. Yeah. Which was a bit of a shame. So the user journey that everybody wanted to do was log in, find their courses, open the courses, find the activity, open the activity resources. That's kind of it. It's very simple. I think that's it. And our lessons learned were focused on the primary tasks that the users need to do. We know what they are. We did the research, focused on the elements user need, making them prominent and better suited for the user needs. Because some of the blocks and things, they're not necessarily suited for UCL students' needs. Be careful of adding content that makes users feel the page is not relevant for them. Be careful of adding miscellaneous content, which just makes the user journey harder, really. This is Brad Frost. This is one of his slides. So this is what he described kind of a typical kind of corporate website as. And maybe that's what users were saying about our design as well. I don't know. But, yeah, this presentation is called Death to BS. And everyone should watch it. So he redesigned, focusing on the user needs. Here's the login page. We've got sign in and then we've got the business requirement of the latest news, which, yeah, it's cool. Everything the users need and everything that they didn't find useful, we put in the bin. That's it. Here's, on the left, the dashboard. And then my courses page, we've got rid of draws, put content that users want to see on the page, made a grid instead of draws. And it also feels more UCL. This pretty much looks like that UCL design system that we saw. Primary things users needed. That was last visited courses, deadlines and feedback. That's it. It's like, how do you find this anymore? Well, if you look with your eyes, it's there. Alia, you felt strongly that the primary content should be the pickup things. Yeah, explain it. Go on. Yeah. So this is a block that we built. It's a bit like the, you know, recently accessed courses or recently accessed items or activities. I would essentially combine them. And it was just a way obviously students were looking for their courses. And we didn't, we wanted to fulfill that need. But also we didn't want the dashboard to just be the my courses grid. We wanted some kind of, you know, surfacing of information that is relevant and helpful to the students. So I mean, this is just the obvious way. And then we've kind of redesigned it to look more like UCL. And, and, yeah, we've got a lot of positive feedback. I think we do intend to open sources. Jason, Jason, Alistair in the room, speak to them after the meeting. Yep. I think that was a yes. There's a timeline. We made it look like a timeline because I think it doesn't look like a timeline in middle. But when you look at other sites, the timeline looks like this. We cut out loads of the kind of redundant information and just put the facts in. I say that, but I'm noticing PAYO46 slash, I don't know. This is the feedback block. Students will like, I can't find my feedback from user testing. It picks up at the moment, turn it in and assignment. I think this is an issue trapper issue in JIRA to put quiz into it. So that'll either be me or Alex. Alex, can we put quiz in? No. Again, hopefully we'll be able to open source this, Jason. He's saying yes. Okay, cool. That's good. So when we showed these things to the students and kind of repeated the first lot of testing on their top tasks, and we asked them, what did you expect to see? Can you find it? Yeah, it's just there. There's no messing around. We've got news block again. That's it. And we started to look at course pages as well. So here you can see the images that people put on the course cards. We showed that prominently. We showed progress, the title, category. There's more information that'll go in here that's UCL specific. We haven't really done much with the next project. Jason, when are we starting formats? I can't wait. October. Yes, get in there. And we'll be interviewing the students and the tutors about how they use formats. But we want to look at other people's formats as well. Cool. I have a tell you. Okay. So second last bit, so on that evaluation. So as we kind of said, so with this benchmarking kind of UX research process, we want to continue it and constantly kind of build it into our daily process. So we've redesigned it now. We want to run through students through this and also course formats and start hopefully showing improvements and also generating new ideas for our team to develop. As part of this evaluation, alongside, I guess in parallel, but also feeding into the UX research, we have a few other things coming along. So this might be hard to see, but Microsoft Clarity Heatmap just shows you for the landing page on the left and the dashboard kind of see the red colors where we find our users clicking through. And so we can just use this to kind of confirm our design and also kind of track its use. So I mean, on the landing page, we've obviously got the login button is kind of the key of focus area, which is kind of what we expect, which is good. And then on the dashboard, we see the recently accessed or pickup where you left off block in the yellow, that's getting all the attention. So users are finding that relevant and using that. I think it'll be interesting in the like assessment period, we can track, say whether users are going to the timeline and the feedback block. So we can kind of see how our dashboard is being used. It's got a lot of other features where we can look at like whether users are scrolling down so that we can show that, okay, those advertisement blocks are working, or if they're not, we can do something about it. So that's alongside our UX interviews. This graph is also another aspect was like reporting from the moodle database tables. So course design includes the course image now at the top of the course page activity completion and also the course summary. So we're surfacing that information in the theme. Hopefully users see the relevance and start using that more, but we are going to promote it through our support and training teams as well. So this graph is just tracking that usage, which we can see is already going up. So hopefully continues to go up. And these kind of elements, which add to the kind of a nice course experience, we can kind of continue to promote. And then we've also got quite effective staff user group. So about 130 staff members. It's a teams group that we provide a lot of asynchronous communication. And they've essentially given rapid feedback on our demo environment and the theme as it's evolved. It's just allowed us to get feedback, ideas, constructive criticism. And you can see some comments here, but just to get a flavor that that's kind of another channel of feedback that we're getting. And then again, we're doing a lot of things. So this is a system usability score based on the standard set of multiple choice questions. And we ran it on the old moodle. And it got fairly good results. People like the old moodle. And then we're going to run it on the new moodle, hopefully show some improvements, because there were obvious areas to improve. So this score is kind of a bit divided. You can see from the pie chart, it was about the complexity of the old site. So users were kind of, some were saying it was complex. So hopefully with the new site, we can, because we have a clearer user journey that the theme is consistent with, hopefully we see improvements in these type of scores. And we can, we can kind of demonstrate it with this evidence. Cool. And that's it for the presentation. Thank you so much. And if there are any questions? Thank you for your presentation. It was very inspiring. You showed us a landing page with a timeline and feedback and pick up where you're left off. And I was wondering how you made it. Is it a native feature or is it a specific development? I'm sorry if you said it that I did, I'm here. It's a, it's a custom block and it should be open sourced. And Jason or Alistair, oh, sorry microphone. Hi, it's a custom block. And Jason or Alist, Alistair, are you able to, to kind of, if, can everyone see Alistair, he can show you where any of the UCL plugins are that are open sourced. And so you can go and try them yourselves. Is that how the links? Oh, we didn't do that. We're not that clever. Okay. Thank you. Oh, and this also shows courses that are hidden. The normal block doesn't, but say if you're a tutor and you're like editing a course, you can go back and you, this doesn't lie like the pick up way, like the Moodle block. This shows you hidden courses and hidden activities. So if you're a tutor and you're editing something, but it's not live to students, it'll be there in that list, which is just like dynamite for me. Congratulations for all this work and this great presentation. I wanted to ask, have you tested your new theme and the overall UX on the mobile devices and whether your students are using mobile devices and if they're accessing it through the app or through a browser on their mobile? We have some stats on this in our Google forms because we asked about this. Yeah, I think we find not a whole lot of usage through the mobile for various reasons. I mean, I guess we ask students and they check grades. They might check grades very quickly, but they're not doing the more intensive perhaps learning activities that show up. But we did test obviously like say the nav bar, which we've worked a lot on, you know, responsibly, that the theme wasn't messing up any of those things. I think that's also been a focus of our accessibility testing as well to be aware of how say, you know, it's a related topic, I guess, how the, if you zoom into 400% how the reflow works and things like this. So we've tried to use Moodle and make sure the theme doesn't change anything about that. Does that answer your question? The specific app, we don't support it. We don't promote it. I mean, you could use it, but it would strip out the whole theme I guess as well. In the cycle of UX design and testing, development and again, how do we work between the UX team and the developer teams? How do what's the pipeline from design system to actually code? I mean, so I guess we're part of an agile team. The main UX team are me and Stuart. So I think we both do the UX design, I'd say Stuart's probably the main developer. So we're, I guess, we're kind of embedded in the process. I mean, there isn't a separation really. So yeah, when we build prototypes, we don't necessarily build them in Figma. We tend to use CodePan or some of the live system because we like to test if the prototypes are accessible as well. So we like to test with live code. So we build live working HTML prototypes of things. And then when we once put them into Moodle, you just go copy paste into a mustache template and you have the same code that you're testing very quickly. Great. Hi. Thanks for the session. It was very nice. I enjoyed it a lot. We are going through the same situation basically, but with Moodle workplace. So we're moving from 3.12 to 4.2. And now we are in the prototyping phase. We're testing or we are in several iterations. And I wanted to ask you, did you fail anywhere along the way? Any tips? Would you do anything differently? Yeah. How did the whole process go? Because we've been eight months now in this. Thank you. Well, I guess our initial design had good intentions, as we said, and there were failures there. But I think a lot, some of those ideas, we have to flesh out more probably as well. So nothing was really like a mistake or we do differently. Isn't it failed fast? Yeah. We just failed fast. We could build the most high-value things and then the other ideas we knew. Some aren't going to work, but some just need to be workshopped and thought about. I think one of our challenges was the design system, the beta phase, because that's being changed. So, for instance, we moved, I don't know if I have an image of it. It's just kind of funny. I mean, one of the challenges was to actually just get the UCL logo image to the top right, which took a bit of development. And now we know the design system is actually okay with us putting it on the left. So, I don't know. I don't know. But we've done that now, so we're happy. So, when you're doing your prototypes, we did a thing where what we did was we made, we took a screenshot of something, like from Photoshop or PDF or anything, and then shoved it into Moodle as an HTML block and just said, don't click on it, but tell us what you think of it. And even some like bit, you sketch it on bits of paper, just stick them in as HTML blocks, just put an image in and then fake it until you know it's the right thing. Does that make sense? I don't know if that's helpful. I can't think of any other things at the top of my head. In that page, screen grab or in the other one where they have the overview of the courses, et cetera, and none there's the right-hand side block draw extended. What did you take on block draws or did you get rid of it? Are they important? What does your user testing show? Yeah, also in the former one, but maybe I just looked wrong. We took the block draws off the dashboard because they didn't test very well. That's it. I don't know if anybody else has tested with them, but on this page, certainly they didn't add value in it, as best as just being normal web people. I think when people visit Moodle, especially students, they think it's a website, so it doesn't behave like a website and it has weird things like block draws and whatever, then they get a bit confused and they don't use them. But if it looks like a website, then they're happy. It's this idea of prototypicality, why I'm on the web, just been to this website, go to another website, expect it to behave the same. If you put things in there that are a bit weird and not the same as other websites, then they're not going to like it or use it. Yeah. Does that help? We're keeping them on courses. We're just removing them on the pages where they are not useful.