 Hello, everyone. Thank you for joining us. Welcome to the Moodle Academy webinar, improving Moodle through UX research and data analysis. I am Sandra Mads, Moodle Academy Educational Events Coordinator here at Moodle HQ. My colleague, Mary Kud, Education Manager from the Moodle Academy team, is also joining me today and will help facilitate the webinar. Today, I would like to welcome Elliot Hovey, Senior Learning Technologist and Stuart Lamour, UX researcher and developer, both joining us from UCL. UCL Moodle is one of the biggest Moodle installations in Europe. It provides an essential arena for the teaching and learning of UCLs, 51,000 students and 16,000 staff. It is also an incredibly complex Moodle that combines third-party cloud infrastructure, third-party development, around 200 plugins, some local and some from the community, and internal agile-based development. Today, our invited speakers will outline the methods of UX and data analysis used at UCL in order to upgrade to Moodle 4.2 in 2023 and the results. They will further show how these methods can be adapted at other institutions looking to improve their Moodle. So over to you, Elliot Hovey. Thank you. Thank you very much, Sandra. Yeah, thanks for that. And hi, everyone. Bring up my last slides. So I hope you can all see them on the screen now. So, yeah, as Sandra said, so myself, Elliot and Stuart, both at UCL. And today, this is kind of a run overview of what we're going to run through. So we'll talk a bit about the UCL context that we're working in, a bit about the design system that we used in our building our Moodle theme and then the kind of research that we did. So top task analysis and benchmarking, testing the initial designs we'd learn as running design, focusing on user needs and then how we're kind of evaluating this going forward. It's going to cover similar territory to our Moodle Mood talk, but we're going to update you on some of, I guess, some of the progress. And then at the end, we can have Q&A. But as we go, if people have questions or thoughts, do put them in the chat and then we can answer them at the end, probably, and just respond to them. So hopefully that sounds OK. So the UCL context, if you haven't heard of UCL, essentially, we're a massive kind of blended learning, some online learning university. We need to update our Moodle to 4.2 in July 2023 from 311. And it was a good chance to kind of update the theme, the landing page and the dashboard of my courses. And this is kind of what we concentrated on. We've also just upgraded to Moodle 4.3 in January. So it's part of our kind of continual development. The business requirements that were kind of major when we started this project was to use the UCL design system. And we say it's in beta because it's still like an evolving document or resource. If people have their own design systems that are using, that'd be really interesting. So do share that in the chat if you have one or maybe even share it if you have the link. Or if you don't know what a design system is, we'll kind of explain that in a second as well. We also had a requirement to have marketing spots, like to advertise events and news to students on Moodle and to alert users when things are broken. So kind of basic requirements that we wanted to include in our upgrade to Moodle 4 and in our redesign of the theme. OK, so I said Moodle, you said Moodle is big. So these are some kind of, you can see a sense of the sale. So about close to 40,000 active users per day, 70,000 sessions per day, 5,000 courses in each kind of up to a year, and 150 plus plugins. So the UCL design system, hand over to Stuart. Cool. So this is a UCL design system. It's UCL's design language to use across all of its websites. Design systems are really helpful. You have one basis for your code, for your HTML, for your JavaScript, for your CSS, so that every time you build something, you're not starting from scratch. It means that everything you build is consistent, which is really useful for users. So that when they see a button on one page, as they see a button on another page, they recognize it's a button. They can see it has cards and a grid. It means that accessibility is built in whenever you build a component or an element on your page. And this was the UCL design system designed by Clear Left. It's in beta at the moment because UCL's design systems evolve to incorporate different needs. But we thought with UCL Moodle, we kind of want it to look like UCL. We want it to look like the students are still in the same system. So we wanted to apply this design system, this design language to Moodle as best we could. Next slide, please, Elliot. So on the left-hand side, you can see what the Moodle did look like that we started with. That was Moodle in 3.11 or 3.9. I'm not too sure. It's been three something, possibly two something, for a very long time. And as you can tell from the juxtaposition of what it looks like on the left-hand side of the Moodle and what it looks like on the right-hand side, which is a page from the UCL website, they're very different. Next slide, please, Elliot. Top task analysis and benchmarking. OK, am I doing this one or are you doing it, Elliot? I've forgotten. Are you? OK, cool. Top task analysis and benchmarking. So this is an industry standard process you kind of do. Basically, you ask people what their top tasks are. Why do they come to the website? And you get a large sample. You can also do data analysis and see where people are going in the click paths if you've got kind of that kind of recorded. And the idea is to improve those top tasks if people find that they struggle doing them. If something's difficult, let's fix it. But the top task analysis allows you to focus on the big problems, the ones that the most users have difficulty with and go, you know what? Those ones are our priority, which makes sense because you've got to spend money on doing these things. So what you do is you choose what to measure these top tasks. You find out what the main problems are for the majority of your users. You choose how you're going to measure the successiveness of your design. So it might be people can see the information clearer, people have less difficulty finding things, or it takes them less clicks. And then we rinse and repeat. You just re-measure it with your new designs all the time and keep testing and iterating. Hopefully that makes sense to everybody what top task analysis is, and benchmarking. Next slide, please, Elliot. So yeah, we surveyed 100 users. We asked them how often they visited UCL Moodle, what the main task they did when they visited UCL Moodle, how easy it was to do this task, what their favorite thing about UCL Moodle is. That's a bit of a general question, but just for fun, really, and what the worst thing about UCL Moodle was. I don't know if anybody's done the same with their users, but it'd be interesting to see. So this is what we found. Yeah, on the form you could see from the responses that the students love UCL Moodle. It was really nice to see. And this is with the old design even. So hats off to Moodle generally. Learners have Moodle open all day. So with everybody we interviewed, we found that basically they always had a tab or two or three with Moodle opening it throughout their entire day when they were learning or when they're doing work. We found out about the top tasks. So they're all based around course content, finding course content, doing activities and feedback, looking for feedback and grades. Here's some quotes, what people liked. It has all the information I need, ability to locate all information, and tasks in one place. And then we asked them how easy it was to use with a rating one to five. And we got a 3.96, which isn't too bad. I mean, that's pretty amazing considering lots of the content is user-generated content where busy lecturers are having to go into Moodle and build things that they might not be experts. But what they didn't like was how it looks and feels. So Climkey, I think that's a general thing that Moodle gets quite often. I think it's got bear with four. So we'll see. Old and inconsistent was the other one. Yeah. I don't know if that's the kind of feedback that other people get about their Moodles at all, but that's what we got. Next slide, please, Elliot. Testing the initial designs with learners. So we had some initial designs for what it could look like. And we wanted to test them with learners to see if the designs fitted the needs of our users. We did once one moderated distance user testing similar to this over Teams, though, because UCL uses Teams. Quick sessions, 30 to 45 minutes, conducted remotely through Teams, they were recorded. Ideally, you also want people observing the call who aren't necessarily on it to make notes for you. That's always really good. And we want to capture qualitative and quantitative data. There's a graph here on the right-hand side. What it shows is the amount of usability problems versus the number of users you're testing with. And you'll find when you're doing user testing that after you've done maybe three, four, or five, you start to get deja vu because the users are telling you exactly the same thing as all the other users. Every person struggles with the same things. So it's kind of like the curve. You can see here, once you get to about five, you've got 80%, a few, 90%, 80% maybe, of usability problems documented for every user that you will ever encounter. So I think at one point, we were testing with about seven people, but we were just seeing the same results, really, to be honest. So you don't have to test with massive groups to do usability testing. Five is enough. Next slide, please, Elliot. Here's what it looked like. So this is the initial design that we started with, the UCL Moodle. It was a theme that UCL had bought and it had lots of different bits in it. As you can see, there's me in the top right on my screen. There's our user who we've masked and there's Elliot observing. And we recorded all the sessions and watching the user click around and seeing what they struggled with, seeing their facial expression, hearing their tone of voice is really useful for getting empathy with your learners. Next slide, please, Elliot. So yeah, what we did was that we had a facilitation guide and you write this down and you could go through the same script pretty much with every person that you're testing. We tested three most common tasks from the top task analysis. We did, designed to go through the main pages, landing page, dashboard page, my courses page, course pages. We didn't delve too much into activity pages. We looked at the elements on site, so blocks, navigation, another familiar part. So the system to allow us to of where participants struggled or endured using it. And at each page, we asked participants to stop and not click on anywhere. Tell us what they could see and think out and we asked them, whereabouts would they go to complete the task? What would they expect to see there? And then they would click on it and would say, is that what you expected to see? See if their expectations match reality. Next slide, please, Elliot. Afterwards, we did a debrief survey with the participants. A series of quantitative and qualitative questions in Microsoft Forms. For each Moodle page, we asked participants to rank the elements on the page in order of importance or usefulness for them. Did they find it useful? How often would they use it? And we also asked participants, what was missing? Next slide, please, Elliot. So this was our landing page. As you can see, it contains a login link. That's the thing at the top right there. It's got massive carousel. Course library link. That's the one on the left. Free course link, help and FAQ. New courses and then there's also a bit underneath about site announcements somewhere. What did users think of this? Should we have a look? Next slide, please, Elliot. Jump in for the next slide. Okay, go for it. So this is part of the exit survey on the landing page. And so we found that users, so the students we tested were primarily looking for their courses and focused on elements which may lead to this, which often lead them, led them to the wrong place. Which kind of makes sense, what are students looking for? And overall, so the landing page content did not seem useful to the demographic we tested. In that sense, the design kind of had all these other elements in front of the student, that they were kind of either getting confused by or just expressing that that didn't make sense. So the other thing is like this course library, so these are the rankings in which people ordered how useful it was. And you can see course libraries number two. But that course library was like a library of every course that UCL, so thousands of courses. So the students were going there, but what they really wanted to obviously just to log in to get to their own courses, rather than see thousands of courses which isn't gonna be helpful. Similarly with free courses, UCL doesn't really have, well it has some free courses but not in the way that it was promoted on the front page. So again, that's why students kind of favored that. And then if you go down the page, these kind of other elements like a carousel of images, which is quite common in different Moodle themes or a Moodle that was ranked less useful. So this is like the login page, so it's quite clear that students want a login button and a way to get to their courses is kind of the key or the top task they're trying to complete. We then looked at the dashboard. So once you log on, brought to the dashboard a Moodle four and this was the kind of design that we tested and our initial design had a kind of site announcements at the top, items accessed by peers in kind of the drawer on the right, which would show like, here are some activities or items that your fellow students have accessed. Maybe you wanna access them too, so that was the kind of idea. We had last access courses, a timeline of deadlines, which is just the Moodle core kind of block. My courses updates, which would be like a my courses grid, but also showing when a course has been updated, some kind of indicator there, and then a searching course catalog. So these are kind of just the initial ideas that we had and we did kind of a basic mockup. Not everything was interactive. Some of it was just pictures on the dashboard that we tested with users. So how did this dashboard perform? So students ranked it in this order. So last access courses was the main thing, my course updates timeline. Items accessed by peers was kind of, we thought it was, well, I thought it was an interesting idea, but many students kind of fed back that they weren't sure how useful that would be and like if everyone accesses the same kind of assignment detail at the start of the year, does that just always show as the most like recommended? Item. So they kind of expressed that they didn't think that would maybe be that helpful. And some of the other elements going down and it's quite clear that again, like accessing courses is quite the key demand here. Just to summarize that. So yeah, for the dashboard, so overall participants just wanted to see their courses as the primary requirement. Anything that hinted at showing courses was rated highly. The second most useful element was the timeline, which we thought was great because that's something Moodle provides, so we could just use that. And then participants were able to find their feedback grades from the dashboard. So that was, again, we were thinking about in terms of whether we could serve as feedback or grades on the dashboard and whether users were familiar with this or would favor that. I think we'll talk about that a bit more. I'll hand back to Stuart on this one. Yeah, so this was a quote from one of the users. Okay, so I don't think this is for me. Maybe it's for someone else. So thinking about all those elements that we'd put on the dashboard and kind of like when students were going through it, they'd quite often skim an item. They'd go look at it and go, okay, this isn't for me. And then just skip over to the next one and go, maybe this is for me. And then skip over it and go, maybe this is for me to try and find the information they were looking for. And I think when you do that to anybody on your website, you kind of, the trust degrade as you go down almost. If people seeing irrelevant data, the next bit of data they're less likely to think is relevant or trust it. I think that's really important. Quite often we might overload people with data and choices and information just because they might need it. Whereas actually it's not good. It doesn't create a good user experience for them. Next slide, please, Elliot. So yeah, for the tasks we asked participants to show us the user journey they showed us was always exactly the same. They went to log in as quickly as possible, painlessly as possible, pure as clicks as possible, find their course, open the course, find the activity and open the activity and resource. And that was it, very simple. Next slide, please, Elliot. So these were our recommendations from the user testing. Focus on the primary tasks, users need to do. Focus on the elements, user need, making them more prominent. That are suited for the user's needs if they don't work very well, improve them. Be careful of adding content that makes users feel the page is not relevant for them. Be careful of adding miscellaneous content which makes the user journey harder. I think it's everywhere a choice paralysis where when somebody's given two choices, three choices, etc. the more choices or the more links you have, the more information on your page, the more likely it is that people will not make any choice at all or will just leave. I mean, with Moodle, they have to use it, but we are still doing the same thing to them if we just fill the page with loads and loads of redundant information. And next slide, please, Elliot. And yeah, this is Brad Frost. So he's a really great developer and he did, he works on lots of huge, huge websites and kind of this is what he found quite often. So this is from one of his talks called Death to BS. And yeah, this is something we see quite often. That's the content that people are looking for, which is quite small on the web page. And then the web page tends to get filled with lots of business requirements or those kind of political arguments that people have. This has to be really prominent. And instead of asking the users what they need, do you end up with this kind of thing? Yeah, just if you can control your content on your Moodle landing page, the sign in and the dashboard in my courses, don't do this to your users. Next slide, please, Elliot. OK, yeah, so. We'd seen what users thought of the initial designs. We'd evaluated it and we needed to redesign focusing on those user needs. So this is kind of what we did. Should we have a look? OK, so this is or was a few weeks ago, the homepage of UCL Moodle. Yeah, we've got the business requirement of using events at the bottom. And apart from that, there's kind of just one big call to action, UCL login and no carousels. There is a photo underneath. Next slide, please, Elliot. On the left hand side, you can see what the dashboard design looked like. We'll go into each of the elements in detail and how they help users achieve their tasks. And on the right hand side, you can see what my courses page looks like, which is probably similar to most peoples. Next slide, please, Elliot. What we're looking at here, we'll find out in a minute. Might be the pick up where you left off block. Let's find out. Yeah, so OK, this was a block we designed called pick up where you left off. It's kind of a replacement for the recent courses block. We found that the recent courses block has weird things with a carousel. Don't even want to go there. It didn't show people courses that were hidden. So if you weren't enrolled in the course as a newer teacher, it wouldn't show here. And that wasn't very useful. And we just wanted to make it kind of responsive, feel like it was part of UCL's brands because this just exactly the same as the UCL brand system and provide the information that students needed. Next slide, please, Elliot. So we had two options on the pick up where you left off block. We've got courses at the top, as you can see, shows the image from the course and your progress on it. And then there was a second link underneath called activities, which allowed you to carry on with an activity or if you were a cheater, just find very quickly what you were last editing or whatever. Next slide, please, Elliot. So it's a pick up where you left off block. We've got two other elements on the page. So on the left-hand side, you'll see what we did to the timeline. This is the middle timeline block. Students actually commented this should be called deadlines, not timeline. But there you go. We took out lots of the information that people didn't need and just kind of... I think everybody's familiar with what this is. It looks like a timeline. And we wanted to make sure that it looked like a timeline to students. Yeah, I don't know what more to say. And then on the right-hand side here is a custom block we made called feedback. And it says feedback for Stuart because this is a screenshot from Stuart's page, which is me. What it does is it goes off and it gives you a link to the activity that you've received any feedback from. It would show you the name of the cheater in the picture if that's allowed by permissions because students really like getting personal feedback from cheaters and having meant deference to their teachers and things. And it just makes it feel really personal as well, which is important. And then underneath that, you've got a link to your full grades page and things like that. But clicking on each of these links just takes you straight to the activity. Yeah, so we wanted to solve the issue that students couldn't find the feedback. We think this might do it. Next slide, please, Elliot. And using events. Skip, go. It looks like UCL's using events, that's fine. We did it. It looks like a design system. That's all we need to do. Okay, so once you get to a course page, we haven't done much with the course pages at the moment, but well, we've just done a massive research project into formats, which was pretty interesting. And that's a whole another presentation. And maybe next year at Middlemoot, we'll see. But what we did do to the course page was add a little bit of eye candy so people can put in a big picture that shows. We show the category, the name of the course, the course description, and the progress. And then kind of tidied up some of the other bits to make it match UCL's brand a bit more. I think that was about it, really, on that one. Next slide, please, Elliot. Evaluation. Yeah, so I'll jump back in. So just on evaluations. So as Stuart said, so we've got this kind of, we're trying to really embed this benchmarking process. So to run these user interviews with at least five students regularly in our, I guess, in our year. And that's part of our agile approach to the development. So we're constantly iterating and releasing, hopefully improved and new kind of developments on Moodle. So we ran our initial session with the kind of mock-ups of what the theme could be. Then we kind of did the theme work and the dashboard work. And then we ran a second round of interviews in November last year. And so instead of the kind of mock data, we had the live Moodle site with the new login page, the dashboard page and the course page headers that Stuart has taken us through. So how did we do? So this is an interesting one. So this is like a poll. So similarly we polled about 100 students and we asked them a series of questions, one of which was like, what's the worst thing on Moodle? So in March, this is kind of a word cloud of their free text responses. And then we re-ran that thing in the same question in November. And then you get a different word cloud. And you can see in the first one, there's kind of slowness being logged out and time is kind of a key issue. And then there are things about kind of lesser things about maybe the old thing being a bit ugly, for instance, or inconsistency. When we re-run that survey, the slowness and logging out kind of disappears, which is good. Now, I think that was a suggestion that Stuart maybe kind of led on about increasing the session time out on Moodle. I think we had it at two hours and we increased it to 48 hours, something like that. So kind of soft. When we were doing the interviews, there's a student with three tabs open or with Moodle open and then they're on a call with us and they go back and they're like, oh, it's logged me out again. And it's kind of like, wow, we're really doing that to students. That's bad. Let's stop that. Yeah. So part of these interviews, we were focused on kind of the theme, but then other issues kind of emerged. So it was really valuable to just sit down with users and talk to them. And you can see the issues in November. So again, the worst things are around pages and courses. And that's hardly why we have, or fits very well with our course format work. So how can we, you know, improve course formats so that they're more organized, easy to navigate, more usable to students. So this, I guess it shows we still, we still got challenges that we're working on. But this has given us key feedback that we can then put into our, you know, development and our projects and things. People mentioned that we left off block. We had quite a bit of feedback from students in our November interviews. The main one was that we initially showed three courses, but students mostly have four courses. So they wanted to increase. So this is like the new design that we're going to launch very soon and then evaluate it. And then now the feedback was maybe that the activities people didn't necessarily realize the activities was there, so we made it more prominent as these kinds of buttons at the top. So yeah, again, we're trying to reiterate on this approach and then evaluate and see how it goes. In the, so the exit surveys for our new rounds of students, we had some quite positive feedback. So obviously the visual appeal, which a lot of the work we did kind of related to kind of increased in their score that they gave it. So 3.75 in the March interviews to 4.25. And so the content designer, the front page and dashboard page had been updated since the previous test. Navigation and findability remained the same at 3.25 and we'd like to improve this. So while we're looking at course formats and other little things around moodle. And then the ease of use for elements increased from 3.25 to four. This could relate to the participants' increased familiarity with moodle four, given they had more time to use it. But yeah, that's pretty promising. We also did this kind of visual design, I guess adjectives that the students rank, whether the design is more or less creative, calm, busy or all these different adjectives that you got up on the page. We're happy to find that the highest move was on inspiring, which is good. So from 2.75 in the old surveys to four. And then this table kind of breaks down. So the items on the left, the adjectives went up. So people kind of ranked them. The new design is more creative, more cutting edge, more fresh. I think someone put in the chat fresh as well, which is good. We think it's fresh. And then down some interesting things. I mean, calm went down, which is interesting to interpret. But people didn't necessarily think it as cheap or old or unprofessional or inconsistent. So they thought the new design was, the opposite of these things, so it improved. And there was no change on busy and trustworthy. I guess this gives us a sense of the kind of how students are perceiving the visual design in terms of descriptive, but somewhat kind of emotional level. So it's quite helpful to see. Generally quite positive there. One of my projects at the moment is around data analysis. So we are using different tools to try and see how users are responding to the new Moodle design. So in addition to the interviews, we have Microsoft Clarity, which is a heat mapping tool and also does different web traffic analytics. So this is a heat map of the dashboard. You can see where the red is indicated is where most users are clicking. So we find that the first thing is that first card on the pickup block and the second card. And then our users are going to view all courses and then the third card. So we would suggest, it kind of confirms that the key task is to get to courses on this page. And we're gonna be evaluating this because it's clear that kind of students go into the initial courses they've accessed, but then also go to the view all courses. We just need to be careful that we're allowing them to access their courses as quick as possible. And that's why we're doing the four courses here to hopefully get the balance there, right? Also quite promising is that you can see the red further down the page. So some interactions with the timeline block. This is obviously gonna occur at a less frequent rate than courses because it's the timeline blocks gonna show the deadlines when students have them. And there's also some interaction with the feedback block on the right that you can see, which is also promising. So again, we've got evidence that when students are getting their grades, they're starting to go there and see content. So we're gonna keep an eye on that. This graph is just showing that, so in the theme we kind of promoted in the course page header that it showed an image, it showed a progress bar if completion tracking was enabled on a course and it also showed a bit of the course summary. So we're promoting that in the theme. And this graph just shows you that these elements are being used more for this academic year, which is good. So particularly that kind of orange or tan line that's gone up, which is more users using course images in their courses. And that's something we promote at the top of a course page. So that looks promising. So if you have any questions about how we're evaluating it, do put in the chat. I might hand over to Stuart if you wanna talk about course formats. Yeah, so we've been looking at, as said, we've just finished a big project on course formats. And this is kind of a preview of what our course format might look like in the feature. So that was it really. We might have two course formats because we might have one for those kind of courses that people put in middle that are really how to use the library and those kind of information things. But this is kind of gonna be probably what we do for blended learning. And is the next slide back to that to the UX workflow top testing thing, Elliot? It's Q and A. You put the UX testing workflow back one wherever that was. It's fun to meet. Yeah, so I just wanted to say, like on this one, I think we kind of missed it somehow and when we presented, but with each of those elements, the pick up where you left off, the timeline, the feedback, you don't just put them all live and then see what happens when you're developing each of those individual elements, you test it with users as you're going along. I hope that's clear to everybody from the diagram that you kind of start with something that's a simple wireframe of what you think is a good idea. You test it with users to see if it works, if it gives them all the data that they need if it's easy to use and if it isn't, you fix it. And so then when each of those elements is, yeah, they can all be improved iteratively and we test before we put it all just go, it's all live now. Yeah, you test during design as well, if that makes sense. All right, so. All right, thank you. Nice page. Thanks everyone. I hope you enjoyed the talk. We can go through the chat from the top or don't really mind. Maybe do that. If people want to jump in on the mic, we can, or what in the chat, we can answer those questions. So I'll just look back over the chat. And the first question was from Russ Ward, who asked, so when you say 100 users were surveyed, is that all roles or specifically undergrad students? I can answer that. So we put our survey just up on Moodle as a link to Microsoft Form, so just on the homepage. So anyone could kind of access it, but we advertise it as for students because we really wanted to target them. And then we asked them what kind of, what was their enrollment at UCL? I'll perhaps look up the stats of people interested, but it was mostly, yeah, undergrad students, but with a mix of post-grad, post-grad and some research students. And then we asked them as well, series of questions, and then if they wanted to attend a follow-up interview. And so we were targeting very much a different range of students. And then that goes well with the next question. So Xavier has asked, did you have different personas or just cleverly picked your student sample? Do you want to answer that, Stuart? Yeah, we just tried to be clever with picking our student sample. Cool, and then Fabian's asked, where are you using Boost or another theme? Would you create your own theme? That's a really good question actually. So we did start with a different theme. Might stop sharing, because maybe we don't need to. So we worked with Titus Learning on their Norse theme. That was the initial theme that we were using at UCL. And they were super helpful in working through our ideas and developing those initial mock-ups. So we worked with them to do that. And then we have now undertaken that Stuart can perhaps talk more about of building a UCL version of our theme. Just because we found that our requirements have kind of maybe diverged a bit from what the Norse theme does. So we're kind of very much in a lucky position to build our own UCL theme. Which we will open source. Yeah. And similar to most of the plugins, I think all the plugins actually you've seen today, will be open sourced pretty much. Certainly the pick-up block, for instance. The back one is, there's a few people using the pick-up block already, I've noticed. So yeah, that's good. So another question Vanessa, we might have answered it, but what theme do you have on your Moodle? Are you using a custom theme? I think we've answered that. What else we've got? What system did you use for analyzing your qualitative data? So we mainly looked at ourselves and I guess categorized it into different themes, top usability themes based on frequency. So we didn't use any other tools per se. I like using the Rice methodology. Google it. That's a really good one for prioritization. That means ask kinds of extending session time. None that I'm aware of. So far everyone's happy with us. Yeah, I guess if you leave your computer open and you're using like a computer in the library, a shared computer or something and you forget to log out. Xavier's asked, are you using Iotracker? I thought you would. No, we didn't use that. We might use it in future because some of our computer science departments would lend us their software and systems, but no, haven't used that yet. If you've used that, it'd be really interesting, maybe put in the chat what you've used before or found. That'd be helpful. Katya has asked, how many designers, developers worked on the new theme? That's a good question. So, I mean, we initially worked with Titus, but I would say most of the work's been done with myself from a kind of product owner of the theme, planning, user engagement side, and then our main designer developer is Stuart. And then we do have a development team, that they've kind of chipped in on different elements. But yeah, so here's the pickup. We are a third block available to the public. Yes, it is. Are you trialing the course format first for any particular faculties? Are you expecting a period where all courses switch to a new format over summer, automated or manual? Might of my own assuming that. So, we've done a research project where we again interviewed students on a series of course formats. We targeted the most used ones, which are topics, kind of the tabs format, and also kind of grid, more visual display formats. And this was more of a research project, which out of that, we produced a presentation and also options paper for senior leadership. That still has to go to senior leadership to approve. So, we haven't necessarily agreed at UCL. We very much hope they agreed to building a new course format because we can see how it'll improve a lot of the issues students are having. If we do, if that does get agreed, I think we would then make it part of our automatic course rollover process so that staff could convert their old courses to this new format. And we provide support around that. And we would be trialing it. We would make it available when people do that course rollover. So yeah, summer and then leading up to the new academic year. If people have moved to a course format, that would be awesome too. It's like a single course format, that institution that would be good to share. How you communicated that, because we very much haven't started that planning. We're still waiting for approval. Okay, Vanessa's asked, do students land on the dashboard after login is the home page before login? Login takes into the dashboard. Yeah. And the home page is before login. So what we called the landing page was before login. But yeah, once students are gone, they go to the dashboard. Leslie's asked, what were the students' thoughts on the course index and block drawer within course pages? We're finding that our students are not using these or they don't know about them as they are not open by default. You wanna answer that, Stuart? I think it's another talk, isn't it? We will show our research on course formats at a different time, if that's okay, Leslie. Yeah, I think in general, we found similar results to what you were saying. Maybe the other way around, because I think they are open by default for some reason in all moodle. So then students were getting tied up in them and not really paying any attention to the main content and going, I'm meant to use this. It's open and kind of failing. They're open by default, but then they change to the user's preference. So if a user closes them, they don't reopen when they go to the page. So I think we found many students, most students probably didn't use them, probably closed them when they went to a course and then hadn't, like in the interviews, certainly didn't reopen them. So yeah, it would suggest they're easily lost, but yeah, we could go into this a lot. The block drawer having like stuff for power users and admin in, I think is really good. You know, it's stuff you don't need all the time. You see how we've got loads of custom admin teacher, editor blocks and things and having those somewhere. So it's not just taking up room, distracting for the users is really good. And the course format page we showed briefly did have an idea of maybe having a menu, course menu on the left-hand side, which potentially it would always show. So students don't lose it and would only show kind of section headings rather than all the content on the course. I think a lot of the issues with the course index menu has been like we have lots of content on our courses, which it's perhaps not designed to do and it just shows them, shows all that content and becomes, some people have said like a second scroll of death. So yeah, we're definitely very aware and interested in that topic and trying to improve it, I think, these two elements. Amy's asked a good question. So I'd be interested to hear how you address accessibility and inclusivity in your design. Did you include a diverse range of students in terms of neurodiversity disability, for example? Stuart, do you want to access really this action of yours as it's- No, for the second part, we didn't test with any users with any range of neurodiversity at all for accessibility and I'm really passionate about accessibility and I've done talks at quite a few accessibility events in the United States. Used to work hand in hand with a developer who was blind and yeah, when you build things, you test for accessibility as you're building them. You don't make them inaccessible by design and then when you launch them, they're accessible and that's it, but you kind of just build. Accessibility, accessible by design is our solution generally to things, I think. There's also a really good team at UCL who do kind of will run the lighthouse checks, kind of that can run saying things like, the area role is different from the button name and the contrast isn't high enough for muted text on a dark background and I think UCL generally try and report those to Moodle Tracker or get them fixed by Catalyst if they're kind of, because there's lots of them and more core issues. But when we're building stuff, we just build it accessible by default. Yeah, so our accessibility team did audit the theme and then we either resolved those issues. Most of them we resolved or logged with Moodle and the issues we couldn't are in our accessibility statement on Moodle and we've tried to, through the accessibility team, I think, tend to liaise with students with different needs and kind of feedback to us and then we also have a staff user group that we've encouraged a range of different students, so staff, so in needs to join and provide feedback through there. I think it's always a challenge, we could definitely improve, but so far feedback's been good and our accessibility statement is pretty short, actually, there's not too many issues on there, which is good. So Xavier's asked more concurrent users more expensive to host, yes, about the session timeout. Not sure, we haven't heard anything back about that, we could look at it. But yeah, it's good, possibly. He's asked, is there any plans to create the course module visibility as a looped format? Students are able to view the course modules after the finished slash previous configured one and the accessibility to the modules is vaguely currently. Don't quite understand, don't know if anyone else does, but if you wanna ask your question and the mic, if that is a question, not sure. Tim's asked, was there a reason you concentrated your time on the initial part of the user journey rather than in course experience? I think so we wanted to, so we say with Moodle 4, we're quite kind of, users to get into the courses have a clear journey that we found, so that was the login page, the dashboard. So we concentrated on those initially, just because if you get those right, that's the first start of most students' journeys, most staff as well. And then now we're very much looking at the in-course experience, so this is what that whole course format interviews were and survey sessions that we ran. Hopefully we can report on them in another meeting or at Moodle Mood, because yeah, we followed a similar methodology, but also kind of streamed our surveys to a group of maybe 20 to 30 staff members that could watch and see students navigating through a course. So that was kind of really impactful and led to a lot of really healthy discussion with our different kind of digital education, e-learning support folks and staff around UCL. Yeah, formats are more difficult. One doesn't simply mandate to a university that every tutor must use a format or you'll get a riot going on, won't you? You have a rebellion. You've got to kind of manage it and I think one of the best ways to do that is to get the people who are building your courses, designing your courses, department heads, anybody else to actually watch how users use the Moodle courses, where they're struggling and understand the pains and complexities of their end users. And then you might get memento towards moving to better course formats, but until you've done that, maybe it's a bit difficult to kind of mandate everybody has to do one thing. Yeah, nine formats to three, that sounds great. Wow, more than nine formats, I didn't, but maybe we can answer in the chat now. Nine, oh my God. Again, she was paralysis. Did your tutors have a degree in Moodle course formats to be able to choose which ones to use? Yeah, I believe we have five formats, I'm right, but they used to be six. I'm not sure. I think we have to hand back to Sandra and Mary, but people can go to contact details. We'd love to continue the discussion. I might just answer two quick questions. Russ said, did you use a Moodle app? We don't really promote the app. We have our own UCO app where people use the browser on their phone. That's a whole nother talk as well. And then we do have a separate Moodle instance where we tested these things. So we have tested instances. Cool, thanks everyone for joining. Really, really good to hear feedback as well and to share our findings. Thank you so much both of you for being here. It's been great and thanks for all the interesting questions. I guess we could continue going on for a little bit, but it's good to know that, you know, you might have some more content to share with us, maybe at a future Moodle Mood Global. So that would be good to see. But if you have enjoyed this session, we'd love to, we'd love you to consider getting involved further and help us grow by contributing to the development of Moodle Academy. You can do this by visiting our Get Involved course, which you'll find on the front page of the Moodle Academy site. You can suggest ideas for new webinars and courses and you can vote on ideas that have been suggested by others already. We're always in the lookout for community members to help present webinars and co-create short online courses. And we'd love you and we'd love your help, basically making Moodle Academy more inclusive. So if you're able to please jump in or translate Moodle Academy course and get started with helping us translate our courses and webinars into other languages. And of course, please help spread the word about Moodle Academy by telling your colleagues about the courses we offer and the events we run. When you complete the course, you will earn a badge. You will also receive a badge if you present a Moodle Academy webinar like today's session. Educators might like to think about getting involved with the Moodle Educator Certificate. You can take the are you ready for the MEC quiz and one of our certified service providers will support you through the certification process. So that's all from my side. Thank you so much for joining us today and I hope you find this useful. And I also hope to see you on the Moodle Academy course forums and in our coming webinars. So thank you again for joining us today and thank you to Elliot and Stuart for being here today. And we hope to see you soon at future Moodle Academy webinars.