 Good afternoon, I'm Sharon Woney. This is Tim Hunt and we're going to talk you through some of the things we do to look after our Moodle base learning systems Tim introduced me I am blaming Tim to the free as in puppy metaphor a little while ago as I'm sure you're a Appreciate if I take my new puppy Mootie home install him in a comfy bed with some water and some snacks nearby and leave in He's not going to keep going for very long He's not going to be a usable pet. He's soon going to get all sorts of bugs and he might well fall over We depend on Moodle as the base platform to our learning systems The Moodle license is free, but our investment in our people and our technologies needs to be managed We're going to take you through some of the task systems and tools that work for us We've been using Moodle for a long time and our ways of working are continually evolving So where do we start managing Moodle I? Work in the online student experience team who alongside our learning systems team are responsible for the product management Tim works in our IT development team We have numerous hats from research to development from design to support We work closely with each other every day as one large multi-skilled collective There are very distinct roles and responsibilities and jobs across the teams But Moodle administrators like Tim and I cross both as do our Moodle managers We share the responsibility of the administration tasks with admins partnering up on life system changes Basically, it's an extra pair of eyes Most admin naturally sits within IT But some things like settings and roles and permissions actually sit across both teams We develop and enhance our learning systems using our user-centered design process Which includes user sent user testing usability and accessibility testing This has been really successful for us on our larger projects such as developing a new theme and course format But these stages are also used for small and medium changes like installing a new plug-in such as student quiz Monitoring all of our new changes and tools as they're adopted and used is vital for us We use the ad hoc database queries plug-in to keep track of a number of different things For example, where new features are being used and how many people are using them I use the automated daily and weekly reports a lot and they're now a vital part of my job There's more to administering the systems than well more to looking after our systems than just the administration We run several instances of our learning systems from community and staff only servers to the open science lab To our VLE for students in secure environments Our main server learn to delivers all our core teaching and learning We use 55 roles on that server to manage over a hundred and eighty thousand active users a year We have 450 live courses live at any one time and alongside our Qualification tutor only and learning related websites and all their content and tools Those roles organize who goes where and does what? 55 roles I admit does sound like a lot But managing our courses website and content Involves a lot of people to make sure that our end users get the best online learning and teaching experience we can provide Most of these teams and roles have actually existed in the organization for a long time from the days of TV correspondence courses and online Conferencing this isn't an organization chart staff across many units have a number of skill sets and responsibilities that are evolving as our use of technology in the learning systems increases For each of our courses we've got a variety of staff in editing roles creating and updating the content Our system roles work across all of our websites with our managers looking after Configurations and settings and some very specific roles like only being able to administer assessment tools or update library content and Non-editing roles are available to our help desk and support staff Moodle is a core part of some people's jobs for others. It is quite a small but very vital part We can allocate users at system category course and activity level Most of our users will have multiple roles in multiple levels to make sure they have the right permissions in the right context For example our tutors will have an automated system level role that gives them read-only access to all of our courses and their content They will also have an automated course role and group allocations on specific websites Which will then give them full access to all the group and collaborative and moderator tools Manual roles allocations are focused around editing roles at the course level and Support and administration roles at the system level Our roles always give us the opportunity to fine-tune all our activities through configuration settings and permissions in order to deliver the perfect learning activities. I love roles capabilities and permissions simply because of the flexibility they give us for all our different uses I do worry that sometimes I may irritate our developers. I Constantly ask is there a capability for that every time we look at a new piece of feature or functionality? We review our roles and permissions as part of our quarterly Release cycle. This is now semi-automated with a simple set of emails and a shared changelog This is grown from manual data extracts and some Excel magic in order to place spot the difference with roles and permissions across our servers The edit roles by capability plug-in is the perfect way for us to manage these updates it allows us to quickly and accurately make these changes on the live system and You can update all roles for a single capability at one time Over the past 13 years Our learning systems have grown around Moodle. We've got a large number of plug-in summer Moodle summer our own We've got a large number of internal and third-party system integrations And we've got real-time data Populating our systems 24 hours a day seven days a week We do have some key tools and processes to help us manage this at scale Our students have access to their course websites for an additional three years After the course has finished in read-only mode So for us, this means each new cohort of students for each new course requires a new clean website We have our one-step roll forward feature Which allows us to back up and restore a course with some additional automatic configurations For example the users and their content are removed, but the groups remain They're automatically renamed for the new course and remain attached to all their activities The roll forward also adds a new workflow to each website, which Tim is now going to take you through Thanks, Sharon So as Martin will be the first to say an important thing is we want to empower our Educators to use all the power in Moodle to set up the teaching and learning that is what you know right for their students However, Moodle is also a very powerful tool and not necessarily easy to use and we want to avoid industrial accidents where people lose a finger or a vitally Expensively created activity in the gears and the tool we have to manage. This is our workflow block it's a block that sits at the side of the course and Manages the flow between our admin staff creating the shell of the course and then our academic staff setting up Staff and then the activities and then You know people checking it before it goes live to students The all of these steps in the workflow are configurable So normally the workflow that we have for each faculty is a negotiation between the admin team and the faculty about what's needed And the workflow system can make some changes automatically So for example when the checker says they've completed checking the site It's then automatically made visible to students The system can also send people emails to tell them it's their turn to do something on the site now and Of course for our online services admins who are having to manage the production of all the modules in a faculty We give them a report so they can see it to glance the state of the workflow in all those courses and they can make sure everything's on track So we've started from the people and then we've gone to the Learning content and we're going to sort of head further and further in the techie direction So this is the learning. How do we develop the system behind it? And you know, we use all sorts of tools For doing that development, but what we develop comes out of the user centric design process that Sharon talked about That could be a huge project like develop a new set of websites for each qualification Which would be hundreds of tasks down to a single small task like add a new button in in a particular place in the forum If you've not seen how modern agile software development work, then basically it's all powered by post-it notes Each each sort of request that's come from Sharon's team gets put on a post-it notes and tracked across the board as it's developed review tested and then released We also use tools like Azure DevOps bug database to track this So as Sharon said we release three times a year four times a year every three months And it's really hard as we heard earlier for example in Caroline's talk to find any time when people are not teaching In fact, our course presentations do overlap So people are teaching pretty much every day of the year students logging into our veerly on Christmas Day for some reason So this rather amazing chart is our attempt to look at all the critical factors that affect when it's better or worse To do a Moodle release and as I say, we've just about found chinks in there every quarter To do releases now As Caroline was saying you need to prepare these things having done a very agile development process We then have really what's quite a waterfully final release process where we do several test upgrades We test all the new functionality. We test all the old functionality to make sure it hasn't broken We do the process of auditing all the role definitions and particularly any new capabilities and then hopefully On release day it all goes smoothly I've said there's no good time to do a release and we've now got it down the on the morning We tend to release his first thing in the morning before the start of the real working day and the main VLE is now offline. I think for less than two hours And of course because we're doing a new version every three months Well, every six months, there's a new version of Moodle core and we want to integrate it And what we found is invaluable for doing this is all the automated testing built into Moodle And we make sure to build automated testing into all our plugins. So this graph We're going to release Moodle 3.6 in June. So about a month ago We merged Moodle 3.6 into our development version and a whole bunch of automated tests failed Well actually 70 tests failed out of about 1900 So that's not too bad and the point is that tells us immediately. What's breaking so we can go in and fix it And as you can see it only took us a few days to fix almost all the breakage that came from that code merge So if you're not investing it if you're doing development and not investing in Moodle automated testing You're missing a trick and Moodle HQ have made all the scripts you need available So continuing the move in the direction of technology. Let's now get down to the servers This is where we've got to today and these are just the servers for our main Moodle instance where all the course websites live Of course, it's by far the biggest and this is just the servers for the live site We also have other servers for test instances again We heard earlier how vital it is to have test instances where you can test upgrades and other things It's lots of servers But in a very standard configuration with multiple web servers and one big chunky database server And it's no good just having these servers with the software on you've got to know from minute to minute Is it working for your students? So you need a lot of automated Monitoring and again all the tools you need are out there things like Nagios to monitor the very low level What's the CPU load on the server? You also want to measure the Experience your students are getting so the bottom graphs are showing sort of average page load times Which are fairly consistently under half a second which we're quite pleased with The thing about this monitoring is if something starts to go wrong it very quickly shows up in wiggles on those graphs And if something is seriously going wrong We have a major instant process with a major instant manager who deals with major instances across the whole IT department Who will coordinate getting all the right people together to fix it Which normally includes server people and application people like me Monitoring the live system is another area if we go back to the ad hoc database queries report We've got some of these automated reports that run every day and just check that certain business processes in the system are working properly And flag up any exceptions and if one of those exceptions happens it's going to email and admin So they can work out what's actually gone wrong and fix it So that's it really We may have just completely terrified you and we should point out we've been doing this for 13 years And we didn't do all this from day one This is a set of processes that have grown up over the years Basically you hit a problem and you think about it and you come up with a solution and you add that to your suite of processes But really it's just about having the right people in the right teams with the right tools And you can build it up over time And also you're not on your own If you engage with the Moodle community, if you come to events like this you can hear about the issues other people had and how they sold them So the Moodle community is a really great place in which to develop all this good practice So thank you very much for your attention Thank you Tim and Joan Does anyone have any questions for these two? Just raise your hand and Bob or Helen will come and bring the microphone to you Oh we've got two, we've got one over there as well Bob maybe you want to have the microphone over there Tim just a question, you had lovely pretty pictures up there which I've always drawn to But with your graphs and you had your load times and all that Do you have a set of standards that you'd be willing to share? A set of standards that you'd expect, for example on page load times and all that sort of bits and pieces? Yeah, so something I had to gloss over For every release we will do a load test that compares the old version of the code to the new version of the code And we basically expect each new release should not make it slower Most of our big performance wins have actually come from other places When we move from PHP 5 to PHP 7 that got us a performance increase And we're about to move from Postgres 9 to Postgres 11 And we're thinking we're hopeful that's actually going to give us another little performance bump So we kind of know the average page load time should be under half a second And if that started drifting up we'd want to know why Our actual service level agreement counts the proportion of requests that have taken more than three seconds So you're looking for the really slow outliers and that's supposed to be less than 5% It's actually less than 1% of requests And again if that took an uptick people would notice And we'd be asked to look into it So basically we don't ever expect it to get slower because computers are getting faster But in terms of actual baselines like I don't know CPU being 67% or anything like that Like you don't have thresholds Well we've got all those servers and they're split between two computer rooms And if you looked closely the CPU load was kind of peaking at 50% That's because ideally if one, well ideally one computer room wouldn't go down But it should be the case if one computer room falls over the other one can handle all the load Thank you We've got another question over here Just to say thank you to you because as another institution it's always nice to see behind the sort of What we present to the outside world about how we're actually doing it And I think you guys shame the way you're doing it helps other people validate that there is A right way to be doing this and where we should be moving towards with our own hosting I don't think there is a right way For instance there's always things you look at and think that could be better So we're currently thinking of changing from Memcache to Redis We're thinking about moving the whole thing to the cloud You always have a kind of wish list to think you know wherever you are there are always things you could be doing better But you just as I say you have to pick your battles one thing at a time Yeah and I think just to say it's really helpful to see that So we can have our own right answer based on what you guys answered So thank you very much Just while we're getting another question ready Brett and Peter if you could make your way down here Helen there's one in the middle of that Hi there. I was just wondering what you're using What software are you using for your regression testing? So built into Moodle is the Behat Automated Acceptance Testing System So that's basically all we're using And then we do have manual we have human testers who are also doing sort of exploratory regression testing Because while Behat will test the functionality it can't notice if like the CSS has changed And suddenly it looks wrong But it will tell you if it works Okay Martin go to the question No I just wanted to comment and say I just thank you again for the really nice way you work on these blocks and plugins And publish them and they're like nice standalone useful community things So I just want to say thank you that is the right way Thank you very much so Any more questions? Okay well in that case let's thank Tim and Sharon again