 Good morning everyone My name is Steve Bader, and I'm the associate director of open source applications at the distance Digital education learning technology applications at NC State. We like to call ourselves Delta for short So that's pretty long. I started with Delta 10 years ago when we Delta needed help upgrading from 1.9 to 2.3 And every year since then we've upgraded Moodle with our upgrade process, which we've worked on During my tenure, I've had the privilege of working on the local course copier, which copies courses across instances of Moodle a web services that that push Student grades into our student information service and several plugins around gamification within the course and the event system As a part of a grant recently I've gotten to program the Moodle course roadmap, which there's a poster presentation on tomorrow Recently we've adopted the mod zoom plugin and my team is the maintainer on that and we've been pretty excited about that Oh, no, I'm a pretty big fan of Moodle So I'm here to present to you the guide that we use to upgrade Moodle It's a refined process that changes every year because we usually get a Throwing a curve ball of some type and it either proves that we need this type of process or that we need to add More more to our process This year we learned that there were millions of questions in our and our questions table that were growing out of hand There were random questions So now we look for crazy growth in our question table. So starting with our timeline we Begin our process around the end of September and we started with a very important planning process and then we get when we get our Hands-on Moodle in the November release. We'll start making it our own. So we'll add plugins. We'll add our our commits and We will add outside plugins from other vendors and our core modifications And if the train's running on time at the turn of the year, we'll look at config The new settings the remove settings. We'll look at capabilities the ones that been added and removed and Sometime in January, we'll get our first version set up for our our department to start looking at and creating support materials Hopefully by the end of February, we'll have a working system ready to go and we'll stand it up All right We stand up one instance of Moodle a fresh instance and then after that we work on an upgrade for our two perpetual service And we'll practice that from February to May so we're going to look at the details of this of this plan a little bit and The first step that we're going to take is selecting the version Our plan is built around that November release Because we don't like to ride the wave of a new release So we won't immediately grab the latest and greatest and put it into our production systems We'll grab this November release and we'll put it into production sometime in March That gives us time that gives the community time that gives the tracker time to find those bugs Which we're happy to help with but we don't want to find them in production or have our instructors encounter them We also like to respect the supported date range so the security date range is very important to us So we ought to make sure the lifespan of that matches the matches or exceeds the lifespan of our production server within the year And then finally we're going to look at new features New features since our last install so our previous install was three nine We'll look at three ten three eleven and four oh, and we'll look at the features within Moodle A lot of times that we find too many changes is actually a negative thing for our faculty and students So we kind of we kind of trying to find out that great balance This year we really tested them with the 4.0 release With all the interface changes that we had we had some folks that had some struggles After we decide on our version, we're going to go to the infrastructure We're going to make sure that our PHP version and whatever database version We're using is going to be up to date and its life cycle is good with ours We chose PHP 8 We did not go to 8.0. Just 8.1 because it's a it's a very new release And we've got to respect all the vendors that we rely on and make sure that they support the current PHP version We couldn't stay on 7.4 because seven port four's lifespan is going to run up this year And we wanted to make sure we were supported for the full life cycle. So we chose eight We did have some struggles getting some of our vendors some of our plugins making sure that their their code was ready for eight The the big issue there was the ordering of parameters within functions your optional parameters had to go last so we note all these changes and we Document them as we prepare for our upgrade. There we go Next we're going to look at the important code modifications that Moodle has made and the best place to go to start with is the Moodle documentation they got a very nice new website at Moodle dev.io We're going to go and we're going to look at that 310 release that 311 release and the 4.0 We're going to look at the major tracker items that are listed and if you get into that tracker you can you can filter that tracker by Release so you'll find not just the major but the minor tracker items and sometimes the minor is major to you But there's also a section to play close attention to it's called the for developers section It'll list out the core plugins that were Removed in Moodle it'll list out the deprecations which are important to your code Make sure that you you're also removing them in your code And it's also going to list out component APIs that were upgraded if those are important to you You want to make sure that you pay close attention when you revisit your code? So we're noting all these for later Also, I like to look in the lib deprecated lib PHP It gives you kind of the history. It also gives you a little look into the future of deprecations Things that are going to start throwing warnings as opposed to errors You'll see that coming and you can kind of prepare for that coming down the line Also lib DB upgrade is a great place to look for a running history of the database changes So you can see the way the database is changing in The current version and some things you won't find in the tracker that you say why are they doing that? So we'll look there as well And then finally we're going to review last year's data So we're going to go into our current database We're going to look at activities and we're going to look at the usage there. We want to see that Just got weird We want to see how much our instructors are using what they're not using And we want to decide, you know, maybe it's time that a plug-in retires because it's not being used or it's not Because it's not worth our time for maintaining it. I'm sorry all right that will lead us up to the November release and We will get into our development review The first step in our development review is getting a place to work We will get our dev and test servers set up Our dev server is a place that all of our developers can spin up their own instance of Moodle They can tear it back down and spin up another one We like to test our plugins or our Commits in a clean environment so that there's no cross contamination between what we're testing And we're going to go through all of our plugins our commits and stuff there And then we'll have a test environment that our larger department has access to so that's where they're going to build their support materials and test for us It's fun to note that we have two protected branches that we maintain once the test for a branch Where our developers will create pull requests and we'll approve them And then there's the main for a branch which I integrate the approved changes into Our main repository It's something to note this year There were so many changes visually in Moodle 4.0 that we set up a Moodle preview server It's a production like server We set it up early and we allowed anybody on campus to opt in so they could see those changes and kind of get rid of that Shock that we're going to see when they saw the changes in Moodle. So next we're going to do the tedious Core commit walk-through. We have about a hundred commits that we make to Moodle. It's core changes They're not necessarily bug fixes because we try really hard to give that back to Moodle But these are behavior alterifications that work better for us at the University and we'll walk through each one of those commits instead of just doing a Flat rebase and hoping it's all gonna work We go through and make sure that the change is still needed that there are there are no new conflicts because of the code changes From Moodle and we'll make sure that the patch still works as desired We are also working really hard to get our own set of the hot and PHP unit test for these commits So that this step isn't as tedious in the future Next we'll look at the plugins that we maintain So the course copier that I mentioned before the course roadmap our gamification plugins And we have some plugins that we've adopted because the original maintainer kind of flew the coop And we needed to take over because our instructors didn't want to see that plug and go away We will look at the updated component APIs that we mentioned before look into those Into those plugins and make sure that everything's up to date there will make sure PHP 8 works for it And we'll also look over the interface because Moodle has changed so much We don't want the plug in to look out of place. So we will update the UX UI in that as well And that was a significant process this year Then we're gonna do our community and vendor plug-in audit You would like to think that when you Get a plug-in from a vendor that it's gonna install and you're not gonna have any issues There's no bugs or anything, but that's that's It's rarely the case a lot of times we have to give some code back to them. We have to test it for them There's a lot of PHP 8 that wasn't ready in our vendor plugins So we go through the same steps we did with our own But we also do a health of plug-in check with our vendors or with our third-party plugins We'll see we'll go to the repository see if there's a community still alive and around that plug-in We'll see if it's well maintained or if it's gone stagnant, which is kind of a warning sign and we'll see that if if there's any issues being reported and if are there are those issues being fixed and if the health of this Plug-in is not well is it one that we adopt and start maintaining if it's something worth our time for the instructors looking at those usage numbers Or is it something that we can finally let go there we go So now we get our we have our code base pretty well intact. We've got our commit review done We've got our plug-ins installed We probably have some loose hanging ends, but we can we're moving into the start of the new year and we want to start looking at our Config setting changes and our capability setting changes and what we want to do is we want to extract those changes so Years ago. I developed a script that goes in and pulls out the Site-level settings and the plug-in settings and then some settings from other areas in Moodle And I put them into a data file and then we'll run that on the our old version We'll run that on our new version and with a secondary script. We'll find out the differences so we get a full list of new settings a full list of Settings that were removed maybe because we deleted a plug-in and then a very small list of settings that had their defaults changed Which was a little hocus-pocus we ran in two years ago when we thought we our settings were right But because the default change we kind of got Bambuzled Yeah, so we'll pull those out one thing I would want to mention that was really cool this year when we we upgraded using the CLI command of 4.0 Moodle did this really cool thing and it spat out all the new settings So it's starting to do what I've been doing by hand. So I thought that was really nice of them Then we're gonna do the same thing with our capabilities except we have a local plug-in that's on our repository. It's in public view This local plug-in will pull out capabilities both roles and capabilities in the system and how they're set And it'll push it out to an XML file We originally created this plug-in so that we can grab that XML file and sync all of our instances So they were identical that way no one you know one instance got really weird or anything. They were all the same Well, I take those XML files for the old and new and I compare those as well So we have the new capabilities with the new instance of Moodle the change capabilities And the ones that were removed. We'll take all of those both Config settings and capabilities to our tactical committee their governance and let them decide How do you guys want this set and when they decide how they want it set? I put it into our config script that runs against the server and sets everything the way we want We've got some LTI Considerations if you're just upgrading your server It's not so bad because your course IDs are the same your user IDs are the same the configurations stay in place So the LTIs are kind of still set to go But if you're standing up a new instance like we do every year and We're moving our population into that new instance every year We kind of run into some issues with those course IDs restarting and the user IDs getting jumbled because of them loading in differently My advice here is to approach the vendor Tell them your scenario and have them handle it on their end because we're going to set up that configuration We're going to connect to them. We need them to know what our issue is Fortunately, most vendors have some type of solution whether it's pause the server sever the data and start us over or give us a new configuration detail whatever it may be Be upfront with them and let them know and work with them My biggest advice is whenever you set up an LTI Contact that vendor and say hey, I want to walk through this with you I want you to look at my config settings and I want you to show me You know set us up an instance live real fake real fast either on test and and or production And most vendors are completely open and happy to do so because they want to avoid issue as well And that's been very successful for us one thing to note whether you're upgrading or you start new instances every year There is a move from 1.1 to 1.3 LTI And if you're on 1.1 most vendors go into 1.3 you're gonna have that little hiccup where the instructor backs up their course It's gonna have that 1.1 LTI in it. They're gonna restore it It's gonna have that 1.1 LTI and you're gonna need them to move over to the 1.3 configuration And then there's not really a way to avoid that that I know of So finally we test our integrations. We've got that course We have course creation Automation we have user enrollment scripts. We have great extraction. We have the process of back-uping and restoring When we get that into test we test that with our our unit tests So we have a test that we can run against that test server and go through all the functionality for each of those and make sure They work because testing and production isn't that great? So That's our last step make sure all integrations are working after that We can set up our new installation of Moodle So we stand up our new new installation during our spring break It will go live in the summer, but we let allows our instructors to get on board and get in But if you're upgrading your Moodle server, there's a few more steps I want to mention for us when we practice our upgrades we go ahead and take a snapshot of production and we put it on a replica server and We we change Convig values like divert all emails to an address so that you know this the server doesn't send out emails to anybody and We we run the url replace tool Against the server to take it from the production url to the staging url to that replica url Once we get our replica set up so that it's running on its own and it's identical to production We'll take a snapshot of its database and we'll take a snapshot of its Moodle data folder And we'll set those aside and now we can run it a practice upgrade if things go wrong, or we don't like it We restore that snapshot and we can run it again So we can run it over and over until we're really happy with the output a lot of times There's there's plug-ins in the background that don't like going through the upgrade process. They may spit out some errors And we want to make sure that you know, we're we're getting through those and working out to get a smooth upgrade Sorry, I skip that So the Important thing here is if you're removing plugins remove them before you upgrade That way it doesn't have to go through that upgrade process Just make sure that you know that when you remove that plug-in You're gonna remove the data with that plug-in too because that that plug-in is gonna take away the tables and along with the Tables goes the data for that plug-in One thing that we had to do a few years back We moved from form ng to Moodle cores new forum because we really liked it and form ng wasn't wasn't right for us anymore So this was the step that we had to create a script that moved that data from one plug-in into core so we had a little conversion script and We got that moved in one thing to mention on our code base the way we Have it and get our git repository our main Moodle core is one repository and then we have Roughly 40 sub modules that are a part of that Repository each one of those sub modules represents one of our plugins that's managed in its own repository So if we have changes and we do pull request against it Those happen in those repositories and then I as a maintainer Manage the main branch and which one that main branch is looking at my advice I always upgrade via CLI you get the output you get all the output. You don't have to worry about any browser Confusion or caching which Moodle still really good about but I trust I trust the terminal more than anything I Like pipe in the output so that I can review it later You know if something crashes and it dumps a whole bunch of stuff out I can dig through it So that small bit of advice there making sure debugging is on What am I supposed to point this at and then post launch a Few things here after you've launched my best advice is be available almost So block off some time be ready to respond fast This is your first impression as soon as that server goes up and instructor has formed an opinion So if there's something going wrong Jump on it as quick as you can Be responsive Engage with the Moodle community It's so important that if you run into errors or confusion Reported on the tracker even if it is confusion if a lot of people are having the same confusion. It's probably actually a problem The trackers open to anybody not just developers actually we probably need more non developers More end users supplying your opinions on what our direction should be Help us write test cases if you know how it should work write those steps out If you if you know how it's how it's not working and how it's busted write those steps out And if you have code, of course Help us out by tossing it up there And then my last piece Advice is to actively monitor We use you know look into those errors server error logs. Just take a glance. You'll see stuff in there You'll watch database growth is our advice this year our questions table started growing way out of hand really fast and it was a sign that there was something wrong and Server infrastructure monitoring we use a program called Nagios Just tell tell signs that we can we can identify an issue before someone else does I Hope something in there was helpful for you today. That's all I had So we have a few minutes for questions anybody have any also if there's a if there are any empty seats Could somebody like stand up and point to if there's place available. I don't know if there's empty chairs anywhere, but yeah, go ahead Development processes your configurations reviews Do you implement any automation to your? your stages So we're playing around with it I think we're a little behind in that apartment and that might be my fault because I'm a hands-on guy Okay, I really like that idea But we we're working more of that on the production side and not on the development side, so After I as a maintainer get those pushed up We're getting to where production will start updating itself More automated but on the developer side. We're still old-school. I guess yeah Thank you just a quick question How many people are responsible for doing out this work and do you think that number is adequate? Are you happy with that? so Developers there's four of us However, the whole Moodle upgrade process We have we have an all-in department that that works on it and that that is all of Delta There's a presentation tomorrow. I think it is that is the Moodle for upgrade on a whole that encompasses our entire department so the support materials the ID team the instruct instructional development team The design team so there is a lot more than just us developers, but this is more of a technical thing Thank you very much You talked about your code base and git and adding plugins as subplugins I guess you're doing that instead of sub modules. How are you handling? plugins that have subplugins themselves So that works. I think HIP has subplugins as well and you can run a recursive command that pulls them We don't actually maintain their subplugins, but we do keep a copy of the plugins repository in ours because a lot of our plugins we have commits on top of theirs we We give ourselves the right to change anything In in our install, so we've we have some modifications to plugins that we bring in as well Do you have to handle also integration with the public clouds like Google Google classroom or Office 365 for education similar thing. So our campus is a Google campus, but we don't use Google classroom but we do use File embed and the repositories and we connect those So the integration is just a sharing at the Google Drive level. Yeah, and so forth. Yeah Any other questions? Nope up here. I guess that's a little bit out of out of touch with the update process but I have a question about the copy processing and since I worked with the copy process I Would like to ask if there's any planning on simplifying the copy process so Moodle cores process is As far as code goes is not simple it's rather involved, but it makes it easy for the end user We simplified it by a Another program we run. It's called wolfware. It's it's our portal to our Our University and the instruction can go in there and say I want to copy this course to this course and Our our code does all the work. So it'll run the backup on one server It'll it'll pull it down into a shared shared server and then I will restore it on another server for them Because we wanted it to be easier But I don't know if moodle is gonna make it any simpler you said at that you said that when you test the new release you get the learners to have a Sneak peek at the new version for example version four How does you handle that do they sign up to a firm? Do they just get the URL or do they create a new user or how do you handle that? So we're also a shibboleth campus So we run a shibboleth plug-in. So if a user Hits our site the shibboleth will it'll push them out. They'll log in and it'll create a user on the fly So we didn't have to worry about the create user But we did set up a form that if they wanted to try moodle in the moodle preview server if they filled out the form and agreed to the Beta version of our moodle then it would it would build them a course and add them as an instructor So yeah, we automated a little bit there