 Hello, everyone. Welcome to the Mural Academy webinar on upgrading a large model site. My name is Rajanith Gautaram, developer educator at Mural Academy. And I'm joined by my colleague, Sandram Mardz, who is going to help me facilitate today's session. Our speaker today is Cameron Ball. He's a senior architect at Catalyst IT. And in today's presentation, Cameron is going to share his experiences on upgrading model sites. And in particular, in the session today, he's going to talk about upgrading a large model site. And for those of you who do not know, Cameron was the winner of the best presentation at the MuralMode Global 2023. So we are very excited to have Cameron as our presenter today. And with these words, I will welcome Cameron to start with his presentation. I know that he's got a lot to cover. So over to you, Cameron. Hey, thanks. Yeah, thanks for the welcome. So this is a reimagining of the talk I did at the Global Moot called Upgrading the Largest Moodle in the Southern Hemisphere. And I've got a few slides to talk about the company that I work for, which is Catalyst IT. Got my red catalyst shirt on. We're a Moodle partner, and we're all over the world. And we're pretty good at what we do. And this slide is pretty interesting, because I can show you where I currently am, which is right here. I'm from Perth in Western Australia. And here's another nice picture of our team doing team stuff. Here we are at some conferences and hacking mainframes and all that kind of good stuff. And here's a little bit about myself. I usually do this slide at conferences. My shell is ZShell. My desktop environment is KDE. My terminal emulator is console, and I use Tmux. My text editor is Emacs. And the percentage of larger sites in the hemisphere upgraded is 50. David Integer is 1729. Bia is pale ales. Arms is two. And here's a picture of me. The double hurricane thing is a reference to Global Mood two years ago when I had to fly through two hurricanes to get there. Right, so about the talk, the technicality is moderate. It's not super technical. I don't really do a deep, deep dive into all the technical issues we face. It's more of a high-level overview, but you do need some technical knowledge. Is it a case study? Yes. Is it as glamorous as the title makes it sound? Is probably not. It becomes a lot less glamorous when you realize that most of the world is not actually in the southern hemisphere. I want to set some expectations about the excitement level of the talk. The first Global Mood talk I did was about queuing algorithms, and that's about as exciting as it gets. Birth of your first child is a nice 50 on the excitement scale, and this talk is a little bit lower than 75. This particular instance of the talk might be a bit lower because I don't have all the flashy videos and silly stuff I included when I was at the Global Mood. If you want to see those, you'll have to watch the recording that's up on YouTube. But still, we'll try and make it exciting and fun, right? OK, so prologue, let me give you a bit of background to how it all began. It all actually began at the Global Mood in 2022 when I was having a chat with my colleague, and it went something like, hey, have you ever upgraded Moodle? And I said, I was so smug, I was like, yeah, over 100,000 times because I used to work on Moodle Cloud, and I was in charge of upgrading the sites. But the big difference is those are very small sites, not really big ones. And then, hey, would you like to upgrade the largest Moodle? And my reaction was sort of like, oh, yep. I like that emoji. That's why that's like that. But I said yes. And then throughout 2023, that's what I was doing until I came to the Mood to talk about it. So the question is, how big is big? There's lots of ways to measure bigness. Here's one way, which is 5.44 terabytes. And that, it's probably bigger now, but that's the size the database was just before I gave this talk the first time. So it's a bit of a hard number to visualize. So I've put it in slightly different terms. This is the Nintendo Entertainment System, and it has about, well, it has 248 bytes of memory. And so if you express the size of the largest Moodle in the Southern Hemisphere in terms of Nintendos, you get $2,656,250,000 Nintendos. So it's a lot of Nintendos. And if you stacked all the Nintendos on top of each other, I calculated it would get you more than halfway to Mars. So now that you all know how big it is, and how it's easy to visualize way, let's talk about our approach to the upgrade. We had two paths we could go down. One is actually running the Moodle upgrade script on the largest Moodle in the Southern Hemisphere. And the other one is delete the largest Moodle in the Southern Hemisphere and start again. And obviously we didn't really delete it. Like, that's not what we're talking about here, but it's sort of building it again from scratch and migrating the data over. So these are some things we had to think about when making that decision. The existing site has loads of content. Obviously with the database that size, there's heaps of stuff there. The database is full of lots of nonsense as well. Having existed for as long as it didn't being as big as it is, the database is full of weird DB state and things that are just orphaned. And if you've ever worked with the Moodle database, things often get left behind and stuff gets weird pretty quick. And we actually inherited this site from another vendor and there's quite a bit, was quite a bit of customizations in there that we really didn't know what they did, but we hadn't had the time to sort of go through it and clean those up. And here we've got another picture, the state of the largest Moodle in the Southern Hemisphere visual analogy. Sorry, that font should be different. I went through and changed them, but that one I must have missed. Oh, no, no, I wasn't supposed to change this one because there's a little video game thingy. So this is the kind of conversations we were often having with the client here. So, hey, Catalyst, the Moodle's behaving weird. Could be an obscure bug. Can you help us out? And it's like, yeah, yeah, no worries. Catalyst, world-class support and solutions to any issue you may have will solve it. And so later, but not much later. Wow, Catalyst, that was amazing. You solved it in a timely and professional fashion. It's like, yep, no problem, just a bit of invalid DV state, easy as when you're as competent as Catalyst IT. But then the follow-up to that was always, what, how did it become invalid in the first place? And our response often went something like, oh, well, it, we don't really know. Then that's like I said, it's old and big and lots of stuff has happened to us. The database is really, really confusing and hard to untangle. So with that in mind, the path we decided to go down was, oh, sorry, some conclusions here. The existing system presents too many challenges and risks. The technical debt has become unmanageable and I've got an updated visual analogy here. It's not fine, no reason let it last this long and get this bad, revised analogy. Right, so with that in mind, the decision we made was to go Greenfield, start again with a clean database and obviously migrate all the content over. So I wanna take a quick second here to sort of discuss the upgrade philosophy and talk about this word here, amortization. Like there was a, there's the definition of amortization, the usual one. But I like to use it in sort of this context where you, when you're working on these kind of things, it's really tempting to try and automate everything, like as programmers, we really want to write scripts to solve all our problems. I know I do. And sometimes often you end up spending more time writing the script than you would have if you just did the job manually. So throughout the entirety of the project, this was in on my mind and I gave myself some constraints. So only write scripts when it makes sense. So the first thing to think about is how long would it take to do the task by hand? How error prone is the task? How hard are the errors to fix? And how simple would a script be? So I gave myself this limitation. If the script doesn't work in two hours, give up and just do it manually. And I'm talking about things like moving over cohorts or assigning roles and all those kind of things. Yeah, script will be easy, but I might make a mistake. And in reality, it's simple enough to just go into the UI and look at how it is in the old site and then just copy it into the new one. So there aren't that many sites in the Southern Hemisphere, largest Moodle sites in the Southern Hemisphere to practice on. So we were a bit naive when it came to this and the expectation that we had was that we just get rid of Moodle. Something, and then everything would be fantastic. There we go, there's a nice recap of that. In reality, it wasn't that easy. And what you're seeing scroll past the screen now is a checklist of all the things that we had to do to make sure that the new site was working as expected. And there's that one again. Okay, so here's the slide where I start talking about what we actually did. So phase one was core customization analysis. I mentioned there was a bunch of core customizations who didn't really know what they were. So we had to go through those. The requirement here is a way to manually audit every single customization in the code base for each customization, work out what it is, decide if we need it in the new code base. And the strategy for doing that was just identify every piece of code in our code base, which is not part of vanilla Moodle 3.9. And there's a really good tool for that. You might have heard of it, it's called Git. And so there's the Git command and just tack on the WC-L there. And you can see we had 1042 customizations on top of vanilla Moodle 3.9. This is not plugins, this is Moodle core, like stuff that had been done to Moodle core. So the tooling to deal with this was a spreadsheet, keyboard shortcuts to focus windows, small bespoke shell script, which I'll show you in just a second, copy paste and eyeballs. So here's a table that like we used a spreadsheet to keep track of all the customizations. And this is kind of what it looked like. So on the left there you have the commit hash in the 3.9 site, the commit message, a column to track whether or not it's needed in 4.1. If it's an MDL, which some of these are, they're MDLs that have been backported or MDLs that may not have been finished yet, the status of that, and then ultimately the decision on whether we needed it or not. So the workflow was you select a shelf, a cell from the spreadsheet, control C, the commit hash, use the keyboard shortcut to switch to the terminal, run the shell script by pasting the commit hash into it, use your eyeballs, and then use a shortcut to go back to the spreadsheet, move to the appropriate cell and paste the commit hash if it exists in the new code base and check MDL status if necessary. Here's an example of what that shell script looks like. So you run it with a commit hash, you can see it's found it in the largest moodle in the Southern Hemisphere. And then it also found a very similar looking commit in vanilla moodle 4.1. And you can see some of the lines are a little bit different like file lib down, if you can see my cursor down the bottom here, lib file lib has a few different changes to it, but it's mostly the same. So chances are that this is the same thing, maybe with slight refinements, but if these numbers were wildly different, we'd probably go and investigate a little bit more. So I have a demo of that, which I will try to show you real quick. So here's the spreadsheet, not the real spreadsheet. And the workflow is, okay, we've got this commit, so I'm gonna copy that to my clipboard and then I can go down to here, type checkback port, whack it in there and it found it, but then in 4.1 it didn't find anything. So I know from looking before that this, so there's no commit in moodle 4.1, the MDL status of this one is that it's still open and the conclusion is we need it. And so then I can go to the next one, go back here, have a look, same thing, it didn't find it in 4.1, so I come back to here, NA, open, still needed. Then we can have a look at this one and we can say, hey, look, that is pretty much exactly the same in moodle 4.1. In vanilla moodle 4.1, this change exists. So it copied the commit hash of it in 4.1 to the clipboard. You can see down the bottom, this one. So I can just paste it in here so we can track what commit, so it's closed and not needed. And the last one we have in this list, okay, so this one looks like a core hack, right? It doesn't have any MDL, it's not in 4.1. And it says work around. So that's an NA for MDL status and yeah, we need it. Well, maybe we don't need it, we need to go and look at what the hack is in more detail. And so that's how we did that, and that worked really well. So discoveries from this process. We had a list of stuff that we would need to cherry pick into the new code base. And we did find a lot of weird stuff. We found some craft strings, all of those. There's an MDL now to get rid of those. These strings exist in moodle still, I believe, and are just not used anywhere. And generally not much else. Moodle's pretty good here. Like if we had to cherry pick something, it just kind of worked because moodle doesn't really break backwards compatibility. So that worked out really well in our advantage. So the experience for this phase, 7.49 out of 10, I would do it that way again, if I was doing an upgrade project like this with that little script and the spreadsheet, it was super quick, even though there was over a thousand, you know, we just go through, dum, dum, dum, dum, dum, make notes of the ones that need investigating, go investigate them, get rid of them, fix the problem properly, whatever. But yeah, overall that was a really good experience. So phase two, which is plug-ins analysis. The requirement here is we need a way to manually audit every plug-in in the code base. For each plug-in, work out what it is, decide if we need it in the new code base. The strategy was get a bunch of humans to do it. So tooling was a spreadsheet again. Zoom calls, collaboration. And here's an example of what the spreadsheet looked like. We just have the path to the spreadsheet. So path to the plug-in, the type of plug-in, whether it's one of our internal plug-ins, whether we forked it from a community plug-in, whether it's just a vanilla community plug-in, the branch it's on, all that kind of stuff. And we just had to go through. There was about four of us. We just went through the spreadsheet, checked what the plug-in is, if it works in Moodle 4.1, or if it needed any work done on it, then we'd fix it up. Or sometimes just remove the plug-in entirely because the functionality is now provided by core or just not needed anymore. We made some contributions throughout this process. The most significant of which was to Q-Type Stack. We fixed their GitHub actions because it was broken. The tests weren't running properly for different versions of, for Maxima compiled against different LISPs. I'm an Emacs user. And so that was an unexpected joy to get to play with LISP as part of upgrading a Moodle site and get paid for it. Then we improved this project called Maxima Pool Docker, which is abandoned, but we are currently still using it. There's GeoMaxima now, which is a better choice. And we'll probably look to moving to that. But if you use Maxima Pool Docker, we have our own fork called Dockerfile, we have our own fork with a branch called Dockerfile improvements that makes it easier to build the Maxima Pool against a specific version of Q-Type Stack. So you might wanna check that out. Then we had to have some tough conversations about things. It's like, hey, we identified this plugin that we think you don't need. Can we leave it out? It's like, yeah, yeah. And that's to be continued. We'll come back to that later. This right here is a commit in the Q-Type Stack plugin that I'm very happy that I authored. I mean, just look at, that's the commit message. Just look at that. Better normalized floats for cross compatibility between SBCL and GCL Maxima. Doesn't get better than that. And in that commit, I fixed a comment and I changed it to say, Oilers number. Instead of saying the base of the natural logarithms, it now says Oilers number. And you can go look at that in Q-Type Stack, if you want. But for me, that was a fantastic thing that I got to do. And there's Oiler himself who's very pleased with what we did there. And no worries, big E. And also good job coming up with functions. They turned out to be really useful. And even 18th century mathematicians know how competent Catalyst IT is. The experience for this part of the project, 7.71828 subtract E over 10. It was just okay. But I have no idea how to do it better. I mean, it's a very manual process, right? You just have to go through the plugins. There's no real fancy tricks to automate that. Now we're on to the third phase where things start to get a bit interesting. We had to figure out how to migrate configuration. So sure, building the code base is one thing, but then there's the whole Moodle configuration table plus the plugins configuration. So the requirement here was that we had to manually audit every single config value for core and plugins in the database. And for each configuration value, we had to work out what it is, decide if we need it in the new database. And we also need to bring across other things such as user profile fields, roles and the role assignment matrices, like who can assign what role to other people, web services, grade scales, scheduled tasks, cohorts should be in that list as well. And the strategy here was automation. We did start doing some automation here. And there was also a lot of manual work too, exactly like I was talking about before. In some cases, we just decided quicker if we just go into the user interface and click, click. So naive again. One thing we thought we could do was we thought it would be really clever if we took the existing database for the 3.9 site, upgraded the code to Moodle 4.1, ran the upgrade script and then we'd have a new config table with all the appropriate values after the upgrade scripted ran. So then we just restore that back in, just restore those tables right back into the new Moodle and everything's gonna be great and profit. That's what we thought would happen. But in reality, it didn't really work that way because of things that are really good about Moodle and have never caused headaches, lack of referential integrity, back up and restore. Sure, there's more, but that's all that I had written down. But the reality was that there were things in the config table like IDs and there was just like a string of comma separated IDs. And of course, if we bring that back into the new site, the IDs that is referencing might not exist and all sorts of stuff like that. So we were back to manually auditing stuff again. So what we used for this is a thing called Report Custom SQL which is developed by the Open University. Hello to any Open University people who are watching. And PHP scripts and eyeballs again. So for a basic case, all right, so this is an overview of the whole thing. For basic cases, if it's feasible to simply look at the UI in the 3.9 site and click, click, copy, copy, paste, paste things over to the 4.1 site, we did that. For example, grade scales. If the thing has import, export functionality, we'd use that. So roles is a good example. You can just export the roles and then re-import them. If a core API exists, we could use that to make a script. Cohorts was one of those where we were able to make a script. And here's an example of one of those scripts. Very simple, took five minutes to write. I just got all the cohorts out of the database in the 3.9 site, put them in a JSON file, pass the JSON file, looped over them all and ran this script on the new site which created the cohorts in the new site. Pretty easy, pretty straightforward, worked really well. A more complex case is comparing configuration values. This is the CFG table and the plugin CFG table across different versions and environments. For the initial migration of the 3.9 site to the 4.1 UAT, that's like staging environment, we needed to go through every config value in the 3.9 production environment and compare it to the value in the 3.9 UAT environment, compare it to the value in the 3.9 staging environment, compare it to the value in the 3.9 production environment which obviously would be the same because we're going through those values. The value in vanilla Moodle 3.9 and the value in vanilla Moodle 4.1. And by doing that, we sort of gave us an idea of what these configs are and what a sensible value might be in the new site. Oh yeah, and we did also compare it to the value after upgrading from 3.9 to 4.1. So that clever idea about upgrading the 3.9 database to 4.1 turned out to be useful after all because we could see if the upgrade changed the value. And so what we did was we used that report custom SQL plugin to export JSON files of the relevant tables, MDL config and MDL config plugins. And then we wrote a magic script that can help us audit it. So the magic script has the following features. It can create placeholders. So I should say what it does first, it generates a shell script at the end of it that you can just run and it will set all the config values to what you want them to be. And so it could create placeholders in this script so you could go back and fill in sensitive information like database credentials and stuff like that. Ability to create variables. Oh, there you go. That's what I was talking about. Ability to stop and resume. That was cool because there's so many config values. I could sort of do a bunch of them, stop, come back to it later and it would pick up where I left off. It intelligently can skip irrelevant configs. So for example, if it sees that the value is the same in all the environments, it's like, okay, you probably don't need to worry about this one. Let's just set it to what it is everywhere else. And it has a beautiful UI fully compatible with NC escape sequences. That's just a fancy way of saying it works in a terminal. And it was only about 250 lines of code. So that was nice. It sort of fits all the ticks, all the boxes. So we can do a demo of that real quick, hopefully back here. Okay, so in this directory, I have the script generate let me make the text a bit bigger. I have the script generate config commands and I have the various exported JSON files. And all I have to do is run the generate config command script and tell it the name of the shell script. I want it to make config Dutch and then I go. And you can see it's already skipped one of the config. So I've got 850 conflicts to go through. It already determined that the first one I didn't need to worry about. And here we go. We have a config that's different. So I can pick one of these and choose the value I want to fill in. Say for whatever reason I want it to be C. I can just press C. I also have the ability to grep code bases to kind of see where the config is used. That was pretty useful. But for this one, I'll just press C. And then you can see it skipped another one. It's gone to four out of 850. So in this case, what I'll do, I'll just skip it because maybe it's irrelevant. In this case, maybe I'll add a variable. If I press V, I can type variable. Then in this one, I will use skip this config but come back to it because maybe I'm not sure maybe I want to revisit it later. And this one doesn't matter. I'll just skip it and I'll just stop for now. I skipped a whole bunch there, went up to 86. So now if I have a look here, I have config.sh. And it's generated this nice CFG script. There's the variable I made. If I search for it, you can see it's used here. So later, when I know what the value of that's supposed to be, I can fill it in. And you can see it's got all these little things here to set the values. And when I've decided I want to come back and continue auditing the configs, I can just run the script again and just jump back to 17 because that was the one where I said, skip it but come back to it. Now I decide maybe I want it to be A. And then it jumped all the way back to 86, which is where we left off. So that's how that script works. And that was, it's very much a bespoke sort of script. If people are interested in that, you might want to contact Catalyst IT and see if we can work together to get it in a more generic fashion so that other people might be able to use it because the way it is right now, a lot of things are hard coded to our specific use cases. Okay, back to the presentation. An interesting case was the role matrices which I mentioned earlier. So there's quite a few roles on this site and the thing where you can say which roles can assign roles to which other people and that kind of thing was big. And we couldn't find any, maybe there is one. I still don't know of it. There's an easy way to migrate this across. And so what I did, at least from going for 3.9 to 4.1 UAT environment was I just manually did it like I had two screens open and just copied it. And then I did it again. I did that again to go from the UAT environment to the staging environment. And I was thinking a bit like, oh, I sure as a shame that no one invented a way to write code directly in the web browser that would have saved me a lot of clicking. And then I remembered JavaScript exists. And I felt really silly and I figured out a sort of funny way to do it. This JavaScript code you can just run that in the browser on the roles page and it will spit out a big JSON string that has the states of all of the checkboxes. And then on the 4.1 site, you can run this piece of JavaScript code with the string from before and it will set all the checkboxes and then you just press the save button. So I'm gonna, I tried to demo this at the moot and it didn't work because it was really difficult to see the screen, but I will try again. So we need to go back to this slide. And I copy this. Okay, so here I have a more basic example. I've made a nice little pattern here of how the roles can be assigned. So I'm gonna open up my console and I'll just paste in that. And there we go. It spat out this JSON string, which I can now copy and I should be able to go to here. Here's the 4.1 site and you can see it doesn't look like the 3.9 site did. So when I go back over here, I can grab this. Actually, first I need to make this checkbox info JSON. So paste that up and it didn't work. Oh, I know why. Nope, still didn't work. I'm gonna try if it doesn't work this time I'll give up again. No, okay. I don't know why that's not working. I had it working just before I did this, something weird. No, in back ticks. No, I don't know. Wait, do I just need a semicolon at the end? Is that the problem? No, I'm gonna give up. It does work. It sets everything and it's all super nice. Maybe I'll make a separate video of that where it actually works. And the experience of that, I rate it at eight out of 10. The configuration migration script was great. Really, really liked using that and I would do that that way again. And now phase four, which is quality assurance. So the requirement here was to triage issues and fix the real ones. The strategy was to keep fixing things until they're fixed. And the tooling for that was a spreadsheet and patience. So here's the continuation of the tough conversations from before. So catalysts, the functionality provided by a plugin that we wanted removed has stopped functioning. And then of course, that's because we removed the plugin. But oh, we need the functionality provided by the plugin. It's like, oh, so should we install it? And would you like all the configuration for the plugin migrated? Yeah, even after we finalized it and migrating is more difficult now. Yeah. All right, so of course we did it and it was fine. But probably the lesson learned here is we should have just brought across all the plugins and then removed them at the end rather than removing them earlier and then finding out later that they were actually needed. Catalyst provides you with enterprise grade patients. So we made some discoveries throughout this phase. We found plenty of bugs. Here's a few listed. There was a sort of backwards compatibility breaking web service change. You'd have to go look at the MDL, but it's not really, or was it? Actually, I can't remember exactly what that was. We might maybe just go have a look now. What was that one? No, it actually was a backwards compatibility breaking web service change. There was another one where changes to the nav system made some nodes completely inaccessible. This one was really weird. There was something odd that changed which made the embedded question filter go crazy and just start duplicating the question over and over again. I think that one's fixed now. Some minor issue with a format plugin. And an honorable mention to debugging curl headers and proxies. We use proxies and we had a really, really interesting bug. This MDL sets this curl opt and that's fine. But if you're using the header size thing from curl info, that includes the size that it would be before the headers were suppressed. So even though the headers aren't there, this value is as if they are. So any code that uses that header size to pass headers will break on sites with proxies configured. So I followed this MDL. It's not really Moodle's fault. It did bite us with this block panopto. So if you're using that and you have a proxy and things are behaving weirdly, that could be why. The panopto people did respond and I don't think they fully understood what the bug is because they just closed the issue, but the bug is still there. There's a workaround which is documented in the issue. And here's a little summary of how that felt to deal with. So in a debugging curl request, it's pretty fun. Everyone likes it. That should say debugging curl request with proxies. I don't know where that text went. Then reading libcurl documentation. And then of course, reading the libcurl source code. I'm a serious person. This is not sarcastic. I really, really enjoyed taking a deep dive into libcurl to figure out what was going on with a Moodle plugin. So the experience, it's five out of 10. It's not the most fun thing in the world, but in QA never is and there's not really another way to do it. You just have to keep fixing the stuff until it's fixed. So that was it. What's next? We've got to do content migration, as mentioned. Big site, lots of content. They obviously want it in the new site. A lot of that work is done now, because this slide is as if it was in the past, I didn't update it. Working on updating integrations with the student management system, that's still ongoing. Heaps of cool plugins being developed. Some might be open sourced, I'm not sure. And that's it. Contact Catalystite. Multi-region IT service group that provides enterprise-level technical support for open-source software. Thanks for listening. I think we're doing pretty okay for time. Yeah, we've got heaps of time left. So we can do some questions. Do I need to hand it back over to you, Rajneel? Yeah, I think there's some questions in the chat. So this one says, this one says from Matthew, a very interesting presentation. Thanks. Your choice to do a new installation and migrating rather than running the upgrade, model upgrade process on a large dataset implies that the later is bad idea. What are the main issues you were trying to avoid in the model upgrade process or script? Yeah, good question. It's not necessarily a bad idea. It was just a challenge for us because as mentioned, the site is so big. The database is so big and the data in it is not of the highest quality. So running that script is a big unknown. And of course, after upgrading it, all the weird stuff that's in the database is still in the database. One of the main things we wanted to do was sort of get a clean slate. Running the upgrade script wouldn't give us that. The only way to get that is to just start from a fresh site. So I hope that answered your question. There's another question from Celine. While upgrading this large model site, did you have to restore existing courses? And if so, how did you proceed? Yeah, so we guess, yes, we did. And we did some magic where the old site still exists, right? So we just use backup and restore and we wrote a plugin which I don't think is public that writes the backup file to shared file storage that both sites can see. So it gets backed up to the shared storage and then the new site can see. They goes, oh, hey, I've got a course to restore and it restores it. So I mean, it basically backup and restore, right? Another question from Vigendra. How long did the whole migration process take? I think from start to finish, close to a year, like eight months to a year, I can't remember the exact timeframe, but it took a long time to get through everything and test everything's working. Another question from Rohit. What is the downtime in the whole process, including migration? Well, that's part of the magic of doing the greenfield thing, right? Because we had two sites, the 3.9 site never went anywhere. It was always there, always live and students could just be using that. And as soon as the 4.1 site was ready, just tell everyone, hey, please stop using the 3.9 site. So they have different URLs, both still exist with different URLs, just say like, hey, students, use the new site. So no downtime, basically. Okay, there's one other question from Luis. So did you upgrade from model 3.9 to 4.1? If so, why did you choose 4.1 and not one of the new versions of model? I think I did see that in the chat. It's to do with LTS. Yeah, so Moodle 4.1 is a long-term support release as was 3.9. So that's why we went to 4.1 because it will continue to get security fixes whereas some of the other new ones weren't. The next LTS is 4.5, I believe. So at some point we'll be moving to that as well. One more question from Santosh. What do you think about the blue-green approach for the upgrade? I don't know what blue-green approach means, Santosh. I'm sorry, could you rephrase the question? Maybe can answer another one while Santosh rephrases. So there's one more from Vijendra again. What would be an estimated downtime if you did not start with a new site? So if you had to upgrade that existing site here. Yep, so normally we do these kind of things. We're continually upgrading Moodle with security but even the 3.9 site, there's downtime. There always is and we plan for it and we have a window of something like five, six hours, something like that. So it would have been the same for the upgrade. We would have tested it, heaps, and then we would have said, hey, for this eight-hour chunk of time or whatever, site will be down, run the upgrade. If it works, then go live. If it doesn't work, roll back. So yeah, like a eight-hour-ish kind of window. I think Santosh has come up. So he says the blue-green approach means upgrade the site in the backend while the site is live with minimal downtime. I mean, that's, yeah, you could do that. Like upgrade a separate, like have it, upgrade it, and still have one not upgraded and switch it over or something like that. I suppose that's a thing you could do but the way we generally do it is you upgrade and if it doesn't work, you roll it back to how it was before. So I don't really have much experience with that particular approach, sorry. There's a question from Sebastian. Where will you publish the demo of the JavaScript? Where will you publish the demo of the working JavaScript? Good question, I don't know. Maybe I'll get in touch with Moodle and we can sort that out and put it on Moodle's YouTube channel or something like that. I don't know. It won't be on my personal YouTube or anything like that. I'll try to find somewhere more official to put it. There's a question from Louis. I'm not sure if you want to answer, but what did the big upgrade cost? I don't have the details of the costs of it. I'm sorry. I don't think I probably would share it even if I did. A question from Tarsos. Did you run a final upgrade at some point after freezing the old production instance? Or how? So we never... Yeah. Sorry, I finished repeating the question. Yeah. So did you run the final upgrade at some point after freezing the old production instance? Where did you ensure that you had the latest data in the new version? So yeah, as I mentioned, we didn't really upgrade the old site. It's still there. It still exists. All the content is still in it. So we never had to freeze it. We just told people to stop... Well, the institution told people to stop using that and all the academics stop using it and everything. Regarding having the latest data in the new version, we... All of the students' information is stored in an external system, so it's not that critical. What matters is that the cost content is there. And I touched on that earlier with the content migration stuff that we did. I hope that made it a bit clearer. Yeah. A question from Paul. What type of database did you use? Postgres or MySQL? Yeah, it's Postgres. A question from Sebastian. Can you please give us a link to your personal YouTube channel? You can look up camera in 1729 on YouTube and you'll find me, but there's nothing there about Moodle. A question from Vichendra. What professional development was done to get the users familiar with the new site? So, unfortunately, I don't really have any information about that because we, Catalyst, just handled the technical side, handling sort of onboarding, all that kind of stuff was done by the institution. So, I don't know. Sorry. Another question from Nicholas. Fast forward to Moodle 4.5 LTS release. So, will you simply be upgrading a direct upgrade using the upgrade script? Yeah, that's very likely how it will be done going forward now that we've sort of got the database in a much cleaner state. And we've got sort of a handle on everything that's going on in the code base. Yeah, we would be doing rolling upgrades rather than getting rid of everything and starting again. A question from Paul. How big was the DB instance, memory, CPU? Sorry, I don't have that information off the top of my head. I'm not actually a sysadmin, so I'm not in that every day. 5.44 terabytes. Big. Okay, one more question from Santosh. How did you manage migration if the data is continuously generating on the 3.9 life side? At a certain point, it was just said, stop making stuff for the 3.9 side. That's it, just stop. That was handled by the institution, not by us. A question from David. Maybe I've lost something at the beginning, but how did you guys find out the inconsistencies of the previous database? So, I think I understand the question. It's just often we get reports from the institution saying like, hey, this thing's acting really weird or hey, there's an error saying invalid ID that's popping up on some course or some module or whatever and we go in and look at it and we see something as referencing an ID that doesn't exist. So that's what I meant by that. There would have been heaps of stuff, like we didn't find every inconsistency, we just know they were there and cause we were getting like reports every week or so about this kind of thing and just cleaning up, cleaning up, cleaning up. And that was one of the reasons why we sort of wanted to go with the Greenfield thing and have a fresh DB. Okay, we've got two more questions and then we'll wrap up the session. So there's one question from Danielle. Is the old side switched off? The old environment switched off? No, I think it still exists. I'm not 100% sure on that, but I'm pretty sure you can still access it because there could be something there that they want. I don't know what the institution's plans are for the old environment if they're gonna eventually just turn it off completely or what? And the second part to that question is that does the old data have to be retained or is this old data in the new environment? So you've got that old system. Does the data need to be retained there or now that it's migrated, you can just use the new system? Yeah, again, that's something that's up to the institution. Their choice, we just handle things from the technical point of view. I don't know what their particular policies about data retention and all that kind of stuff are. Okay, and the final question from Vigendra. What theme is used for the new site? It's a custom theme specific for the institution that we developed. All right, thank you Cameron and thank you everyone for all the interesting questions. So we had a very active question and answer session. So thank you very much. So we'll wrap up the session now. And yeah, if you've enjoyed the session, we'd love you to consider getting involved further with Moodle Academy. You can visit our Get Involved course which you can access from the front page of the Academy site. You can suggest ideas for new webinar presentations or courses that you'd like to see on Moodle Academy. And if you want, you can also contribute by co-creating our courses or you could be one of the presenters just like in this presentation. And we're always also on the lookout for community members who want to help us translate our Moodle Academy courses. So if you want to help do the translation of our courses, you could jump into our Translate Moodle Academy course and get started and help us translate our courses and existing webinars into different languages. And you can also help spread the word about Moodle Academy. So you could tell your friends and colleagues about the courses we offer and the events that we run. And when you complete a course, you could earn a badge. And if you're an educator, you could consider completing the Moodle Educator Certification. Are you ready for the MEC quiz? And once you take that, one of our certified service providers will support you through the entire certification process. And thank you, everyone. Thank you so much for joining today. And thank you, Cameron, for your wonderful presentation. And thank you for the very active question and answer session. And I hope that everybody found the session very useful. And we hope to see you in a future upcoming webinar on Moodle Academy.