 Hey, hey, hello Hello All right, cool cool just checking everyone's paying attention So yeah, here we go. Let's do it. Let's do the talk So just little hello. I don't know how many people remember. I was here last year So if you remember me like hello, if you don't nice to meet you Maybe you can see your hello everyone greeting there I don't know the languages of everyone here. I tried to pick a nice variety That last one that is the Australian English way to greet plural people in American English The you can be singular or plural so it's confusing But we invented this yas how he is going So that's how you address a group of people in Australian English So the next thing we have to tick off the list is the quote for the conference like every good talk Needs a quote to like feign that you're cultured This is mine. Hello. Hello. Good to be back Which is attributed to the 20th century poet Hans Peter Baxter. I don't know if anyone knows who that is We've got there he is the man himself and a few translations of that quote Need to do a little bit of a spiel about the organization I work for catalyst it This is a fun slide because I can point to where I come from which is So it's a long way Last time I struggled through hurricanes and stuff. There were no hurricanes this time So I had a very relaxed trip here and I'm happy to be calmly presenting today There's a few picks of catalyst stuff you may have met some there's you know Sam there from the catalyst booth Um, so a little bit about me Um, like I said, I was here before and you might remember me as like the double hurricane guy or something like that This is the same slide from before. So just go through it quickly Uh, yeah, that's my favorite shell. That's my development. Sorry my desktop environment That's my terminal emulator That's my text editor The percentage of largest sites in the hemisphere upgraded is 50 outstanding statistic there Favorite integer is 1729 last time I said I checked them all and I found that was the best one I checked them again, and it's still the best one Pay alleles arms Me cam double hurricane ball Um, okay, what's next So Yeah, this is about the talk Uh, the technicality level is is moderate. I'm sorry if you're a dev and he came expecting to learn like Wicked dev spells or something like that. That's it's not really like that Is a case study. Yes Is as glamorous as the title makes it sound probably not the uh, Allure wears off as soon as you realize most of the world is not in the southern hemisphere So I just want to set some expectations for the talk because last time I talked about queuing algorithms And that's pretty much as exciting as it gets So I've got a graph that I want to share Of the how you can expect this talk to go. So Birth of your first child, you know nice 50 excitement level queuing algorithms. Obviously it's 100 This talk, you know somewhere about 75. Okay, so Just don't want anyone to expect too much But I'm gonna do my best to make it fun because I don't like boring conference slides Um, I have a slide about boring conference slides and a picture of a boring conference slide um And my notes say ha ha ha say something funny about programmers and recursion I didn't figure out the joke yet, but yep boring slide. We don't want that Um, so This next part there is a bit of flashy colors and possibly a bit of loudness. So Do you know which knob to turn in case it's too loud? I've prepared a little video to sort of Make it more fun. Um, we've got the party later, which is 80s themed and I like video games from the 80s So if you'll allow me I made a little introduction video That's 80s video game themed about the talk. So Hopefully it works Hey, so I want to point out everything used to make that video was open source software So when catalyst say expert open source solutions, we we mean it Um, so yeah, it's you know, I mean the northern hemisphere come to the southern hemisphere. Everything's upside down for me Um, and that my friend told me that was how to translate the title of the talking to spanish Um, so I hope it was accurate Um, okay, so let's uh, let's start talking about the actual stuff Uh prologue so now we've got video game font. All right. I just want to give a bit of background about how I ended up doing this stuff Uh How it all began was actually this time last year at this conference Um, I have a little bit of a summary. I was having a chat with my colleague And he asked me did you ever upgrade moodle before and um, I said yes Over a hundred thousand times because I was at the moodle cloud talk It was mentioned that you know, kovat was crazy I was the guy that upgraded the moodle cloud sites during kovat So it's like, yeah, I'm all smug like no worries. I got this and it's like, hey, do you want to upgrade this moodle? and sort of A bit reacted like that. Yeah, okay. Yeah, I'll give it a try dramatic Yeah I put that emoji in there and I liked it so much that I had to It just looks funny So I wanted to have a cool slide with little bits and pieces like statistics about this site but My colleague joey already did a really good talk called maintaining mission critical moodle systems at scale And he had a cool slide with stats. So if you saw that just imagine that slide here But I wanted to put the interesting like one number about this site and that number is 5.44 terabytes and what that is is the size of the database of this site So it's pretty easy. This is big So that number is hard to grok like hard to comprehend So I wanted to put it in a bit more, you know meaningful measures So this is the nintendo entertainment system Which has 2048 bytes of memory So if we express the database in terms of nintendo's it is 2,656,256,250,000 nintendo's So that probably is easier to understand, right? I calculated it if you stack that many nintendo's you get like roughly halfway to mars. So there There you go. Now you can visualize how big it is. It's big Okay, so I want to talk a bit about the sort of way we decided to tackle the problem There's two main ways you can do it, okay? One is to run the upgrade script on the largest moodle in the southern hemisphere The other is to delete the largest moodle in the southern hemisphere and start again from scratch And there was actually a secret third approach from the catalyst ai research labs And I have some leaked footage of that that I'd like to share now as well This has some loud noises. So if that affects you just be aware Hopefully it's not going to be too loud Oh, sorry should tell you what is this thing is get ai to do it So that's the approach and here's the video of trying to get the ai to do it Okay, so that's the last funny video for a little while it gets more serious now But yeah, even large language models are aware of the competence and sheer brilliance of catalyst it That really was chat gpt by the way if you can figure out how I did that Like I'd like to hear your thoughts about about what I did to make that happen Um, okay, so yeah time to be serious Um, a little bit about this site We are actually the third vendor to look after it and coincidentally the only one that they've decided to keep looking after it Um, unfortunately that presents a whole host of challenges that I've summarized in a little simulated dialogue here So these are technical considerations. Oh, sorry. Let me just go through this Lots and lots of content lots and lots of db craft Lots and lots of code. We don't know what it does and that sounds bad, but it's big Moodle's big, right? Like it's not like we don't know what we're doing. It's like there's lots of it and it's weird So, oh, yeah, I have this What is it? Yeah, it's a visual analogy of the state of the largest Moodle in the southern hemisphere Um, this is the simulated conversation. Sorry. I got my slides mixed up there So here we go. They come just like, oh no catalyst. It's a it's a bug What can you do? And you know, so oh, yeah, we'll provide you support solutions and solve all the problems In other words, we're gonna solve it. Yeah And later but not too much later because we're on to stuff. We're quick Yeah Wow catalyst you solved it in timely in a professional fashion. Everything is good So, yeah, no problem. Just had some invalid db state and it's a piece of cake when you're catalyst i.t But How did it happen? How did the db get that way and the user response to that is a much more professional one than this Which is uh Uh That's unfortunately, that's the reality right like sometimes the database is just weird and we fix it But we don't know how it got weird or if it's going to get weird again Um, so that's obviously not ideal the conclusion that we came up with is that the existing system presents too many challenges and risks Tech debt has become unmanageable And I have a updated visual and analogy which is This uh An account revised. There we go. So that's sort of where we got to Um, which led us to the decision that we will go greenfield So that's the delete everything and start again Um, that just seemed like the best way forward. So that's what i'm going to talk about now Uh, we decided to go with 4.1, which is of course the lts the previous version of middle. They were on was 3.9 also lts So Um, I would like to take a brief moment to discuss the philosophy of the project So here's a nice word amortization, which is not normally applied to software, but I like to that's the usual definition of that word Um, but I think about it more like in this context where it's like You want to write a script to do something? But actually writing the script ends up taking longer than it would have taken to just do the task Um, and so as as a developer like I like to write code Hey, but I had to keep this image in my brain while working on this progress project to make sure I didn't sort of go down the The rabbit hole of of coding. So put some restrictions on ourselves, which is Only write scripts when it makes sense and you ask what's makes sense Like how long would it take to do it by hand? How error prone is the task? How hard are the errors to fix? How simple would a script be? constraints if I start, you know thinking like oh, yeah script time automation And it doesn't work in two hours. Then I just no don't do it. Just find some other way And Here's the slide where I talk about How we thought it was going to go. So this is another nice word I'm sure people know this one. It's like yeah lack of experience wisdom or judgment Which sounds like a bad thing, but there aren't many biggest moodles in the southern hemisphere to practice on So I think you can forgive us, but now now Catalyst it has all the knowledge about upgrading the largest moodle site in the southern hemisphere So get in touch about upgrading big moodles Um, so our expectation was something like this delete moodle So something Yeah, yeah, that's what we thought was going to happen But the reality, you know the summary of the simple three step plan What actually happened was something more like this and what's about to show up is the actual Checklist of things we did like the last final checklist of steps to take to get the site configured It's a lot more than three steps, right Yeah There was over 100 like individual steps that we had to follow in the end Yeah, okay So, uh, this is this is where I start talking about what we actually did instead of silly videos and dumb stuff like that So phase one was the core customization analysis. So as you can imagine, like I said, there were plenty of Like hacks and stuff that were were in moodle core So we had to do an audit of all of them to figure out what was needed. What wasn't what the code did Um, so the requirement as I said for each customization work at what it is decide if we need it the strategy Identify every piece of code which is not part of vanilla moodle 3.9 And we're very lucky that there's a tool for it called git So a bit of an obscure one. I don't know if you've used that before Um, so that's the the magic spell that we used to get the differences. Um, I can't show the output of that But if I tweak it a little bit, I can show you that so there were 1,042 Customizations that we had to audit So last time he shows up, I think Um, so tooling for that operation was a spreadsheet Which my colleagues were talking about before spreadsheets are invaluable when doing this kind of thing Keyboard shortcuts to focus windows. It sounds a bit silly But it was extremely useful to be able to just like quickly toggle between windows and check things And a small shell script And copy paste and eyeballs So I think Yeah, so here is a sort of what the spreadsheet kind of looked like This is the catalyst collaborative documents suite That's largest moodle in the southern hemisphere on middle 3.9. That's the commit hash that's Not present in vanilla moodle 3.9 That's a commit message associated with that These are the things we had to start filling in so like is is that commit in moodle 4.1 If it's an mdl, what's the status of it like for example? We backport things frequently and sometimes the mdls just don't get closed So that's useful to know and then finally obviously like is it needed or not? So the goal is obviously to come up with a big list of commits that we need to Put into the new 4.1 codebase So the workflow was select a spreadsheet cell control c the commit hash Use my keyboard shortcut to switch to the terminal Uh, you run the shell script control shift v is how I pasted what was on the clipboard And use use the eyeballs go back to the spreadsheet And you know fill out the fields Check. Oops. Sorry. Check the mdl status if necessary So i'm gonna oh here's an example of the output of that script. So it what the way it works is You show the commit in the largest moodle in the southern hemisphere Then it uses the commit message to see if the corresponding Commit exists in moodle 4.1 because of course the commit hash will be different Um, and it uses the numstat thing Which is useful because you can see if it's identical and in this case you can see it's not So the that server dot php that the real version of that has five changes So something is different about what's in the 3.9 codebase versus what made it into the 4.1 codebase So normally it was okay. No one's just like, oh, yeah, maybe they amended a comment or something like that But that's the kind of thing we were looking forward to see like oh, maybe we need to go and investigate like there could be some subtle difference here That could be a problem I'm gonna I'm gonna try to do a quick demo of that for you so this is Not the real spreadsheet but close to it Oh, it's quite so if I go there and I can bring my Terminal over and I would you that's me using the shortcut. Oh wow, and then I can go check backport Paste the commit hash And in this case it didn't find anything in moodle 4.1 and I know that's because that mdl is still open So in this case, uh, we go back to the spreadsheet Um, whatever commit message doesn't matter the commit in 4.1 is n a the status is open. I checked that ahead of time So that means we Need it. Yay And then we can go to this one and do the same thing again And this time again, it happened that it's not in 4.1 But in this case it was interesting. It's because the commit message is different in 4.1 It doesn't have the full stop in 4.1 for some reason And then the last one is That one is in 4.1 You can see you've got the status there and it's identical So wow well at least in terms of the amount of stuff changed So we can jump back to the spreadsheet and you'll notice it automatically copied the commit hash in 4.1 to the clipboard So I can just paste it in there. And so we just did that Thousands of times to end up. Oh, sorry. I should set that to to know there we go Did that thousands of times filter the list so that we only had the ones with y in that column and and that was that That's what we did So let me try to get back to the presentation And it worked great. It was it was good a bit tedious but Having the shortcuts made it Um go a lot faster Okay Um, I just want to make a comment here. The tiny mce guys have this poster saying that they have 99.9 percent accurate copy paste But with this uh bash script technology Catalyst it has 100 percent accurate copy and paste so They said it couldn't be done Sorry tiny mc guys. I know there's more to it than that Okay, yeah, so when all that was said and done We had a big list of commits and we found some some stuff that was fun We found craft strings. Um, if you're looking for some simple Stuff to work on a Moodle core you could work on removing these language strings because They're just still there and they're not used anywhere and you can see from the numbers the mdl numbers They've been around for some time And generally not much else. It went really smoothly. Um, and Moodle Yeah kudos to Moodle in this case because the policy of maintaining backwards compatibility Really worked to our advantage here Um experience it's a 7.49 out of 10. I'd like to be accurate And I would do it. I would do it that way again Like it was fine. You have to audit the code base and now we had a good understanding of what the code base is like So that was great Phase two is plugins analysis. So of course they have their own plugins some work on 4.1 some don't Um, so the requirement is a way to manually audit every plugin in the code base for each plugin Work out what it is Decide if it's needed And the strategy was get a bunch of humans to do it So we just you know, made a spreadsheet and got four developers on it and you know, just whatever we worked with the spreadsheet Uh tooling a spreadsheet Zoom calls with the client uh because we needed to know if they actually needed that plugin anymore Collaboration there's what the spreadsheet roughly looked like. I'm not going to do the same thing like I did before You know, it's like the path of the plugin on disk the type The, you know, what branch it's on all that kind of stuff. So Um, we did have to have some tough conversations with the client, which is like, hey, we identified this plugin Can we leave it out? And the answer was always. Yeah, you yeah, you can And we'll come back to that later Um, so contributions this was cool in doing this we um We're able to fix some stuff in various plugins out and about Um q type stack was the main one that caused us grief I don't know if anyone's familiar with that plugin. It's a question type plugin where you can specify mathematical equations and stuff like that It's very cool. Um, we fixed their github actions because we needed that to work for our continuous uh integration continuous deployment pipelines, so They were grateful for that And this is I did not expect to be doing this. We wrote a patch to our unit tests to pass Using different maxima's maxima is a thing written in the lisp programming language that actually does the computer algebra stuff um So that it works compiled against any lisp So there's different flavors of lisp whatever you want to call it and as an emacs user It was a joy to get paid to play with lisp Um, and we also improved this project called maxima pool docker It's an abandoned project, but we have our own fork of it and we still use it. It um makes a Process pool of these maxima things and it speeds things up a lot and if you're interested you can That's a link. So when these are online, you'll be able to click it. You can check out our docker file improvements branch We just made that a whole lot easier to work With different versions of the q-type stack plugin. So that was nice. Good Good feeling giving back to the community There is a commit that I made in the q-type stack plugin, which is the highlight of my professional career Just look at that commit message better normalized floats for cross compatibility between sbcl and gcl maxima like And it gets better. I changed a comment in the project from the base of the natural logarithms to oiler's number And you can look at the commit if you want and Euler was thrilled I get little e would be more appropriate maybe but whatever And you know just give credit where credit is due functions super useful And even 18th century mathematicians know that catalyst it is the place to come for all your expertise Okay experience 7.71828 subtract e Out of 10. That's five out of 10 It was okay, but I don't really know how to make it much better e is oiler's number Phase three Oh, man one minute left. That's not going to happen configuration migration We had to manually audit every single config value from core and plugins work out what it is the site if it's needed Bring across other things like profile fields roles web services grade scales schedule tasks Automation was the solution here Naivety again We had this idea that we would with our nice new shiny code base run upgrade dot php on it locally on a smaller database Restore it And then everything would be great in reality. It didn't work out that way because Of things that are really good about moodle and have never caused headaches a lack of referential integrity in the database and back up and restore So the tooling there was this report customer scale plugin which the open university makes a shout out to them php scripts and eyeballs So for basic cases If it's feasible We simply looked at the ui for the 3.9 site click click copy copy paste paste into the 4.1 site Grade scales good example. There's a few grade scales. So that was easy to do If export import functionality exists, we'd use that so roles is a good example And if a core API exists, we use that to make a script. So cohorts is an example of that There's some code like the script is small. We used the Custom SQL plugin to generate a JSON file of all the cohorts in the db and just run the script makes the cohort easy peasy a complex case is Comparing the configuration values across different environments. So obviously they've set things and we need to figure out if that's a non-default setting And whether or not it needs to be set that way in the new environments Um, it's not important, but we wanted to compare these different values the value of 3.9 ui t staging production The value here here here Does does it matter? There's a coffee break. Is anyone want me to finish? Okay Thanks very much, sorry, it's too fun. It's too fun And the value i'm going to slow down now so the value after upgrading from 3.9 to 4.1 So we use the ad hoc db queries to export the fields of the relevant tables mgl config and mgl config plugins Then we use the magic script The magic script has these great features. It had the ability to create placeholder values and the ability to create variables For example, exactly what i mentioned before when you need to specify id's that could be different in the new system versus the old system The ability to just stop and resume because there's like several thousand configs to configure so You know, you want to go make a cup of tea or something just stop it and come back and it picks up where you left off Um, the ability to intelligently skip irrelevant configs that was pretty pretty useful Like if it's just the default value, for example, you don't need to look at it. It's fine And a beautiful ui fully compatible with antsy escape sequences And it fit into a very satisfying 250 or so lines of code So it's pretty easy to write ticked off all those constraints that set for ourselves in the beginning And i'm gonna try a quick demo of that one so So you run the script it's a php script and then you tell it the name of the file you want it to make So it makes a shell script And i mean look at that ui that is outstanding So this one it's see found cfg 15 out of three seven 60 it wants us to tell it what to do So it's like hey, this is this is different. You need to figure it out And so i can pick any one of those so for fun in this case i'm going to add a variable Just called hello And then we go to the next one and we can see what's happened here This one has that value in 4.1 ui t and staging so it's likely that they want that in the production environment So i'm going to go ahead and pick a And you know you can keep going this one they've done this one's weird They had something in ui t then in staging it's different And then in prod it's back to what it was in ui t So who knows maybe we want to come back to that investigate a bit more so That for that one we press ah Okay, so i'm going to close the script here and show you the what it produced So i produced this plug-in cfg thing It's it's just a big old shell script that you can give to your friend this is admin to run And you can look at it and see what's going on And if i start this script to generate that again, you'll see up the top it says 113 It goes back down to 40 because we said skip that one and come back to it so it knows And maybe i've decided okay, just going to skip this entirely and then we jump back up to 113 where we left off So this script was awesome. I got so good at just sitting and be like Getting through all the configs. I really liked using it So back to the presentation Interesting case was the role matrices. So these are like what roles can be as other can assign other roles They had a lot of them And Maybe there is some cool way to do it. I came up with my own cool way to do it I'm not sure if it's the best way but We'll go through this is what I was doing. So If you're sensitive to that kind of stuff That was dumb. That was a really dumb way to do it and I did that like four times Um before having this thought so oh no if only there was a way to write code in the browser that ticked boxes And then I remember this thing exists And I had like a I had a really interesting revelation which was we can just do something like this In javascript get the state of all the boxes and then just spew it out as a json string And then on the other side read that string and apply the state to all the boxes on that site Um, so I'm going to attempt to show you that real quickly So if I just go back to here and copy This So this is this is the 3.9 site got like this interesting pattern of of stuff in the matrix and I paste my code here. I get a nice string nice json string that tells me the state Of every box Then I can copy that Over to the 4.1 site actually I should copy The code here first So I can copy that Go to here where it's different paste that in This is quite challenging because I can't easily see What I'm looking at Then Didn't work, but whatever it does work That's too too hard for me to see I totally did No, it's not working. So at least one of the demos had to fail, right? And it's that one But yeah totally works. Um, and was a Cool solution to that problem. Um, because the cool part was once the checkboxes are all set You can actually like just visually look at it and before pressing the save button So nothing's happening until you press that save button And you don't need like a sysadmin to run a script or anything like that. I was really pleased with Coming up with that weird hacky solution that definitely works Um, so experience 8 out of 10. This is probably my favorite part of the whole thing The config migration script was a joy to write news and I would definitely do that again I would not do the click click thing again. That was ridiculous Um, I hope the boss isn't here because he might be happy that I was spending hours of the day clicking checkboxes Phase four quality assurance. We're getting close to the end now This is the part where we triage issues and fix the real ones Strategy keep fixing things until they're fixed Uh, so the tooling was a spreadsheet surprise patience And that's it. Uh, this is the follow-up to the tough conversations It's like no catalyst the plug-in we want to remove has stopped functioning. It's like, yeah, because We removed the plug-in But we need the functionality from the plug-in So it's like, okay, so you want to install Yeah She like all the configuration migrated Yeah Even after we finalized it and when we said, you know, and it's difficult to do it now Yeah And so we did it the you know, whatever it's the biggest mood in the southern hemisphere. You only get to do it once And they were happy Um So enterprise grade patience from catalyst it here Um, so we made some discoveries bugs You can quickly read them some backwards compatibility breaking stuff was found during the qa So most of these are fixed now. I think so This one Tim, did you know if that got fixed? This was that one where the dom nodes would duplicate this one's bizarre Yeah, and we have a look at that one. It's fun. You set up filters in a certain way And questions just start duplicating themselves in in the dome. It's wild Some minor issue with selectors in that project Honorable mention to debugging curl headers and proxies So this mdl sets this curl opt thing And this is what it affects it affects the header size attribute Because this this header size thing includes the suppressed headers So even though the headers aren't present in the response The header size attribute Like is as if they are there. So if you've got any code that's splitting headers, which you shouldn't be doing Um, that relies on this header size thing and you have a proxy It goes just all busted like it chops the headers in the wrong place Um, and so I filed this mdo. It's not really a core problem It's the problem in this penopto plugin that shouldn't be doing what they're doing if you're using that just something to be aware of Um, and so that was a really great experience that I can summarize here. So yeah debugging curl requests pretty good debugging curl requests with proxies even better Reading the lib curl documentation is a bliss obviously and then reading the source code of lib curl That's what we had to do to figure out what was going on Um, and I'm a serious person and I'm never sarcastic catalyst is just Overjoyed to go to this level of pure dedication reading lib curl c libraries to figure out what's going wrong with someone else's product Um experience five out of ten. You know, it's not the most fun thing in the world But qa never is right. So I don't really know how we could improve that Epilogue very close now Um have another silly video of the state Yeah So it's fixed forever until it's not right Uh, what's next? We've got content migration to do because there's some content in the old site They want the new site. We didn't really delete it. It's still there Um, we've got to update integrations. We'd like their student management system Uh, some plugins that we're developing are in the works. I don't know if they're going to be public They might be but they're going to be cool either way. That's just who gets to appreciate them me or you Um, and that's it. Uh, thanks for listening. I'm really sorry that went over but it doesn't seem like you mind