 All right, so a little bit of technical difficulty, but that's to be expected This is how to run a reporting and disaster recovery rule instance with Docker I'm Matt Rice from mid-Michigan Community College. Just to give you some idea of scale. We have about 5,000 enrollments per year serving about 4,000 unique students. Can y'all hear me okay? Okay, I got some nods. That's good All right, so Some day. All right. So Audience participation time. Who here has ever ever lost some data lost a picture? I'm really some people are very very fortunate It's been zero days. I someone has lost some data. Maybe not in this room Someone has lost some data somewhere in the world and it's our job as IT people to try and help them stop doing that Single points of failure come in all forms. I mean everything from file getting corrupted to server room catching fire I hope none of your server rooms ever catch fire. It's a terrible terrible thing that can happen to somebody Moodle is no different. It's software, but it relies on hardware or relies on people to very very fallible things So we want to take good backups and we want to be able to help these people that do do bad things and have bad things happen to them I Moodle kind of on the back inside is built up of three pieces You have the PHP code that the developers are working so diligently on and updating all the time and then there are two persistent Persistent storage mechanisms. We have a database of some flavor and a file repository. That's commonly known as noodle data Each of these should be backed up somehow But backing up isn't enough You also need to verify that your backups are valid so that when you need them you can go get them And you need to validate your restore process. How do you get from a bad state back to a good state? So I know I promised Docker so let's talk about the elephant in the room or as my boss said that the whale in the room I Haven't really come across a good way to integrate Docker with backing up noodle data It's it's essentially just just files. It's important files, but they're just files And that's not really as I understand it the way the Docker is designed to run It's more around micro services and running applications Our PHP code a little bit of foreshadowing here We are already running production containers for our moodle So we have containers that are running on a server somewhere that the students get to just get served the web traffic and We'll come back to that a little bit later The database layer it's files But you need some kind of a database engine to run those files for this backup instance We're running Docker for for ease of use We didn't want to set up and configure yet another database server So having the files there and having a Docker image that's Pre-populated from the people that make the software and just have it run is very very very very very effective and Part of Docker one of these orchestration tools as Paul had mentioned There's a related tool called Docker compose which you can put all of your configuration Into one file and then you don't have to remember Oh, what's the exact command to link up this port on this machine to this port in this container When when your server room has just been put out but with a fire extinguisher you really don't want to have to remember very many things So Paul touched on this a little bit I'm gonna explain a little bit more why why we use Docker Consistency when you build an image in a container from that image you can build it on your machine You can test it on your machine or other machines and after it's past testing You can actually deploy that same image out to production and that eliminates the problem of well, it works on my machine Why doesn't work on yours? It's a very very common excuse from several years ago that we're trying to eliminate with use of Docker Infrastructure is code. I mentioned the single file that remembers all of your port mappings and all of your directory mappings Because it's a text file We can use all of these neat tools for revision control like get and we can keep track of the of what changed by whom when and very frequently it's very important to know why something changed and Using a text file helps us do that ease of use like I said, I don't want to set up a whole another database server and Maintain this database server and security updates just for this this backup thing It's important to have but I don't want to take away time from the rest of my job to be fiddling with this server all the time and then FOS stands for free and open-source software Moodle is very good at being open and My my CIO really likes free. He always says it's a very good value to cost scenario Just to throw this out there How many how many other LMSs do you think would let you run just another Moodle instance or another instance of their LMS? Just for funsies All right, so this is kind of what our production infrastructure at MMCC looks like this backup instance that we're talking about is on a single Server it's a VM hosted somewhere else and it tries to capture a little bit of the essence of our infrastructure But I mean it's a flowchart that the users they come into our load balancer Which is we use a software load balancer called paproxy To then direct to each of our application servers most of which are containerized and I'll come back to that in a second and They each access our persistent data. We have a database cluster for great uptime and Our Moodle data gets served off of a content server So that's kind of the who the what the when the why now to get into the how this is really what you all came for this is how you make Docker work and There's a lot of configuration and a lot of examples that I couldn't really jam onto these few slides So at the end I've linked to a get hub repository where we pushed out a lot of example scripts and that and it's about 300 so lines of shell code mostly That will enable you to hopefully get started on your own very similar process if you want So I talked about each of the three things in Moodle that you should be backing up And this is how we're backing them up So the Moodle data as I've said is essentially just files You need to get them from somewhere to somewhere in the Linux world the de facto standard is our sync and On the Windows side of the world you can do very similar things. There are other utilities There are actually packages which bring our sync to Windows under a GUI of some some flavor We pull new files From our production server to the backup server About hourly via our sync and we do it that way because when we're pulling the database backup We also don't want to pull the entire 24 hours worth of files that students have uploaded You know you get towards the end of the year and that's a lot of projects Excuse me a lot of very big projects, and we just want to spread that load out throughout the day The PHP code as I said we're already running Moodle in production containers so we can reuse that container on this backup instance That's what Docker's designed to do. We're gonna use this container somewhere else because we have it as a container We can do special things on our backup instance We can add a few more lines to the moodles config dot PHP and prevent emails from going out from this backup server So students aren't seeing Yesterday's email multiple times. We can also enable debugging on justice server automatically So that leaves just the database And this is wildly different between your your system in mind probably, but how we do it is we have a multi we have a cluster that we back up the physical files and In data backup terms a physical backup you back up the files that live in var live my sequel and that's Different composed to a logical backup where you run my sequel dump or whatever And you get a whole bunch of sequel statements that you can then import into We decided to do a physical backup because prokona provides an excellent tool called Extra extra backup which does a physical backup. It's non-blocking So the students that are trying to access moodle for whatever reason it to in the morning Don't see a downtime It's fast and it watches throughout the time that the backup is running so that Transactions that happen in the middle are reflected in the backup at the end and When that process is done you have a bunch of files that we can then our sync back to this other server So we have all three of our main moodle components. We have Docker that's orchestrating all of these So we've taken our backups. We've tested our backups. We've restored them to a different place all while i'm sleeping And that makes me very very happy So what can we do? I know disaster recovery was in the title Uh Mid is a little bit unique in that we have essentially two locations So our disaster recovery scenarios are mostly pointed at well what happens if we lose exactly one of these I mean that's that's bad, but not as bad as if we lose both of them and that's a whole different discussion So we have all of our production web applications including moodle running at one location and we have our backup appliance And this backup instance running at the One that's 30 miles away if we happen to lose our Backup location. Well, that's bad because we've lost a location but all of our Web production traffic will still be okay because it's in the other location If we happen to lose our production location. Well, that's that's bad, too But we can very quickly manually fail over to the slightly stale data Uh, there are ways to automate the failover But we decided that if if this situation were to happen We would prefer that students see a little bit of downtime rather than Seeing a grade from yesterday, which may have changed We found that students are very very upset if they had a grade and it changed to the better And then they see their grade go down again. That generates a lot of traffic to our help desk So other use cases you had this entire You had this entire instance That's set up that's running and it is moodle. It is your moodle as of About 2 a.m. When these backups completed. So what can you do with those kinds of things? Because it's a vm you can do nifty things like take a vm snapshot and just test an upgrade When you do an upgrade you definitely want to try to use your production data because your production data is different than your testing data Is different than your dev data is different than the toy data that you can create and Sometimes the upgrades only fail in a very specific Configuration that your production data might happen to have So if you do an upgrade and it fails for some reason you can revert to your snapshot Change a few things to your upgrade and just try again and you can iterate very very quickly I guess going from the bottom up here I say ol ap like workloads because data lakes and data warehouses and whatever Data model you have for a querying they tend to use very different Different models in the transaction model that moodle is designed for But because you have this other instance You know it's it's still you're running a query against a database And if you have this backup instance that's separate from what your users are seeing Your queries can be very complex and take a long time and the users don't care because they they don't know It doesn't impact them at all We really like The use of moodle's automated course backups and what this is if you're unfamiliar is moodle can take a snapshot of a course and put that into essentially a zip file Which is unportable two different moodle instances You can restore into the same instance and depending on what you choose you can get the same data into a different course Now this is helpful because i'm sure anyone that has ever restored into a course has had this panic moment Did I restore into the wrong course? And that that certainly does happen Especially in the middle of this semester after you've wiped out some student data. You get a panicked phone call How do I get this data back? If you run moodle if you run these automated course backups You can have this file somewhere and you can very quickly get back to at least some semblance of where you were Without having to restore your entire server somewhere and lose possibly a bunch of other data So as I mentioned, there is a github repository that includes is it should be A fully functional docker environment if you install docker on your server you can run a couple of simple commands and run A version of moodle. I'm not going to say it's a good version of moodle, but it should be a running version on docker Uh and you can do things like that my contact information will be at mid michigan's website Please get a hold of me if you can't catch me after the conference after i'm done presenting here I would love to hear from you if you have any questions as I said This is a very very technical problem that I didn't have a lot of time to cover in technical detail So, uh, are there any questions out there? I hit them all wonderful Thank you Oh, there was a question. Yes Yes, uh, just do you have a Just an overview of the system Okay, so there is uh, there is some place in moodle to configure Uh automated course backups and you can specify I only want things that have changed in the last 30 days or something And it will write them out to a file Uh, if you if you configure it, it will show up in the moodle interface And you can go to restore from that file into a course And it gives you the option to restore this backup into a completely new course So it will create a course shell for you and put that content in there you can Merge with a current course So if you have two sections of a lab and you want some some activities into the other one You can do that or you can and this is the mostly destructive one You can delete the contents of a course and put what's in the backup into that course Uh, we have Some instructors that use this They take from one fall Course and they put it into a different fall course and it's slightly different from the import functionality that exists in moodle Which is essentially just a merge with it existing But this backup it can include things like users and logs. So if you want to Yeah, so if you If you want to know that Johnny was in here two weeks ago and he took a test and he's since Unenrolled from the course, but we still want to see what he was doing We can go back to this backup from two weeks ago Restore it into a simple into another course without affecting the course. It's still running and see what was going on Did that cover your question? Anybody else? Yes Well our our moodle data file our folder is about 700 gigs currently More or less. I think we're trying to shrink that through some various things Our database backup includes more than just moodle And that's about 60 gigs, but it's fairly fast The day-to-day changes between the moodle data is not that great So we have our moodle data our sync runs about every hour and usually completes within a minute The database backup it is much more involved, but still to change To get to get the delta between the existing backup and the one that just completed it. It usually takes less than an hour So so about an hour of downtime on this backup instance That doesn't affect our production instance it because it's essentially just copying files in the background Yes It really depends Depends on what's changed We have had it We had this automated course backups running in production For a while and then it just got to the point where it took too long And it took too much load against the production server that students and faculty were negatively affected So we we turned it off even though we really like it And that was one of the things that we really like about this backup And since we can run these automated course backups And they can take 10 12 hours and eat up all the resources on this on this machine And they don't care because the students aren't using it Trying to think I don't remember the telemetry from the last time we finished this But it seems like it was about four or five hours And we have a lot of courses our moodle instance our production one is about seven years old now It's getting a little long to the tooth Any other questions Yes Yes Yes Okay, so the moodle data what we do hourly is we pull new files So anything that students have uploaded then and then after the database has has run its version of the backup And we've run some work on it There's another Another slightly different r-sync that we run to delete the files that have gone away to kind of clean up after ourselves And we do that after we've done our work to make sure that the files are there and we don't need them anymore yeah, so The example scripts will show this right before we do our our database backup. We tell a doctor All right, we'll take yourself down We don't want to actually be running against these files as as they're in transit and then after it completes successfully We tell it. Well, just run this one simple command and use the information in the docker compose file to bring yourself back Up into a running state Yes We get, you know Back the members delete data and assignment they didn't need to But uh Of course he is Yeah, and I've I've heard of your system before and I'm very interested but we just we didn't have enough hard drive space To be able to work with it. I mean Yeah Yeah, so again oakland college not quite on the same level as mid michigan community college a little bit a little bit different But Yes It is it is completely in the GUI for the admin side. I think it's site administration courses Backups automated backup somewhere in there But it runs through prawn But there is also another way to run it on the command line on demand like we do here I think it uses the same settings Any other questions? Like I said, I'll be around. Please come find me. I'll be happy to talk to you Thank you