 I don't remember what happened, it wasn't like that, so yeah, as it says, this is a relatively short talk on solving moodle storage, a year of object storage in the wild, so I'm going to have to talk a little bit about what sort of data moodle stores, so I don't know, is it working? I just need to get closer. Is that better? Yeah, okay, cool. So I'll talk about the sort of storage that moodle uses. So it has a database and a file system, and the database contains all the text and moodle, so your course and activity information, your forum posts, user information like username, first name, last name, passwords, things like that. And the disk contains images, videos, PDFs, that sort of thing, so it's kind of documents and data mostly, that's mostly the separation there, so the file system has got documents and the database has structured data about things. So with moodle disk storage, we've tended over the years to store a lot in our moodles and multi-terabyte moodles are quite common these days, and once you have this amount of storage, it starts getting a little bit difficult as a system administrator anyway, things start getting difficult because if you're going to have redundancy, you need to double it or even triple it if you're going to be storing data in a completely different region as we've been doing on Amazon. And if you're doing it internally, when you start having to back up and store many terabytes of information, it can get a little bit difficult on the IT staff, so I'm sort of used to dealing with maybe just megabytes of information in their on-premises equipment. And when you're storing information on the cloud that can be quite expensive, one terabyte on Amazon costs $100 a month to store, whereas if it's in cloud storage or object storage, as I'll get onto it, that costs $26 per month to store. So as you can see by this diagram, this perfectly explains the difference between object and block storage. OK, OK, it doesn't. So the difference between object and block storage, mostly I think can be boiled down to object storage is entire documents and block storage is individual parts of files. So if I go back to this slide, we could see that perhaps our database probably works better on block storage and our documents are probably better stored on object storage. So block storage is better for seeking small pieces of information than object storage is better for retrieving whole streams of information all in one go. So you can use object storage to retrieve ranges within files, but basically that's what it's best suited for. So I've got this great slide from my colleague Matt, he prepared the presentation also. I'll blame him for the slightly weird slides. So cloud block storage has some limitations. If you've been using Amazon or something like that, you'll notice that the latency is a lot higher than what you're used to if the disk is actually connected to your computer. So in an Amazon data centre or something like that, you'll have the computers at one end of a data centre and perhaps the disks are stored at the other end of a data centre. They could be hundreds of metres away. They could be in a different room. They might even be in a different part of the country. So the latency is a lot higher and there's also challenges around managing, attaching and detaching the storage. So if your virtual machine is starting, some sort of virtual connection needs to go between your virtual machine and its virtual storage somewhere else in the data centre. So there are challenges with block storage on cloud computing and the challenges aren't really there if you have your disk directly attached to your virtual machines or your real machines. So when we started moving Moodle to running on cloud hosting as opposed to on-premise hosting when we had full control over everything, some extra challenges were presented about storing all this information. So my colleagues in Australia came up with this Moodle Object Storage plugin. And I'm sorry, you may not be completely familiar with the way Moodle is storing information but inside the site data there is a directory called FileDur. And we have moved this directory, so the content users have uploaded. We have moved this directory into Object Storage. And our Object Storage plugin supports multiple cloud providers. So by good luck really, everyone has copied Amazon's API for Object Storage because they kind of got there first. And so that made it easy to adapt our plugin to work on different cloud storage providers. It might also work on on-premise Object Storage you may have if you have a SAN or something like that in your on-premise hosting. You may be able to connect Moodle to it with this. So our Object Storage plugin migrates in your documents from Moodle automatically. Once you've set up the plugin, you tell it to start migrating things to the Object Store. And away it goes in the cron that starts pushing the files over to the Object Store and starts deleting them from the Block Store. And as an example here, the use cases would be you offloading large files or old files to save money because of the huge difference in cost on cloud storage between Block and Object. And it also let us discover, well, we discovered that we could point our staging and production in UAT Moodles all at the same Object Store. So no longer when a developer wants to set up a copy of the production environment, they needed to get the whole file store as well. Now that our Moodle is talking instead of to a locally attached disk but to some other store over there using HTTP, we don't need to copy around the site data anymore when we're doing our staging environment. It's great. And it's a bit small but this is the output of the report that our plugin generates and it gives you an overview of how much is stored on the disk and how much has been moved to the Object Store. So as a case study, Catalyst Australia moved two terabytes of their file storage from disk which added up to six terabytes because they had wanted replication at three different places. And this will have saved them hundreds of dollars every month. So lessons that we learnt, well, unfortunately one thing that would have been nice is that we still didn't quite remove the single point of failure that the site data provides. But we learnt that when we were creating the Object Stores on Amazon at least, we had to set up the data retention policies or the replication policies at the beginning because once we introduced these policies onto the store, we realised that these data retention policies weren't being applied to the files that were uploaded earlier. And so when we wanted it to replicate, we realised it was only replicating the files that were uploaded after we told it to start replicating. So we had a bit of a challenge to reupload the whole lot of things back into the Object Store. And I don't know what a contraindiction is. That might be a train that goes backwards. I'm not sure. In the first place, Moodle wasn't designed for storing its information this way. So unfortunately, we needed to write this plug-in. I know there's some core changes that were made in Moodle not too long ago to allow this, but alas, this is kind of tacked on. But you can download the plug-in and you can install it in your own Moodle if you like. We're happy to help you set it up. And yeah, any questions? Yes. So the question was, or rather the question was, why didn't we use a CDN? Because instead of putting the files into sort of a local Object Store, we could have put them on a kind of remote Object Store that is sort of in front of Moodle. And I believe the reason was because we needed to work out how to make the magical link because for security reasons, otherwise, if you just randomly upload all your Moodle files to a CDN, there's no security on them. So you need to put some security on what's a CDN. I know you can do it with Cloud Front and you can do it with many CDNs, but we didn't solve that problem. And we could have, but we didn't. Does that answer your question? Otherwise, the files would be just there for anyone to download if they knew the URL. Because there'd be no moderating security provided by Moodle if the file was available to the requester or not. Does that make sense? It is implementable, but we didn't do it. Yeah, but we didn't do it. Any other questions? Hiya. Hello. I'm assuming since you were able to put the same set of data onto your test and production, if you actually deleted a file completely from Moodle, it didn't delete it from the Object Store? Ah, so the way we have the Object Store running is actually write only. Moodle only has permissions to write. So if you delete the file from your Moodle, it doesn't actually have any way of deleting the file from the Object Store because it doesn't have permission to do it. And there's no facility for deleting at the moment from the Object Store. So yeah, if you delete something in your stage in your production environment, it's still available if you have an older database looking for that file. Cool. Hi. I've actually just bought an Object Store and was considering using it for Moodle, but I think the thing that's holding me back at the moment is just whether there's going to be much of a performance hit when I move the directories over and I just wonder if you've found that. Performance? The performance is pretty good because mostly you don't mind waiting a few extra milliseconds for a file from your Moodle like a PDF or a Word document. Those few extra milliseconds aren't really mattering. If those few milliseconds were for a database, you'd be crying because every database query would all be stacking up and those several milliseconds would be adding up to thousands, but in the case of getting a file, performance hit doesn't seem to be really an issue. So yeah, we've not noticed any issues with performance since we started using this. I'm glad there's lots of questions because the presentation was pretty short. Oh yeah, down here. Okay, fair enough. The question was how does write only work? Therefore you can't read, he's right. We can't delete from the Object Store. We set up permission so we can't delete from the Object Store. So yeah, it's write and read only, but no deletes, you're right. So we can only put an object. We've not given Moodle permission to delete the object. Yeah, that makes sense. Well, we'll have to. Well, we'll just have to go through the database at some stage, write a script that goes through the database and checks, okay. If this file is in S3 or the Object Store and is not in the database, we'll delete it. But the reason we didn't give permission to Moodle to delete is because in case there was a security problem with Moodle or with our plugin, we didn't want to use it to be able to go and delete from the Object Store, that would be terrible. Yeah, yeah, we're going to have to do that. But because of the storage price difference, we're not, you know, we're not rushing to do it at the moment. It's cheap. So when you're looking after multiple Moodle sites, do you set up separate Object Stores for each of them or does your plugin sort of know is it possible for data to end up on the wrong platform if they're pointing at the same Object Store? So do we need to have separate Object Stores for each site? We're running the same ones for the same, the same Object Store for the same customer in our case. You could use a different one for every Moodle, but we're using the same. And we can do that, we think that's safe because of basically the file name that Moodle generates should be unique for that file. If you upload the same file to two different Moodles, they're going to, well, they hash the same. They end up with the same file name. It's the same file. So it's okay if you have two Moodles reading the same Object Store because the same files end up with the same file name. And they should never collide in theory. But for your separate customers, you would always use separate Object Stores? Definitely because if you wouldn't want customer B to be able to list the Object Store of customer A, even though all the files are just gibberish file names, you could just start downloading them and see what they are. And then you would be getting someone else's information. That's great, thank you. Anymore? Oh yeah, one more up there. Yeah. Yeah, so it's not actually the site data directory. It's the file data directory inside the site data directory. So it's that one that's full of AA, AB, AC, that directory, which should only contain the uploads and no cache information and no temporary information and nothing like that. So yeah. So it's maintained by the Moodle file table, that one. Yeah, it's the content hash, I think. Column is the one that we're mapping to the file names in the Object Store. Okay. Oh wait, we've got one more. Let's back to Neil. He has a question. Does it also link to the deleted files directory that Moodle keeps around? Pardon? Moodle's got the directory that stores all the files it thinks it's deleted. Does it put a link into that when a file's deleted? Actually, I don't know what it does with that one, to be honest. So we'll have to look at the code and find out what it does with that. No one else? All right. Thank you.