 I'm Luke, I work for Floriam. We basically build online learning partnerships, so we do everything from campus development to content development to actually selling the content under other people's brands. This talk is basically going to walk through some of the problems that we had actually getting into deployments at Floriam, so this is going from single servers to really large scale deployments. We're going to look at the things that we've learned, what we've taken away from the different challenges that we've hit, the tools that we've built and a few ideas that we've got to take those tools forward. So pre-2015, we were maintaining around five to ten instances. This number is slightly off. I wasn't at the company then, so I don't have anything too accurate. But essentially we were checking in all of the components for our sites into every single instance, so we were working out of one super repository. These were essentially just based on the upstream Moodle code, and we would be checking in all of our plugins into a single branch, and that would be what we'd deploy. This meant that almost all of the development that we were doing across our different platforms was not shared. We couldn't just take a plugin and install it on multiple sites. We were maintaining the same plugins across multiple sites. In early 2015, things were quite simple, though. We had a single application server, nice and easy to maintain. We weren't having to do synchronised deployments. We had a few problems with downtime. Doing maintenance was difficult. We couldn't take an individual server out of rotation to do maintenance. It's difficult to deal with traffic surges in that environment. You don't have any degree of redundancy, and one site getting a sudden spike in traffic can have a really nasty effect on all of the others. It also just became a general performance problem. We were seeing really bad disguise, and it was very challenging. We then switched. We were doing all of our deployments with subtrees, so we'd switched away from single repositories with everything in. We had around 100 separate repositories for each of the plugins that we were maintaining, and we were then just merging all of those in production on the servers into a single branch. This presents challenges. Subtrees are not really an appropriate tool for deployment. The subtrees that we were using for releases were essentially like 50 to 60 plugins being merged. We were merging changes from different branches all in together in one go, and we didn't really have much visibility of what was going live until it happened, so we had a lot of problems. It was also really, really difficult to version because we weren't doing this offline. It was all happening online. Late 2015, things got more complicated because we introduced multiple servers, and now, as well as the challenges of keeping the plugins up to date across the sites, we were also having to keep these sites in sync across multiple servers. We were trying to deploy UI changes, so anything that didn't trigger the Moodle upgrade process, so things like cache updates, database changes, role changes, we're trying to do all of this without downtime. It's incredibly difficult to synchronise those changes across servers, and we hit a lot of problems where certain caches would fall out of sync, so you can have one application server that thinks it's running this version of a plugin and another that thinks it's running another one, and they both compete in the cache. It does mean we can mitigate traffic surges a little bit better, though. We're still doing our deployments with subtrees. We didn't make the change to centralising this work, so our release process, it basically involved logging into each server sequentially and performing a merge there and then. Servers could diverge, and as I say, this leads to some serious problems with things like caches which are shared between servers. In hindsight, this stuff's all really, really obvious. We basically learned that we must be able to promote only the code that's actually been reviewed, so once we've had code UAT internally, that's what we want to be releasing, and we only want to promote that code once it's past testing. We want to know ahead of time what we're actually deploying. The deployments needed to be coordinated across the application servers, so we shouldn't be relying on developers manually logging into servers. We should have tools which can take a release and just apply it taking each server out of rotation one by one. We want to get into CI, but because we have this problem of not being able to do reproducible builds, we can't actually guarantee that what we're testing is what's going live again. We also need a means of making it easier for developers to take a site that's in production and actually get a local copy of it. Taking the latest stable code out of every plug-in branch doesn't necessarily work if that's not what you're deploying, and we have no guarantee. In early 2016, our plan is to go into offline deployments, and we've started this with one of our campuses already. We're not relying on the state of any of the servers. We're aiming to treat all of the servers as like a clean thing from scratch every single time we do a deployment, so the files that are in the tree, all of the source files, we throw them away and we start from scratch. We coordinate the deployments across servers, deploying changes in series either, so depending on what we're doing, if it's a UI change, we can take one server down at a time, apply the changes and bring it back up so we have no downtime, but if we're doing a change which involves database upgrades, it's too risky and we want to take the campus down every time, so we take all three servers down at the same time and we do it as quickly as we can. We've changed that development process as well. We've got like a lightweight get flow, so we don't make a distinction between hot fixes and features. We just have a separate branch for every single ticket that we're tracking internally, and this means that we can deploy individual changes one by one. We don't have to wait for the code that's on a branch to pass UAT. We can just merge these changes offline. We've also added a separate integration stage in so that when we're going to go live with something, we have another round of UAT on that to make sure that no regressions have gone in, so this mitigates a problem about us just deploying the latest stable code again. If the merge conflict occurs and it's resolved by a developer who wasn't originally involved with that feature, it will get caught in UAT. So Component Manager is a tool that helps us facilitate this. It's an open source tool. We've published it on GitHub from the beginning. It's essentially a package manager for Moodle components, and what we're going for here is something sort of similar to Composer, but which actually has an awareness of Moodle plug-in types. It has two main operations. One will install all of the components listed in a configuration file, much like Composer install, and it will just drop those into the source tree. The package operation actually goes to Moodle.org and will download either a zippall release or a copy from Git, and it will actually then install the plug-ins on top. So if you're doing a clean build of Moodle, you can do your full installation and it takes about 90 seconds to get all of the code and build everything offline. The big thing for us was pinning the versions as well. So we generate a lock file much like Composer, NPM, and a lot of other package managers. So we're actually able to reproduce those builds later by just grabbing a copy of the lock file. And it means that we don't have to check the whole site into Git anymore. We can just keep track of that lock file. And ordinarily what it will contain is like a Git reference if you're cloning from Git, or all the information about where we downloaded the file, the checks some of the file if we downloaded like a zippall release. So this is just a walkthrough of what a component manager configuration file looks like. It's quite minimal. So if you're going to use a package operation, you're going to want to set up the actual Moodle version. There's a detailed explanation of these on the GitHub project, but there are various different specifiers for versions. This one will grab the latest 2.7 weekly release. So if you're on the long term support release, this is probably what you want to do. And it'll always grab it from Git. We don't want to check it out from, sorry, it'll always grab it from zip. We don't want to be using anything that hasn't been released. Then we throw in resources. So unlike Composer, where you have one central package repository and packages, we wanted to go for an environment that let you use your own plugins internally. So Floriam has a lot of forks of Moodle community plugins and a heck of a lot of bespoke code. We're up to around 150 plugins at the moment. It's not feasible for us to release all of these because some of them reveal details about our configuration that we don't want public. So we want the ability to grab plugins from Moodle.org, but we also want to grab them off our internal server as well. And then components. And this is literally just the name of a Moodle plugin. The desired version, which can either be a release name or a version number. Or if you're using Git, you can use a Git branch name there instead. You say which repository you want to fetch it from. So in our case, we're pulling it from Moodle.org here. And we can specify if we're using Moodle.org that we want to use the Git branch or the Git reference that's been specified by whoever published the plugin to Moodle.org. A lot of developers don't fill this in, so please, please, when you're making releases on Moodle.org, try to tag them. It makes it a lot easier for us to download plugins. But we're using Zip releases in here. And it's just really, really simple to use. We just go into the folder that we want to install these components into. So this is assuming I've already got a Moodle install. And I just say that I want to go and fetch all the information from Moodle.org about the plugins. We cache it locally because it's a huge list. And then I go ahead and install those plugins. And this takes a matter of seconds. It's around five seconds. Or you can package a whole instance from scratch. So again, I'm doing a refresh. Here I'm specifying that I want to use this specific configuration file. And I'm saying that I want to package that whole site. The package format there can be Zip archive or MS deploy at the moment. Fortunately, we deploy on Windows, so wouldn't recommend it. We also give it the name of the file that we want to dump it to. And again, we say the name of the project file that we want to use. So what we've got at Floriam is a repository of all of the lock files and all of the project files for every single one of our sites. And we just specify the one that we want when we're doing it build. This is really, really good for CI because we have one really, really simple definition of basically running BHART and PHP unit. And all we need to do is change the project name once for each of the sites that we're testing, and we can reuse the same set of steps over and over. It's not perfect, though. There's still some things that we're not doing well. Dependency resolution is something I really, really wanted to get in. But it turns out to be a really hard problem that you can't really solve in a matter of days. So we want to be able to infer from the plugins that you specified in that file what dependencies they have. So we need to be able to download a middle plugin, read the version file, understand the dependencies and then try and infer which versions of those to run as well. We want to be able to have more complex version specifications at the moment. You're only able to say I want this specific version or this specific branch of a plugin. We'd like to have more complex build steps as well. So one of our problems is the middle less compiler is a little bit temperamental if you have a really big theme with a heck of a lot of less files. And one of our sites takes around 30 to 40 seconds to actually build the less. If you're trying to do a deployment without any disruption, that's just not okay. And it can actually take an application server down because it's so backed up on requests. So what we'd like to do is have this build take place offline once we've installed all of the plugins. So we don't have to use the middle less compiler. We can just give it a CSS file. And again, it means that you don't have to check in those static build assets. Another use case might be if you're using soap for anything, you can actually build your client offline rather than have to check in all those definitions. And we'd like to do some work on doing some more CI integrations as well. I'm going to release our team city one as soon as I can get it approved. And I have one working on GitLab as well. But we've got a lot more work to do on those. So I think we're taking questions at the end, but I am finished. Thank you very much.