 So, this is a quick talk. It was triggered a couple of weeks ago by a Slack conversation about having to update a whole bunch of sites due to a course security release and I think the conversation was kind of like, you know, sad face, sad face, I have to update now. This is really painful and then we kind of got it discussing that, well, we've been doing, at previous next, we've been doing kind of automated updates on a regular basis for some time and I thought it was probably just worth sharing our experience and our approach to doing things and I kind of tried to make it sound really interesting by the top title just because the actual solution is really, really simple. It's not complicated at all. So, hopefully, this will make it more enticing when you're looking at it like that. So, yeah, so just to kind of go through the first points, I guess, for a lot of people, updates can be time consuming. You know, if you've got, you know, 10 sites or 20 sites, even just a handful of sites, when you need to go and do updates, it can take a good chunk of time. It's not usually an interesting thing to do, just to have to do security updates. You know, most people would much rather spend their time on, you know, building your features or delivering value to their clients rather than kind of just maintaining their sites and updating them. So, it's not interesting work and then updates can also break things. So, you know, if you do an update, it has an unexpected change in there, or your code was reliant on a particular feature that's changed and, you know, you want to kind of be aware of things that break and those things that, when that happens, like that can actually be time consuming in fixing things. So, yeah, we want to try and solve that problem. But updates are actually becoming more important now than they were in the past. You know, there are work colleagues who used to, back in the day, like, once they finished a project, never updated the site. If there was like even like minor security updates, they would basically assess them and go, well, it's not applicable to us. Let's just skip it regardless of all the warnings and so on going on inside the Drupal status report. But now we're kind of being forced to update more frequently. You know, the recent Drupal 9 release really showed that it wasn't just security updates that were forcing us to upgrade. It was removing deprecated code so that the modules could be compatible with Drupal 9. And also, we've got underlying frameworks that we're relying on, and they're actually increasing their release cadences as well. So, we rely heavily on symphony, that's being updated at a more frequent pace. PHP is being upgraded at a more frequent pace. That's forcing other things to update as well. So, things like PHP unit, they're just being updated more and more frequently. So, we need to be able to manage this somehow. So, yeah, you don't want to fall behind. You want to be able to keep your head above water and kind of roll with this new flow of updates that are required. And we actually have a bit of a philosophy or a mantra internally here, a previous next, which is always be updating. And every time someone's complaining they've got to do something and they haven't updated, it's like, just go to try and get into this mentality of just be updating all the time. And of course, we want to remove the busy work, the kind of labor out of that, so that we can always be updating without having to carry all the baggage and just streamline the things as much as possible. So, we've had a product called Patchy for a while. It began its life as a kind of a product. When I say product, I mean it was a tool. Back in the day, when we're doing Drupal 7 updates, we wrote something that would kind of scan module files or info files and call APIs and do all this kind of stuff and then generate like, actually do the updates for us. But since Drupal's all moved to Composer, it's actually now just reduced down to a bash script. So, it's actually more of a bash script than a product these days. But we still call it Patchy and it's more of a concept. So, whenever we're talking about builds, it would be like the Patchy build or the Patchy script just so everybody knows what we're talking about. So, the overall goals of Patchy is that updates are automated. So, we want to remove any of the manual toil that's involved as much as possible. We kind of need to rely on automated tests to give us confidence that updates haven't broken the site. So, if you're not doing automated testing, this kind of workflow kind of requires some level of automated tests just to be able to catch things that might break your site just from doing automated updates. And instead of just updating security releases, we just update everything. And the reason I've got the little asterisks there is because like when I say update everything, I don't mean like jump major versions all the time every time you're doing it. It just means that you need to be more careful about what you put in your Composer Jason to your requirements, your version requirements, to make sure that you're confident that you can safely update to newer versions of that, whether it's like you're saying I can safely update Patch versions or minor versions or even major versions, but just make sure that you focus your attention a bit more on those version constraints. And lastly, update often. So, we know that core security updates are Thursday mornings, once a month, contrary of updates, the other two weeks, we basically just have like a weekly schedule on a Thursday morning that just goes and runs patchy. But you could do this, you could do this daily if you really wanted to, depending on how much overhead that brings. And the idea, of course, is to fail fast. So, one of the arguments I've heard from people is, well, for updating all the time, it's just going to break things. Well, the reality is it's going to break things anyway. It's just going to break things earlier where you can catch things and have time to kind of decide what you want to do with things. As in, do you want to, do you need to fix your own code or do you need to, you know, there's a contrary module, need a patch or whatever it might be. But you know early, it's not going to be when you're trying to, you know, you're hitting a hard deadline to get a feature out onto the site and you run into, you know, an upgrade that you need to bump up the version. And then you fail, then you've got to scramble to try and fix things. So, the earlier you kind of hit your failures, the kind of smooth the road is. So, I'm going to just, I'm going to get into the actual script a bit and just talk about how it's all set up. So, we're relying on CICD. So, we use CircleCI here. We used to run our own Jenkins. It doesn't really matter. You can, this will work on, this approach will work on any CI platform. A couple of the key things that we use are, you need a GitHub token with push access, because we're actually going to be, you know, creating a pull request on each project. And the Hub command line tool is GitHub's command line tool. I think they'll bring out a new one soon, but this is what we're using. Because that just makes creating a pull request a bit easier. We're obviously, everything relies on a big composer now. A couple of other things that are really handy are a pre-built Dev database image. So, Nick Santa Maria did a talk on this a while ago, I think. But it's basically what, I don't, I don't think everyone will have this, but we're taking our database image from Dev and then nightly taking a snapshot, sanitizing the database and then creating a Docker image out of it. And then that means that when we run our CIC pipelines, we're actually using a recent snapshot of the database that is safe to be used on something like CircleCI. And then finally, there's another tool called ComposerLockDiff. And that just is a nice way of printing out all the changes that you've made by doing the Composer update. So for setup, as I said, we're using CircleCI. You need to, typically you'll need to generate a new private key. Well, that's the safest thing to do. And we've got like a bot account on GitHub. So you need to add it to that. And then you need to add your SSH private key into CircleCI for that domain. That just allows you to push. And then this is the CircleCI config. I'm not sure how many people are using CircleCI, but it's quite a flexible tool. This is just simply the CircleCI workflow part where we're basically saying there's one job and we're just going to run it every Wednesday morning. Oh, sorry, every Thursday morning at like eight o'clock in the morning, just running on the master branch. So basically, all that patchy builds are just whatever's on master, which is our local Dev branch. And then the job itself, again, is really simple. Some of this stuff is more details of what we're using, but it depends on what images you're using for your own builds. But all you really need to know is that we will need an image that can actually connect to a database and a database image. So instead of having a pre-built database image here, you might pull in an SQL file and then load it or do something like that. But the general idea is that we'll need a running Drupal site to be able to run our update on it. And the rest of it is not in CircleCI YAML. It's actually in a bash script, which I'll talk about in a sec. So this is essentially just a run-through of what the bash script is doing at a conceptual level. So there's not that many steps. Basically, it fits on one slide. So we just need to set the user to that bot user, one that we've set up the SSH key for. We check out a new branch. We run ComposerUpdate. We just check if that generated a diff in ComposerLock, as in, were there any changes? Because sometimes there's not, depending on how many dependencies you've got, there might not be anything new. If there is, then we'll just commit that to that Git repo. We'll then, this is where we need a running Drupal site because we'll run DrushUpdateDB. That basically is there for, if there's any config schema changes, those kinds of things. That will happen in the database and then we'll do a DrushConfigExport. So the idea is that we want to try and capture any of those Drupal updates where the config has changed. Then we just check if there is, once we've done our export in whatever your config sync directory is called. If there's any changes in there, then we'll commit that. Then we just do Git push and we create a new pull request. That's pretty much it. That's patchy. I do actually have the bash here. I don't know whether how useful that is for people to write, but there's a bit more detail in here. You can see there's a couple of nice little tricks in here. Like the checking if the composer lock has changed. There's a Git files command there that you can use in bash to let you have that control flow. We're creating a commit message into a little temp file. And then we're using that composer lock diff command to print out like a nicely formatted markdown version of what's changed. And then we use make as a script, but whatever you're using, we're just running this command's config import update db and config export again. And then you can see we're committing our changes and then creating a new pull request. So yeah, and that's what you get. So every morning at eight o'clock, you get a new pull request on your GitHub repo and you'll get something that's printed out like this where it's essentially here's all the changes. I probably should have mentioned that when I do an update, I propose an update. Yeah, it's updating everything. So every sub modules and whatnot get, or sorry, dependencies of dependencies will get updated here. We just do everything and you can actually go and see exactly what's changed. If there were config changes, then that would obviously come up as another commit that would be in here. I'd say nine times out of 10, there are new config changes and usually it's just simple dependency updates like that. Yeah, and that's pretty much it. So I don't know if we've got any time for questions, but I thought it was worth showing you that like there are commercial products out there that do this for you. I know that GitHub's got dependabot now. It does something similar, but more on an individual dependency basis. But to actually do this is not terribly complicated and everyone can have automated updates on a weekly basis. And yeah, don't have to go through that pain of when security releases out, having to feel like you've got a huge burden. So thanks for any questions. What about automated testing if it does break? How do you then discover that there's a problem with the site? Because, okay. So because we're just creating a pool request, we're not running tests in this patch you built. Its job is basically just to run ComposerUpdate and then create a new pool request. And then we're assuming that our pool request has its own, every pool request has its own, the set of automated tests that run. So yeah, often, I mean, if your ComposerUpdate breaks something, then your automated tests should fail. And then you've got to obviously jump in manually and resolve things or work out why it's failed. But the idea is that you should be running this regularly enough that simple updates shouldn't break things all the time. I don't know if that answers your question. Yeah, yeah, that's it. And as you say, the little updates rarely break things. Yeah. Kim, is there a cost across all of the CircleCI instances because they're private? Is there, sorry, costs? Is there a cost for you for the CircleCI? Uh, look, I don't know. I think, I don't think so. I mean, we out, the CircleCI plans are by the compute unit now. So yeah, there is some cost but minimal, really. It's not much. Yeah. So I mean, the costs of CircleCI, the benefits of that far outweigh the manual effort of going in and having to manually, you know, do updates and push your own pull requests. So yeah, the idea is that we have project maintainers and they'll come in on a Thursday morning of a course security update. There should be pull requests all sitting there waiting for them. Most of the time, they're just green and they can just go, yeah, there's a small change. It's past all the tests already and just merge and then get on with doing deployments and things like that. So, there should be minimal effort. Thanks very much. Thanks, Kim. Yeah, thanks, Kim. That was good. Just, uh...