 Hello everybody, and thank you for coming. I know it's getting to the end of the day So I will try to keep this talk short and sweet My name is Evgeny. I'm a head of engineering at Systemseed. We are a proud boutique agency who does tech for good We work with governments with NGOs charities Academic institutions as well as other social impact projects So let's begin Someone said there are three things you can watch forever Fire burning, water falling and other people working However said this clearly I haven't experienced automated software updates Otherwise this list would have one more item. I would like to show you something really cool Something I'm personally incredibly proud of and every time I see this it feels like Christmas to me That system seat we use Slack for communication with our team members and clients and here's my favorite messages chain in Slack First we get a notification that security update is released Soon after that updates bot picks up software and creates pull request with proposed code changes to apply the update a Few minutes later our infrastructure creates a clone of the production environment with updates applied Soon after that we get notification that automated regression tests have passed for the created environment When checks have passed The bot can finally merge pull request with changes to the main gate branch The code change get automatically deployed to the production environment with the latest software updates and a cherry on this pie a Notification that post deployment production tests have complete has successfully completed and we know that production is not broken Neither from the functionality perspective or from visual perspective So within an hour after security update had been released We've got it already on our production of a website. How cool is that? Does anybody want this in their own organization anyone? You're in the right place Let's talk about this So a bit of history and how system seat got into software updates automation It all started about three years ago With a automation of security updates. We as an organization as most of you do as well want Wanted to update to automate security updates because it kind of makes sense We all know that we do this from time to time and it was our dream as well So we experimented with the soft security software automation tools. It was generally successful however There was one really annoying thing quite often Automated updates failed due to way too outdated versions of software of other software or dependencies That the update got So and that required some manual involvement this kind of This was killing the whole idea of Securities updates automation because what you want from that system is not only to save some death effort to pick them up But to be able to deliver those security updates immediately to your production environment kind of keep your project secure So we ended up changing our mantra for from if software works then don't touch it To actually keeping the Software up to date all the time And that's how we ended up updating all software And not just security Releases I'm sure you all do software updates. You all do security Updates from time to time. It might not be perfect. It might take more manual effort than you would want It might cost some of the Hiccups in the past in the history, but it kind of works, right? So the question is that you might wonder is does it really worth it investing your time and energy into building things like Like that, let me explain what are the main benefits that we discovered as the team first one is security I Personally like so much telling our clients that as soon as the world has just learned about a security vulnerability It's all your projects are already safe It's a very big deal and it cannot be underestimated The second one which is obvious time and money. I'm pretty sure that I Didn't even have to put this item. However, the question you might ask is how much actually time And effort it saves There's one example. I can give it to you at system seat. We have projects on support Some of the projects not all of them, of course I'm not going to pretend that this solution will solve all of your problems with all of your software updates forever For the whole time being it's it's not the effect. However, we still do have projects who we haven't touched Manually for years and if you open them today, they all have all the up-to-date software and all the latest security updates, obviously I hope that it sparks some interest in desire in this topic next one is reduced risks because having more Frequent in the more and smaller incremental updates give you gives you more time to respond to breaking changes So when an urgent update has to be released, nothing gets in a way Then reduce technical depth So having up-to-date software makes it kind of natural to use the latest features latest best practices following the latest documentation and so naturally there's going to be less technical depth As you're using though all the time the latest The possibilities and the opportunities of the software Last but not least feeling good because the confidence that your projects with hundreds or thousands of dependencies Are always get late latest updates Makes the team feel proud of what they do of the company they work in and It makes them stand out. That's incredible feeling There are obviously some more items in this list Indicate team will find your own benefits of software update automation. A good thing to mention here is that Moving towards software updates automation disrupts your Delivery processes on so many different levels in a good way, of course that it just takes the delivery Practices to completely next level Let's talk about practical aspects of software updates automation how to achieve this actually If you ask a developer how to automate software updates They might start talking with you about Automating writing a script to automate composer updates and PM updates or whatever package manager they have updates kind of thing If this was it, I would not be here The whole complexity of software updates automation is in building Automated deployment process to the hosting infrastructure and building confidence That whatever you're going to deploy to your production environment is not going to break it Software updates automation stands on four pillars Detect apply test and deploy Let's start with the first two There's a huge variety of ready solutions offering to look after the updates Much more than logos I put on this slide really so you can look into Drupal updates initiative You can pick any third party service available in the market really it comes down to your team choosing the right tool that works for you There are tools that help you update automate updates of Drupal like WordPress your React a npm dependencies docker images versions and containers anything you can think of basically there are tools for that So basically my only duty here is to warn you from not Reinventing the wheel and not going the custom route. There are definitely solutions that will work for you Testing is the deployment is far more interesting problem to solve The ideal flow you should aim for is when an update is detected your infrastructure should create a clone of the production environment run Automate Deploy the changes there run tests to ensure no visual or functionality regression and deploy changes automatically This requires a couple of things First of all hosting solution should be ready ready for this It should be able to spin up clone of the production environment or synchronized data from the production environment to get the fresh version of The development environment you're going to be testing this on and Allow automated code deployments the great news here is that most popular Drupal solutions Solutions known in Drupal community. Sorry. I ready to handle this problem for you I'm sure I haven't put all the hosting solutions that even present on this Drupal con. That's that's not the main point We at system seat even ended up building our own Infrastructure with AWS on AWS with Terraform and Kubernetes It's long story. It's definitely not for this talk However, if you're interested if it happened that you also need to own your own infrastructure and that's outside of your domain Feel free to get in touch with me We've built this infrastructure using the infrastructure as a code approach and so it's easily replicable on your own AWS cloud Next thing is automated testing At system seat we embrace Combination of end-to-end testing and visual regression testing and to end tests They spin up a real browser and emulate real user behavior who navigate through the website. They click links. They Fill in the forms submit the forms. They can check Text on pages and so on So basically end-to-end tests. They do exactly what an end user can do On your screens, you can see a video of actually me pretending that I'm a test bot during Doing this thing because it does test bots They do exactly this thing apart from the fact that they run it in the headless browser And so there is no visual thing that I could put into the slides So I ended up cheating a little bit Also, there's going to be a buff tomorrow on automated testing around by our team member So if you're interested to get more into this subject, you're welcome to join it starts at 9 15 ish I am and It's called automatic acceptance testing. I've put a couple of tool names here on the slide So that if anybody is curious to explore They know what names to look for at system seat We standardize on code section for the past decade and very extremely happy with this But not only for this reason there are a couple of fathers However, if you are new into the end-to-end testing, I would recommend starting with something like Cypress first. It's more modern and it has more Inspirable first user experience and you will like it first more What visual regression testing does it takes screenshots of the website pages Before and after the change is released and compares them automatically if it finds any changes it then interrupts the whole deployment workflow and asks an Engineer to review them manually those tools are configurable So you might be wondering what if a certain part of the website as opposed is supposed to be dynamic You can configure those areas and the inside schools. It's not a problem So it's very flexible and configurable, but at least it gives you a confidence that Changes that you're deploying will not break the visual representation of the application Same with with other pillars There are also plenty tools that help you with the visual regression testing. We use tools called Diffie and Percy We prefer Diffie. It's a great tool built by our Drupal community member You regress them off and we find better developer experience with that. However Percy that is part of browsers tech now. If you know this service is also interesting for its own features especially for testing in Different browsers operating systems and so on so forth for those companies who don't do testing automation yet It's important minds of chip that has to happen to start relying on tools for QA for QA instead of humans The rule of thumb that we have at system seed is if automated tests pass then the code change is safe to deploy You should treat automated tests like it's your QA engineer If they say changes are safe to deploy you do the deploy you don't ask for a second opinion if Initiatives found you don't hire a second engineer and fire the first one, right? You help the first one to get better and only this approach will help you Will get you the results that you're looking for The last but not least is automated deployment of tested code changes to the hosting infrastructure This is where you find a holy grail of engineering automation that is called continuous deployment Tools like circle CI Travis CI Jenkins or similar they give developers tool to create consistent deployment experience developers create a script How code is deployed to the hosting infrastructure and the tool just follows the exact Instructions to deliver the code to the right place Sometimes when I speak with my friends from other development agencies I ask them why don't you implement automated software updates? Their answers can be split into two main categories They say our clients don't have budget for it Well clients who come to system seed also Don't start begging to write tests for them or do the automation of the delivery processes and the Codepointments and guess what they all still have it The truth is the clients who say that they don't have a budget for automation Are the clients who don't have budget to waste on manual quality assurance on manual deployments on manual software automation? The whole purpose of automation is to be able to achieve more for the same budget not the other way around So they definitely wanted they just don't know that they wanted Second most popular answer is our developers have haven't written tests before it feels complex time consuming They are not interested at all. I Remember myself like 15 years ago starting with writing tests and in Drupal Those tests took insane amount of time and energy with quite debatable outcomes really and I almost hated them likely these days There are tools which make test writing very easy and enjoyable experience I can assure you that once you invest some time into building a good testing infrastructure As a company you can start asking your parents or kids to help write you some tests. It is that easy these days to give you a feel of how much time it takes at system seat to Write tests when a new project kicks in So let's say if it's a three month ish project then it would usually spend up to week maximum to build a solid QA foundation That covers the essential user journeys on the website It's efficient to run to start doing the automation like out to be automating software updates And then when the project grows we just build on top of this foundation Like one week is literally nothing within a project, but it helps you do all of what I just described So to summarize my talk if you want to run software updates automatically. Here's your checklist hosting infrastructure that supports the creation of the development environment and code deployments Automated detection of outdated software. Remember we agreed no custom solutions Automated automation of deployments to be able to deliver the updates to the hosting infrastructure and the last automated regression testing of features and or Visual representation of the website. So for each of these areas There are tools that can help you with this journey You just need to make sure that You just need to actually make the first step Start playing with them and then you will start Making sure that they play well together if you need any help Guidance or want to just chat about that feel free to catch me or other systems users at our booth we have Lego you will not miss it out and Yeah, that's it. Thank you your questions are welcome any questions. I hope it's enabled Hello, hello, how many times it failed because when my developers Updates a website There is always always a bug at least a bug and then I wondering when it's Automatically Is it really better? Oh That's a very good question. So How often it fails, let me start the first one I was thinking if I should include stats or not like I've mentioned for small projects That are simple enough. We literally don't touch some of the projects for like years It just works and that's quite a cool cool time saver saver. They generally meet each size projects They fail Maybe 10 to 15 percent of the time big projects 20 to 30 percent of the time due to the dependencies the complexity really increases However, what I've mentioned also in the talk is that still the beauty of this sort of updates even when they fail is still that This smaller but ongoing incremental updates like it failed fine We can assign a person like within a week or two or three even weeks to solve that problem But at least within three each weeks your software will be up to date again And so when something real kicks in and you need to update immediately Nothing gets in a way. That's still important Part of the software updates automation. It's not only coming down to the fact how often it fails sometimes it also how often it forces Developers to keep the code base up to date and the second question was Doesn't really help Oh, yeah, it does it seriously does we're especially for us We are a small team you have more projects than we have developers and we're always had this problem like we don't have capacity to handle an update and When first of all started to take in out of our hands some of the Simpler stuff that's first relief and then it started to Take some of the more complicated projects updates Sometimes they handle them automatically and that does work and still in majority of the cases for us it more I'm just thinking what's the best way to put this Sometimes you need to fine-tune The solution that would work for your team for your capacity to make it work Because it's not like you put an updates in place while I'm you having a Automates working it's not like that We spent a couple of years polishing it fine-tuning it and we got to the stage where we can pretty much rely on them Thanks. Thanks for the question. Hi. So, um, hello, we are Actually developing Kind of the same at our company at the moment So I'm pretty much loaded with questions, but I reduce it to two of them The first one is How do you act if or how does your automation act if there is a stage or a feature stage system With unapproved change and feature requests by the customer still laying around On an older version. So, uh, how does it act if that's the case and the other question would be is your Solution a business secret or will it be open sourced in some kind I Will start from the second question first if you don't mind because there's no Secret sauce as such you see in my talk. I've been mentioning tools tools for each of the pillars there's What we've done is a little glue that makes them play together really It's not a thing that you can just open source or lease as a product So we've not done anything smart apart from going quite far in our automation processes And that's why I'm saying that it's you should try this too. It's not that complicated at the end of the day if you do If you do it right and the first question was stage The way we work is we don't have this problem because of the different processes how we deal with those How we deal generally with the development we don't keep anything on stage at all the way it works we have For each bug or a feature we create its own standalone development environment and So if client did not approve that development feature on that development environment We just don't deploy it in our case staging works not like Pre-production environment that would hold a feature for a while it uses it's used only for one purpose to test how Automated deployment works to make sure that it does the right thing that automated tests pass And when they do the feature goes live direct goes to the production straight away. We don't hold anything on stage They do test new features, but on the development environments So just to give you an example imagine there is a feature Big feature one and we put this on a standalone development environment We're just building for each feature we work on or bug or release We create its own development environment You might have five ten fifteen development environments and as soon as the client has tested the one feature They accept Story and it goes to stage and then the production if nothing breaks does it make sense? So we are not holding anything Exactly for the reason that you mentioned because we don't want anything to get in a way of deployments as soon as it's accepted It just goes live From its own standalone environment Does it help answer the question