 Hi, my name is Jeff Sykes. I'm the director of WebOps at VML. And today my talk is entitled, Escaping the Black Hole of Release Management. So I wanted to pick a subject of serious gravity to talk to you guys. Sorry, I was... Ha ha ha ha. So a few qualifications. First, I'm not an astrophysicist. Second, I'm trying to distill what's been a four-year journey into 20 slides. And then third, your mileage may vary as you go through some of the things that I'm talking about here. None of these experiences end up being exactly the same because we're all in different places. So the problem that we faced was that my team spent about 65% of our time doing release management. That was across five different languages, across 10 enterprise web content management systems and at least five database technologies and probably across 40 to 50 projects running concurrently. If you don't think that release management can be like a black hole, then you likely haven't been responsible for it. Release management can suck the joy out of life. It can, sorry, if there's another joke. It can collapse your team. It can do all kinds of terrible things to you. I found this great, I made up this great quote, manual release management is super fun. I love it and it gives you great joy and a real sense of fulfillment. I love scouring configuration files to make sure I update them correctly. I especially enjoy having my weekends and evenings destroyed, rolling code over and over again. No one ever, right? So the first step is to admit that you have a problem, right? Like if you don't admit that you have a problem, then you're never gonna make any progress, right? Hopefully I'm not insulting anyone here with a slide today. So once you admit that you have a problem, the thing that we have to do is realize that unless you're delivering a CI tool, your release process doesn't have any inherent value working software does, right? So the focus needs to move from how we're delivering to what we're delivering. And the way that you can do that is to begin to standardize how we deliver what we're delivering, not what we're delivering. Because at least in my case, I can't change what languages are in use. I can't change the underlying technologies, but I can begin to standardize how we deliver on those things. So once you start doing that, then you begin to realize really quickly that release management, automating release management is a team sport, especially because there may be times where your administrators, your ops teams don't understand the build tools and then devs and ops end up working together and then QA as well. So a couple common pitfalls that I think you can really fall into. The first one is control, making sure that you have controls around when releases go. The second is around configuration, that will really trip you up. And then the final one that can be really toxic is culture. So when you're keeping it under control, that means that you gotta be careful who gets to push the buttons, right? At the very least, you need to know who's doing that. You need to make sure they have enough skin in the game and then you need to make sure that they are aware of things like demos and QA because otherwise you're gonna make them hate you forever and ever. Don't forget about configuration. Configuration should be treated like code. Configuration is code and it really does need to be maintained by a wider team. If you aren't accounting for configuration, you'll never be able to get very far in automating releases. And then you're gonna run into cultural blockers as you go along. These are the biggest problems to usually work with. Some of those are like, this isn't my job. I don't have time for automation or even factors beyond those like, forgot what that one was. So the automation snowball, oh, I didn't have time for automation is that take one task that you have to do every week and find that task and automate that. Then the next week you begin to use that time that you would have spent using that manual task to automate the next task. And then in suing so you begin to build momentum. One of the things that you'll learn as you go along here is that you will, as you automate things, you'll find new discoveries that you didn't know you had. Your swordsman looks really good until you run up against somebody that has tanks, right? And so you begin to discover things like static code analysis and automate QA. And so we will switch and talk about a couple of lessons we've learned. The first is always attack the bottleneck. Whatever the bottleneck is, is what's gonna prevent you from getting more throughput. And so you always end up attacking that. So the first bottleneck was our team. But then as we automated things, we began to discover other bottlenecks like how we hurt QA. So another thing is that culture is much more important than tools. The tools can be switched in and out. The relationships it takes to build the software, the relationships between developers and operations teams or developers and QA and operations, everyone else are more important. Another one is to keep an eye on the big picture. It's really easy to step in and look really close to that one blue dot that you're seeing. And then never be able to step back and see the whole big picture of what's going on. And then finally, don't ever give up. Like manual release management is like a six finger man that killed your father and then you're not gonna ever give up on that. You can't ever let that, the setbacks that you will invariably face ever prevent you from continuing to go forward. And finally, to say thank you to you guys and thank you to all the folks at VML that have helped us make this journey of success for us. So thanks.