 She's on the ball, or they are on the ball. Excellent job. How are you? If the talk gets boring, I might just do some more jumping jacks, and we'll see how we go. Keith and Mario's guide to continuous deployment. It says Keith and Mario's guide, the keen observer in here would notice that there was only one of me. This is Mario, he couldn't make it. He stuck in Melbourne, but couldn't make the conference. He could all send in some heart emojis on Twitter. I shouldn't put his handle up there, but it's Mario who's in the bottom right hand corner, so send him some love and it'll be great. So that's Mario, this is me, Keith Pitt. That's my Twitter handle. That fluffy thing on the left isn't part of my beard. That's actually an animal. Once you get a beard, you start fascinating more about beards. And the other thing is, this guy, his name is Harley, he's from Epic Nealtone. I like his beard a lot, except the outsides are a bit rough for my liking, so I cleaned it up, and I think that's how I would wear the beard. When I rehearsed this, I was under time, so I thought we could just look at some beards together. This dude's beard is quite nice. This is Data from Star Trek. He can't wear a beard, but this guy can. It's a lot of gel in that beard. If you want to know more, I would check out beards.org. It's just the form of beards. I like eating apple products, it's pretty mean, but I'm not growing a beard, this is what I do. I also like this one. And more information about me. I won South Australian edition of the year in 2007. Those birds weren't there when I won. There was the one that was photoshopped in afterwards. But that's what I did when I won. My owner and I used to work for a company called Envato that make theme forest, Coke Canyon, a couple of slides like that. Together we worked on a slide called Envato Studio, it was microlancer, it helps freelancers sell their digital services online. I work in payments. My owner and I also make this thing called desktop together. It's a site that lets you download desktop wallpapers to a Dropbox, so as you scroll through, wallpapers are like, I like the visual cap. You download, it will send you Dropbox, then you change your background thing to come into the Dropbox folder and you get instant wallpapers, it's magic. Plug, I work for Buildbox now, for time, recently, it's having a full time on it. It's a continuous integration service, but it's a little different. All the tooling is hosted, the UI is hosted, but you bring your own service and you get complete control, flexibility, security, you run it behind your flower wall, you don't have to deal with Java or Team City installations. And if you go to the stink, you get two months free of Buildbox, buildbox.com. Anyway, plug's over, let's talk about Australia. I'm from Australia, it's a little island off the coast of New Zealand. You say, we pronounce it straga, drop the A-U. I'm from the left-hand side of Perth, and this is our central business district. It's kind of what most of Perth looks like. Plug, another plug, Rouse Camp Perth is happening in November 14th, 17th. I would feel like to come, come find me afterwards. Pretty much, this is the campsite. We're kind of in the middle. We've got a climbing wall, got some archery, we've got a flying fox, snorkeling. We may have some student tickets available. So if you're coming to Rouse Camp Perth, come find me afterwards and I'll give you some more details. So, well, that's about Australia before I get on to what you're actually here to see. This gentleman here, 1954, this is Bob Hawke. He's sculled 2.5 pints of beer in 11 seconds. He later became our prime minister. The first police force in Australia was made up of the most poor, ahead of criminals. You're doing less murdering today. Go Chief of Police. The community and the kangaroo are at our crest because neither of these animals can walk backwards. So they can't pull off the little stand-time. The average Australian will consume 165,000 eggs in the lifetime. That's very interesting, I know. And Australian billionaire politician, Clive Palmer, wants to build a new Titanic, a replica you call a Titanic II. Just, these are not jokes. These are 100% that are actually serious facts. And Google Clive Palmer is politician. All right, everyone. Everyone put your hands up for me, everyone. Thank you so much. All right, put your hands, don't put your hands down yet, I didn't say anything. Thank you. Put your hands down if you don't write tests. It's okay to put your hands down. I won't shame you much. It's okay, put your hands down if you don't have a CI server. Put your hands down if you don't automatically push your master branch to production on successful builds. Who's got the hand up? Ah, yeah, you get, high five, high five the guy next to you because he had his hand up. Are you? State? No. It does not count, unfortunately. It's just production. So a few people do it. And you may have heard the terms continuous delivery, continuous deployment, they are different. And it's easy to get confused between the two. In continuous delivery, testing is automated but the deploy to production is manual. In continuous deployment, the entire pipeline is automated including the deploy to production. So you might be asking yourself, what is this continuous deployment Voodoo you speak of, Keith, and why would I want to do it? Well, because of YOLO. Well, well, not really YOLO. The process is pretty YOLO but the code is not. I found when I was doing continuous deployment, I would be writing some code and I would knew that in about 10 minutes time this code would be on production. So I really changed how I approached particular problems. I was much more careful than what I did. I was much more thorough in what I was trying to achieve. I wasn't just throwing some code out there and whoever deploys that next week it's going to be their problem to deal with because I know it's not going to work. Deployments are tied to my name. So Keith pushed this, it went to production and now he's deleted all the users. So I tend to be much more careful in the way that I approach problems. There are three qualities of continuous deployment that I say are important. One of them is fast feedback. We want deploys to be fast in this new world of AB testing and those other things that require fast deployments. You want fast feedback from your production environment. So doing continuous deployment makes that work because your entire process is automated. You commit, you push, time passes, production and there's really no human involvement. You're much more confident in your deploys because these things are automated. You've written scripts to handle the entire deployment process. Migrations are handled for you. Testing is handled for you and if you trust the tests then you should be able to deploy a production of master straight away. And I used to do a lot of manual deploys and manual deploys now scare me because I forget, have I tagged this thing? Have I not tagged it? What's the tagging structure? Like have I committed this thing to staging, right? Have I run the smoke tests? I've stuffed up a couple of deploys in my time because I'm trying to rush something out and I just forgot to do something. So it gives me the confidence and it's scalable. If you've got, if you're doing maybe one or two deploys to master a day you can scale up to maybe 30, 30 deploys a day quite easily and when you commit to master and the deploy goes out you end up doing much more deploys anyway. So it scales really, really well. I like Venn diagrams. Let's turn this thing into a Venn diagram. Great. It doesn't really need to be a Venn diagram but I thought it could be one if it was good. And this is the thing there in the middle. So everyone take a photo of this. Show it to your boss. I'll be instantly convinced that this is a good idea. Everyone's got photos? Great. Okay, so continuous deployment. As the famous saying goes you can't do continuous deployment without breaking your few omelets. So let's talk about what we're gonna need for a continuous deployment. All these things. There's many of them but they're all very, very easy. Let's get into the first one which is feature planning. Okay Monday you come to work. Your goal is to make this new feature. You should talk with your team about how you're gonna write this feature and deploy this feature all in the same conversation. You talk about what's the smallest thing that you can ship today. Let's say this feature is quite large. Try and break the feature into much smaller features. So in ones that you can ship today. Add this link here. Boom. Add this thing here. Boom, boom, boom. And by the end of the week you've done 10 deploys and the feature is complete as what the customer originally wanted. But you've got lots of smaller deploys and small increments in the code base. Can it be deployed with zero downtime? Maybe you're adding a new piece of infrastructure to your production servers. Maybe you're adding Redis or changing databases or something. You should probably talk about if, does this feature that we're about to embark on have any implications to zero downtime? And we'll learn more about what zero downtime is in a few slides time. And how are we gonna handle database migrations? If anyone here has tried continuous deployment before, you know that database migrations are a pain in the bum. So we're gonna learn about how to deal with those later on. And you've seen it talk about them during those conversations. So code reviews. This is a picture of Bill Gates who's in the 1980s version of GitHub. He's just looking at paper. Code review is really important to continuous deployment. And GitHub plays a big part of that. The pull request system is completely amazing. It fundamentally changed how we do code reviews. When Mario would do a feature on desktop, he'll write some code, submit a pull request, I'll plus one it, I'll hit merge when it, sorry, he hits merge when he's happy and I've plus one it, goes to master, master goes production. Boom, very, very simple. But sometimes, this is a picture of a pull request of someone adding a flux capacitor to a project. Yeah, that happened. But sometimes PRs can get a little off topic when you're talking about white space or quotes or like hash and tax. So there are a couple of gems to kind of just make those decisions for you. One of them is cane by square. It does some static analysis. It looks at line length and it looks at the ABC complexity of your project in error if it doesn't look quite right. And this one here, Rubocop, is a robust Ruby code and analyzer based on the community Ruby Star Guide. It makes sure that you're using the right hash and tax and you're not doing anything crazy looking. And those two gems will just make the CI process a little bit easier for you. Feature toggles, one of my favorite parts of continuous deployment. The idea behind a feature toggle is just a mechanism to flip a feature in production on or off without doing a deploy. Now how you really do that is up to you. It could be a database table, it could be Redis, it could be environment rebels. Doesn't really matter. As long as there's no real deploy to production involved. I like to think about feature toggles as just being really ghetto. Let's say there's a feature that I want to implement and to get to that feature, you have to go through some button or some link. I would generally just hide the button in the feature toggle. I won't really bother about hiding the page or anything like that. If the user finds the page, they found a broken feature, that's great. It's nice little surprise, treasure for them. Nice broken feature. FetLife made this gem called Rollout which works with Redis. That's how he does feature flippers. This is the one that I use personally. It's very, very simple. You can define groups of users. So I'm gonna roll this feature out to developers only or this group of users only. The other one is called Flip by PDA. It's a bit more enterprise version. It's a lot more features and it's got a web UI that allow you to control stuff. With Rollout, you have to go into the production console and do it manually. Automated tests. Automated deploys need automated tests. It's just kind of the way it goes. Mario and I write just enough tests to make sure that we trust what we're gonna do. There was a fad a while ago to test everything. We don't really test everything. We test just what we trust. We're not sure about this, let's test this. Just because we've embraced yellow in the way that we do our processes and not writing tests for everything is a part of that. We also don't write all the tests because we can keep the tests fast. Now, I know you're wondering, yes, I just said I write less tests to make them fast. Yeah, I did. But there are other ways to make your tests fast aside from just not writing them. Corey Haines did a talk on fast Rails tests. You can watch that. You just Google fast tests by Corey Haines and he'll show you how to write them fast. Another type of test is called a smoke test. Now, the idea behind a smoke test is that it runs on your production server after a deploy. So if you're doing a deploy manually, you go type cap deploy, you'll watch the output stream by, and then when it's finished, you'll generally go to production, hit refresh, great, that's all working. The smoke test automates that particular process for you. It can be pretty easy. It can start by just checking to make sure that the name of your product is on the homepage. It doesn't have to be too difficult. Just something to hit the page or hit your key flows, like maybe it'll go purchase something with a real credit card or maybe it'll, I don't know, sign up. Just something to make sure that your main flows in production are working. The way that we do that is by using Capybara. Capybara makes this really easy because you just give Capybara an endpoint and you start, like, click this button. It's this link there, that sort of thing. It's even better if you can run those same tests locally against your development machine and on production. So if you're just taking to make sure the name of your product is there, you can run these tests locally, point them at your local environment, it'll make sure that they run, and then when we put this script as part of our actual deployment process, it'll also check to make sure that it's on production as well after a deploy. Zero downtime deploys. Now these are probably, this is one of the more harder bits of continuous deployment. It's just the act of deploying code to production without a maintenance page. Because we're committing to master a lot more, every commit to master will cause a deploy to production. That will just mean more deploys to production. And if you've got quite a bit of traffic, seeing this all the time is gonna be annoying for your users. So deploying to production without the users noticing that a deploy ever happened is really important. Now if you're on Heroku, you'll notice that after a deploy, the next request is quite slow. The way to fix that using Heroku Labs Pre-boot thing, if you enable this option, your users won't even know that a deploy happened if you don't turn on maintenance. So that's quite a useful thing to know. If you wanna know how to do zero downtime deploys with Unicorn, there's a great Rousecast about it. Just go to Rousecast and search for zero downtime deployments, and it'll show you how to do it with Unicorn. Now, database migrations. This is the hardest part of continuous deployment. Changing the database while the code is still running is very, very, very tricky. And there are really two ways to do it. The easy way is to use the maintenance page, like I told you not to do just before. So the flow would be write some code, push the master, in your deployment script, it would turn on the maintenance page, then do the actual deploy to production, then turn off the maintenance page. That's the easy way to handle database migrations in a zero downtime process, but the hard way is by doing two deploys, and you'll know in a second why it's so hard. So if let's say you're adding a column, this is one of the more common ways of, so one of the more common things you're doing in a migration, you add a column. You start off by deploying the migration out of column. Great, and then after that, you deploy the code to use the new column. That's how you would do add a column in a zero downtime deployment process. Now, removing a column is actually very, very tricky. So you would have to first remove all the code. If you wanna remove a column, you have to remove all the code that uses that column first. Great. Secondly, you include some dirty, dirty hacks, and now you're about to find out why database migrations are very, very tricky. This is me removing the nodes column from the user's table. This is pretty innocent. I deploy this and I start seeing this everywhere. Even though I'm not using the column anymore. Like why is this thing still inserting into the nodes column? I do not understand. It's because of active record. When active record boots, it actually looks at all the columns of a table and then has a case of them internally. And when you insert a record, it will use those columns. If you don't provide a value for one of those columns, it will just use nil. So we've removed the column from the database. We haven't restarted our Unicorn workers. We haven't restarted Rails. It still thinks it's a nodes column and we're gonna get this error like we see now. This is what makes zero downtime deployments very, very, very tricky. And so the way we solve that is just by hacking active record. We just get it to ignore the nodes column. So the process of removing a column with a zero downtime migration is remove the code that uses the column, include the dirty hacks, deploy that code to production with the hacks, deploy the migration that then removes the column and then remove the dirty hacks. That's a real pain, but that's the way that you remove a column with zero downtime. There's a great article by Pedro Bello and he'll show you how to, it goes through every single migration type and it will show you how to do it with zero downtime. Also be aware of database locking. If you add an index to a table in MySQL, there is a good chance that the table will lock and if you're adding a table to your user's table or your sales table and those tables lock, then in production, no one's gonna be able to write to those tables for the period that it takes to run like at the index. So if you're adding index takes 10 minutes, if you've got a lot of data, then you've just completely broken signups and purchases for 10 minutes. You can use the Large Hadron Migrator Gym from SoundCloud that'll show you how to do that. That'll show you how to set that all up. Now, in production, stuff sometimes breaks and so really rolling that back to ploys is a very important part of continuous deployment. Stuff catches on fire occasionally. Servers catch on fire. You know, people's wallets catch on fire. Stuff of it. I'm sorry, Aaron. Should I do it again? Stuff catches on fire sometimes. Servers catches on fire sometimes. And so, what I do when stuff catches on fire is I generally just try and roll forward. I don't freak out too much about it because I know I'm gonna make a mistake eventually, so what I tend to do is I, the first thing I do is just turn the feature off. Because we've used a feature flag, it's a very, very similar deploy. If we've introduced a bug into this feature, I'm gonna think, oh, crap, this is not what we wanted. I'll just turn the feature off, go back to the code base, fix it, deploy again, turn the feature on, and I hope that it works. Another way to do it is just to get revert and then get revert, commit to master, automatically deploy to production again. You'll essentially just roll back and commit. GitHub have made this really easy recently with the revert button on pro requests. I'm not sure if you've seen that, but you can merge a pro request and revert that particular merge commit in the UI. So you can roll back a deploy from GitHub. And the great thing about GitHub is that if you use pro request, the merge pro request button actually becomes your deploy to production button. It's very, very, very cool. And in the bare grills worst case scenario, you can always just turn on the maintenance page. Even though I'm telling you not to do it, you can always do it if you really screw up a deploy. So you can always just do that. Now, monitoring is really, really important. You wanna know someone's looking at your stuff when you're not. And so, exception tracking is very, very important. I'm gonna assume that you're all doing that already. Performance monitoring with New Relic is, oh, sorry. This is bug snag. This is my favorite bug tracker. I don't really follow them. I don't really know them, but I just like it a lot. Performance monitoring is very, very good with New Relic. And you can also do business magic monitoring as well. So instead of looking at how fast a request takes, you can start looking at things like sales per minute. And those are very interesting stats to monitor. So if you do a deploy and there's a change in your sales or signups, maybe they've dropped by 5% after this deploy, you may have introduced a bug and not know it. And PagerDuty makes it really easy to get notified. You can plug all those services into PagerDuty. So you do a deploy, so you commit to master, or make this deploy to production, you go to lunch, and then you get a phone call because you've just introduced a bug. It's pretty powerful, but a bunch would be kinda stuffed. So these are all the ingredients you need for continuous deployment. Those are all the things we're gonna need. So let's mix all these ingredients, and I'll tell you a story about how I use continuous deployment to make a feature. This is my current deploy to production script. I don't do anything. I just pushed a master on Heroku. This is probably the most yellow way of doing continuous deployment. I wouldn't recommend doing this, but this is really good for Rouserumble. Next thing we're gonna add to our deployment script is running the quality checks using Kane and Rubocop. We're gonna run our tests, we're gonna push to production, then we're gonna run our smoke tests that I told you about before. Now, this is even more safer, but to make our deployment script even better, we're gonna push to staging first. So we run the quality tests, we run our actual unit tests, we push to staging, we run our smoke tests on staging, we push to production, then run our smoke tests on production. And I think that's a pretty good deploy to production script. And if we take that deploy script, take all our ingredients, and I'll tell you the story. This is me at a conference. I had a great idea, social links, social buttons were very popular these days. I wanted to add them to my side project desktop. So I did some code locally. This is what I come up with. We've got Twitter, we've got Google, we've got Facebook, and we've got two LinkedIn buttons, just for that extra business wallpaper sharing. Those recruiters need wallpapers too. So I was pretty happy with this particular feature. It was great. I pushed it to a branch and then created a pull request. This is Mario. He's very busy working or reading a book of ponies in the bottom. He's pretending to lag, but I know what he's doing. I submit the pull request. I usually include a GIF with all my pull requests. It's part of continuous deployment that GIFs are included with all pull requests. So that's the GIF that I chose. Mario was pretty happy with it. He tested it locally, and I'm sure that it worked fine. So... Ignore that. That was a separate project, I'm sure. So he's happy with it. I'm happy with it. He's plus one the pull request. I hit the merge pull request button, which is now deployed a production button. This script runs, and plug billbox would say maybe do it for you. Or do the specs, deploy to staging, staging smoke test, production, production smoke tests. And if all those things are green, then desktop will go to production and nothing looks any different. No one knows I've deployed. No one knows anything's changed because I use feature flags. I turn on the feature flag for developers only. With rollout, you can scope to particular groups of users. I've scoped it to developers only. I then turn it on. This looks great. It's exactly what I expected. I'm happy with the feature now. I'm gonna then turn it on for all of the users, and then boom. We've both achieved continuous deployment, and that's a nice gear for you. One more time. Right, right. You know how that is to do. It's actually very tricky because you need to look at the camera and do it. No, anyway. You're welcome. So we've got our three things. We've got fast feedback, confident scalability. Have we achieved them? Yes, we have. We've got the fast tests. We've got, you can actually make your deploy as much faster by the way, by only doing things that have changed. One thing I didn't cover, but I didn't have time to, was in your deploy process, it's generally push code to production, bundle, run migrations, and then restart. If you haven't changed, oh, you've also got asset compilation. You can actually change your deploy script, so only do the asset compilation if the assets have changed. Only bundle, if bundle has changed. The deploy script for one of our side projects takes a second to get code to production, just because it doesn't really do anything. And that's essentially to production and get pools and just restarts unicorn. And those fast deploy are very, very cool. So we've got that. We've got the confidence because everything is automated. We've got scripts that handle everything. We've got monitoring in place. And the solution is scalable because it's scripted. We don't have to worry about it. We doing 10 manual deploys a day is gonna get really annoying really, really quickly. Because this thing's automated, it's really, really cool. I have some science to show you. We asked a team that does continuous deployment to give us a Git log of, and the deployment log for a two week period. These are the same size teams, the same size project. And this is the results that we got. We found that the team that does continuous deploys actually does four point seven deploys per day. And the other team only does one point nine. And interestingly enough, the changes per deploy were much lower on the other team than it was for the manual deploy team. So it's more science for you to show your manager. So that's kind of it. That's continuous deployment. You should always be continuously deploying. That's kind of my Mario's motto now. So you should do that. It's always be shipping is another motto, but it's not as good as this one. So that's it, continuous deployment. Keith Mario's guide, thank you very much. Thanks Keith for the great presentation. Any questions? Well. It's because I answered everything they were gonna ask. If not, let me try this. I have a question. Ah, sure. So when you were starting to do this continuous deployment process, it was, I assume, an incremental process. Did you face any challenges implementing this? Like were there a lot of things which started breaking when you first tried to do that? Do you have to change your workflow in order to accommodate these changes? Yeah, absolutely. So in this particular project where we tried this on, we actually started on a Greenfield project and we decided to do continuous deployment out of the box. The biggest problem I would say we had was actually Heroku. Because we're using Heroku, it's really hard to run migrations on Heroku. The migration needs to exist on Heroku before you can run it. And so we had to do two deploys for everything, which was a real pain. The way that we solved that was actually running, shouldn't be telling you this, but I will. We actually downloaded the database URL from Heroku, set it locally, ran the migrations from our machines and put it at production, so we didn't have to deploy the migration. But there was a lot of pushback from people in the team as well because they haven't really done this sort of thing before. But if you ask anyone in the team now, I'm sure they'll swear by it and manual deploys scare them. So that was, yeah, the biggest problem was probably Heroku. How do you coordinate infrastructure changes? Like do you have, if you're on Heroku, I guess not, but maybe you have experience with teams that are using Chef or something like that. How do you coordinate with Ops if you need new infrastructure for a new feature of the developer that you're rolling out? Yeah, so this is the process that we use for most of our commits, like maybe a typo change, maybe we're gonna add a new form. But you've got staging. There's nothing saying that this week we're gonna do manual deploys. You've got these scripts you can run them whenever you like. A couple of times we said that we're gonna turn off order deploys this week while we work on this big infrastructure thing. So this is just a tool. You can use it however you like. But no, I haven't had any real experience with that sort of thing because we're on Heroku, but I'm sure it can be done. Akita, have you ever come across a case where you've had a racing deployment? Instead somebody commits a branch, it's deploying when somebody else merges another full request. Oh, so yeah, so there's two deploys happening at once. Yeah, at the time we were using Travis CI and I think we only had one worker, so anyone built it right at once. So that's how we solved that problem. Was it the fact that we're gonna do one deploy at a time? But I'm sure there are other ways to do it. With Jenkins and TeamCity, I think you can configure your pipeline steps to be only one of these can run at one time. But yeah, you just only, if you split up the pipeline to multiple steps, you can just control when that step gets run. But in that script that I show you, you would definitely have that problem if two deploys run at once. I'm not sure how much of that a problem that is on our review though. Hello, after all of this, what do you find to be the most fragile or continuously fragile part of the process? That's a really interesting question. All the fragile bits we ironed out, migrations was tough, but then we figured out how to deal with them and so they weren't really fragile anymore. The fragility with me would be doing it manually again because I'd have to look at the script and be like, okay, so we tag it at this point and we deploy this tag to this one and there's that tag, particular naming scheme. But there's really nothing fragile about it. Like once you've got those tests in place and all those checks, then it's really, really simple. And manual deploys scare me much more. Not much, sorry, I'm just saying the same things every now and again, but yeah. Cool, if there are no other questions, thank you, Keith. Thank you very much.