 Cool thanks everybody. Issues getting the display going. I am going without my notes here so hopefully I remember everything. Really quick about me. I'm the founder at Lincoln Loop. Here's a shot of a few of us at PyCon last year, this year. And at Lincoln Loop we do Django development and consulting and everything that goes along with building Django sites like DevOps, JavaScript, UX, design. Today we'll talk about deployment which is something I'm kind of interested in at the moment. I'm also the co-author of this book High Performance Django and it's all about how to build Django sites that scale and are fast. You can get a copy at highperformancejango.com and there's little coupons in your swag bag as well. So today's talk is Django deployments done right. But really I probably could have called it Python deployments done right. There's not a whole lot of Django specific stuff in here and honestly deployments done right probably would have fit most of it as well. Deployments for web applications all look kind of similar. So whether we were doing Rails or Node or Python or Django or whatever, a lot of these things will still apply. So first off, what is deployment? I think of it in two ways. One is the noun. The deployment is kind of all your stuff you have set up, your servers, your software, how it all interacts and then the verb, like the actual act of getting all that stuff onto your servers and keeping it up to date. So today we're mostly going to be talking about the verb but I may interchange these a little bit. So why do you want a good deployment? First off, uptime. This is obvious. You don't want deployments to take down your site. That's a problem. You don't want to be scared to do deployments because they're going to take down your site. But maybe less obvious is the humans that are working with the deployments. One thing I've noticed is having bad deployments sort of feels like this. It's like showing up at a messy office. You can't find anything. It's stressful. There's probably one person that knows where the thing that you need is and there's these gatekeepers and it's really bad for people. I do a lot of consulting work and I work on lots of different sites and I notice the difference at the end of the day when I'm working with a site that's got a really poor process and not well managed and kind of a mess versus one that's kind of clean and tidy and does what I expect. It's stressful. It tends to lead to burnout and burnout leads to people leaving your company. So there's a case that the humans are very important to the deployment process. For me, this is a good deployment. It's kind of neat, tidy. You know where everything is. There aren't a lot of moving parts. It's just simple. Good deployments are non-events. It shouldn't be hand-ringing, palm-sweating like, oh gosh, I hope nothing breaks event when you do a deployment. It should just happen and nobody really notices. And good deployments are also empowering. Good deployments let all your users in the sense of developers get software onto servers where it's in staging or production. That's our job as developers. We build software and we want the world to see it and sort of not letting people manage that process of getting their software onto production and having gatekeepers in the middle is really kind of limiting. So empowering developers to do this stuff is pretty powerful. And then finally, there's a business case for good deployments. So if you can't convince your boss that burnout and stress is a bad thing, good deployments, you ship faster. You get new features out to production quicker, you fix bugs quicker, and you recover from failure more quickly. In my notes, I had something from the State of DevOps report by Puppet Labs and they did a study on the differences between high-performing IT organizations and low-performing IT organizations. And the numbers are really fascinating. If I had my notes, I would tell you what they were, but it's like they deploy 30 times more per day and they are 60 times faster to fail or to recover from failures and all this stuff. So there's a really big business use case here. And then also having happy people that are at your company, happy people are more productive and they're going to leave less and you're not going to have brain drain and be training people all the time. So there's a great business case for good deployments. Now what makes a good deployment? First off, it works. It's surprising how many times we see people get this wrong. They've got a deployment process and they deploy and we had an issue a while ago where we made a bunch of changes to somebody's salary worker queue and they deployed it and things broke and nothing worked and we said, well, what happened? And the one guy who was in charge of the deployment said, oh, I have to go in and restart salary manually after we do deployments. Nobody else knew that, but the one person did. And so this kind of stuff is really common. So when it works and it's reliable, it works every time. And in the off event it doesn't work, it doesn't break things in the process. Next up it's boring. This is kind of the part where I go on a little rant about Docker. Docker is really cool technology and I think it's interesting for a lot of stuff. If you're just getting started with deployments, it's very likely not the best choice for you. Docker solves a lot of interesting problems, but it also kind of brings in a lot of problems of its own. So I prefer kind of using technology that's tried and proven. And Docker you end up using a lot of kind of bleeding edge technology. It's great for a lot of use cases, but for your standard, hey, I just need to get a Django site out on the web, it's probably overkill for your purposes. Next up it's user friendly. So everybody should know how to use your deployment system. That doesn't mean they need to know all the internals of how your system works, but they should know how to get the code from their laptop to get to a server. Next up it's fast. I'll give you an anecdote here as well. So we worked with somebody who had a deployment system and the idea was all their tests had to pass every time before they could deploy their software. Sounds like a really good idea. We want to make sure our test suite passes before we put it in production. Their test suite took something like 45 minutes to an hour to run. So when they wanted to do a deployment, they needed to wait an hour before their servers ever got updated. Well, a problem snuck through their continuous integration. It didn't get caught in their tests and they deployed a change that took down their sites. They knew what the fix was. If they could push it out, it would have been a 30 second or a minute long issue, but instead it had to go through this hour long pipeline to get into production. So fast deployments are really important for recovery and they're also important for closing the feedback loop with developers. We write code, we test it locally, we write tests, we put it on staging environments, but truly we never really know how it's going to operate until we put it into production. So being able to give your developers that quick cycle of, hey, I built something, it's in production, it works and I can move on to the next thing is really important to productivity. Next up it's not disruptive. You shouldn't have to deploy after hours. Your deployments shouldn't take down the site. You know, even for a few seconds, people shouldn't get 500 errors when deployments happen. And then finally, a couple of $5 words here. It's idempotent and deterministic. In deployment, these words are kind of blend into each other, but basically it's idempotent. You can run it over and over again and you get the same thing and it's deterministic. Given the same inputs, you get the same output. So when you deploy, you should expect that if you deploy on one server, you're going to get the exact same thing on another server. So to me that's a good deployment. Now how do we build all this stuff? So first off use configuration management. It's kind of key to this all working. If you are not using configuration management today, look at Salt or Ansible. They are Python based and they also, you're going to probably be most comfortable with those unless somebody in your shop has experience with Chef or Pop It or one of these other systems. Fabric is not a configuration management tool. It's a remote execution tool and you can do configuration management like things with it, but you're going to paint yourself into a corner and you're going to have this really messy system. There's better tools out there, so don't use it for configuration management. And then use it for everything. You shouldn't configure your servers with your configuration management system and then update your software with other thing and then do deployments with another thing because that's kind of violating the don't repeat yourself principle and you're going to end up in situations where those systems diverge and one stomps over the other. So once you kind of get everything codified into configuration management, that's what you want to use to sort of drive all this stuff. That includes setup, updates, deploys, everything. Next up, pin your dependencies. This sort of is that deterministic thing. When we deploy our software, we want to know exactly what we're getting. So that doesn't mean in your PIP requirements you put install Django or even install Django less than 1.9. There's a chance that a minor version update, if there's a security issue, could be backwards and compatible. So make sure you pin to specific versions. One thing I would consider is actually just using setup.py and putting your requirements in the install requires there. That sort of gives you a constraint that you can't link to crazy third-party GitHub repos. You don't want to count on some random person on the internet's code or repo being up or existing when you're doing deployments. Now, there's certain scenarios where you can't use PyPI. Maybe you've got proprietary code that's shared among multiple projects. Maybe you need to fork something and it's not available on PyPI. In that scenario, consider setting up your own PyPI. It's not that difficult to do a simple index, which basically is going to get you what you need. There's a project called PIP2PI that will take a requirements file and dump out a directory structure that's compatible with PIP and you just point PIP to that and you can install stuff from it. So copying that to a server that has Nginx or pushing it up to S3 is a pretty viable solution. Next up, if you've got code that is you've forked some project and you're diverging significantly or the other project is dead in the water and not getting updates anymore, consider just vending it. Put it in your project. The fewer external things you need to manage, the better. So just drop the code right in there and take ownership of it. Then finally, if you insist on installing out of Git repo or something like that, make sure it's when you own and don't count on other people's stuff. Next up, this may seem obvious, but I see a lot of people that don't do this. When you deploy, you should reload and not restart your services. So the difference is during a restart, you basically tell your WISG server or whatever to shut down and come back up. You're going to have a couple seconds during that time where all the requests coming into your application are going to get dropped. So what I would recommend doing instead is graceful reloads. What that does is typically your WISG worker is going to have multiple processes that are serving requests. It's going to let the old processes finish. It's going to bring new processes up alongside them. Once the old ones are finished, they die, and the new ones are already serving new incoming requests. Your front end proxies probably support this as well. Nginx, varnish, HA proxy, and that just looks like a service Nginx reload on most Linux variants. It works out of the box with upstart if you're using the current Ubuntu LTS, and if you're on system D, putting this line in there, the exact reload, which is going to send the HUB signal to your WISG server, which most of the WISG servers will do the graceful reload when they get that signal, that'll cover it. So next up, making it user-friendly. If you're using configuration management, you might have a line like this if you're using Salt that is going to SSH into your Saltmaster server and update all your web servers. That's not good. And Ansible, you know, if you're doing Ansible, it might look like this. Also not good. It's kind of exposing all the details of your configuration management system to your developers, and you don't need to do that. Much better is something like this. I said Fabric is not a configuration management system, but it is really great at remote execution. So write a deploy function, and all your deploy function does is call these other ones. And if your users are coming in and have used anything like Heroku or something like that, they're going to be much more comfortable with a script like this, or a Fabric command like this, instead of the other options. While you're in there, consider adding some other stuff for your users. And most of these are wrappers around really simple, you know, kind of bash command line options. So, you know, hey, I can deploy to production. Can I easily get a branch out to a server for other people to look at it? That's a super common scenario. You know, hey, I'm working on something. Can you guys look it over? Making that scenario really easy is great for them. Another common thing, you know, hey, I need to just check the logs on the server. Something weird is happening. They shouldn't have to SSH in. Anytime somebody is SSH-ing into a server, consider that an opportunity for improvement. And you know, the status, hey, what version of the code is running on the server and, you know, is, are all our services up and stuff like that. So, Fabric has this really nice dash L flag that will list all the Fabric commands. Here's an example from our Lincoln loop website. And it has a bunch of handy stuff. And we went through most of them. It also has like, hey, I need to open up a Django shell or I need to see what release has happened or roll back to a previous release. When you run those, you get something like this as a user. There's the server status. It shows you like the load and who's logged in and what version of the code is running and when last time you whizgy restarted. So, next up on how we do, how we do a good deployment is isolating each build. So, this was something that wasn't really feasible a couple of years ago and wasn't super easy up until a few months ago. So, what we're talking about is basically every time you build a new, or you do a new release, you're building a fresh virtual M. You're putting all your dependencies in it. You're putting a copy of your code in it. Probably even a copy of your static files in it. And you sort of have this bundled directory that has everything to run this version of code. It's really nice for a few reasons. One is if you do have something that breaks down during the deployment, you're not doing it in your running version of the code. So, the typical process is, hey, I've got my code on the server. I'm just going to update it on the fly and install any new requirements I have and then reload my server. If something breaks during that process, you've now broken your production site and are scrambling to fix it. So, this way you can kind of build one in the background. Once it's all ready, you switch over to it. It helps with that determinism. You don't have any craft left over from previous builds. Has anybody here been bitten by a stray PYC file that you deleted everything but the PYC file is still there and imports aren't working? Yeah. So, stuff like that. Old versions of stuff that you've now removed from your requirements but are still in there. All those types of things can kind of bite you and prevent you from having kind of a true build. So, the other nice thing that this creates is it's really easy to roll back quickly to previous versions in the event of an issue. So, I'll show you that in a second. Here's again as an example from our linkinloop.com website. It's really simple but the same thing applies for larger sites. So, we have this sort of home directory for our site. It's got a data directory in it. In that goes any sort of user generated content, stuff that isn't version. So, user uploads and stuff like that. It's got a deploym.json file. I'm not really going to go into this but that's kind of a 12 factor type application thing where we store secrets like API keys or your database IP and password and all that. So, that comes in from the configuration system and then the app reads it. There's a package cache which is sort of the secret to making, being able to create new virtual ends every time really fast. So, the change that happened a few months ago is that PIP now when you install dependencies will build Python wheels. Wheels are like binary kind of blobs of Python modules. So, if you have C extensions like common ones, Psycho, PG2, the Postgres driver, or Pillow, your image manipulator, those usually take really long time to build like on the, you know, minutes. So, that made it really hard to build fresh every time because your deployment got really, really slow. The nice thing is that PIP by default now creates these wheels and sticks them into a cache and we'll check there first for, if you have an updated versions it's almost an instant install. So, using that package cache and keeping kind of outside of our versioned repo we can reinstall everything really quickly. There's a checkout of our repo. Uwizgi cache, you can ignore it's, we're actually using Uwizgi's caching mechanism. There's our Uwizgi configuration file which tells Uwizgi how to run and then there's a virtual ends directory. So, this is what the virtual ends directory looks like. Those are commit hashes of our code and the date, you know, each one's a directory. It's actually a virtual end and you can see the date on there that shows the time when that one was deployed. And the trick to kind of making this all work is you don't run out of a commit hash, you run out of this current directory. The current directory is just a sim link to whatever is currently running. Uwizgi points to that. Any sort of like static file serving points to that. And so, updating to a new one or rolling back to a previous one, all it is is you change the sim link, you reload your services and you're on a new version. So, this is really nice for a bunch of reasons. One is if you have, here's an example I ran into. So, we had a site and it had dependencies and those dependencies have dependencies and one of the subdependencies got updated and broke one of our dependencies and that dependency wasn't pinned down the chain and so we had a working version of the code. We deployed a new version and things broke and we weren't sure why. And once we finally pinned it down, you know, you try to like release a new version or roll back to the previous version and it's still broken. You know, you're using the same code but something else down the stack is broken. So, using this, in the event when those things happen, you can flip back to a previous working version of the code even if your dependency tree has broken, even if, you know, PyPI went down or GitHub went down or whatever, you always know that you have some good working version that you can quickly flip back to and that can be priceless. You know, other scenarios we ran into like, hey, you deploy code and somebody put something in it that broke and that person's not available right now and you don't have time to dig in and troubleshoot it. Hey, let's just punt and roll back and, you know, we know we'll be safe and we don't have to muck around with our get history or anything like that. So, the other interesting thing that this kind of gives you is the ability to build once and deploy everywhere and that's kind of one of the big things that people like about Docker, you know, the promise of Docker that you build and you know exactly what's in there and you can ship that image to multiple servers. So, what you can do is have a build server. Maybe it's Jenkins or something like that. A new commit comes in and it starts building all your dependencies into these wheels. You can build it all into, throw it all into a tarball and ship that archive off to all your servers, which simply unpack it and run it. You don't need to install all your development headers on your production servers. They just live in one spot and everybody else gets pre-compiled wheels. There's a project by Armin Honahair from, did Flask and Jinja. He's over here with Century Today called Platter and it's a really small Python script. I don't know, it's probably four or five hundred lines that does all this for you. So, you can also stuff all your static files in there and it's a really nice way to kind of isolate Python builds and be able to ship, you know, a single file out to your servers which you can easily unpack and run. Next up is database migrations. So, some of you may have noticed, like, if we're going to easily roll back to previous versions, how do database migrations play into that? You know, what about backwards migrations and all that? Quite frankly, I've never seen people really use backwards migrations effectively. The idea that you can, you know, move forward and then move backwards. What I would recommend instead is just maintaining backwards compatibility for a little bit until you know that your current code is good code. So, if you need, if you think you're going to delete a column, don't release the code that doesn't depend on that column and delete the column in the same deploy. Release the code first and once you're sure everything's working, then release a migration which is going to delete that column. So, you always maintain kind of a buffer that you can roll back. If you have a really large database and get a sufficient amount of traffic, you can run into the risk of locking your database with a migration. So, you know, maybe you're adding a new column and it needs to have a default value or something like that and, you know, your database starts ripping through all the columns. It puts a lock on the table and nobody else can write to the table until that migration completes. So, if you have a scenario like that, there's a post that describes it all in detail much better than I could. It's there, it's by also in my notes, Ludwig Hahn, I think is his name and it basically gives you techniques on how to identify migrations that are going to be problematic, how to split those migrations up into multiple migrations that will allow you to do the same thing without locking your database and it's a really great resource. And I think this is the last one, tracking releases. So, you want to, when you release new code, know that new code was released and know when it was released and what was released. This is like hugely important for regression testing. It's really common on big high-traffic sites that you're moving fast and you're releasing new code and a week later somebody checks your app metrics and holy cow, we're 30% slower than we were a week ago. What the heck happened? And you've put in tons of new stuff in the interim. Being able to go back and see, you know, hey, on this date, we released this commit and performance degraded immediately after that is huge. And it's frequent that it's not immediate. It's like, hey, I need to know from a week ago or two weeks ago or something like that. So getting this data really close to where you're tracking performance is helpful. If you pay New Relic a bunch of money, you can get them to, you can do this. Op-Beed includes it and you can even just push it to a Slack channel. It might not get it close to your performance, but at least you have that data somewhere that you can reference it later. And usually these are just like a curl API call and really easy to add into your configuration management system kind of at the last step. Like, hey, new code's deployed, make a curl call to this REST API and it'll handle all the release tracking for you. So to wrap up, the thing I've noticed with building our own configuration management system and building it for clients over the years is probably the biggest difference and the biggest, you know, thing that helped us was that we were always chipping away and making it better. If you're not using configuration management today, you don't need to sit down and stop everything and spend a week or two to build out the perfect configuration management system. Start small. Next time you need to add a new user to a server, do it with a configuration management system. Next time you need to change a configuration file, put that into your configuration management system and slowly chip away over time. Think of the humans that need to use this system. That anecdote I gave earlier about, hey, you know, oh yeah, we have a configuration management system, but it doesn't restart salary when you deploy. That's going to come back to bite somebody, like, you know, soon. So when you run into those issues, fix them. Figure out how you can, you know, any time you do have a failure, figure out what went wrong and how you can fix it and make that fix. It's not something that you need to invest tons and tons of time into, but you need to kind of chip away at it and make things better and eventually you'll get to a system that's really, really nice and really reliable and has all those features I talked about earlier. If it's slow, look into why it's slow. Can you make it faster? Can you, you know, do some caching or something of some files like we talked about with the wheels? You know, that's one thing we ran into with our deployments. We were doing some node stuff for front-end files, you know, building stuff with gulp and it took, I don't know, 20 or 30 seconds and I spent an hour looking into it and figured out how to, you know, cut that time in half. All that time adds up and if nobody ever looks at it, you're going to end up with a system that's slow and people don't want to use. And then finally, make it easier. When people ask you questions, when people are confused, figure out why and figure out how you can improve and make it better so, you know, people don't have questions in the future that they can empower themselves to do this, to build their own software and to put it onto your systems and that will make people happier both on the sys admin side that they're not, you know, constantly answering the same questions and on the development side that they're, you know, they don't need to have a gatekeeper to get this code out. So that's it. Thanks. I have a few minutes left. So I'd be happy to field a few questions. And I also am at the booth for Lincoln Loop over here. And yeah, feel free to come talk to me. Thanks. Thanks for a great talk. I come from a sys admin background. And in the sys admin world, we're starting to learn all the kind of similar stuff that that you've just been describing that we've we're coming from a perspective of, you know, things systems that don't change for a couple of years. And oh, hang on, you want to deploy twice a week or twice a day or twice an hour, whatever. The big movers and changes in the sys admin world, people like Gene Kim, who has written a couple of things, the Phoenix project, I'm just wondering if you're familiar with that. I'll come and talk to you later. And Tom Lim and Shelley, who along with a couple of co-authors have has written a book, which is cloud administration. And they don't use the word cloud anywhere except on the front cover, which is great. But basically, again, coming from a sys admin perspective, these similar kind of things. So I was trying to ask a question rather than just deliver a comment. But you know, if you heard of those things or not, but I know I'm not familiar with them. And I do have a was originally a sys admin, probably not a very strong one. And then became a developer. So I do have some empathy from the sys admin side of things. But you know, I wasn't like down in the trenches for 15 years type guy. So I'll have to check those out. Thanks. Thanks. Can you talk about the the sim links used for the virtual ends that you talked about using the current sim links seems like a really good way to solve a lot of problems. Is there something similar you use for the actual for the code also so that you mentioned not leaving craft like the ricey files? Does that is how you do that with it also? So yeah, there's this repo directory here. And that's a directory above the virtual arms. That is our get checkout. So we literally do a get pull on that. And then we copy like a flat copy of that code directly into the virtual and we are our projects we put a setup.py file in. It's like super simple. But then we pip install that that copy of the code into that virtual and so it's all sort of baked in there. And it's independent of the repo, even if you do like a force push and blow out the commit, like that that is sort of, you know, baked in there and you can't mess with it. So yeah, thank you. Hi. Again with the separate virtual ends. Does that mean that your Python processes need to be restarted? Or can you reload those right when you're switching? So what we're trying to do is yeah, that's an issue if you're if you're, you know, your you was the process or your salary process or something like that is running from within that virtual and you can't reload it, you know, because it's it's still that copy. So what we've done is sort of taken a different philosophy a little bit and looked at you was getting salary more like we're looking at engine X or Postgres. Those are like system level services. So you was he and salary are going to get installed kind of on the system at a higher level. And then they just point into those directories and those virtual ends, which lets you kind of do all that stuff. So we don't have long running processes that are getting kicked off from managed up high. They're getting kicked off, you know, by external kind of Python. So this morning I did the Azure Microsoft challenge and I pushed to GitHub and boom, it went live. And I was able to do that. And this feels like a really big, you know, learning curve. So at what point will I not want to just do the Microsoft Azure or Heroku or whatever method? As a beginner or as just like a kind of when you're doing small scale, smaller scale development, platform as a service is amazing. I strongly recommend it. Like you as you don't want to do this stuff, they probably do this stuff in the background for you. But at some point, if you know you're you're learning more, and you're using new services or you're starting to, you know, get lots of traffic and you're starting, you know, those services can get expensive quickly. When you're, you know, running large scale stuff. And at some point, you start to kind of feel pain, like you're, you know, I'm too constricted using this, I want to do something that's not possible on here or, holy cow, I'm paying a ton of money. And I think I could do this, you know, cheaper on my own. At that point, you know, that's when you start looking at, you know, how do I kind of manage all this stuff myself. But I'm very, it's kind of sad to me how much stuff there is that developers need to know to get a website online these days and anything that kind of makes that smaller and prevents people from like just getting so overwhelmed that they give up is a good thing. So I'm a big fan of pass and all that stuff. And it works great at a certain point. And there's points where it doesn't. This might be a kind of out there question. If you have multiple projects that sort of depend on each other and depend on versioning. So what you were talking about, like being able to easily roll back, but then what would be your suggestion for if you have that, where you're deploying multiple projects completely separate, but maybe one of them messes up, you know, how would you handle that would be the best way to do it. Right. So if they're not dependent on each other, you just create like two of these directories and serve them separately. The challenging part is when they start to depend on each other. And there's a lot of talk on how to how to kind of gracefully, you know, you start talking about them like you're in a service oriented architecture, and there's multiple services. And how do I ensure that this service, you know, is talking the same, same, you know, messages that this service understands. That's really hard. And my take on it is like the whole microservices stuff is don't do it until you like really, really feel the pain of the big monolithic project and stick with, you know, there's, there's, you know, really big organizations that have, I think like Twitter and maybe Google to have like monolithic repos that are huge that encompass like multiple services and everything. But then they still have like, you know, hey, there's one commit that represents a certain state of the world. And, you know, it's gets really hard. You know, I've done this sort of system. Sorry. Stop. I've done this sort of system, you know, with like multiple projects. And yeah, you can do like two commit hashes as your unique identifier. But that breaks down pretty quickly. So I don't know, hopefully that answers your question.