 with Twilio and I am joined on stage by the amazing Kate who I will let introduce herself. Hi, I'm Kate Hiddleston. I'm a software engineer with a company called Runscope, which this is our sweet t-shirt. Everything's going to be 200 okay. And this morning, we're going to present to you choose your own Django deployment adventure. And we were really excited that Brandon Rhodes kind of almost plugged us this morning with some choose your own adventure slides because that is essentially what today is going to be. And we're going to ask you to do the exact opposite of what most speakers want you to do. We want you to pull out your phones. Everyone in the audience should get your phone out because we're going to need audience participation for this. So there are going to be many points and we'll give you a phone number in just a second where you're going to text in your vote for which path we're going to go down as far as this choose your own adventure story. So yeah, so you'll be texting in. We'll put the number up on the screen. It's going to be the same number throughout the talk. So that'll be easier. We are doing a choose your own adventure story. And this is a bit of an experiment. We're also relying on technology, which is a very dangerous thing as all of us in this room know. But basically you are going to be the protagonist in our story. You're going to be the hero or heroine or young startup. So you're a software engineer. You're living in the hotbed of technology, Silicon Valley. You're young, you're hip, you're fashionable. You have your tech hoodie, your backpack, or your Timbuktu bag. Maybe you shop at Everlane even. And you have an idea. It's the single greatest idea since Pied Piper, the completely fictional compression algorithm from the show Silicon Valley, which is actually a very funny show. You're really excited about your new idea and the repercussions of your new bad application could change the world. So did you build Tinder for Pets, the hot new application casually connecting otherwise distant pets to each other? Or did you build Mellonhead, the application that determines the size of your head and which fruit is closest in size to your head? So what you're going to do is you're going to text either the word pets or the word melon to the number on the screen and you'll start to see votes show up. So this is a reveal JS presentation and we've used WebSockets. So after every single text that comes in, you'll see that on the screen and in about 15, 20 seconds we'll cut it at that point if it looks like one is winning over the other and we'll go down that path. So get your votes in now, it's 503-946-3662. Oh my goodness. Wow, we've got a head-to-head race here. It's a little terrifying for us because we don't actually know what the audience is going to pick. Oh, Mellonhead. Do we have 70 people in the audience? I don't know, it just keeps going. Those are tests of our application, come on, application that provides services for pets to meet other pets. Before a tender for pets, discovery was ad hoc and sporadic. Stamps of meeting other pets at parks, going out for walks or in the case of house cats, not at all. This is unacceptable, as you have discovered. Now your app is changing the peer-to-peer landscape for pets in new and innovative ways. Stamps to another pet profile. If both say yes, they can then connect for a play date, if you know what I mean. And it turns out soccer is very popular for pet play dates. Though you've built your tender for pets MVP, you've raised a million dollar seed round, obviously, based on your awesome idea and you're ready to deploy this to the public. So you've built your minimum viable product, as it says in the Lean startup, which you just read. And now you need to get your Django application out into the wild. You need to get it into the hands, the excited hands, or actually paws, of all of your users everywhere. So this is your initial deployment. So you do some Google searches. How do I deploy my Django application? You watch Kate's PyCon talk of full-stack Python deployments, and this is roughly what you come up with. You have, okay, this is the structure of the WISGI app, but there's a lot of pieces here. So you take this basic diagram and you say, okay, one step at a time. The first thing that I need in order to put my application out there is I need some sort of server. You could buy some components off the shelf and then put it into your home and then serve up your application, but it doesn't seem like it would be rock solid for a production application. So you decide, I'm just gonna get a virtual private server from one of the many companies that provides them out there. But then on top of that server, you need an operating system. And you've read that while you can do your development on Mac or Windows, you're going to have to use Linux in order to do this production deployment for your Python application. So on top of the server, the virtual private server, put Linux, and then you realize, well, okay, Linux itself doesn't handle web requests, so you need a web server. So you decide between Apache and Nginx. Now, the missing piece here is you've written your application in Django. A web server by itself, Apache Nginx, has a no concept of what Python code is. And that's where Whiskey comes in, web server gateway interface. There's two sides to this. One is the application or the framework side. This is actually in PEP 333 or 3333 that is specified. And what this specifies is that there's an application framework side. That would be Django in your application in this case. And there is a server side, which is what is at the container that is going to be running your application. So the web server, you configure to pass certain requests onto your Whiskey server. Now, you need a place to put your Python code. So you read about virtual M and that that is some way to separate and isolate your environment from other applications that you may deploy in the future. And you put your application source code. That is what's going to execute the application source code. And then you also have your Whiskey framework, Django in this case, and your application dependencies, any other thing that you may rely upon for your deployment. A few other pieces here and there's actually one critical piece that we're missing. We have all this data about all these pets and we need to store it somewhere. So we've decided we need a database. And for this case, we're gonna pick Postgres. Postgres happens to be the one that most of the Django core committers are using to actually test their own code. So you say, all right, I'm just gonna stick with Postgres as opposed to MySQL or something like that. So you put that as a part of your architecture and then you have some JavaScript files are using jQuery. You may wanna move on to an AngularJS or a backbone front end at some point. So you have some JavaScript and CSS files that you need to deploy. Those are served up directly by the web server. They're not going through your Whiskey server. You have those things and then you have one other piece which is there are some things that you have to handle outside of the HTTP request response cycle. So for example, if a pet hasn't logged in in over a month, you've decided you're just gonna get rid of that pet's profile. They've probably found a pet that they really like and they're never gonna come onto your application again. So every night you run a function in task queue that cleans up all the old pet accounts. So this is the architecture that you've come up with but these are the general concepts. So let's take a look at the specific components. So we're gonna, for example, use Linode as a server, a virtual private server provider. You install Ubuntu on top of that. Use Nginx as your specific web server using Postgres as your database. Celery is for your task queue. Green Unicorn as your Whiskey container. And of course, there are many options. You just decide this is what I'm gonna go with for my initial deployment. And then you have Django as your web framework. So that is our architecture. And now we're live. We're available to the world and we've launched a TechCrunch disrupt. It's amazing. We are getting so much traffic, so much press, so much publicity. But now, despite your successful launch, we have one of two problems. Either we get so much traffic through our publicity that hackers or that our web server is starting to fall over and we need to handle that situation or hackers are hitting our web application and breaking it down and stealing our data. Which one of these two is the situation that we find ourselves in? So the initial running looks like, wow, head to head, 33, 35. Hackers are starting to come up. We'll cut it in about 10 seconds or so. Actually what we'll do is we'll do first to 100 if we're gonna keep going. Otherwise we'll just cut it in a few seconds. Hackers, hackers, hackers it is. I just went down to five. So hackers have broken in to our new web application and they have stolen our user data. We're not sure they may be Russian pets. They may actually be people that look like this. They clearly look like this and dress like they're from the 90s. Regardless of who broke into our system, we decide to hire a security consultant. The security consultant says we've got four problems. We're using self-signed certificates. We've unencrypted, our unencrypted cookies have been sniffed off our server, off of our user sessions. There's been SQL injection into our database and we've exposed our data through something called Heartbleed. And we didn't worry about all this stuff with our initial deployment. We were just trying to get it up and running so we could launch a TechCrunch disrupt. So here's how we handle each one of these. The first one is a self-signed certificate. So if you have a self-signed certificate, your users are gonna get an error like this. So if you do your deployment and in our case we're using nginx and we sign our certificate ourselves, we're gonna see something like this probably not the site you're looking for. The browser gives warnings. What we need to do is go out to a reputable set certificate authority like DigiCert and they will help us sign the certificate so that it's actually something that has been shown to be, okay, this site is actually who they say they are. So we have full encryption. So that solves the self-signed certificate problem. The second thing is user cookie sniffing. So here's what happens in a Django application. This is the scenario. Our homepage, we decide, we have to worry about encryption on our homepage, but everything else is accessed through HTTPS. So a user logs in and okay, they're working through the application, they've got their session, but if they go back to the homepage and that's unencrypted, then a tool like Firesheep, which is a plug-in for Firefox, can sniff those cookies and if you're on an unencrypted wireless network or just a wireless network in general, other people say you're in Starbucks, other hacker could sniff your cookies and then impersonate you and your session. And so what you need to do here is have SSL everywhere. I've also had SQL injection. I really like this picture. It's just kind of weird. So what happens with SQL injection is if you are not using the Django RM, we don't have to worry about this with the Django RM, but if we have direct SQL queries and we are taking in user input and we're putting those right and we're concatenating them with our SQL, then we can be exposed to SQL injection. And this is one of the top website vulnerabilities. The Django RM protects us against this, but in our case, we're using direct SQL queries for certain cases. And in that case, someone could specify drop tables as a part of their user input and that is actually executed as a part of the database. And then Heartlead. Heartlead was something that happened recently. What this is, is a vulnerability in older versions of SSL that allow hackers to read, direct read memory from your server. So the way that we address these, again, use proper cert authority, SSL, all the things, if we are going to be using SSL on for, to encrypt our sensitive data, sanitize all SQL input. So making sure that we escape that input if we're taking it in directly from users and use the latest version of open SSL with our Django server to ensure that we're not exposed to the Heartlead vulnerability. All right, we've solved the hacker problems for now. For now, such thing is a totally secure setup for our Django application. But we're running into another issue. Either users are reporting bugs and we don't even know that these bugs are happening or VCs are starting to access about this thing called traction and we need to show them actual metrics and statistics around it. What do you guys think is the situation we find ourselves in? So it looks like bugs are winning for now. We'll go first to, you cut it when you're ready. It looks like bugs is winning and I really don't think we have over 100 people in here. All right, bugs are in. We might have some, so this section might be good for us. All right, bug tracking and exception handling. So one of the first things you need visibility into is going to be exceptions in your application. So users are now hitting bugs and issues, but they're having to report them to you. So basically, pets using your application have to email in and be like, I can't do this with my profile and I have no idea why. So with Django, there's the debug setting and most of us know you don't want debug set to true in production, but you can set debug to true in development and test environments. This might help you figure out where exceptions are happening and what they look like, but you need to get the exception stacked trace from production as well. So there's a couple of ways that you can do this. You can have Django generate an email every time that there is an exception. So you basically get an email anytime someone hits some sort of internal 500 error or even 404s, you can set the settings for that. You can also use freemium services like bug snag. They have free tiers where you can plug them in with your application and that will do a lot of the exception, exception tracking and aggregation. Finally, you can use a service like Get Century and Raven. So Get Century was built by David Kramer, David Kramer who's been active with Django and very active in Python. And the reason you want to use some of these more robust systems is if you have lots of traffic and people are hitting exceptions thousands of times or thousands of people are hitting exceptions, you don't want to get 4.7 or 4,700 emails for a single exception. It's a lot of noise. And so a lot of these tools will aggregate these exceptions so that you can see what exceptions are happening the most, when are they happening and you can see the stack trace. So now you have this stack trace. You're like, I see where this is broken. The next step is going to be reproducing this bug in your development or test environments. So the Django debug toolbar is great for this. You can inspect SQL queries, you can take a look at your settings, you can take a look at your static files and templates. If debug is set to true, you can see the stack trace but basically you get to go be a bug sleuth and look around and try to figure out why this thing is breaking. Hopefully when you find the bug you feel like an idiot. This is something I always tell new developers but there's two types of bugs that you run into. There's the bug you run into and you're like, I'm an idiot. I cannot believe that I put this bug into production. The second type of bug is the type of bug where you're like, oh my gosh, this bug is super complicated. We're actually gonna have to rewrite our entire architecture because this whole thing is just toppling as a result of this bug. So basically, hopefully you find a bug where you're like, I misspelled this, it's just a key error, it's really simple to fix, I'm an idiot and then you write a regression test because let's be honest, you don't want anyone else to make this simple mistake that you made. And now you're fixing bugs like crazy. Users no longer have to email in every time they hit an exception. Whenever you deploy to production, you can see if there's a huge spike in the number of exceptions. Did this feature break somewhere? You can fix bugs hopefully before users ever have to complain to you about them. But now one of two things happened again. The first scenario, customers are still reporting issues but there's no stack trace. There's no exception that's happening. So basically data is missing or data is inconsistent and you realize you're gonna have to implement logging. The second situation is that customers are loving your application. They think it's awesome. All of these pets are like, oh my gosh, I love hooking up with other pets in the area. We really want new features. I would love to be able to share hot pet profiles of other pets with my friends. And so now you're like, we're gonna have to start tracking features. Looks like data is far and away the winner. So basically data is missing or inconsistent. This is all of our favorite problem, right? Oh, there's a problem. We don't know why. It's completely invisible and we can't figure out what's going on. So you realize you're gonna have to implement logging. Django has built-in logging. This is great. Import logging, just kind of cut off, but logging.getlogger and you can logging.error. You can actually just output log errors. There's five levels in Django. There's debug, info, warning, error, and critical. So depending on the type of log statement that you have, you can choose the level. So if I'm just debugging something on my dev environment or in test, I can set the level to debug. This allows me to just output the word here 100 times like I do here one, here two. Oh yep, my code made it this far. Info, warning, error, and critical are more production level logging statements. Basically info is like, this is information that we think we're gonna need to be able to read through. Warning, error, and critical are, these are things that are really issues. Perhaps you log whenever data is in an inconsistent state or if you have a request object that is missing, a header that you would really expect it to have. In your Django settings file, you can actually set all of your logging settings. So you can set the handlers, you can set the level. Basically what you can say is in development and test, I would actually only, I would like it to output all of the levels. Debug all the way up, but in production I really only want to see info and hire. So Django allows you to set these things so that you can see different log outputs in different environments. The really simple way to get started with logs is just to have it write the output to log files on each server. The second simple solution is to pay someone like Logly or LogEntries to do this for you. So Logly aggregates all of your logs, you pay them. They give you some really robust search tools on top of the logs that you have. A lot of data, you can look at cool graphs of what people are using and where they're going. And you can search things like session keys and search for keywords. If you want to get more advanced and you want to maintain the logs yourself, you can aggregate the logs yourself, but you're going to need to do things like log rotation, backup, and cleanup. So log files can get very large very quickly, especially if you're outputting a lot of information. And if you have a lot of production servers, so let's say we have, I don't know, 10 front end production web servers, you might even have a database server, or two, or three, or replication, or all of these different types of servers, an asynchronous server. And these are all going to have separate logs. Celery is going to have separate logs, your Django application is going to have separate logs. So you're going to have to aggregate these yourself, either in one location, or again, through using a third party tool. You're also going to have to rotate logs. When they get really large, you're going to have to zip them and rotate them out. You're going to have to back them up if you're worried about keeping historical data. And you're going to have to clean them up when things get too large. So you can't keep them on these boxes if they keep getting too large and taking up memory. So logging fixes the data problem. You're now logging output. You can see what's happening. You're like, oh, hey, this is why data's missing. It's not actually getting saved, or we're getting these requests coming in from third party APIs that aren't working very well. But now, again, we branch. And one of two things happens. Either deploys are taking a really long time. They're inefficient. You can't bug fix quickly. You only ship maybe once a week because nobody wants to fight fires Monday through Friday. Or your team is scaling really quickly. You're hiring a lot of engineers, but you're having a hard time integrating them into your development process. This also includes management, deploys, but development environments largely. So you realize that you're going to have to focus on one of these two things, either automating your deployments or automating your dev team. So deployment got off to a quick start. We'll see if team catches up here in a few seconds. The deployments, it is. Deployment automation. So like most of the sections in our talk, any one of these could be expanded out to be a full length talk. Deployment automation is a huge topic and it's actually a really large space. There's a lot of things to automate when it comes to deploys, when it comes to infrastructure management in general. The first thing to automate is going to be your infrastructure configuration. So tools like SaltStack, Ansible, Chef, and Puppet are all really good tools, very robust tools, well documented, for automating your web infrastructure configuration. And the reason that you want to do this is because you really don't want all of the domain knowledge for what's installed on each box in one person's head. You also want to be able to automatically install the things you need for different types of servers. So if you had to go to every single front end server and add a new installation every time you had to add a new installation, you would go nuts. It would also take a really long time. So these tools are great for doing that. SaltStack is master minion, Ansible is YAML files, Chef is Ruby scripts, and Puppet is its own declarative language. But these are all basically for things like installing Postgres, installing Redis, installing Python, setting up user accounts and permissions. So each employee at your company theoretically has a user on the different production boxes just in case. And that's what you use these tools for. There are also some platform as a service companies or solutions out there. Heroku is the most famous. Heroku and Elastic Beanstalk will deal with all of these things for you. So Heroku allows you to build your app locally. And then when you push to Heroku, it does all of the server configuration. It installs all of the application dependencies. It has a seamless deploy because it just uses Git pushing for deploys. And it manages all the scaling for you as well. Elastic Beanstalk is Amazon's solution to this. It's much younger, so it's a lot newer of a solution. But the idea is to be a platform as a service solution for people. These are expensive. Heroku can get expensive really fast. But if you're making a lot of money and you don't wanna manage this yourself, there are great ideas. Infrastructure and DevOps engineers are some of the most expensive engineers and some of the most difficult to find. So a lot of companies have opted for using tools like this to solve their infrastructure problems. Automating your deploys. So deploy is a big one. I was actually talking to someone earlier about no downtime deploys and what that means like several seconds versus zero seconds of downtime. But deploys can get really complicated, especially if you have one really large application and you're deploying changes to it. I don't want that. So one of the things that you can do is decouple the deploy logic from the infrastructure management logic. Infrastructure management involves what's installed on the box. What's installed on the server? Do I need Postgres? Do I need Redis? Do I need Python? Your application is actually separate. So because we have Viet, like virtual environments, where we can install all of the application dependencies within that virtual environment, deploying Python code can be separate from updating your actual infrastructure configuration. While one might need to run before the other, and that's important to note, decoupling this logic and keeping it separate makes each task simpler. So rather than trying to do too many things at once and having too many dependencies that are very tightly integrated, what you have is you have separate steps for deploy that are each very simple and easy to maintain. This allows you to deploy more quickly, to update things, basically to get your code out to production without things falling over and having huge fires. Automate your tests. So step number one, write tests. Step number two, you can use continuous integration tools, that's a thing, to automate the testing of your code. So every time you open a pull request, you can have test suites run on the code that's been opened. So you always know, does this code pass tests? You can create test coverage tools. Those are very popular to make sure that you have at least a certain percentage of test coverage for any code that has been pushed to production. And finally, one of the big things that you can do to make sure that your application is as simple to deploy as possible is to think about service oriented architecture. And for those of you who are slightly new to service oriented architecture, the idea is to separate different pieces of logic into services. So one Django application could be a service. Maybe you have a really robust tool for emailing customers, like a notification center, and that can be separated from everything else, and you can deal with it through an API. You just send emails through this API, it's your own internal service, but that logic is decoupled. So whenever I need to make changes to this one email notification service, I just deploy to that one service. Having an abstracted and decomposed system like that means that if one piece goes down, it doesn't necessarily topple your entire system, which is a good thing, and it's actually one of the fundamentals of building computer systems. So thinking about building service oriented architecture, separating out logic, having small pieces within your larger system is good. One of my favorite quotes is, there is no such thing as a small change to a large system. So the idea behind this is to have more small systems, so you can have small changes to small systems. All right, you now have continuous deployments, but now we come to our most important decision point. We're doing really well. Company is scaling, got continuous deployments, but now the VCs have asked us, they've said we're getting a lot of requests from Fortune 500 companies. They want a white label version of Tinder for pets. They want their own GE branded Tinder for pets. Capital One branded Tinder for pets. Do you build that or do you clearly deny that as your votes have indicated already? Who said build? Do you stand up and fight against the VCs? Or do you just acquiesce and say sure, we'll build whatever white label version of Tinder for pets do you want? Clearly you've chosen wisely. Have so much money coming in because you've decided to focus on your core audience. Instead of just dealing with dogs and cats, now have a separate product line for hamsters and gerbils. The money is pouring in to your new application. There's a really large customer base, rodents. And you are able to raise more and more money. Massive funds flow through your coffers. You are a great success, but in the back of your mind, you wonder, is it all just a bubble? And that takes us to the end of our story today. This is a totally separate number. So obviously this talk here was pretty experimental, a bit different than some of the other talks where it's just a linear path through the topics. If you could send a text message to this number, it's, as I said, this is a separate number from the other one. And on a scale of one to 10, would you recommend this talk to a friend or a colleague? 10 being extremely likely, one being not likely at all. And just from one to 10, whether you recommend this. And what this is going to allow us to do is have a quantitative measure and you'll get back our text message response with a blog post that we've written with all sorts of resources on all these topics, some of which we did not even cover today. So you've only seen actually a very small portion of this talk and we'd like to continue to iterate and approve on it. You'll also get back our Twitter handles and if you could tweet us with whatever improvements and suggestions that you have, be super beneficial and we'd really appreciate the feedback. So again, 503-476-3063 with a one to 10 score. Thank you all for your time today. Kate Hettleston, take a bow. Thank you. Yes, this has been a really fun time to present this to you today. Thank you very much.