 All right, thanks everyone for joining. Again, this is performance testing for modern apps, and we're gonna be talking about some tools of the trade for testing performance on the server side, but also understanding client side performance. So the ton of content, there's a bunch of notes at the bottom of my slides, I'll make them all available online. So if I go a little bit fast, know that there's plenty of notes for later. So the reality is that top engineering organizations think of performance not as a nice to have, but as a critical feature of their product. And it's because they understand that it has a direct impact on their business's bottom line. Most of the time developers really don't think of this until you go to launch, and really I'm here to help change that. So you can find out a little bit about me, follow me on Twitter at DustinWiddle or DustinWiddle.com. So why does performance matter? Microsoft found that Bing searches that were two seconds slower resulted in a 4.3% drop in revenue per user. And when Mozilla shaved 2.2 seconds off their landing page experience, Firefox downloads increased 15.4%. So they got 60 million more downloads just because the page was a bit faster. And making Barack Obama's website 60% faster increased donation conversions by 14%. But the most impressive metric that I've come across is that by decreasing the end user latency of Amazon.com's retail operations by 100 milliseconds results in a 1% improvement in revenue. So whether it's Yahoo, Shabzilla, AOL, Amazon.com, all of these engineering companies, all these engineering organizations understand that ultimately performance directly impacts the bottom line. So the question is how fast is fast enough? So 0.1 seconds, it feels instantaneous. It feels like you're flipping a page in a book or 100 milliseconds. So you should really strive to keep your load times in this range. But one second allows you to think seamlessly and after 10 seconds you really start to lose the attention of your users. So there's been a bunch of performance studies to understand basically the attention span in applications and user experiences. And what they realize is that performance really is key to a great user experience. I think everyone's probably had the experience where you go to check out an e-commerce store and you click the check out button and then it just waits for a long time and you really quickly start to lose faith. I think the engineer and all of us knows to just wait it out or you're gonna get charged again. So again, how fast is fast enough? 100 milliseconds is again instantaneous. It feels like flipping a page in a book. 100 milliseconds at 300 milliseconds, the delay is perceptible. So your users are gonna start to notice. And after about one second you really start to interrupt the user's flow. So users expect a site to load in two seconds and after three seconds 40% will bend in your site. And this comes from a Nielsen performance survey that they did a long time ago. And the reality is that mobile applications are even worse. This is really hard to do because modern applications are really complex. When you have 100 microservices talking to each other and you're making calls to external providers, a shipping provider, a payment processing provider, a fraud detection provider, it's really hard to have great performance. So with application complexity exploding, how do you manage this and how do you test for this? I think most companies treat performance in really uptime as a critical differentiator. And if you look at some of the major enterprise companies, they all treat uptime as a critical metric. And I think we've all encountered this if you provide a service to others. It's enforced by an SLA. So what's the goal here? The goal is treat performance as a feature and we're gonna talk about some tools of the trade for performance testing to do exactly that. So the first thing I like to start with is how to understand your baseline performance. So just raw Python is gonna be pretty fast. When you start to add Django, you have a framework layer that's gonna add a bit of overhead and then you're gonna have your application stack that's gonna add a bit more of overhead. Really what you wanna understand is what's your baseline performance on some hardware? On a specific set of hardware that you're gonna run in production. What's the performance of a static asset being served, not from Python? What's the performance of Python Hello World? So a very simple script. What's the overhead of your framework and then what's your actual application? Because oftentimes what you'll find is that the business transactions in your applications have very different performance. The homepage is gonna be highly cash whereas the checkout process is gonna talk to a bunch of third party providers so it's gonna be inherently slower. So you should understand the performance of each one of these transactions and really understand how that affects your users. So I like to do that by understanding what the static threshold is, what the Hello World baseline is and what the application benchmark is. So if you've ever installed Apache, you probably have access to Apache events. If not, you can app get installed Apache 2.0 tools on most Linux platforms and it becomes available. Apache Bench is a very simple tool for benchmarking the performance of applications. This isn't specific to Django, this isn't specific to Python, you can use most of these tools across any different application platform. So Apache Bench is really crude and dead simple. So if you just wanna get an idea of how fast a particular transaction is like your homepage, it's really easy to test a single concurrent user. So in this case, what you're seeing is Apache Bench-G for concurrency. So I wanna test one user going as fast as possible for 10 seconds to acmedemoapp.com. So we'll run Apache Bench-G concurrency of one for a time of 10 seconds and dash K is benchmark mode. So I want no delay between the request. And what you'll start to get is a response that looks like this. So you'll get the request per second, which is a very useful metric, but more importantly is the latency. So what's the average response time per transaction? And the balance to figuring out how much capacity or infrastructure can support is understanding when you max out your request per second and the latency starts to rise. So you can always serve more transactions just very slowly, but you don't want your users waiting 10 seconds just because there's a thousand of them showing up. So you really need to understand the balance of those. And with Apache Bench, it's really easy to start to increase the concurrency. So in this case, I wanna start testing 10 users. So we'll just change the concurrency level here to 10 and test again for 10 seconds. And what you'll see is that we have more requests per second. In this case, we have 65 requests per second, but the time per request has gone up to an average of 151 milliseconds. Now Apache Bench is great. It's quick and dirty. It makes it very easy to load test a server, but I prefer Siege. So Siege has a very similar format. If you can get install Siege on most platforms or port and brew install Siege, it's pretty straightforward. And it has a very similar format. So you can run Siege-C, again, concurrency of 10 users for a time of 10 seconds. Now what you'll notice is in these examples, I'm load testing one endpoint. I'm only load testing the homepage, but that's only so useful. And we'll get metrics that were very similar to Apache Bench. So for 10 concurrent users, we get about 65 transactions a second and the average response time is about 30 milliseconds, I think. Or 150 milliseconds, rather. So you can keep increasing the concurrency until you max it out and really, until you can start to see the latency start to skyrocket. And really what you wanna understand is what's the maximum request per second that the machine can get where before the average response time starts to increase. Now this is fine for a very simple application, but most of the time applications aren't one endpoint. It's many different endpoints. You have functionality to, you have the homepage, you have login, log out, add to your cart, and check out process and order, et cetera. So there's a quick tip to be able to call the entire application to discover all the URL endpoints because most of us inherit applications and we don't have the ability to just build them from scratch every time. So you often don't know all the functionality that exists. So there's a tool called Sproxy, which is a transparent HTTP proxy. Basically what it allows you to do is make a request through a proxy and it'll log the URL that you're requesting. So if you wanna interact with an application and you want to crawl all the endpoints in the application, it's very easy to use Sproxy and WGIT to basically emulate a search engine spider. And the goal here is that we want to find all the URLs of the application. So it starts with Sproxy-o and all it's going to do is run a HTTP proxy on port 9001 and all the URLs that we access are gonna be put into a urls.txt file. The next thing we're gonna do is use WGIT. So WGIT has a spider mode that allows you to emulate a search engine spider and it'll go to the homepage and recursively crawl all of the links. So if you really want a quick and easy way to discover all the functionality of the application, you can run Sproxy and WGIT and discover all the public URLs in your application. And what you end up with, we'll sort the list so we have a unique list of URLs at the end and you end up with something that looks like this. This is just a simple e-commerce application that I'm using but there's an about page of view carts, change the currency, log in, register for an account, view by category, view by tag, et cetera. And really what we wanna understand is what is the, how do we benchmark traffic across all the unique URLs so that we can understand the performance of each transaction? Because the processing in order is gonna be a lot slower than going to the homepage, especially when the homepage is highly cached. So you can use Siege again, this time feeding in the list of the URLs text file so that we can benchmark traffic across all the unique URLs. So we're gonna run the same command that we ran earlier. So this case is gonna be Siege-V for verbose-C is concurrency. Again, we wanna run 50 concurrent users for three minutes across all the different URLs in the URLs.text file. And what we'll end up with is something pretty straightforward to the output that we had earlier, which is the average response time for each one of these transactions and the transaction rate. Now this is only so useful because oftentimes you don't wanna just go through a list of URLs, you actually wanna perform a transaction. In that case, let's talk about multi-mechanize. So I'm a big fan of multi-mechanize. It's an open source framework for performance and load testing. But what makes it really useful is it allows you to script transactions. So I can actually post these credentials to the login form. I'll maintain state by creating a cookie jar and maintaining the HTTP session along the way. So I can log in, then I can add something to the car and then I can check out. And I can actually manage the entire HTTP transaction or a set of transactions and then I can start to load test those as well. So it's very easy to install. It's a Python tool, so you can simply pip install multi-mechanize. And then you'll have access to a command line library and you can just bootstrap a new project pretty straightforward. Now I presume everyone here is pretty familiar with Python, so this should be pretty straightforward. The idea here is we're gonna import the HTTP request library. So it's a HTTP client for Python. And then the only thing that multi-mechanize provides is a transaction class. And what you do inside of the transaction class, it doesn't matter. It's just gonna run many of these transactions at a high level of concurrency. So I can do this for a single URL endpoint or I can run a series of URLs. So this is gonna be the same example that we just ran earlier with Siege or Apache Bench, where I just wanna load test acmedemoapp.com as a single endpoint. So here we'll import the request library, we'll define the method run, and then we'll simply make a git request to acmedemoapp.com. But oftentimes you wanna do much more than that. You actually wanna script a user scenario. So I actually wanna go log into acmedemoapp.com, go to the cart page, and then start to script this out. So here you can import multi-mechanize. Again, the only thing that matters is what you put inside of the run method. So in this case we have a transaction and we just want to run this transaction at high levels of concurrency. So here we instantiate the multi-mechanize browser. This is what's gonna allow us to emulate the HTTP state and maintain a cookie jar. And we're gonna disable the robot so it doesn't block public endpoints by using the robots exclude. And then where it has a custom timers API. So this is where it gets really useful because based on the custom timers, it will generate fancy graphs that will allow you to understand for each one of these transactions the average response time and the number of requests per second. So for each one of these we'll create a custom timer and we'll create a custom timer for the homepage and a custom timer for the cart functionality. And obviously you can script this for the specifics of your application but it becomes quite easy to script complex transactions. So here we create start time. We're gonna use the cookie jar to open AcmeDemoApp.com. We're gonna read the response. We're gonna record the latency. So the start time minus the end time gives us our latency. And then we're gonna add that to the list of custom timers. And then we're gonna do the same thing but go fetching the cart. We could very easily do a post, post the login, fetch a bunch of products by their tag, add all those products into the cart and then process an order. But that doesn't fit on a slide. So once you have the script you can create as many of these scenarios as you want. So this is just one transaction. So I could have a homepage transaction, I could have a buy stuff from a shopping cart transaction, file support request transaction, all these different user scenarios. And then once I want to run these scenarios I can use the multi-mechanized command line to run these at different levels of concurrency. So here you'll simply run multi-mic run demo. So this file is called demo.py and it just has the transaction class inside of it. And once you run that, it'll run to whatever level of concurrency you've configured and then it'll generate a list of output that looks like this. For each one of the custom timers you'll get the response time plotted out. So the average response time versus the request per second. Now by a show of hands, so this is pretty easy for any scripted transaction. But what you'll notice is that you're running all these transactions off of a single server. By a show of hands, how many of you have a real production application that's more than one machine? Right, how many of you live in the cloud? Okay, so we're gonna be talking about my favorite open source project, bees with machine guns. Big fan of talking about this one. One because it comes with a notice that says it's a felony to do this against any other site except for your own because effectively it's a distributed denial of service attack. It's brought to you by the fine folks at the Chicago Tribune. So what is bees with machine guns? It's a utility for arming mini bees to attack targets. We're creating micro EC2 instances in the Amazon web services to load test web applications. So when you live behind a load balancer and you have hundreds of machines behind a load balancer, how do you generate enough traffic to actually test that? You're not gonna do that off a single machine. You're not gonna do that off your MacBook. You're not even gonna do that off an EC2 instance. You'll max out your network throughput before you can generate enough load. So the idea of bees with machine guns is allows you to run a distributed load test. And it's very easy to use. So again, Python, so simply pip install bees with machine guns. No reason this can't be fun folks. And then it uses the AWS photo library. So if you're not familiar with Amazon web services, you can go to aws.amazon.com slash free and get a free year's worth of computing power. And then all you need to do is configure your access key and secret here. And that'll give the Boto library AWS credentials. So bees with machine guns relies on the Boto library to spin up EC2 instances. Once you configure this, you're good to go. And you can pick whichever region you want. In this case, I'm running it off the West Coast. Now you can spin up two machines or you can spin up 200. It just depends on the level of concurrency that you wanna generate. So in this case, it looks pretty fancy, but the only thing that really matters here is you call bees up dash S2. So I wanna spin up two servers so I can run a distributed load test. There's some other stuff in here. Basically, I wanna be in the default security group. I wanna spin up these two servers in the US West 2B data center. I wanna use this AMI instance ID and these SSH keys and username, but all of that other stuff is irrelevant. What you really need to know is bees up dash S2. So I wanna spin up two servers to start load testing. These people have a great sense of humor, so they're gonna connect to the hive, attempt to call up two bees. They're gonna wait for the bees to load their machine guns, and then they're gonna be ready for attack. The swarm has assembled two bees. And then you can check how many machines are actually up and running and ready to be used. So you can call bees report, and it'll tell you that you have two bees from the roster and the IP addresses. So we're gonna start the load test with just a thousand requests. So in this case, we wanna load test a single endpoint. So we'll call bees attack, and then the end, the number of requests, I wanna file a thousand requests with the concurrency level 50 at a time against this URL endpoint. So I wanna basically throw 50 concurrent users until I hit a thousand requests at acmeademoapp.com. So something very similar to what we did with Siege, just with a bit more concurrency. And then we'll get this response. So it's gonna read two bees from the roster, connect to the hive, assemble bees. Each of the two bees will fire 500 rounds, 25 at a time. Seeing the URL, so it'll be cash for attack. So whenever you use a framework like Django, whenever you use many web application stacks, you should prime the cash because you're gonna do a lot of things on the first request that you want to, when you're doing a load test, you wanna understand the real world performance. So you wanna have one cash request that gets fired, and bees with the machine guns will automatically do this. So it'll create one request to that URL to prime the cash, and then it will start load testing. And then we'll see the request per second at 50 concurrent requests per second. We get 306 requests per second with an average time of 163 milliseconds. So here again, the goal is that we wanna increase the level of concurrency until we start to see the latency start to spike. And that will tell us what this particular hardware running this application can support. But you can very easily increase this to 10,000 or 100,000 or 500,000 very easily. So in this case, we're just gonna crank up the concurrency to 1,000 requests a second, and we'll call this again. So bees attack, this time I wanna fire off 100,000 requests, 1,000 at a time, at acnedemoapp.com. So bees attack, 1,000, 1,000, 1,000. So concurrency of 1,000. And this time we're gonna do the same thing. And what we'll find is that the request per second, now we support 502 requests per second, but the latency start to spike. So now the requests are averaging out at 360 milliseconds. So that just means that yes, while we are not gonna fail over, the users are gonna wait longer. So you decide how much money you wanna spend and how much the business can support on the threshold there. But ultimately you want to be able to start to add hardware when the latency starts to spike, especially if this is gonna be sustained normal workload. Again, they have a great sense of humor, so sometimes things don't always work out. Each of the two bees will fire 550,000 rounds, 500 at a time. Once the offense is complete, actually we missed it, the target crush would be offensive. So the server survived basically. And if things don't go so well, no bees completed the mission, apparently your bees are peace-loving hippies. So the swarm is gonna wait some new orders. And then the benefit of the cloud is that costs go linearly with demand, so you only pay for what you use. So you can call bees down and it'll tear down the two instances, you just spun up. So if you really wanna easily run a production load test, you can spin up a couple of machines for an hour or so and then tear them right down when you're finished. So bees down is simply gonna turn off the two instances that it's spun up to create the load test. All right, so we talked a little bit about patchy bench and siege when you wanna load test a single endpoint. We talked about multi-mechanize when you wanna script different endpoints and we talked about bees with machine guns when you wanna generate massive amounts of concurrency. But sometimes you wanna do all of those things at the same time. So enter locustio. So this is relatively new, open source framework for load testing. I'm a big fan, it's again written in Python. Go to the website locustio, pretty much all the slides are directly copied from there. So you can pip install locustio because again it's a Python tool and it gives you the benefit of both worlds. One, it allows you to run load tests from multiple servers but it also allows you to have complex transactions and then even better is it has a bunch of great reporting and ships with the web application so that you can do this very easily in an automated fashion. So the reason I talk about performance testing is this isn't a one-off. Most of the time the problem with performance testing is you only wanna do it right before launch. How many of you know about healthcare.gov? So this is the only time I've seen a president go on live TV and apologize for a broken website because they didn't do any testing and they spent some seen amount of money on it. They did some testing right at the very end to realize they were, it wasn't gonna go well. And it didn't and a lot of people got in trouble for that, right? So the goal is don't just do this when it's launch time. This is something that should be part of the software development life cycle. It's a continuous process. Everything that you do affects the performance of your code. You should understand that. And the same way that you have a continuous integration setup and you write unit and functional test, you should write performance tests because you should understand when you're going to significantly impact the end user experience as part of every release process. So we'll get more into that later but for now more about Locustio. So the idea behind Locustio is that you can very easily script a website task. So the same way that Multi-Mechanize has a transaction. You can define a set of tasks in this case. Yeah, we'll just go right into this. So you have a user behavior in this case. We just want to log in and fetch the profile. And on every request we need to log in. So on start, just like you have a setup in teardown, you have an on start where we're automatically gonna log in and this is just gonna do an HTTP post request to the log in endpoint. So if you see back here, it's just the call self.client.post and post the username and password to the log in endpoint at the start of every request. And then we want to run the task number two which is to fetch the homepage. And then finally the third task which is to fetch your profile. So I wanna log in, then I wanna go to the homepage and then I wanna go to the profile. Now we can get into some of the rest but the idea here is that it also comes with a great UI because for each one of these endpoints you wanna understand what's the average request per second and the average latency for each one of these transactions. By scripting them with locus.io and by giving you a nice UI to run these tests they make it very easy to automate all this stuff. Very easy to script complex transactions and then generate high levels of concurrency as well. So check it out locus.io. But the idea here is there are many tools. It's not just going to be a bunch of Python tools that I handpicked for this talk but there's galing work. So there's a ton of different server side tools. The idea is to use what works well for you in your use case but that's just the server side. The reality is, what about the client side? Most people when they think about performance they think about only the server side because that's the application stack but what you don't realize is that in modern web applications more latency comes from the client side than the server side. The 200 milliseconds that you're waiting for the response from the server is nothing compared to the two seconds that you're downloading JavaScript CSS resources waiting for the page to execute JavaScript and paint the actual page to become usable. When you talk about the end user experience time it's about users perceive experience. It doesn't matter to them whether it's on the server side or the client side, it's all about am I waiting in my browser, can I use this right now? Can I click on that box to start searching? That's why Google's homepage loads very quickly. It's just the box. I just want to be able to type in the box as fast as possible, right? So when you think about performance testing don't just focus on the server side also understand how the client side's impacted. So Google's invested a ton of money in performance engineering because it's really important for them because they understand that performance impacts their business. And one of the great tools that they've released is PageSpeed Insights. So Google PageSpeed Insights allows you to analyze and optimize your website using the PageSpeed rules. So what it will tell you is, hey, you should be using far futures expires headers for your CSS and JavaScript. You should be using GZip compression for delivering your CSS and JavaScript. But even better, they've made it extremely easy to automate this piece with Nginx PageSpeed and Apache PageSpeed. So these are modules which will rewrite your responses in real time to make them more optimized. So most of the time I don't recommend using Apache or Nginx PageSpeed. This is the cheap, lazy way to do it. Cheap as an engineering time and lazy as in you don't actually have to fix anything. You just install a module and it does all the hard work for you. You should really use automation and front-end automation to make this part of your development cycle so that when you package your application up to deploy to production, you automatically minify your CSS and JavaScript so that you automatically set the Nginx configuration up to serve it compressed. So you can do that very easily with Nginx PageSpeed, but you're not gonna really fix the root of the problem unless you fix it in your software development stack. So it's not just a module for your web server. They also have a great website so that you can go and get actionable advice on how to improve the performance. So you can go to developers.google.com slash PageSpeed and type in your URL and it'll give you tips on how to improve the performance. And this is the end user's perceived performance. This is what usually matters the most. So it's available as a webpage and standalone API. It says available as a Chrome developer tool. So it's very easy to make part of your development setup. And it's also available as a Gulp package. So if you're familiar with Node.js, you can NPM install PSI and this will allow you to automate this as part of your front-end workflow. So what that will look like once you NPM install PSI, you have access to the PSI command line and then you can simply run PSI and then the domain. And what this will do is run PageSpeed Insights using the PageSpeed API underneath and give you some formatted output. It will tell you your PageSpeed score, so 73 is not too stellar. The amount of CSS and JavaScript that you're using, how many are cached, uncached, how much is compressed, uncompressed, et cetera. So this generally is a good way to integrate with your continuous integration setup so they'll fire this off as just another task and then start to do some benchmarks across this. So if our page weight ever goes over 500 kilobytes, then we should probably have something fail like a test would fail. And then if you want to just integrate directly, of course they have a REST API that's available. So sometimes you wanna actually understand what's going on in the JavaScript world. So Wbench is a Ruby tool that makes it really easy to understand what's happening inside the browser. So it lets you understand the page load times. So this is a Ruby tool, so you can simply gem install Wbench. And Wbench uses effectively Chrome and web drivers so it'll run inside an actual browser instance and then record the PageSpeed times. So in this case I'm just gonna run it across the very wonderful dustinwittle.com and you'll see how horribly it's built. So I rely on Google has a Google Fave icon service that they don't really publicly expose but each one of those requests ends up in a redirect which means everything takes a really long time on my website to load. And what you can see here is really the end user timings that yeah, so the user timing APIs. So the W3C has a set of timing APIs. One is the user timing API. So how much time have I spent doing DNS lookups? How much time am I spent doing SSL negotiation? How much time am I spent waiting for the browser to download CSS and JavaScript, waiting for the page to paint and for the page to become available? And then they also have the resources timing API. So if you've ever added a social widget on your website like the Twitter share button or LinkedIn share button you'll realize they really start to impact performance. So that gives you performance timings for the individual JavaScript and CSS that you include on your site. And then they're just releasing a custom timing API so that you can do whatever measurements you want in JavaScript. And let me tell you all of this so that I can tell you this which is to automate your client side performance testing with grunt and gulp. So this talk's gonna deviate a little bit but use Bower for managing dependencies, use grunt or gulp for automation, front end automate your front end workflows and EOMN for bootstrapping a project. So if you're getting started with something new today there's a bunch of EOMN generators for Django that make it really easy to do all the stuff that I'm talking about with best practices built in by default. Grunt or gulp doesn't really matter pick which flavor works for you. Okay, so by a show of hands how many people understand, how many of you would call yourselves professionals? Yeah, hopefully all of you. And then how many of you understand that you have a problem when a user complains directly to you or via customer support? Which says you have no level of performance monitoring at all built in. You find out when users go and say, hey, I can't check out into your website. So really the reality is that most people don't measure performance. I work for a performance company so I'm not here to pitch that. My goal here is to simply say that this is really impactful and affects your businesses more than you might know. So you should be tracking performance and development and production. Even in those companies, the few companies that do track performance they only track performance and production. They use some variety of APM tools out there. But if you're gonna do load testing the whole goal of load testing is to generate useful information. You can only capture that useful information if you're monitoring. So before you go and start doing the load testing and the client side performance testing, instrument everything. Insurance your code, instrument your databases, your cache is queues, third-party services in your infrastructure. Because oftentimes it's not the application stack that breaks. It's not the Django app that's the issue. It's logging errors with some error provider making API calls to some payment provider or the cache hit, miss or database lookup. So unless you're instrumenting that once you generate all of this load you're not gonna get any meaningful insights out of that. So the whole goal is you should instrument your applications, your infrastructure so that when you run a performance test you get useful insights. So there's a ton of different open source tools pick your favorite stack but the reality is that you should be doing some level of performance testing and some level of monitoring. I'm a big fan of Chef and Sensu, StatsD, Graphi and Grafana so that you can have a cool dashboard that looks like this. I know I'm covering a lot of this stuff very quickly. But the idea here is that if you can instrument your applications and you can instrument all the API calls that you're making all the database lookups, your cache key hit, miss ratio with your talking to Memcache or Redis you can really start to understand where you need to focus on improving performance when you do run a load test. Running a load test to understand like capacity planning and whether you're gonna fail on launch day is one thing but making it a part another level of functional testing for your applications is really the end goal. And you can only do that if you actually have some level of instrumentation to get meaningful insights. And then the last part is again it's about the end user's perceived performance. So you should track performance of your end users. Because if you have an application hosted in Amazon Web Services in the West Coast and you have a bunch of users in Australia well a bunch of users on Telstra service in Australia might be having a very serious problem and you're never gonna be able to see that sorry to be specific. You're never gonna be able to see that unless you actually monitor the end user's experience. There's a bunch of open source tools that make that possible. So there's the episodes framework from one of the Steve Souders one of the guys behind Google PageSpeed and what it essentially does is it captures the JavaScript timings from inside every one of your real users browsers. So you'll see very quickly if users in a specific location or on a specific carrier are having performance problems because that wouldn't show up on the Python stack that's not gonna be an exception in Century or whatever error-logging platform you use. It's just gonna be a user who has a bad experience who complains to customer support because they think it's your fault. Even when it could be their carrier it could be a particular browser. So check it out. And then webpagetest.org. So I'm a big fan of webpagetest.org for a couple of reasons. One, it allows you to very easily check performance across a different array of browsers. So you can check Chrome performance on the West from specific locations over specific network conditions. So I wanna check Chrome from New Zealand over 3G. How does my application load? And more importantly, what does the rendering process look like? So one webpagetest.org you can just fill out a URL and tell where you wanna run the test and from what browser? And then it'll generate test results. So one of the things that I really like is this is a collaboration amongst a bunch of technology companies. So it's available free. And it'll give you a, both you can use it for a QA purpose. So it'll give you a screenshot of the final rendered form but it'll also give you a video of the rendering process. So oftentimes what you want is to optimize the critical path. So the key part so the browser can paint the page as fast as possible. That's what matters most. And you'll very quickly be able to see, oftentimes what you'll see is the sudden flash and then the page gets painted and that can be a one second delay or you can see immediately the page show up and then it gets progressively built beneath. It all depends on how you architect it. Oftentimes it's very hard to test those things. Webpagetest.org makes it very easy to test and QA. And like I said, if you've ever used the developer console on Chrome or Firefox developer tools, you probably used through the waterfall view of loading your client side resources. This will allow you to see where you're spending time. Like I said, I steal the Google Fave icon service which ends in a bunch of redirects which kills my performance. And without this tool, I would have never realized that was happening. But it basically makes two requests for 20 different icons on my homepage. And my homepage is only 20 icons so it's a bit of a waste. Okay, so I think we've covered a lot here. The final thing we're gonna talk about is sitespeed.io. So sitespeed.io is very much like Google PageSpin except for it's much more automated and much more comprehensive. So again, you can MPM install sitespeed.io. And what this will allow you to do is analyze your website speed and performance. So the same way that PageSpeed will give you actionable advice on how to improve the performance. Like you should cache these CSS and JavaScript files or minify them. There's a bunch of rules that sitespeed incorporates and will test for. So this is really great because when I talk about automating all of this you wanna integrate this in your Jenkins flow your Travis CI workflow and you wanna be able to have some real metrics to use. Sitespeed makes it very easy to capture each one of these as individual metrics and then fail based on that. So you should fail if the average end user experience time is over two seconds or you're including more than 500K in JavaScript or whatever the conditions make sense for your site. It's a pretty comprehensive listing of tools so it's not just CSS and JavaScript, cached or uncached, you zip they're compressed or uncompressed. It's also how many DNS lookups are you using SSL negotiation times, it's very comprehensive so take a look at it. And then again, you should be monitoring production applications. I think a lot of you are familiar with probably New Relic. AppDynamics is a company that works with enterprise applications. There's Compuware, there's another one, there's Ruxit, there's a bunch of these performance management companies. The goal of all of these companies is to effectively map out both the server side performance and the client side performance and allow you to understand exactly what's happening. I've talked about a bunch of open source tools that allow you to do various pieces of this. The goal of all of these tools is to tell you where your traffic's coming from in real time and then being able to capture the individual metrics and then give you visibility into the client side. It doesn't really matter whether you roll open source or you buy a commercial package but you should be doing some level of performance monitoring. And then oftentimes I've talked about a bunch of open source tools for writing all of these load tests but oftentimes it's much easier to buy this than to build it yourself. So we'll just run through a couple of load testing services from the cloud. So I'm a big fan of Fipeka just because they have a global footprint and they make it very easy to just write to load tests and say, hey I want you to go to this URL and send 50,000 users and give me a report on what that looks like. Blitz.io is probably one of my favorite in terms of just easy to get started, blitz.io. It's they give you free service. It's paid and free but they give you a decent free plan where you can just say, hey I want to spend up 500 users send them this URL and then plot the response time versus the level of concurrency. And then Blazemeeter which works well if you have complex interactions or you want to test mobile devices. So sometimes I know it's in the vein of open source we try to write all of our code ourselves but oftentimes it's a business problem that you're trying to solve and you should remember that. That's not always, code isn't always the solution. I can't emphasize this enough but tests for failures. The reality is that most people test under ideal conditions but at some point in life you realize that it's an imperfect world and things will happen. So you should be prepared for them. So Netflix understands this very much which is they've released a suite of tools called the Simeon Army and one of those tools is Chaos Monkey. What Chaos Monkey does is it goes and creates chaos inside of your AWS setup. So what happens if you lose your caching layer? How does your website respond if you drop your memcache instance or your reticence instances? What happens if PayPal's API that you rely on slows down 3x? Do you retry and exponentially back off? Do you queue things up or do you block on them? You don't really discover those issues until you have a production outage. Oftentimes that's when you learn the most when things are failing and it's the most painful to learn the lesson is like, hey, we really probably should have queued that up and not blocked on that or we should retry that request if it fails, not just ignore it. So you should really test for failures. There's a bunch of tools that make this very easy to do but what happens if you run a load test and you kill off your memcache instances? Does it surge on the database? Can your database handle that, et cetera? So performance matters. Hopefully I've emphasized this enough. So tree performance as a feature. There's a bunch of tools and tips on how to improve front end performance but basically use the 14K rule so you have instant loading so you wanna make one small request and get everything you need to paint the page and then progressively build the page from there. There's a bunch of tools to do this but the core of this is use task runners to build and deploy production code. So grunt and gulp or there's a ton of tools for Django that make it just as easy to incorporate these best practices like minifying CSS and JavaScript and combining all this stuff. And then best practices for performance testing. Capacity plan and load test the server side. Optimize and performance test the client side. Again, you spend more time in the client side than you do on the server side so don't just think because the server can respond to a request that your users are gonna have a good experience. Understand your starting point. You can't understand where you're failing unless you know where you're starting from so instrument everything. And don't just instrument and monitor production. Instrument and monitor your development and production environments because you don't wanna fix it when it's red. You wanna fix it in the yellow. You wanna fix it before it goes live. It's very easy to learn these lessons when the executive team's yelling at you because you're losing all this money and revenue per hour because everything's broken. You get a lot of resources but that's also the most painful time to do it so try to do it earlier. And then measure the difference of every change. It's not just upgrading from Django from one seven to one eight that's gonna impact performance or upgrading for your version of Postgres or Linux or moving from one data center to the next. Everything affects performance and everybody cares about performance. They just care about a different perspective. Business people, they care about revenue. The operations team, they care about uptime. Engineers, they care about the end user experience. They just need to carve off the perspective that's useful so you should measure out the difference of every change. And then automate performance testing in your build and development process. It should really be part of the software development process and something that's automated that happens every time. And then ultimately understand how failures impact performance. Now there's been a lot of developments recently. HTTP2 specifically, a lot of the best practices for the HTTP1 world are actually going to hurt you in the HTTP2 world. So sharding, concatenating all your CSS and JavaScript, spreading a bunch of images into one file. These are all hacks around shortcomings in HTTP1. HTTP2 eliminates the need for most of these hacks so just be conscious that the protocols are changing and that you also should adapt your development techniques accordingly. Oogorgorg has a great book on high performance browser, high performance browser networking that I'd highly recommend if you're interested. So again, integrate automated performance testing into continuous integration for both the server side and the client side. Understand the performance implications of every deployment and package upgrade and monitor the end user experience from end to end in development and in production. And that's all I have for everyone. Any questions? Thank you very much for the talk. I had a question about whether you think, given the complexity of modeling involved in creating a load test and an enormous possible investment of developer time, whether there's any value in taking a feature ramp approach with regards to, and I noticed you didn't mention, I don't know if you have opinions about that. So most of the time what I try to do is take the basically replay production traffic, like don't model them out based on what you actually, what you think your users are doing because your users will very quickly surprise you. Go and take a look at production access logs and find the most common user paths and then replay that traffic. The feature branch, when I write a new feature, test out this feature does work quite well, but what you're gonna miss is the complexity of the overall. So the most common use cases are things that you built originally. Anything net new that works well, especially as a way to have sustainable performance testing, but I definitely think that it's something that you should replay production traffic versus just what you think users are gonna do. And I think when you go building sort of the feature branch approach or when you build individual features and you try to just test the net new functionality, the users often use that in ways that you don't expect. So I think you rely on the source of truth which is those access logs. So if you use some sort of load balancing solution, pull it off of that. I'll ask for a follow-up as long as no one else has. So, when I say feature ramp, I meant something very specific. So releasing to say 1% of your users. And then change. Okay, yeah. Yeah, so most companies, well, a lot of the smarter companies end up slowly releasing features through that. That's a great way to actually test that out, especially in multivariate testing. You see that a lot with very substantial changes where you just wanna slowly roll them out. Either way, I would write some sort of automated performance tests, but you can very quickly learn by doing custom instrumentation around metrics that matter for those scenarios. So if you're gonna introduce a new feature, you can use like stats D to create some custom metrics around that. And that's what I would, that's generally how we release new features is when I want to release something new, we'll track some important measurements for that specific feature and then we'll slowly roll it out to the users. Yeah, that's a much better approach. Anyone else? Let's not all ask at once. All right, that's our time. Thank you very much, Justin. Thank you very much.