 It's officially good afternoon. Good afternoon, everybody. My name is Eric Matthew. I'm an Enterprise Unboarding Manager at Pantheon Systems. And obviously it's been a long four or some odd days that you will come because my voice sounds like my 12-year-old son's voice does as he hits puberty. So please bear with it as it sort of cracks and scratches and generally just goes downhill as I talk. So with that aside, one of my main jobs as Enterprise Unboarding Manager is load testing websites. I really, as I mentioned earlier, I really wanted to name this session destroying websites for fun and profit, but I wasn't so sure that the Drupal Association would take that title seriously. So I went with load testing Drupal at scale with Locust IO, Casper JS, and Elastic Beanstalk. So as we all know, as Drupal 8 comes out and they start making plans for Drupal 9, Drupal is becoming a much larger mainstay in the Enterprise world, right? We're running bigger and bigger sites. Larger presences, we're running huge farms of stuff, right? And the websites themselves are getting infinitely more complex. I personally have worked on aeromexico.com. I've worked on patch.com. I've worked on Southern Poverty Law Center. Really large sites that require a massive scale of testing to make sure that the code actually runs functionally. Which means that as we develop stuff, as teams develop stuff, it becomes more important to actually not just test your work to make sure that it works, but to make sure that it works at scale, right? That one function may work great if you're just like one person testing it. But if like 25 people are hitting it at the very same instant, does what that's executing actually also scale to the SQL queries scale? Do those calls out to, let's say, Twitter or authorize.net or the back end Sassus and do those things also scale? Because if they don't, obviously you have problems, right? We need to rewrite it. In my, going on five years now of being at Pantheon, we've used a lot of different systems. Like we used Blazmere, Loadstorm, Siege and a bunch of other things. And it came to a point where we needed something that we could scale infinitely, right? And also run not just a variety of sort of automated, like just simple like posts and gets, but could also interact with browsers, right? Which most, if not all, the commercial products don't do. And so after a lot of research and a lot of patience from my boss, I built, I found a system and started building out something using Elastic Beanstalk, which gives us a scaling we need, Locust.io, which gives us me the test bed that we need. And then for the browser-based things, Casper.js, which is a headless browser testing, right? Which is built over Phantom. So real quick question, how many of you or your teams load test new features? All right, actually I'm not. The guy in the back corner. I told you I wouldn't forget you. The guy in the back corner, load test system's cool. How many of you talked about it or thought about it or wanna start actually load testing your stuff at scale? Sweet, excellent. I have an interested audience, cool. Because obviously we've always talked about it, right? But almost always, traditionally from a QA standpoint, like smoke tests, regression tests, that sort of stuff. But again, at scale, things that pass with a simple, like click through with one or two of us, they don't pass if you have 5,000 people hitting it. A really, really good example that I almost based this talk on is a non-profit client in Florida. They have this one time, it's once a year, giving thing, right? It's a united way, right? And so all the non-profits, it's a competition to see who can get the most. And the one that gets the most gets everything matched and doubled. And so they go to town, they have literal war rooms just like this filled with volunteers who just donate at the same time. And so they'll in the first five minutes of their day of giving, it's called in five minutes, they'll take like somewhere between 10,000 and 15,000 donations in five minutes. Just at that same time, as soon as people are done, they go to the score page and they hammer the score page to see if they're in the lead or not. And if they have to like rally more people to donate. So it wasn't just like, we're gonna do 10,000 financial transactions, right? With a full checkout flow in five minutes. But we're also gonna like do like 50,000 people hammering on the score page to see who's in the lead. Because of course they didn't have it with AJAX to like pull it asynchronously, that'd be too easy. They just had to actually manually HP refresh. And so the site got crushed every time. And then the development agency came in and I was asked to go in and load test a thing. And I was like, I need something that will load test this at scale and also mimic browser interactions because they had anti-spam stuff, which is where Casper came in. And I ended up spinning up something like 50, double max EC2s. And just like, it was one of the most entertaining load tests I've ever done. Because it's just like taking out the hammer and just like, it was so much fun. And also doing it affordably is important, right? Because that sort of test through a commercial system is gonna cost you a lot of money. Whereas elastic bean stuff, when you're done, you just spin it down, you're done. You can get rid of the EC2 nodes. Like it's as expensive as you need it to be. And then that's it. So because these are all open source projects with the exception of elastic bean stock, I was able to put together a really affordable, flexible and scalable testing solution, right? I can run five concurrent users. I can run 10,000 concurrent users. I can run more with Amazon's permission, of course. They sort of max you out at a certain point to make sure you're not doing something really nefarious. And while loads on a blaze meter are there and they're really powerful and they're really extensible, they're also really expensive. And in their own way, they're limited, right? You can write your scripts for them, but you can't go really any deeper. And so the things I've actually been able to test, besides shopping cart purchases, site searches, API calls, phone apps, processing workflows, publishing workflows, like all of that stuff, are things that you should load test at scale. So just to run through the pieces, right? Elastic bean stock is an Amazon product. It allows you to take an EC2, an AMI, right? And spin up as many of those nodes as you can and they're all load balanced. So as Locust is making those calls, it sort of shuttles them out between all those EC2s, right? Locust IO is Python, open source, well-developed, well-maintained, very extensible. And Python, if you haven't worked in Python, is actually not a bad language to pick up. It's a pretty easy language to pick up, especially if you've already been doing PHP or any other type of programming language. Python's actually quite simple to pick up. It's not that big of a deal. And this framework, actually, I can't take full credit. I picked up in 2016 here from a blog post where a guy was like, hey, I work at Amazon and I decided to combine elastic bean stock with Locust. And I was like, yes! And I grabbed this, forked it into our own private repo and Pantheon and started to get to work with it and then added Casper to it. So I can't take full credit for a lot of the underpinnings, but this project right here is the starting point. Back to my slide here. Additional libraries that you will need, just so you know for the board I'm gonna show you are PyQuery, which is like a jQuery type library for Python. You're also gonna need Casper.js, which is built over Phantom.js, so you'll need both those things as well on your EC2, your AMI. Otherwise, this will fail spectacularly. Obviously, you need to have the elastic bean stock and Amazon command line interfaces installed on your machine so you can fire stuff off in the command line. And then as well, there's also for Casper, for recording tests, if you don't want to write them from ground up, there's a plugin, open source called Resurrect IO, that works on Chrome that allows you, and I'll do a test record, that allows you to just like step through a site, like an e-commerce site and just record the test flow, spit it out into JavaScript and then say the JavaScript file is a Casper.js test file. And that, while it's not always like Selenium and other libraries, they're not always like perfect, it gets you in the ballpark with all of the objects and everything, and you sort of sometimes have to tweak it around from there to actually make it work perfectly every time and robust it out, right? But it's a good way to sort of get that base going. And all these, obviously all of this except for elastic bean stock is open source, which I'm a big fan of. So that's pretty much like the introduction, this isn't a long deck because I'm actually going to like pull down the repo, change some tests, record some tests and walk through an actual load test because I'm pretty sure you'd rather like see it work than like listen to me yamble along and stare at a slide deck all day. So I'm gonna take a walk through the repo, we're gonna look at the repo, we're gonna initialize elastic bean stock instance, we have two of them, we just sort of keep around, right? We're gonna make some tests, we're gonna pull down a Casper test, and we're gonna modify the basic locust file and pull some tests in. And then I'm gonna unload on a website. And then we're gonna sort of watch how the performance of the website functions at different levels inside of, let's see, should we use? We'll use New Relic Pro, right? Because that's a nice deep dive analytics tool, so we'll use New Relic for this. And then we'll go through it and see what we managed to break. Because that is really most of the fun of this. I have to admit, it really is almost all of the fun of this. So let's get down to business. So here, right here, is our repo. On a side note, functionally, right, I'm just gonna sort of like talk a lot of these things through. Functionally for us, we do a branch per test, right? So for every client I sell for any of our other EOMs load tests, we do a separate branch so that their tests are saved forever. And if we ever have to rerun a load test on a client, they come back and they say, hey, we've got this new feature, we'd really like it if you could reload test it, right? I can just pull down that branch and just rerun the tests, right? So the master, we just keep clean and then we just branch it off for every client. We have a run and then we have a full archive. And if I'm not around, another one of my team could just grab that branch and run the test, right? So functionally, the first real thing you should do is make sure you always branch off per test, per client, per whatever it is that you're doing, right? Which is why there's X number of branches and all this sort of fun stuff. And sometime in the next few weeks, I will have, oh, don't you, there we go, thank you. It's a problem with projectors is they hide things. All right, so while this is pulling down, which doesn't take very long, let's open it up. So the functional parts of the repo itself, the things that you're gonna play with, like so this is the policy file for elastic beanstalk, you're pretty much gonna leave this alone, just to let you know. The functional parts that we have built out, locust requires a locust file.py. This is sort of the master file that locust by default wants to go to. You can change the name if you want to, but this is the best way if you just wanna fire locust off remotely without anything special, locust defaults to locust.py, right? And then what we've built out are in here some standard tests I just have commented out and some sample tests. I believe we're talking, there's so many people now have lost you. We were talking about API calls. So we have form post samples, generic Drupal services tests, right? That essentially mimics just a constant stream of hits to a service and gets the results back so you can load tests like just a Drupal 8 service bus or Drupal 7 service. I have samples for generic Drupal logins, form post samples, and then also in here as well. So what I've done, what I typically do is I put my helper functions on top, things that are called constantly all the time, and then I put my specific test classes farther down, right? So for example, going to one of the tests that's actually a bit more finished and I'm gonna run, right? So I have, for example, the anonymous index page that takes a path, default path is essentially just top root, and what it does is it substantiates a HTTP client, it loads the path, so in this case the home page, it grabs every href, every ahref on the page, loops through to make sure it's not a telephone number, right, because that will break things, make sure it doesn't go into external domain, right? And then essentially puts it into the Python equivalent of an array, and then it comes back here, and randomly grabs 10 of them, right? One at a time and fires off an HTTP request to that path, right? And the thing to note, the thing to keep in mind with tests like this when you're running it through something like Python or C, just something like that, is these execute as fast as the programming language can execute them, right? So when people, when you're talking to someone or you're deciding what the load test is gonna be and they're like, what's your peak users? And they'll say, we get maybe 100 a minute, right? Which is fairly substantial, okay? When you break that down, 100 a minute divided by 60 is how many per second, right? And you break that down to per second because locus runs on users per second. And those users are going extremely fast. They're not normal. It's not like somebody like clicking a link and reading the page and clicking another link. It's clicking the link, loading the page, it's like this. So the traffic is a massive factor higher than a normal person, a normal user would be. Which for my purposes is fine, right? My purpose is to break the website. My purpose is to be a complete and utter jerk to the website because I wanna break it. I wanna find every edge case. I wanna find every performance bottleneck. I wanna find every SQL query that is what we call a SQL query of death that doesn't scale in SOD, right? That's my point. So like, I built this to sort of run on turbo. You can throttle this if you want to. It's Python, you can do anything you want. I haven't because I don't want to, you know? I know what application containers on Pantheon and other places should be able to take with a well-built Drupal site, right? Regardless of how much traffic they actually get. And I know what that performance metric is. And if that performance metric isn't hit, it's worse. Then I know something's wrong and I can start digging, right? Cause I just sort of have a, I've done this long enough. I just know a good app, like a good piece of a good PHP transaction on a Drupal site should be 700 milliseconds or less. And 700 milliseconds, I'm still not comfortable. 500 milliseconds or less, right? Hmm? Man, right? And if it's slower than that, right? Then I'm not sure because gains are incremental. I'm sort of going a little bit off, but just philosophically, right? Since I still have like 42 minutes to two your ears off, right? Those gains are incremental. So think of it this way, right? Your servers have a certain number of PHP processes that can run. And if one process takes two seconds to run, or if that process was better, it would take 500 milliseconds to run. That's 1500 milliseconds freed up. That's four times the number of processes that could run, right? And that's an extreme example. That's a huge, you know, but if you scale that for a site that's getting a ton of traffic, if you can get certain transactions down by 100 milliseconds, right? And you're getting 1,000 people a minute. That's that many more people you can click through and keep going and make happy, right? So it sounds really trivial to be like, we can cut 150 milliseconds off of this, but it's not. Because when you're running at scale, 150 milliseconds is an eternity because it stacks up. And the same with SQL queries, right? A good SQL query is fast, 50 milliseconds less. If there's something that's actually trading a slow trace and it's taking like 30 seconds to run a SQL query, that's thousands of SQL queries that should have run with that process in that 30 seconds. And that's really bad. That's a SQL query of death, right? And so that sort of stuff is what we hunt for when we run load tests. It's not just the responsiveness. It's like, where can we make incremental gains to make the site faster at scale? Okay, and all of this is just geared to try to find those out. And so here's another example, right? This one, so structure-wise, locust files are built like this. You create the task, you say how many times this task is gonna run, right? It's hierarchical. So this one's gonna run 100 times. And for every 100 times this one runs, this one is gonna run 30 times, right? And this one is gonna run five, right? And so you can take, instead of a locust file, and you can sort of hierarchically create what hits what and how much proportionally. So if you have a client or a site where it's like, okay, we get hammered at blog, we get hammered at news, not much to mount the traffic on the homepage for whatever reason, and we get one purchase flow every five minutes. And so you just sort of work out relatively speaking, like where those lie, and you start defining your tasks with the weights by that, and that's how you control how much traffic goes to what process, right? I also have tasks built out for searches, right? Site searches, because those can often be a massive performance suck, especially if it's the core search versus running an external search appliance. And if it's an external search appliance, you still wanna check it out to make sure that those are performant, then you haven't made a poor choice in vendor, right? So this one, for example, I have a separate file, flat text file that has like the thousand most commonly used words in the English language, and it just grabs that, grabs the word randomly and throws it through the search, right? So it's that way, also your search isn't repetitive and it doesn't cache. You're always triggering fresh searches, right? You're always punching through the cache with your search, which is important to make sure it performs well. And I also worked out for a pet food vendor, a geolocation. So in this file here is every US city, its name, its county, its state, its coordinates, everything you need. And so I can grab that parser, grab one at random and throw it a geolocation search because some stores say, hey, where's the nearest store? And those can be really, really heavy. What you're looking for, again, is anything that could be heavy, anything that's problematic, anything that takes up a lot of resources or could be like a lot of custom code, anything that calls external services, like anything at all that could be a problem, you hit it and you hit it hard to see what it's gonna do, right? And then down here, this is your locust test class. It gets the, you set an environmental variable called the target URL, but I also typically set them here, so there's a default just in case that environmental variable goes away. The task set it's gonna use, and then finally a min max weight. That min max weight is essentially how a random number that it weights to spin up another locust worker when the previous local worker is spun down, right? So if I spin up 50 concurrent users, once that user has gone through his task set, it's spun down and then between one and three sections later, another one is spun up and it goes back through, right? And at any point in time, there's 50 of those things doing that, dying and coming up and dying and coming up all at the same time, or 5,000 or 10,000, right? So in this case, and then the other thing we're gonna talk about, so further out up here, let me introduce you to Casper, right? So in order to include Casper JS into the locust framework, there wasn't really a good way and it might be just my familiarity with Python. If anybody here is a Python expert who wants to chat with me about it, I'd love it, but it was simply just to send off an OS system call, right? Casper JS test and then the file name, right? And then also grab a time to start that job and a time for the end of that job because as you'll see locust does a really wonderful job of keeping track of how long requests take and then you can spit those out into a CSV to sort of munch up later for data reports. The only catch with firing off a system call like this is I don't get the file sizes of the pages, right? Which is one of the things that locust records because I haven't really found a way to have Casper spit that information out in a way that locust can grab ahold of and throw into its reporting matrix. That's on like a mid to long term like things I wanna get done with this thing in the next year or so. So Casper itself is essentially fired off as an operating system command line call, right? And we capture the start and finish and it goes through and then you can custom name what it shows up as. So at least you know, like in this case, as you'll see it's a purchase flow for a bike pump. At least you know that this purchase flow typically takes this long to go through the purchase flow all the way to the end of the order, right? On average, best worst. And if that's five pages, then at least you know what that flow is like, right? If we're not catching the individual pages. So speaking of Casper, right? What we're gonna be beating on is an e-commerce site for mission bicycles. So Resurrect.io I mentioned earlier is the Chrome plugin. It's really, really quite simple. It's quite nice. It just pops up like this. I had also side note looked at things like Selenium and other testing frameworks. And some of the complications with Selenium at the time was that it didn't fire off particularly well through the command line in EC2 consistently. And it wasn't talking with Phantom.js very well. And at the time, Chrome and Firefox didn't have a headless option for the browser. And so on an EC2 there's no display and therefore you fire up a browser that wants to display and it just explodes. Because it's got nothing to do its output to. Whereas Casper.js and Phantom.js it's a headless browser, right? It doesn't need a display. It doesn't care. It's built for this sort of stuff which makes it very handy. In the meantime, Chrome has actually come out with an option to fire it up to the command line headless. So it's another thing I'm gonna reevaluate is whether or not Chrome can run well on our EC2's headless and be more consistent. Because what we get into here is you'll see is browser compatibility. Remember back in the, for those of you who've been around as long as I have the old browser compatibility does this show up well with IE6 and Firefox 4 and it breaks here and I've gotta do special stuff there and it's just a giant pain in the butt. Right when you're running a headless browser you run into a lot of those things because it's a very, it's a powerful thing but it's really hard to troubleshoot and it's very fussy. And so that's kind of the drawback that I found working with it. So I'm just gonna just to show you what it's like and it's just to record a quick test flow here. When you go to shop with 45 lights. Front only. Add to the cart. Check out. The other thing to also keep in mind with recorders like this is they record every keystroke. So that mistake right there.op is now in script. And I'm gonna have to get rid of it and clean it up. And also if you have form information sort of saved in the browser and you just tab down and grab it the recorder won't grab it. So these are all like the little ADBD gotchas that will drive you nuts that I'm hoping that I'm helping you avoid because they will drive you nuts, right? And also typically these recorders prefer clicks in locations, right? Do a little recording clicks, really the same. I wanna see how the comments come. That's probably gonna break just because I can't remember what the authorized.net test credit card is but it's okay. Yeah, yeah I think it is. I've already gotten one of the scripts I'm just flying through this here just to sort of show you that you stop recording and then you export to JavaScript and here it is, right? And so we can see things like all of my missed typos and everything else. And then all you really have to do from there is just grab it all, we've got a JavaScript test or a Casper.js test. Also typically we'll always try to name mine. Oh, stop it. Siri go away, you're annoying on a phone you're even more annoying on my computer, right? Save it as a JS file and here it is. So you can see all the clicks it goes through it tries to go through and get as much identifiable information as it can get, right? And some of these will fail. Like you didn't need to go back through and clean these up because they don't always, these don't always hold true to actually execute in Casper just as a warning. You can see all of my typos, where's that OP? There's too many P's. Yep, there it is. Test pantheon.opio, right? So like these little gotchas, these typos will get you every time. Cause it doesn't actually want to record the delete for whatever reason, don't ask me why. Okay, so these then can in turn be tested and run on your local with just a Casper.js test. Like this is a pump purchase flow. And the difference is the key thing is with Casper is if you put test in there, it puts a framework around it to look for tests and output the results of tests, right? And pass and fail tests versus just running as just a Casper.js file name, right? So for example, so it'll take this test past, past, past, past, past, past. And the fun part is this bottom 23 past, the zero fail, zero dubious. Like they actually wrote in a state of dubious. This, this test, this, this is a dubious test. So we're just gonna, we don't know about this one, but you can look at it. But we personally, we feel it's kind of, we don't know what you're thinking with that test. It's a dubious one and zero skipped, right? And so this is what you're looking for. It found the input names, it entered the credit card information, it found the completed message and this particular image is the image file. It's like, thank you for shopping at Mission Bikes, right? And so you can also have it look for success messages. And then there should be a new order in the bike shop. So back to Locus Faso in Beanstalk then, what we do, just EB and it, we're gonna go to Northern California. I keep several different environments around. All right, just floating around. These will keep an easy two or two sort of alive and stable, but it's a lot, sometimes these can take a while to spin up and we do enough load testing that for us just having them around for other EOMs to grab and hold up and run with is worth it. But you don't have to, like you can spin these up as you go on demand and that's fine. I think we're gonna go with, this should be on one. And what this looks like in Beanstalk, right, is we have load test one, load test two, what tiers they're on. I have the configuration saved. So when I need to spin up a new one, I can just spin up a new one with the existing configuration, which includes the AMIs and everything else, which is also something I can automate. All right, so at any given time, I can have, I keep two around because there's four EOM, you know, prize onboarding managers on Pantheon. But if I need to have like one per, if I need to have 10 of them because we're all running like four tests each in a day or something crazy like that, we can just sort of spin these up and spin these down. All right, and some of the options on them just to take note of that can be set from the command line. The most important one is the number of application servers, right? So right now, there's only one running, right, in the EC2. But if I wanted to come and just like do maybe scale 10, 10, it's going to scale up to 10 EC2s, right? And later on, I can just go back and when I'm done with this test, I can scale it down to two, right? But you need to preset it. The auto scaling doesn't work particularly well, just as a catcher, because I tried this, because it doesn't grab your custom AMI ID. It spins it up with Amazon's preferred AMI, which doesn't have your PyQuery, doesn't have your Casper, and doesn't have your Phantom installed. And there's nothing you can do about it. It's a giant pain in the butt. So if you're hoping you can like auto scale your instances up as the low test ramps up, it's not going to work. Just sort of pre-calculate these tests through this heavy. I mean, you need 10 app servers and just re-pre-do the containers and just be done with it, because doing it on demand just doesn't work, right? But in this existing one, we have everything set and we're just going to go ahead and it wants to, it links to, right? It's linked to get. So it actually won't push code, it won't deploy code unless it's committed to get, right? So if I did ebdeploy right now, that lights test.js won't go, right? Just for funsies, I'll actually do something functional. We're going to push that, ebdeploy. So this command just like deploys the code out to the elastic beanstalk. And then we're going to go ahead and fire off a load test. By the way, at any point in time, if you have questions, just like ask them, throw something at me. It's totally fine. I have absolutely no problems with that whatsoever. Actually kind of like questions or save them for the end, that's fine too. But yeah, I have no problems answering questions in midstream, so that's not a problem. Hit me. Yeah, but I'll pop. I'm actually in the middle of cleaning everything up so that I can GPL, so I can just put it out there for you guys to play with. But it's a matter of scrubbing all the client info out of it and like genericizing things so that it's okay for you guys to have your hands on it. So yeah, I actually have this forked one more time for public and I'll probably have it ready in like a week. So you can just get a hold of me and I'll shoot it out or we'll do something on Twitter about it, right? So typically, so the thing with Locust, you do eb open, it opens up a web GUI, you can also do it against the command line. Number of users is a peak number of users and hatch rate because they're stuck on the bug analogy is how fast those users come into being. So a hatch rate of one is one user per second. So depending on what you're trying to simulate, like a slow ramp up, a fast ramp up, you change this. So if you want like one new user every two seconds, it's 0.125, you know, something like that. So I typically will just do something small to make sure that everything is working properly, which is important with client sites who may have hard coded links to I don't know their current production environment. Just saying, like going whole hog in there and like blasting their current live environment because they forgot to change all those links is probably not the best idea. Other things while I'm talking about that to ask clients if you're dealing with clients with this stuff is, is a test going to hit sandboxes for like their authorize.net for their internal services for their SAML for all that sort of stuff, or are you gonna hit their production environments, you know, and are those production environments capable of taking the spike? That comes up for me most often with educational institutions, right, they use a shibboleth sign in for like UCSF's internet was a client and they had to simulate like 5,000 students coming in and hitting the thing and at peak all hitting their shibboleth server because every time anybody uses the internet they're logged in. And so I was like, they ready for this, right? They have a sandbox, please? And like, no, they don't have a sandbox so I'm not gonna run it until you have a sandbox or until your shibboleth engineers actually tell me me directly that it's okay because I don't wanna be the one response we're like taking down a production environment. And so these are things you have to, which also brings up the point that we're testing an ecosystem. We're not just testing Drupal on a server, we're testing the ecosystem that makes up the entire website. Drupal in and of itself does to a certain extent irrelevant, right, because it's just part of the whole. We're testing Drupal, we're testing MoriDB, we're testing Twitter, we're testing authorize.net, we're testing all of it because your client isn't gonna care that it's Twitter's fault that the site's slow, they're just gonna care that the site is slow and they're gonna have you fix it, right? And so with this little test, we're getting zero failures, we're running five users, I have seven slaves. The other cool thing about Locust is it's distributable, right? So you can set up a master and as many slaves as you want and the master loads out to all of its slaves and gets the results back in, right? Which is another reason this runs so well on the EB. I can have one master and 500 slaves, which gives me 501 instances of Locust firing off. Limitations is that Python natively will only allow a certain number of HTTP connections. So that's something to check. If you have too many Locust connections going on in a single box, you may overwhelm Python, right? And you'll start to see Python errors. So that's the other thing to keep in mind, whatever environment you set up, you need to change that to something that Python can deal with. Or change Python so it can deal with Locust, six or one half dozen to the other, right? So you can see we've got the caster ones running down here and they're significantly larger because one, they're unauthenticated and two, it's like five pages instead of one. The other things you can do in this interface is you can sort by number of requests, by number of fails, the median time, average time, min time, max, content size. So you can start to use this to really troubleshoot and try to find, like this page is gigantic. Why is this page so gigantic? Why is this page so consistently slow? Why is the median so lousy on this page? And you start to use this to ferret things out, right? The other things Locust has are failures. So like if I kept the telephone thing in there, so it wasn't filtering out telephones, the telephone requests would obviously fail. Exceptions, these are code exceptions. So if Locust starts failing at the code level and start throwing operating system errors, those show up here inside the exceptions, right? And then finally, when you're done with your tests, you can download the data in a CSV file, which allows you to like put it into reporting engines, whatever it is you wanna use. So I'm gonna go ahead and we're gonna ramp her up a little bit, then go to 25 concurrent users. And again, this isn't just like 25 people gently clicking through, this is like 25 people on speed, not reading anything, just like smacking buttons and going crazy, okay? So keep in mind, this is like more like 75 people and you can see that when we come to statistics, like 11 users were ripping 20 requests per second, right? So the rate is what matters, really, okay? And then if we go ahead and go into New Relic, how many of you are familiar with New Relic or New Relic Pro? Cool, it's awesome when I actually see the majority of people because I think it's one of my favorite freaking like analytical tools. So now we can start to go in and see, okay, what's going on? And I'm also gonna go in here and you remember how I said like a lot of the points just to be as big of a jerk to a website as possible. That includes like mashing the clear cash all button as often as you can, often back to back. I'll go in and I'll run the cron over itself, like I will do terrible, dumb things you should never do to a website just to see what I can break and what I can trigger. Like I'll search through the watchdog, I'll do just dumb things that you tell yourselves and your clients to never do underneath peak, right? What do you do? You do not do under peak, you don't clear the cash unless you're me and then you mash it a bunch. Just I wanna see what breaks. I wanna see what happens when there's a cash stamp feed because inevitably someone's gonna trigger one and you need to be prepared. Like what's gonna happen? Does it matter if that's the worst case scenario? That's what you're testing. You're testing worst case scenarios every time. Don't be nice during a load test. Don't spare anybody's feelings. I mean, you can be gentle about the results and say these are some things you should really look at but don't spare them when you're running the tests, you know, because then it falls over and then that person is to blame for destroying the site for bad code and you'd rather have it just fixed proactively, right? So we're currently getting into the meat here. We're gonna see we've got an error. And one of the reasons I chose this site is it's old and janky and hasn't been maintained very well so I knew it would spout out a lot of like errors. It hasn't gotten the data for the errors yet. So things I look for a new relic while we're chatting about and I have like 20 more minutes and about five or 10 minutes and I'll stop shooting my mouth off and I'll take questions, okay? Things I always and me look for in new relic because I always look for transaction traces every time. Right here, I go right to the transaction traces, right? They're my first visitation. So in this one, we've got Drupal entity controllers, we've got load, file, what's our slow trace look like? What's the suck in here? Metatag entity load, shocker. Application code inside the site itself. So this is custom code that's been embedded somewhere that you need to find, right? Lots and lots of entity loads because you know, Drupal 7 loves their entities and we have some theming stuff going on, application code within the plugin render. And then do we have any SQL queries of death? We've got some terrible queries but we don't have anything that's showing a query. So what it would show if there's a query of death here is it would show an explain on the query that's slow and it would give you advice on how to fix it, right? Because new relic pro is awesome and it allows me to be lazy and not have to do that myself even though I'm capable. Right, and so we go through and we start looking for stuff like this and we start looking for patterns. Like in every single one of these traces does it have the same pattern? Is it predictable? If it's predictable, then we know we have a problem that can be reproduced that we can go to the developer and say, hey, I didn't explain on the code, this is your function in the load test. It's consistently slow. We need to talk about how you refactor this before we launch, right? So patterns are key, spikes are key, weird behaviors are key, errors are key. You can take a new relic and we'll just do this by slowest average response time, right? This is where like the pedal hits the metal, good old droop will get formed which shockingly doesn't have any slow grays. Crazy. I know what does though. It's always the shopping cart. Right? No view page. I'm gonna go ahead and while I'm at it, just because I'm that guy, I'll clear the cache again because we're currently, now we're finally, so I'll try to clear cache like every certain percent, every 20%, every 10%, like systematically, if I'm running 50 users, then I'll try to clear the cache at 10, 20, 30, 40 and 50. I wanna see how it reacts at every one of those tiers to a cache clear, right? So that allows us to look at the code and see how is it scale, at what point does the scaling fall over and die, right? And there's all these other things I'll sort of look for. Luckily, currently we have no failures. So while this is running and we have 15 minutes left, I'm gonna stop my yammering about how obsessive I am about breaking stuff and see, are there any questions? Does anybody have? Hit me. Yep. Yeah, yeah. Yeah, so the question was tools like Apache JMeter and Blazemeter, which is a paid-to-play system, how do I feel they compare to like home-rolling a system like this, right? For us, our direct experience is that this has been consistently less expensive than Blazemeter for the tests we have to run. I can, because I have all my functionality pre-built, I can roll my tests out a lot faster on this because I know the tests I need to run and it's very simple to just hit it and done, I can get a low test going in less than five minutes, right? And JMeter, I don't know, I just didn't get the same flexibility with JMeter, especially with adding in something like Casper to do the browser-based sessions. That was a pretty big blocker. And honestly, the vast majority of our tests are just like the anonymous user, front page, grab the URLs and hit 10 of them randomly. Like that's what most of our clients need and that I can get going in like two minutes to like any scale. So you just have to remember to spin the stuff down because otherwise you have 50 servers sitting up there getting charged from Amazon. Like you have to remember to turn it off. You also have to remember to turn it off, by the way, because sometimes it'll run overnight and you didn't mean for it to run overnight and then you're like, I'm a jerk. So there's one back here and then you. So what I've seen with Drupal 8 is kind of fascinating because of Drupal 8's caching, right? And if you have a functioning varnish, right? It's instant, like once the thing is cached it's insanely blazing fast, which is the point. The cache clears on the other hand. If you're like doing what I do and you're clearing everything, the re-caching mechanisms get super heavy on a cache DCL. Like they were ridiculously heavy, either massive spikes. When you don't allow Drupal to re-cache in a granular fashion, Drupal 8 to re-cache in a granular fashion, it's gnarly. So of course I do that as much as I can because I want to see what falls over. But yeah, 200 milliseconds, 500 milliseconds, I think is perfectly acceptable. It would be actually fabulous, especially if it's re-caching in a granular fashion. There's no way a Drupal 8 site will reach that if you've cache cleared everything. If you force a Drupal 8 site to re-cache every single block, every single image, every single piece of text, it's catastrophic. That's excellent. Well because I know your instance. So in his instance, Drupal 8 is running a services bus and returning to JSON. And his developers forgot to actually make it so that varnish would work. So everything is getting hit fresh every time. So that's actually, considering that, that's actually quite phenomenal. But in Drupal 7, a pure JSON return call for some of my clients that run a headless setup like that, like attn.com, for example, it's 40 milliseconds. So, yeah, that's the JSON call uncached. Because JSON is light and fast, right? So, and then back there. I have not, but mostly because I haven't played with Ansible yet, right? I sort of, as a former developer now at EOM, I sort of have to play with the things that I need to play with directly to get it done, but I would like to do that for sure. I think that would actually be rather fun. And it would probably work aces. I just haven't had the time to like putz around with it. But that, I'm probably gonna put something like that on my roadmap after I get the Casper test to talk to Python better so I can get better results in Python from the Casper stuff. Right, yeah. Next question, was there anybody else? Did I worry about your tears? Nope, yes, no? Okay, well, cool. So, I don't know if I have my cards in my, my name's Eric Matthew, and I'm again an EOM at Pantheon. And if you have any questions at all now in the future, come find me, drop me an email, it's just eric at pantheon.io. And look for an announcement on the Pantheon Twitter sometime in the next week or two with this repo being made public so you can just grab it and fork it and like have at it. And I really appreciate everybody coming and like sitting through this and have a great final day at DrupalCon. And I should remember to stop the test. That's literally, the test will keep going if you kill the browser. So if something happens and the browser dies and then you forget you were running the test, it keeps going, like it keeps going, right? And so it's like, oh, oops.