 So this presentation will be talking about performance testing, performance testing, as opposed to load testing, as opposed to stress testing, as opposed to humidity testing, as opposed to all these other terms. How to actually isolate those terms and talk about what each one of those means to us, at least, in a certain way. So first, I just want to introduce both of us. We're consulting that algorithm. My focus is almost entirely on infrastructure performance, scalability, so complete projects, standing issues, infrastructure tuning, really the whole range there. AQUIA for a little over three years now. So my background, maybe to point out specifically to some of you guys, is actually more from the systems side. So I've done group development for a little while, but I actually come into this from the systems administration side, automation. And I'm Jeff Beeman. I'm also a senior technical consultant with AQUIA. My focus is not performance and scalability on a day-to-day basis. And sort of the reason I'm here is to talk about why that doesn't matter, because I have to care about performance anyways. I joined AQUIA in 2010, so a little bit after Eric, I've been working with Triple for quite a long time. And my primary role on projects is usually as a lead-army type. So in that role, I have to care about everything related to the triple site, including how I've been helping just for systems, or just for developers, or just for kind of that final long-shape. It's really something everyone, everyone's going to write tests, but everyone's going to be a part of it. There's a lot of things that go into the plan, and kind of go into these a little bit more, but really important things, like making sure you're representing what the users are doing. Not what you think's important, not what's first or last to the project, but really what is actually going to make the site money if that's your goal, or get the most information out if that's your goal. Whatever the purpose of your site is, that's really what you want to make sure people are able to get to as quickly and efficiently as possible. Making sure that those tests that you're writing, you're actually scripting that in a way that really similar to what a user is. It will come back to a kind of variability in kind of this, but making sure that if the user is performing some kind of path for you to site, they're not going to perform the exact same path every time. Make sure that you're seeing things a little bit differently. You're actually really trying to get ahead of what your users are going to do, clicking on places, just do a bunch of things that adds to simulates what the real world's going to be. And lastly, what are we going to do to actually analyze those results? So it's really easy to look at those numbers and say, well, it looks good enough, or it was better than before, or as soon as you can actually look at those results and know what the biggest thing is, performance testing is for everyone. So I actually want to say in the kind of abstract for this session, I actually kind of put a call out to all those different roles. So I'm actually curious, how many of you guys are, would you say you're developers first? How many of you guys would you say are like system ends back in? I think that's actually, the PMs, you actually want to really kind of feed that from at the end and kind of reach out to. Because I feel like developers, they want to learn the tools and the learning processes, but ultimately it's all about getting the client project help as a whole. Before we get too far in the discussion, it's really important for us to level set on some terminology and talk about the various things that go into performance testing. Because like Eric was kind of hinting at, performance testing isn't just load testing. It's not just loading a page on your site and seeing how fast it is or finding slow queries, things like that. There are lots of aspects to performance testing. So we're going to talk about some terminology and some of it you might use different words to talk about these things, but it's important to have clear lines in your minds about what some of these things are. Performance is about the speed or the time it takes to do an individual request. This can be both from the time a user requests a page to the amount of time that it takes for that page to come back, but there are also other things that go into measuring performance on a page. How long does it take to render? How long does it take for all the assets to come back? Are there JavaScripts executing on the page that make it take a long time to actually fully render? So that's performance. Scalability is the ability of your site to go from one user to 100 users or 100 users to 1,000 users and still have a predictable performance threshold. And predictable being like not the same, but 100 users hitting your site is probably going to perform a little bit differently than 1,000 users hitting your site or 10,000 users hitting your site. But it's acceptable and it's predictable. That's what scalability is about. There was a lot of discussion when 777 came out about how 777 was slower. And the whole idea with 777 though was that it could scale much further than higher versions of Drupal. So just because an individual page request is slower doesn't mean your site is performing worse. You might just have to give up some speed on an individual page request in order to be able to handle an enormous volume of the page request. Next term is scaling. So scaling is probably a concept most of you are familiar with. It's probably the area most people start with when they're looking at performance. Scaling is saying we want to take our two small servers down in the corner there and we want to make them bigger. That's scaling vertically. If you scale horizontally, that's taking your two small servers down there in the corner and just adding more of them. So load testing is about focusing on real-world traffic problems and seeing what happens to your site for measurable amounts of traffic. So in this slide here we've got 10 users getting our site and our server is happy. When we throw 100 users at our site, our server is, you know, it's restraining a little bit. It's having a little bit of a hard time. Load testing is not about breaking your site. Load testing is saying, we're going to throw X number of users at the site and we're going to measure and see what happens. Stress testing is about seeing what happens or how far you can push your site before it breaks. Other people call this like a smoke test and stress testing is about saying how many people can we throw at the site before it crashes and burns. I like to think about a load test being like how much does, I'm going to get a scale and I'm going to see how much 10 people weigh and we measure that and say 10 people weigh 1200 pounds. But a stress test is taking a scale and putting people on it until the scale breaks. Last one that we're going to talk about and we're going to talk about a lot, we'll use this word a lot, is contention. Contention is a fight over resources and there are lots of places that you can end up having contention in your site. We found this to be the hardest thing to draw a diagram of and it's probably the hardest thing to talk about but it's also the most important concept to understand when you're thinking about performance. I think one way you can think about contention is to think about wallmarks on black writing. So a whole bunch of people, a mass of people line up at the doors and then they open this tiny doorway that's about four people across into the store and everybody wants to get in at once and there's a fight over the resources both to get in and then there's a fight over the resources inside and there are many points of contention inside that store. Contention is a big thing that will scale at some kind of measurable rate. You go from one server to two servers, theoretically you should be able to take twice of the request rate. Contention is a place that comes in to say, well you could do the first server just because that's where your contention lies and then the second server doesn't actually help the problem at all. So that's why it's so key that if you can identify where the contention is, that is what is going to block or reaching those goals from reaching that next level of performance. We've got a couple slides, a little bit later if contention is the concept you don't understand. That tries to explain it a little bit more, but for right now just try to think of it as a fight over resources. Hopefully help you understand some specific examples. So let's talk about what your tests are actually meant to do. So we kind of are breaking up this presentation a little bit into what you should do and also kind of lessons learned from projects we've worked on in the past and what we've actually done that we've failed on our projects or succeeded and did really well. So certainly coming out from both perspectives. So the key is that if your tests aren't good, the numbers you get from them are essentially irrelevant. If your tests are fantastic, then you can use those numbers and actually decide how much money are you going to spend on servers? The quality of the test is really the valuable piece. Make sure you're setting technical goals. So don't look at this as what makes a user happy and trying to guess what that is. Actually try these things over time. Look at when you add each of your ads. What percentage of users is there? Is the site getting, is the low-test showing it's getting faster? Is the low-test saying it's getting slower? What's actually happening over time? And what are you doing to actually monitor that change side-by-side? If you're not looking at a side-by-side, it's really easy to get three-quarters of the way into a project and really have no idea what that model and that's usually where you find the biggest amount of time spent on two of these things is you're so far down the line of a project, you end up throwing more human resources to solve the problem that realistically you could have caught much earlier in the process by just taking it step-by-step. There's no regressions that are actually happening. So regressions from the QA, from the typical QA sense, so features aren't breaking other features. It's just as easy that new features could worsen the performance of other features. So anything that you're thinking about in the QA of what makes a good test, how much you're testing, how wide that opportunity is, you should go right into performance testing. It's really not any different. I think this is one area where my sort of non-performance experts opinion into the conversation is important. My role on projects is typically leading architecture, working with partners who are building features for our clients and we've found, I've made mistakes on this stuff just like everybody else, we've found that if we don't do performance testing throughout a project it can be any weight until you've built a massive project that's very complex and has lots of moving pieces. It can be really difficult to impossible to identify where your site is slow. But if you're constantly monitoring performance and as you add features to your site, you're constantly testing them and seeing how they perform. You can set yourself up for a lot more success and also reduce risk of having to spend a ton of money at the end of your project. A ton of money or time or resources at the end of your project trying to figure out why your site crashes and burns when you launch it. Another good thing to keep in mind is when you build a feature and this goes back to sort of the data build a feature and like let's say you're building an article content type. It's really simple. Everybody probably does it on their sites. You don't want to just make a few of them and refresh the page and see how long it takes for a page to load. What happens when you make a lot of them? What happens when you make the number of articles that are probably going to exist in this thing? Those are the kinds of things to think about when Eric is talking about monitoring changes to your projects and thinking about ways that users are actually going to use your site. The system inside of this, the system's engineer is a lot of these tests that run early on in the project and maybe they're bad tests maybe they're kind of exaggerated whatever they're showing. I've seen a lot of projects create all kinds of crazy complexity in their infrastructure because they have one bad test early on and it's really easy to see that and just start scaling out horizontally doing crazy things starting databases early in a project or these really complex ideas that it's so early in the project you can actually fix these things periodically. It's much easier to do that than actually create this complexity and not only at the end do you not have tests to verify what's going on you have an infrastructure that you probably can't even maintain so you've actually created twice that complexity just by not keeping up with things that were talked over the whole course of the project. And also when we look at this one thing that I always do going into client sites I might do a lot of one-week short-autotype engagements because I'm usually kind of committed against on the one side the developers on the one side the systems because those of them are always ready or the other ones are always wrong or it goes always wrong if you ask someone else to get those numbers it's amazing how that changes these two groups because the infrastructure folks can make a quick change see what it does if it's good they leave it if it's bad they flip it back developers the exact same way so just having these kind of indisputable numbers to work off of it makes the entire working relationship of bringing together these very different teams makes that whole process much easier. Talking about the quantifying aspect of this one of the worst things I see when it's actually a stress test we want to see how many thousands of users we can handle if you're only ever going to have a thousand of those you're going to be making all kinds of crazy changes for a situation that's not going to happen what are you actually trying to achieve by these low tests is it a certain number of users is it making sure that your servers aren't calling over is it making sure that each request is only taking a certain amount of time really think about what it means to actually do a successful performance test it's really probably the most common thing is X number of users well that's great but if X number of users are performing badly is that a success or not if they're getting all the results they want but it's taking 5 seconds they're not going to be happy but you haven't defined them you don't know what's a success and what's a failure so it's really important to look at each one of these levels and actually find out what you can do to look at the results of a test say that was a success or that was a failure and move on from there if you don't have that you get lost in the numbers that's where the blame comes in we support the users but it's slow to cause that really set these up at the beginning and I think this is a good example of where the project manager role comes in is maybe it's not to load on maybe that's not what's important to the project but you want to be able to support 5,000 users at a time with under 3 seconds per user and no users over 5 seconds something like that where you can put in a full solid sentence and every time you do a load test go back and just say yes or no and everyone's under the same understanding the developer is a systems everyone can look at those results and know if they're moving ahead on the project or they've gotten ahead of themselves one thing that I want to throw out here is make sure you're gathering all the data so if you're the client don't trust yourself if you're the consultant don't trust the client if you're the client also don't trust the consultant that's the whole point of getting numbers if you're getting numbers that's what you're arguing against you're not arguing against people or roles or responsibilities the whole point is to make sure that whatever the goal is at the site that's what you're accomplishing does it feel slower? that doesn't really matter does it feel faster? that's not really what that defeats all of us doing the test and getting the numbers one thing that's key is if you're actually looking at value so one thing where performance is very different than something like security is one small security issue can really escalate very quickly one small performance or two certainly has that risk but it's a little bit different 80% of the traffic is one user path that's a great place to dive in and try and find the most the most important change you can make in a short amount of time making sure you understand the full use case so when you're not sure what that full use case is if you think it's doing a search, maybe it's doing a search and going all the way through your purchase don't think about individual user actions think about what you're trying to get to because if you're building an e-commerce site you want to know the time when someone gets to the site to the time they make that order if they oppressor it anywhere in between you've probably lost that order so make sure you understand what the entry point is, what the exit point is and what's really making a user happy and successful in this site and also I think this applies to anyone anyone, any of you guys that have taken the time to comment or put on comment this session you're all smart people don't think that you've found numbers that aren't right if you find numbers that you don't think are right write a different test and see if it comes out the same way and don't write one test question it you're all smart people, write the good test talk to the right people see what you can do about getting those numbers where they need to be and then finally move on you're all smart people just make sure you're actually utilizing I think one thing that we try to do it's funny but it's true like the first bullet point never trust the client but there's also always trust your intuition and never trust your intuition always trust yourself never trust yourself and sometimes you do need to trust the client it's kind of like when we were talking about it there is no screwing right or wrong you have to constantly be questioning your assumptions and your intuition the client's assumptions the client's intuition but then you also want to always trust it too if the client is constantly telling you saving an article page it's slow and you can't replicate it there's something going on there you just may not be able to replicate it you may need to dig deeper you may need to go watch them create an article and see what they're doing they may be doing something crazy or they may be doing something that you never tested because every time you test creating an article you just go ASDF we just say we all do that right the other thing I wanted to say about this particular slide is around the important metrics that Eric was talking about where if you've got if your goal is that anonymous users are able to view pages in less than 2 seconds no more than 5 seconds or at least 2 seconds no more than 5 seconds and that is your primary goal if you're focusing all your attention on why saving article pages is slow you're not putting your effort into the right place if there's like 10 authors on the site and they have to deal with the pain of saving an article page that's not really like the business case you have to solve you need to solve the business case for the end users if that's a more severe problem I can't mention e-commerce just because that's kind of the easiest place to see a visiting user has a particular value to the site every site has some value from a user whether it's getting them into some sort of business just making contact with someone e-commerce is a great example of your goals for someone to buy it buy whatever your product is buy whatever your service is it's really easy to monetize that and look at the actual value so I think e-commerce is just a great simple case but I'm sure each one of your websites has something that you're trying to give to clients or get the other thing is don't give up until it's done this is kind of a mantra I use but most importantly don't think that maybe you're newer to Drupal but you know the system stuff but again you're all smart people it's a system stuff that looks good dive into Drupal see what happens get your hands dirty if you need to write a core patch just fix something I was on work on a client that has 5,000 nodes and I think 5,000 taxonomy there's a type of chocolate taxonomy access control and they were building out these huge queries where the query from their perspective each query is taking two or three seconds so they spent probably a month tuning the database putting on ridiculously big hardware doing whatever they can to try and fix it what they didn't do is realize the query was very fast it was taking Drupal 4 or 5 seconds to build the query so it was really easy for them to look at oh it's a slow query let's go one off from here but if you really get your hands dirty and realize what part of the page is slow not just trying to articulate what the big component is that's slow you can actually dive in and look at the profile of the page and save weeks of effort don't get lost in here it's part of a larger ecosystem in your company and Drupal is just one piece of it you might not run into issues that the actual in-user would see so one example of this is a consumer electronics company in New York they were running load tests on a Drupal site no problems at all barnaces working perfectly fine no issues at all what was happening was when the users were coming in they were going to the main.com site that was a Drupal company that happened to match the configuration they were using in varnish for this actual Drupal site so the only way to recreate the problem was to actually go back and think what's the entire user story they go to the.com they hop over to the blog they click around that's the user story it's not just getting into Drupal so their entire fix was to change the cooking their site went down multiple times on the block Friday I mean every horrible time you can think of for a consumer electronics company and it was about 2 or 3 hour change I mean don't get lost in thinking it has to be Drupal because Drupal is slow it might not be Drupal itself it might actually be some external service it could even be a web service whatever it is but Drupal site has other issues and those are all those other people out there make sure everyone's checked the easy stuff so everyone has a great infrastructure team everyone has a great development team but you're going to forget little stuff you're going to realize that if the servers are slow maybe varnish is just missing things you don't think it is because when you click around it's fine but maybe it actually is causing an issue and running tests throughout the day throughout the course of the project you'll actually catch these things and last and this is really key at least from my perspective don't accept anything less than perfect so we talked about the article node type entry that's certainly talking about priorities could you launch with that? of course but in one case I've seen a lot and this was a non-profit education customer was their site would just crawl when qualms were running just absolutely crawl and their solution to that was to see us thinking out why? why don't you ever need that so needless to say they had some slight side effects on that but that's because it wasn't user-facing so they said whatever we'll turn on so these are kind of things that if you really start shipping away at what you're considering perfect what's really making your site work you're going to end up with this just kind of down on a spiral and it's going to take you months to come out with some cool stuff I think along those lines as well something might work but there might be better solutions to it I worked on a project last year that really used feeds and lots and lots of feeds in that site and theoretically those feeds could be processed through ground that's how feeds does its thing but when we were talking about that literally thousands of feeds we had to be more creative about how Drupal processed them because if we ran Drupal Cron every minute we get all this overhead of other stuff that happened so we were able to run those feed thru thrushes through thrush itself and that was one case where it was acceptable but it wasn't perfect so we wanted to make it as good as it could be so we needed to speak up a little so this set is talking about infrastructure so one very common mistake we see is using your dev environment for testing so dev environments have a lot of overhead like you're running additional things that aren't being run in production like xdbug or xhdrop you might be using watchdog and dblog you might be logging to the database so that you have easier access to log messages in dev dev environments are typically on congested networks as well so your dev environment might be on the same server or in the same sort of network as a whole bunch of other dev sites and it's not going to give you a realistic view of what performance looks like so the other thing is to not extrapolate results so in this diagram this is a typical testing setup in the next step we're adding more users and we've scaled up our hardware a little bit and the db seems fine last time but in this case we went back to Eric's earlier example where things were fine with a small number of users but when we add more users we start getting contention and it doesn't matter that we scaled up our servers it doesn't matter that we made our web servers bigger that wasn't the problem in this next step we doubled the db size and we added more servers and now we're better so the whole idea here is that don't assume that the results you get in one environment can be extrapolated to the next step similarly, and this is again related to contention we do a small load test that might be really hard to see so I'll try to talk you through it the blue box on the bottom is the locking query a query that does something that causes a lock in the site it could be locking a table it could be Drupal putting a lock using a lock itself something causes a lock in the process and then we have a couple queries waiting but in the small load test we get about 20 milliseconds of overhead for that last query but as we add more people or we start waiting for that lock to clear stuff piles up and you get basically a traffic jam on your server and that's contention again so our originally what was a 20 millisecond overhead for a query, you start piling additional queries on top of that and you get more and more of wait time if you want to say that if you look at the first example you're really going to look at averages and begin with your test and you're not going to look at how much the slowest query was and how much the fastest query was if you're going to look at this test and realize that this was a third of what and as you actually stack these up because the other locking queries we're going to constantly add that overhead 20 milliseconds in 30 then 40 then 50 again it doesn't matter how many requests are in there, they have to take at least that amount of time so no mad ones, there's a lock and you have to start understanding what that build-up of overhead is to actually realize what the extrapolation would look like and when you really do a couple of these the extrapolation just doesn't work unless there's no points of contention which if you solve that problem I'd love to solve the other ones a good example here is when you clear Drupal's cache you have to start understanding what that buildup of overhead is when you clear Drupal's cache and the next page request that comes through let's say you cleared the menu cache the next page request that comes through has to rebuild that menu cache it doesn't matter if just one request is coming or 5,000 requests are coming you have to build the cache and you only have to build it once but all 5,000 of those requests need that same thing that's being built so they're all waiting on that one thing to be done so this next one your numbers are only as current as your last test so you want to write smart tests don't just write a lot of tests saying that you want x% of test coverage isn't always the best approach what you want to do is write smart strategic tests that truly target your use cases on your site so talk about user scenarios work with your business work with the client to understand what is the path the actual path through a site like Eric was talking about that consumer electronics company if you didn't know that users started over in this other part of the site before they got to Drupal you would never run into the problems use data from analytics this is one area where you never want to trust what your client is telling you about their scenarios for the business get real data from analytics that tells you what the actual most visited pages are or what the most common path through the site is it's not that you don't want to test the scenarios that the client gave you you just want to use real data to inform what you're testing you want to ensure variability so change what you're submitting in forms change the path for a user, change how they navigate to a particular page so you want to also measure test coverage so both components on your page as well as the total page count and as you build new tests or build new features build new tests make sure that you're testing the things that you're about to launch gets to our next big point this is one we see all the time all the time is that your tests aren't about launching your site you don't write tests and run tests just to launch your site you want to be building tests after you've launched your site because nobody's nobody in here I imagine nobody in here launches the site and then just brushes their hands and walks away from it one guy yeah every time you have a new feature you are changing your site and you need to test performance again you can have dramatically unintended consequences when you have new features and you could add any feature over here that does something you didn't think would affect at all user performance over here but it does so testing is about QA and QA never stops so you're testing to either way the one other note here is that when you launch launching is about scale about meeting the scale of the site when it goes live doesn't mean that your test patterns change so continuous integration is for performance as well as continuous integration tools like Jenkins provide mechanisms for you to do all kinds of things that aren't just about building and deploying your code so these tools like Jenkins a lot to do testing so ongoing changes to your site ongoing content entry into your site things like that can affect performance over the long term and by using a continuous integration tool that tracks results and measures results from tests you can see over time trends in your site performance and that's really important because manually running tests every once in a while you may completely miss the fact that every day for the last three months performance has gotten a little bit worse you may not be able to see that but if you can go into Jenkins once a week and take a look at that graph and see that it's going up you can proactively take care of the issue before it becomes a problem also when you're really thinking about more of the kind of pure sense of continuous integration not only are you seeing it over time but usually you're also seeing it when new code is created so you can actually have that correlation of the performance got worse and oh yeah, these are the last three code commands so you're not having to trace back your whole site and figure out where that happened you actually have all that data in one place and you know who to go talk to looking at these milestone tech tests where so much has changed and it's really hard for you to keep that bottled up and understand all those different steps so the last thing and this is really kind of the one piece where we'll get a little bit more technical it's talking about the different types of tools so we talked about load testing and that's kind of what most people think of but when you're really talking about performance testing that bronze a little bit there's a little bit more than just one that you can talk about so make sure you're doing multiple types of testing so it's really easy to look at that final in number of a load test and just use that as your solid single number that you're going to work on it's great and ultimately that's what the real end users are probably going to experience but there's plenty of different levels that are actually going to speed things up for you to work through the process so everything from request profile so for the developers to do things like exit drop or XD bug where you're really looking at the individual pieces of a page load and trying to find those problems that goes back to the the customer I was talking about that had the massive tap light queries they went to a certain page and saw bad performance they jumped on the page they didn't quite jump all the way down in the profile to realize it was the one so maybe use this for investigation and not really as a benchmark but it is the first step while you're doing development service testing so service testing is the way I look at this is if you getting a Drupal request a single page you're touching a patchy or an X you're touching my SQL you might be touching the memcast you're touching the network you're touching all these different elements but it kind of manifests as the time of one request service testing is thinking about how long does it take for a patchy or to run this query on my SQL maybe there's some network lag between my SQL and the web server you're not really going to know that unless you test it separately looking at something like Mib Slack or Mib Cache D all these different things where you're really testing each level of the stack and even testing that work itself these are the kind of things that are going to make sure that you don't make some false assumption about that one and the last two I think are a little bit harder to sort of compare but when I look at a simple response testing meaning something like a patchy bench or siege or services like Condo where you're going to get one single raw number of what the performance is the amount of the amount of opportunity you have to react to that and sort of simulate what it really is you're seeing can be sort of in this so I look at this as something that you really use for monitoring and kind of real quick gut checks but ultimately of course load testing is the plan so we really are starting to jump in quick anecdote I worked with a client who we were seeing very slow performance on queries to the database but we couldn't explain it because we would go look at the query a lot the query that was supposedly slow and we'd go log into my SQL run the query and it'd be fast and Eric kind of hinted at it but some places that we probably never would have checked until we sort of stumbled upon it was the latency between the web server and the database server the time to make that connection and then send the request over and get it back was what was slow and that's what something like that is what service profile is about as well where you would even develop query log may not tell you that that's a problem or you'll see weird things like this query like a cache get which cache gets should be very quick it was slow, slow, slow fast, fast, fast, fast, fast and then one of them was slow and another one was slow that can hint at things like so load testing we split into two different types and I think this is really a key thing from every level to understand what the end goal is first I look at what is usually called virtual user load testing and the goal here is really just to send a request out, get that response back and made a reactive usually it's just a simple transaction a lot of complexity to it or a lot of really interactivity the cost that I have down here is why that's relevant at the minute but these sorts of requests are cheap it's designed to be efficient you want to send out 5,000 requests for the least amount of resources possible that's your goal here maybe this is for stress test or whatever the role is it's really that's a specific purpose secondly I look at what I look at as a real performance real youth load testing so this is using something like Selenium using something more browser based where it's actually loading the page as if it were an end user performing actions with mouse or the keyboard actually doing what the user is doing and getting those results and if you look at the cost difference the amount of resources required to do it is a massive difference and connect some more is usually a common gap a full browser full user kind of full sign off of if this is going to work or not versus I sent out an HTTP request I got a bunch of texts back and now I'm going to do something with it they're naturally very, very different just for instance for virtual users using something like Jmeter you're sending out a bunch of these requests maybe using expat there's some kind of regular expressions to pull something out you read that, you make an action on it Selenium, you're really looking at what is a browser going to do to load this, right? Is there blocking the JavaScript, right? That's not something you would see in a virtual load test or in a virtual user load test you really have to look at this and understand what the actual user is going to see and how browsers can interpret it and maybe how you slowly actually see differences there as well based on the client and I think this gets to a really important part of performance testing that doesn't get talked about a lot except by, you know, people who are sort of at the cutting edge or do maybe like a lot of mobile work is that front end performance is really critical and it's something that as developers, like back end developers and sysadmins we don't like to think about because it's like, well my server is serving up the stuff just fine, it's no big deal, right? My job is done but front end performance can dramatically affect the end user experience. You can get a report from a client that tells you that the site is slow and everything on the server side points to the site being fast if they're waiting to download giant images or there's JavaScript that blocks that essentially blocks rendering of the page for a certain amount of time or a bunch of analytics stuff that's going on that's usually the most common problem if you've got like 10 JavaScript libraries making all kinds of analytics calls that can make your site slow but you wouldn't see that on the back end look at this as kind of a holistic effort of developers have their role in it QA testers have their role in it sysadmins have their role in it but each one of these really fits the whole process so it's really key that everyone's involved and everyone's under the same understanding and use those numbers to make everyone's life make the communication a little easier to hope some people have some questions that put the mic here just while people are going up there I want to mention we are hiring but I think specifically the kind of stuff that we're doing here this is what people like me and Jeff do on an everyday basis so if this is the kind of stuff that we're really interested definitely come talk to us after this session we'd love to meet your assistant if there's any questions left over or things you think of later just reach out to either of us on Twitter we'd be happy to continue Hi, my name is Albert I have a quick question about the tool you're using in Jenkins to create those graphs those performance graphs what expression are you using in Jenkins for that that particular example is using Jmeter and getting the X unit files out of it or X also Java Jmeter Jmeter and Jmeter so it outputs kind of a standard data format and that's what it uses to graph that so it can actually pull selenium results the same way and a bunch of other services it's kind of a common format I want to test the IE so what I have is one machine that's testing with selenium Firefox and Chrome and then it sets if that passes it sets a different branch so that another instance of Jenkins running on Windows is going to test the IE does that make sense to you guys makes more sense than running IE at a certain point that is realistic right at a certain point testing that testing on IE and all these combinations that's what users are going to do so that's going to be the most accurate way to do it this was really nice validation for sort of the direction I've been trying to scale what it came toward I'm really curious what tools you're using to generate the numbers you talked a lot about it about evaluating the numbers but I didn't see too many examples of each testing there's obviously a lot of things and we sort of danced around it I'm just curious if there are a lot under your list that you can basically say here's the top five or six things we're using to generate those test numbers if you're there to talk about evaluating sure so for me request profiling XH process is the biggest thing that's what I use to jump into a project for service testing using just a basic static file for Apache for varnish, I'll actually set up basically a URL of varnish that turns directly and test that using Siege or AV or all those tools for memcash which is a tool that comes with pretty much any installation for memcash for the real user load testing there's things like SOAS though there's things like Keynote Africa there's tons of service out there really a lot of this comes down to almost end there at a certain point so as long as it's performing this role and that's why I kind of avoid it using too many product names or anything as long as you get what it's trying to do and you can sort of map that to a features list and really knowing what you're getting out of the product that's really the key here because there's plenty of tools that will give you essentially the same number that's exactly what I was hoping using because there's just so many options out there and it's nice to know that basically it doesn't really matter the tool you're using it's creating that measurable metric that you can see the trend and maybe even those testing tools you compare the two of those you kind of look at what one's doing versus another and try to see the difference that's the beauty of numbers is you know which one worked better so don't even think that one tool is the the all or false solution so you talk about your numbers are only as good as your tests and it really takes good tests how do you determine a test that's good versus a bad test I think a lot that really comes down to understanding some of these things of what's happening maybe a little bit outside of them or what could a user do or what kind of randomness is there right variation like if I go to the home page am I going to click on the top one every time no I may click on the top one the bottom one the third one the fifth one having that kind of variation that simulates what a user would do so as a good test or smart test I really think of what would a user do what am I trying to understand what would users roll out the site what they're going to do use analytics use these other tools to understand what that is and that's what a good test is so it's about figuring out the realistic actions that's where you're going to make money that's where you're going to get visitors returning that's the sticking point right like if you're running an e-commerce site and your typical user adds a few products to their cart removes the product adds another product and then checks out your test shouldn't be user goes and adds a product and checks out you know you want to test a realistic scenario and you can put randomness in there and that's the beauty of the automated testing is going to someone and saying well today do two products and tomorrow do 46 not going to be a real advocate for testing but if you can automate that and say I wonder what happens if there's 150 items in there or 30 is it scaling linearly is there some pop there and those are the kind of things that say if it's a good test or not is could a user go in or accidentally click buttons or maybe they've done this over a course of time and added all these to their cart really think about what the user's doing because that's what makes that user valuable to your site like I mentioned if there are any follow up questions feedback for us personally feel free to reach out to our either of us on Twitter look forward to hearing from you guys, thanks