 I've been involved with testing for over nine years now but majorly I've been working with Microsoft Stack for the past seven years. So I don't get to work in a lot of open source projects. As a part of POC for finding a better or a much stable or much suited to our needs performance testing framework I came across Locust and just to give you a little context of things currently we how what we do in my team is we use a lot of JMeter anyone uses JMeter here any problems with JMeter any problems you can think of perfect tool great what about how much load can you put from a single instance of JMeter any any any any so I currently work in Autodesk and I work as a part of the identity team we have a huge a very huge amount of requests per second that we need to actually cater to and for that we need and for that basically we have to run a lot of users concurrently to actually simulate the similar loads using JMeter what we were noticing is even in a distributed setup in JMeter we were running into bottlenecks and for that we wanted something that can have a little that is little less resource intensive so what I see I've not used JMeter you can tell a lot you can tell me I'm wrong I've come from as I said I've come from Microsoft background I have used a lot of visual studio performance testing suite so yes that's pretty good to be honest but yes it carries a very hefty price what I've noticed is in JMeter is basically one user utilizes the whole of one thread till it's disposed so unless you have a very strong machine you cannot have a lot of threads and hence you cannot have a lot of users running on those threads I was searching for this and basically on these lines and I Google and it came across something called Locust so Locust is an open open source tool it's been it's made it's licensed to MIT people in MIT made it and they had quite similar experience using JMeter so I thought why not give that a try so here is my experience using Locust let's see how it is so first of all why Locust so basically it's quite lightweight it can be actually run in a distributed manner quite easily though I did not have to try this I ran it on a complete TOC capacity only so it's basically intended to use to test websites and why it's called Locust is if you know what Locust means it's basically they swarm a website as to imitate that logic so that's why it's called Locust so it's completely Python based it's open source you can hack into it you can add your own methods into it can you see properly okay it's open source and you can add your own logic to it add on to the source code do whatever you want with it that's the beauty of it it runs on something called Gwent Gwent that's basically in terms runs on something called greenlets I didn't have much knowledge on it I did a bit of reading on it and found out that they are basically sub tasks so what how it manages resources it one thread is is divided into multiple sub threads and that one single thread quickly context which is between threads and that's how it can manage a multiple users in that one single thread that's how it's basically more resource it's not very resource hungry nutshell and that's pretty much it the features boring part I'll get to the example just to show you what I mean so anyone well versed in Python here anyone and well versed in Python knows ins and outs of Python yeah don't ask me any questions okay this is how a regular Locust file looks like so if you go down at the bottom you see something called website user Locust and basically something called a Locust has been passed into it this this this class over here defines the behavior of Locust you want me to zoom sure okay one second better so this class over here the last one defines the behavior of the Locust so you can imagine that this spawning one insect and this says how that insect would behave so in the sense that that insect would hit this URL it would spawn after five seconds and within the period of five to nine seconds so how Locust behaves it it wants to keep the timing random so it takes up a time between the minimum and maximum and in that minimum maximum it takes a random time and spawns the user to the reason they explain why they do it is to keep the behavior as user like as possible so after you spawn the Locust what do you want it to do this is all defined in these these classes that basically take in something called a task set so these tasks set are instructions to your insect they said how what exactly it should do this defines that so you want it to call a post on your side get or do anything you want this is where you define it so when you actually install Locust or when you actually install packages for Locust you get it comes installed with something called an HTTP client and that basically gives us the facility to call or make calls over the Internet or basically make all the verb calls so just as an example I have two Locust setup here one is actually running off of an API that I wrote for something else that I can actually show you what it does so what I've done is I've written a couple of bunch of APIs and they're running locally I can actually hit those and show you what happens the other thing is I have another Locust which is set up to hit Google because I'm pretty sure they wouldn't go down with like a thousand users maybe can you see this clean I don't know if I can enlarge this can you see this properly the command and everything so the command for starting up a Locust file is you say Locust minus F to signify that what file you want to start or start hitting and then you specify the file name so this is basically for me it's user load dot py and under this if you had multiple Locusts planned then you can hit any Locust you have to specify which exact Locust you want to hit so as I said I have two Locust specified in my file one is hitting Google one is hitting my local APIs so this is this one is actually from a local API and if you started it automatically goes to port 89 89 and if you heard that locally this is the screen that automatically gets loaded now you have two options here you have to specify the number of users it has to imitate and the hatch time so hatch time is nothing but after a certain amount of time if you want a ramp up of sorts this is how it simulates the ramp up so let's start easy so it's basically if you see it started it spawned one thread and one user to be honest and basically it's hitting a certain task that was specified to it which was a post on my local API so it's basically going to keep on doing it till you stop it the other way is when you start Lucas you can specify a time and after which it will automatically stop if there any exceptions or failures they captured here like this one second let me enlarge it so basically what it was expecting is if I get a user if I search for a user I should be getting it back and that should have a status code of 200 but I was something went wrong in that so it couldn't find that user or there was some other error so it got 400 back so this is the kind of errors or failures it it it basically gets here so you can see all the failures it has already failed six times on post itself and this was certain amount of errors that happened if you want to see charts of how how it happens this is a total request and the response times and it measures everything so you can collect this data in this graph automatically you can use this information but it's only available to you till your local server is running as soon as you shut down your local server this all goes away so what you can actually do is you can download this data for later now I'll stop this because as I said it's running all locally I don't want to tax my system too much but just to show you just to show you what I mean by its resource efficient we're gonna hit Google so same file different locust sorry if you see the host from local host will change actual point to Google now we can say maybe so just to make I got confused a bit when it all started so when I say I want a thousand users what it's intelligently does is it doesn't really spawn thousand concurrent users why does it it it spawns thousand locusts but doesn't use them all at the same time what you have to keep in mind here is this counter RPS over here it basically says that how many requests per second it's actually trying to make so we said we want 10 10 users per second up to a thousand users so it's gonna ramp up to a thousand users and basically this is how it looks like so if you see on the resource side of things it's even with this it's taking about 500 mix of memory and about 9.4 percent of processor usage and it's making about 103 now increasing about that request per second to Google you can ramp up as much as you want I'll discuss as a limitation I'll discuss it later and we're done with the demo but this is how it basically looks like in a nutshell let's keep it running for some time so there are no failures the chart looks like this there are the requests per second landing up and this is a response time it was there was a spike in the start and then it's stabilized this is this defines the number of users so basically you see the number of users I've stopped increasing now so with a thousand users making about 141 40.2 request per second I'm gonna stop it okay so drawbacks since it was as I explained before it works on something called wind that's not really optimized for Windows so if you're running this on Windows servers you will hit a very strange bottleneck wherein all your calls after a certain amount of users will start failing so for me this was for about 2700 users so when I had more than 2700 users try to go beyond 2700 users that's that was a limit for me in my Windows system if however you use this on a Linux space system that's much more efficient you would be able to hit much higher loads on us from a single machine not to mention this as I said is distributed you can run it in a master-slave configuration anytime so that's even more efficient it I mean coming from jmeter where and you can actually click and add elements and everything this is a little bit this has a little limited hand holding so you have to write your own logic and everything else so yeah that's that's pretty much how it is and that's it very short any questions let's give it so I think you mentioned briefly we can talk a little bit more about how can you scale this? can you run multiple loads of agents or different nodes? do you want to produce more loads or have you tried this? no I haven't so so I didn't try a distributed setup because I said this was only in a POC capacity but I've gone through videos and people's testaments were in and a read through documentation wherein you can have multiple nodes set up quite easily and they only recommend one master but you can have multiple slaves and if so the way they they they set it up is the master would be running on one node this the the test code has to be copied across all the slaves but the slaves know about the master so basically when the test run the master just dictates work to the slaves whichever slave is free it manages it automatically and runs it you have a single dashboard yeah you can you have just have to read the master's dashboard that's it so when I had used jmeter I had come across some plug-ins jmeter has this plug-in ecosystem right so one of the plug-ins that I frequently used was server analytics okay so what we are doing right now is basically creating load from the client sure yes but we do not know the resources being utilized and the load that is actually getting generated on the server when we are creating the load from the client so does locals have something like that where you can come to know what is happening on the server side excellent question I was gonna say one of the drawbacks of this later but okay anyways frankly I did not look into this to be honest but I'm pretty sure Python has some free library that lets you do that I probably think how it would go down is they have an agent that needs to be set up on the server side that that taps into perform on and and gives out the results for them but as I said I haven't really looked into that no sorry but yeah so we don't use this if it are jmeter just because we are very finicky about what we deploy on our servers so in most of our anything above our development environment we don't even have access to it so we had to go through jump through hoops to go through it in the past when I had used VSO or visual studio it automatically if you are using Windows servers it automatically can tap into perform on so yes that's correlating your number of users versus memory and process and processor usage is amazing on a one single graph how currently we circumvented it is we have something called New Relic running on our on our servers we put the load on it and we get all the processor and memory data from New Relic okay thanks they mentioned that you are investigating docus because you're not happy with the performance of jmeter so I'm actually curious what's the max load you can get from jmeter from the existing configuration roughly about as far as I remember if we are using the UI it's about 200 250 if we stop using a UI just use the console part of it I think you can stretch to about 500 users beyond which we have to use a distributed system which we actually went for later I guess you all didn't change the default setting because default setting comes with a very low JVM I think we all did that I didn't wasn't involved in it so I think we they all did that they tested multiple things and then they decided they want to move everything they had to something called blaze meter I don't know if you know about it or not the thing with blaze meter is you don't have to set up anything they have their own setup you only have to be give them as your test and you can run it from multiple locations so I'm sorry so that was the max load I have seen they might be more but yeah sure so two things to consider here do you have prior knowledge of how so yeah exactly so so locus locals basically offers you flexibility over what you can do with it how you can run it how you can mod it on and takes away basically all the easiness of you know creating tests in jmeter so if you are well versed and you have people who can actually do it then you can easily run locus I would probably recommend it yeah definitely it's much more suited to run on mark it's yeah that's only only prerequisite I can think of yes the one thing you have to be careful on running on Mac and everything usually they come with a max limit of open file headers which is set to a very low limit so if you're specially making calls over a CTP what it does is it every single call opens up a new file header so that limit is low you would see that after that limit is reached you your stress would automatically start failing but that's a very small thing it's just a configuration change that needs to be made on the machine yeah this is as I said this is much the underlying libraries are much more optimized for Linux so this anything anything Linux space this was much will work much better that's for your presentation and I have a question do you know is there any library to integrate web driver with locus because actually we're working with the same example we can integrate we can use plug-in program web driver because you know that the browser used by performance that took a test to normally just simulate the real browser so we don't have a great response time for base rendering that's why somehow we need to integrate with web driver so we have a great response time so I don't know if there is any library that we can integrate web driver has Python specific libraries so you can just import web driver to be honest and if that doesn't work for you there are many frameworks like a robot framework which have which have web driver incorporated within them but by driver itself has a library as far as I know on Python. No, what I mean is that of course the web driver library already exists but in order to make it work with locus it should have some library like with like the plug-in for web driver to run with. Oh I see I see what okay I'm sorry sorry I understand what you mean now so basically what I felt is I'm not 100% on this when what do you specify in a task can be completely up to you it doesn't stop you from saying that don't run you know don't call a web URL or don't instantiate a browser it doesn't stop you in a task you can do whatever it want whatever you want just that some of the matrices which are like client or the response times might not be recorded so in that method it's like a normal Python method you can call whatever you want so if you already have libraries of you know web test written you can just call them directly so that's the excess that's the flexibility part I was talking about you can basically do what Python allows you to do that's it cool thank you guys