 Okay, let's start. Can you hear me well back there? Yeah, great. So welcome to this speak. It's going to be about performance testing in Chameter. I'm going to have this speak hopefully in other events during this year, but you're the lucky ones having participated in the first year. So we're going to be, we're going through some definitions to make sure that we are kind of in the same page about what performance means or at least what I understand what it means and the definition of the users in this context and about defining scenarios in this testing. And then we're going through some other available tools. I'm going to run them pretty fast because there's quite many of them. So don't be, I'm going to run the slides pretty fast so you probably will miss something, but the slides are already, you can download them from the triple cam dot London for my sessions to check up later if you're interested in the other tools also. Then I'm going to give a demonstration of Chameter. I'm going to demonstrate basically from the scratch and I basically hope that I will cover as much of the most important things but I definitely don't have enough time to demonstrate all the cool things. And then I'm going to mention, well it's not written there but a couple of findings I've had in my experience of performance testing, just a couple of them and then I'm going to quickly mention about one tool, about continuous testing tool and then if we have time then questions and answers. So I'll quickly introduce myself. I'm Nikara Wunderth in triple org. I'm using the nickname Inyxu but I'm not anymore an Apple fan. I should probably change the nickname. That's why it's Inyxu. But I like to make really performant solutions. I've been from my student life I remember that it was really exciting to do really tiny changes and knowing at the same time when I hit enter and deploy this it will have an impact for many users and it was really exciting and that's why I'm kind of excited about being or making like performant systems in that sense. Also performance is something that is really hard to explain to clients so there must be some tools or some ways to kind of highlight to the clients that when you're launching a project they know what kind of traffic the system on the infrastructure can handle and that's really if you just give a promise like yes yes we're used to this that it will handle this kind of amount of users or you say that you know it's we're not doing the hosting it's not our responsibility or something like that it's something I wouldn't like to say to client I would like to be in a project where we make sure that we have proper proof of when we launch a project we'll actually handle the traffic that we're expecting. This is also more relevant for like infrastructure systems where it's not that automatically scaled because I don't know it might be actually relevant on that case also if you want to test the scaling it's an automatic scaling but it is definitely more important if you don't have automatic scaling on your hosting. So performance for me what it means or generally what it means for different persons can be basically about page load, performance may mean to somebody that how many concurrent users it handles or in combination of those both there's also a different concept for concurrency and latency also so if you talk to somebody about performance he might be thinking that yeah it's basically how fast the page comes to your browser whereas another person thinks that how many service I need to make sure that 10,000 people can browse the web service. For developer perspective not maybe actually from developer but for maybe SUS admins or something they might think about the CPU memory and input output and from infrastructure perspective you'd think basically the network bandwidth and like the routing network routing for instance but from end user perspective is basically how fast the service feels generally. There's a way, a nice way introduced there's I don't know what it's called like yeah an index score system called aptX and you might be familiar with this if you use neural like there's an aptX score I remember one or two years ago when I was using Neuralink and I was wondering what is this thing and I actually like two months ago actually went to the question mark and it gives you an URL address and it's quite interesting and it's a really nice tool if you want to read more go to aptX.org to get more into the specifics but quickly it's a measure for the user satisfaction basically three elements or three zones of the application like users that are satisfied or users that are tolerating or frustrating and when people are satisfied they are basically really productive so basically when you're using the web service you don't get any time you don't get a feeling that oh I need to wait or like I'm already ready to click the next next element or that kind of stuff you will realize when you are tolerating when you start to click new tabs for instance if you're it's really common that if you quickly click around and then you get really complicated Drupal side and you haven't thought about the performance you especially when doing administrative tasks in Drupal side you might find yourself like yes I will open that but I will open in a new tab and I wait that it will load but I will still you know click another tabs ready for the next task and that means that you're tolerating but if you're satisfied that it means that you are not like tempted to open always new tabs because the load time is too slow. For the tolerating when there's a concept or a value that is we call it T in AppTex and that means basically how many seconds the satisfied user turns into a tolerated user and that's not like a fixed number you can define this value for yourself and by default in New Relic at least I think it's 1.5 seconds and the frustrated users then are the ones that basically are ready to leave but you know I don't care anymore I'll just go to Facebook or something. So this NASCRAP this is from the AppTex.org basically when you specify the T value it will there's another value that is the F value and that's basically T times 4 so if you say like half a second is the T value then 2 seconds going to be the F value if I calculate correctly. So there's a formula so AppTex.org is basically calculated as such so basically you can see that there's no frustrated users here they are like you lost them so you're not getting any points for that. For tolerating users you're going to get like half of the points so for example if you have 100 samples which is the lowest number there it means that there's 100 requests made in this test and 60 of them are satisfied and then 30 of them are tolerating and then the rest 10 requests or samples are frustrated so you get a point of 4.75 and then if you manage to even get like the 10 frustrated users even into the satisfied zone then you get a little bit higher score or if you get everybody into the 100 every 100 samples into the satisfied ones then you cannot get that full one score. There's no number over one. Then this is that so that was the introduction of the AppTex score and it's a really nice tool so in performance when you do reporting or when you do kind of monitoring that's a really good measurement specify the T value for the project and then use that when you communicate with your clients or in general it's kind of a really nice measure but this basically doesn't ask you directly to the question like how many concurrent users it will handle so you basically throw the concurrent users and then see what's the rest of the AppTex score so let's go into that now. If you already know some tools in performance testing some tools are using threads and then some are using a term user and then some are using virtual user. The ideal way is that when you do reporting try not to use that technical term like threads or something so try to talk about users but make sure that your scenarios are the realistic scenarios. A realistic user have a specific mind when they come to the web service they want to do a performance task and do that. It might get a distracted and they deviate from the original process or they have different devices they have limited accessibility if you're doing a site that is for blind people for example the performance is basically measured more from you need to consider that they're using different kind of tools for reading the website it might have an impact on the load pages for instance some tools don't load the images that are used they just scan the dome and that's it. And then a realistic user of course that like the first performance test what I did was that I just used the AV tool up as a benchmark and just load as many page loads as possible to a front page that's not really realistic at all. So the realistic ones actually when they load the page they get the also the JavaScript and then the CSS and etc. they do like realistic tasks in the web service and yeah let's go to define scenarios before that quickly everybody probably now has the same understanding that what performance usually means and what is an ideal user. So let's go now to defining scenarios. I'm using one example that I did recently an example case for the food standards agency in UK and basically I went today it's not yet hopeless it's in private beta it's going public beta at some point I don't know when. In a few weeks. In a few weeks so you can get access to any group beta.food.gov.uk but it's going to be built in Drupal and when I quickly watch the service I found out that probably the users they go for reading the news and alerts around the food safety there are people who are looking for guidance for the business owners and then information for the visitors about the food safety and then food hygiene ratings etc. And the demo I'm going to show you how to try to do as far as possible to cover one of the cases which is the visitor looking for a food rating. But if you go back a little bit so when you start to design the scenarios for the performance test you think this like what's the purpose of the service and then you start to design these scenarios like just giving feedback or reporting a food problem and etc. And an ideal performance test is that when you define these scenarios you actually run them at the same time you don't run like one scenario and then see the results because that's not realistic and realistic work probably 10% of the persons are probably reading news and then 20% are checking the food hygiene ratings then there's two editors that are actually editing the content which is also causing some cash purging and stuff so that's a realistic kind of thing that you would like to do. Of course sometimes you are interested into one specific thing but I know that this specific feature needs to be measured and then that's more likely than to just make one test that gives you that answer but in realistic work you cannot lean only into that measurement. So if we deep into this case for visitor looking for a food rating basically that's in the home page then it clicks a food hygiene ratings link and then there might be maybe some other people are using the advanced options for the search and then some of them are just typing a keyboard and hitting enter so there's a little deviation there and then they click perhaps the researchers of course there's a small percentage where they don't find what they were looking for so maybe we dropped some of them out from the last request when you write or do your scenario. Visually basically means that it lands to this front page and then clicks the link above there and then you land into this page where you have basically the main element is the search for the 20% that are looking specific things that they are only looking for instance the 5 out of 5 ratings they can click the more search options and make a search based on that and the advanced option used to be at least one month ago looks like this probably maybe nowadays it's more accessible now yeah and then you get the search results and then you click on some of them if you find what you were looking for oh yeah and then you land on the page probably if I almost was wondering that probably when you land on this page you probably would click this link that what do different ratings mean it's more interesting when you have some lower ratings on the restaurants for instance it's top 2 out of 5 then you might wonder okay what it actually means that then you probably would click there also so yeah we're now halfway through I could take a couple of questions now maybe yeah I have something to say if you have anything related to previous topics before we go to the tools yes do we need a mic or a question oh we're done I can repeat that question then so for defining the user and the scenario on which basis we can start on the persona that we use to define the different features you mean like profiling users that they yes yeah I mean you can many many times you actually create I don't know it depends on if you create different kind of user profiles and typical kind of devices they use to do some tasks then you would you know configure it that way but the defining user is something that you don't do on the tool on Jamer the tool itself and the definitions I had I was talking here was more like more like what what performance and user means to me and when we talk about this upcoming demos I was kind of making sure that what are the definitions here but practically technically you don't define what's a user in Jamer basically but you define some kind of like things like does it have cookie management stuff you describe the browser the user is doing but it's kind of when you do the scenarios you think of course the different profiles for the user that they perform kind of different tasks and it might affect what kind of devices or browsers they are using and therefore you probably make a slightly different deviation on the other users that they you know perform these requests with these kind of settings I don't know did that answer your question hopefully and I forgot to repeat the question how far do you go in defining these scenarios because here for example you say 80% normal search 20% advanced search but do you go in the 80% these keywords so many percent these keywords so many percent multiple keywords etc so it's the question that I like do we define like the segments by the keywords or yeah and also the different roles concurrency of different tasks being performed well I'm going to answer to the question that hopefully answers that so basically how do I define the segments in Jmetre so the way that I when I think different users are doing different tasks I have thread groups there I'm going to show it to you then in the demo and I specify either a percentable user that will go and run that scenario and within the scenario you can specify also the percentage of the thread amount of users there are some details that I couldn't get in work as I would like but I think it's possible anyway but I find myself just typing it directly calculating quickly somewhere else and then typing it correctly but I know it can do calculations out of that so when you perform the test then you say let's make a 10,000 user test and then we calculate how many of them are the editors and how many of them are the guys who are for instance have accessibility issues and using kind of different setup and etc so you define the thread groups but when you perform the test itself you give it a number of users hopefully that answers your question let's see what the demo is yeah let's continue so other available tools before I demo the Jmetre so this is one tool that I mentioned, this is something like I wouldn't recommend generally but there might be some cases for this one this one is basically just making HTTP requests, it doesn't load any JavaScript or CSS or anything like that that's maybe good for testing on a network bandwidth maybe if you're not interested in a realistic thing but actually checking how the bandwidth goes in your infrastructure that might be good enough for you maybe then there's a Gatling that's good for the Java based web projects when you define the test it looks something like that then there's this project Lopust which is a tool this is interesting if you need a tool that you need a web user interface and this is quite nice for that because I don't know maybe your client wants to do random tests if you need a web UI there's something that is not dependent on somebody pressing a command line or something this is maybe good for that or if you want to kind of make a system internally for a company that we have a template of running the performance test and you want to deliver kind of internally some kind of interface with somebody some project kind of ery down and then press some button somewhere this has a user interface for the web this was written in python by the way if you do test the Lopust you use python then there's a molotov this is funded by Mozilla it was quite like simple python based test script I probably will use this one but there's one option then there's load booster this is actually I think this is not an open source software but it's a service that is definitely you should use brought with this and there are couple of other things that I missed but I brought this because if you're having like a web service where you have actually like Web sockets open and et cetera chain meter is for simulating traffic but load booster is for simulating actions so when you do something with the load booster you basically define what the users do you're not defining how the traffic is simulated there are two things so basically it means that if you if you want to basically make a request that have a web socket open and then make another request or something like that so when there's also some this is a really nice tool because that in this one you just basically say the user presses a button that has a selector of this and then it clicks and then see what happens but it's more expensive because it needs to simulate the browser there and the limitation for the service I saw that it's 10,000 users and I didn't see any mentions about if you need more like if you need to request an offer about that or anything it was just you know that's the maximum top but I don't know it's a technical limitation it's more like the pricing is it's a plan for over 10,000 and they use cost per JS for the tests and then there's now the chain meter that I've been using and this one is really nice it's quite a major mature in English isn't my mother language but it is funded by Apache foundations it means that the Apache foundation has plenty of sponsors and then they have a lot of projects and I consider the documentation for chain meter and also the if you check the stack overflow there's a lot of good documentation if you need help on something and there are also a couple of interesting if you Google chain meter and Drupal you find a couple of really helpful templates also can't remember the guy's name but I was using the template myself also and it was really good it also has some recording feature so if you want to quickly record a session then just repeat it with larger scale you can do that and yeah so I guess like I need to do the demo now I was yesterday thinking that should I do a recording or not just in case or should I just do a live demo and I was like I'm tired I'm going to do a live demo so I'm kind of nervous now how it goes how much time I have I forgot to put the timer 18 minutes I used 10 minutes so I'm not going to demonstrate how to download chain meter it's basically a package you can Google chain meter and you download it and extract it somewhere and you need Java for that I downloaded this directory this one script chain meter usually when you actually then run tests you don't use the graphical user interface but you do when you kind of are working on this test you use the graphical user interface quite often and basically when you do the tests you come up with one a.m.x file that is the test plan file and you can use that for instance in another like there are some source services like Blazemaker or something that are compatible with the file and you can just upload them and you can run the same test designing this graphical user interface into their system or in their system etc so do you see here because there's an option that I can increase the font but there's no shortcut so I need to one by one click no I'm assuming wrong way make sure that you see I wish there would be a shortcut for this there's a reason to not use the graphical user interface so this is now I'm going to put the timer so I don't know so this is the view when you start from scratch you have like a test plan here again I think I never actually use multiple test plans but here basically you start by creating a thread group which means that it will be or well there's many ways to do this but at least I use one thread group for performing one scenario so this was kind of like this is where I put the number of users let's say like 800 are doing this scenario then I would create another thread group that are using something else at the same time like reading a news or something 1000 there or then you can use some variables and then you calculate out of them you can define variables from the test plan or you can use also another for this so basically this graphic user interface is all about basically making this there so you create a thread group I'm going to use an example just one user the run up here means how much time it takes when it is using or when all the users are out there and then you can specify the loop count I usually tend to use forever because I want to run the test for certain time which I specify here under the scheduler I want to perform let's say 5 minutes 5 minutes 5 minutes for one user for instance and I want that the user when it finish the scenario it starts over because when I do when I run this test I want to know how many concurrent users the service can handle and it means that basically when the user finishes it just continues so I want to get the total number that it can handle and therefore I also specify here that when there are some errors for the user or something then it will stop the thread so it means that you let's say you put 1000 users in 100 seconds you'll see that the users are using your service and then you get more users and more users that are accumulating and then basically once you start to get some errors there's probably some performance issues it might be also some software logical issues I don't know but then you start to see that the user is actually dropping instead of like increasing so that's the way I configure this but depending on the actually you can use in different kind of performance test if you're measuring some other things than realistic scenarios these are really helpful then like useful for specifying some other cases but this is the thing I use and recommend to get the answer that how many of the great user handles so now once I have now a thread group of one user that I'm going to use which is run for 10 seconds and we create now under this element the sampler as you can see there are other interesting things here also you can measure let's stick with HTTP things so now you get this form and here I want to specify that it will go to this address and we're here in the front page and then there are some options here that are really small I'm going to read it for you the first one is redirect automatically follow redirects use keepalive usemultipart.form.app for post and then browser compatible headers you tend to leave this as a default the most impact actually has to follow redirects because if you land in the page and it has a redirect of course no user would follow that but on top of that when you do error checking the response gives you 200 ok response status and if you don't have this unchecked and you have that assertion there because you can actually get a redirect response and then for a request you can specify of course there are different methods this is a really helpful also to do a really easy tool to do some REST API performance test for instance so you have different methods I don't know can you even make your own phones here maybe it didn't try out maybe it works and then the primary person you can also specify files to upload so ok now when I hit play here it starts to now it's just for comments to save the test plan which I decided not to save so I have a couple of programs ok I made that I think oh that's taking no I went this through before so let's see what happens so now you see here in the top right corner that it's it hits and then it stops so we have one problem is for instance that at the moment if you go actually to this one this address because it's on private beta I don't know why I cannot basically enter it from this Wi-Fi it's unauthorized so I'm going to give an extra configuration element to this so I'm going to add a configuration element I don't know that's the matter actually the order of here and then I'm going to configure here I need to check from the other other instance now that was because I don't remember this then you're going to see the password and you're going to get access to it it's going to be public soon and it's shared over nobody saw it that was fast enough maybe it works now let's see what happens now I see it's one size one so what it's doing right now it's actually now hitting the front page all the time it's just one user so let's stop there yeah sure yeah so I can demo this also so there will be then for 10 seconds I want that in 5 seconds there's 50 users in 5 seconds out there and the duration of the test is 10 seconds so now when I hit play you see that number increases and in 5 seconds it's 50 out of 15 and then when the duration ends 10 seconds then it will drop off another thing is that it's not really realistic that you go into a page and then there's the loop forever going on so once the page load is done it will make another one so now we can see it so next thing what's happening here we're going to add here a listener this one and remember we're doing the production performance tests like with big numbers it starts to use your CPU memory and etc you must remember to disable this because these use a lot of CPU and memory when you run the tests on your machines and then defy to some server or in a chamber infrastructure you probably set up you need to remember to disable those because you don't want to run the tests with this enabled because it will eat the machines resources for that you should use instead another listener that's the simple data writer you basically just cache the results and write and not focus on the graphical interface because once the test is run or executed you can load the file and then you can view afterwards the results but these other elements like these are working on the run and it's resources of the machine and in this case we'll use it it's just 50 users so you can see that it's basically a lot of requests for 10 seconds but now there's another thing that we don't know are they actually all succeeded requests so to make sure of that and assertion under this hdp request I meant and we want to make sure that well maybe I can make it close so all the request I would make I would expect that the request response code equals to 200 so now when I run this it's now also making sure that it will respond with the 200 ok response so we can actually now test it if I would be expecting something else and see how it works now this is I'm used to actually put the response assertion under the hdp request I'm kind of testing this myself now they are failing and you can see the number tries to push up you can see that it's trying to kind of now it's actually now you can see it's trying to kind of push more users but then at the same time we're getting errors and we'll stop the threads it means like that's the maximum amount of users so try to kind of make a rush there but all of them failed how much time I'm getting confused with the amount of windows I have here I think I need to move on now I would like to demo the other things yeah I'll demo one thing one important thing actually the realistic one here is that when you make an hdp request for and you're getting ready to submit a form you need to take the token from the previous request and use that in the second request in the post request to make sure that cross-site request forgery protection there goes correctly and for that you basically you basically would create and under the request you want to pass it from you make a post processor and you can use either a jQuery extractor or some other things and in here you would use actually I usually use the request and then I would create a variable that I would use like the token and then I would create there are many requests you can use I just copy pasted this from somewhere I can't remember where I never usually do requests from top of my head and if somebody says he's doing it he's lying I always use some tools to check it out because it's I cannot do it so basically I make a request from the request to basically read out the form build id or actually yeah this is the token it's the build id because I need that in the other request and then I get a variable that I can use in another request that I would then use in post requests in the body data for instance or in the parameters so make a post request and then you use the variable and then it starts to work etc I wish I could post maybe an example test plan jmx file for this so I will tweet about this so you can go and test the actual scenario which I introduced there because it's quite useful it will have the csv file you will be using the profiles that you want to use identities there there are tools that you can download csv file out of it and then you can use those in the forms that I was running the real one I was using I asked our analytic team can they do an export of all the search keywords they use so I can use that as a source to make realistic searches on the performance test and they gave me an export and surprisingly and happily there was also some keywords that didn't return any results which is realistic but yeah this is a quick demonstration how to start with it there are many of kind of elements you can use and a lot of resources out on the internet how to kind of make your awesome scenario there and I'll try to share the the one I actually used I can quickly have a show it to you the tree of the actual test that I used so this is the real one as you can see I've disabled a couple of the listeners because well except this one I was using for writing the data but here I'm also enabled the cookie manager then I'm extracting the search keywords then I'm going to the home page and I'm using here some thinking time so the realistic user actually reads the page instead of going right away the second millisecond to the next one so it's some randomness so from two to ten seconds then it goes to the high-chain rating landing page and then from there it will extract the build idea that I will need later and then it again thinks a little bit what to search and then it performs a search with a post request and then again thinking and then there's a if controller so if I don't get search results at all then I probably to stop the test there but if I get some results then I will basically go to the search result page and you can see that I'm using here a variable that is extracted from the previous one yeah this one I'm using yes is take query extractor so basically a run a random match from the search results so I used the selector here and then define it as a search result ad variable and then if I have results then I do that one so yeah that was a quick demo there and quickly then the results that I I've been that's the most exciting stuff where you'll be planning your scenarios and then doing the stuff and then running really exciting and the developers are waiting what's going on with the results and then in place meter you have some graphics out of that but then I like to actually use New Relic and see the actual impact there so from here I was using 2000 user scenario from my laptop and that's like not a lot but this was for testing a minimal infrastructure that we have which is basically one low balancer and two web servers and one database server and these are basically the screensaw from the New Relic and it looks like yes most of the servers are doing something how much time? over sorry yeah but I'm just you know quickly and this like clicking slides on thank you so if you want to talk about testing I'll be in the until the end of the heaven today so roughly some of the performance testing next time but generally it takes me one day to kind of play against like what they actually want and then it also includes that I go to the web server website and see how it works to make sure that it uses a lot of web sockets and stuff and it's like so one day kind of like you know investigating and then doing more simple scenarios like going to a home base which maybe click another thing I don't know what events kind of so that you have to be behind of the end to have a new server and there are other characters and etc so but it's really simple like the example I did is like just like that but the scenario that I