 Well, welcome everybody to this week, the 28th OpenShift Commons briefing and we're really pleased to have Michael Sage, the Chief Evangelist from BlazeMeter with us today to talk about continuous performance testing and we'll be doing a couple more of these external briefings by different service providers over the next coming months. So we're hoping to get a lot more. If you have one that you're interested in hearing from, let me know. If you have questions and answers here, if you could hold them to the end of the presentation, that would be great and enter them into the chat room and if it's burning, I'll try and answer it in the chat or Michael will pop over there, but we've got a lot to cover today so I'm hoping we can just kick it off and let Michael now introduce himself and kick it off. Great. Thank you so much, Diana. Happy to be here. Hello, OpenShift World. My name is Michael Sage, as you've heard, and I am Chief Evangelist for BlazeMeter. My background goes way back to the old Mercury days. I've been doing performance testing for a long time and I've kind of followed the trends from the old days of client server performance testing with thick, heavy sort of Windows based tools into the modern era where we're all about continuous integration and continuous deployment and all sorts of innovative new ways to deliver and test applications. So let me go ahead and get started here and we'll wait for our first slide. I'm waiting and waiting and waiting and waiting so sorry, I had a little interruption there at the office. So obviously, the point of the slide is to induce the frustration that you have when you're waiting for websites to load and these days we know that users' expectations are higher than they've ever been. In fact, we're hearing stats that are really remarkable. Things like 49% of people expect pages to load in under two seconds and 30% expect a one second response time. It's really converging on instant. People really want their devices or their websites and so forth to behave as actual objects do. We expect our apps or our pages to load right away just like if we turn on the television. And that means we have to test and performance testing until now has been generally relegated to a later stage in the waterfall type process and I'm sure many people on the call recognize this kind of graph that's basically a waterfall approach where we had each stage was done before the next stage was taken off. This slow and kind of methodical delivery chain didn't really work. Defects got through pretty easily. They were very hard to troubleshoot, especially because they were discovered too far down the pipeline to remember what that code was all about. And when we did this kind of waterfall approach, load testing was in fact the last of the last thing. It was relegated to this very final stage of the almost final step. And it was generally done as a war room exercise. And some of you might remember this. We get everybody into this big room and it takes a few weeks or months to develop the scripts that represent the load traffic and we get the monitoring people. We get the business owners. We get the developers. Everybody comes together for this big event. Sometimes this is done in advance of some occasion like Black Friday or the up, for example, now we have the Super Bowl coming or March Madness and teams will get ready for this big load testing event to see if their application can sustain the traffic they expect. This is still important and I definitely want to emphasize that as we go. This is an important approach to load testing. It still has its place. But it really doesn't help us in the modern era because basically continuous integration has proven itself and we know now that continuous delivery pipelines work. We discovered defects earlier. We can fix them more deeply and more easily and we can release features to our customers that are satisfying our requirements, whether that's performance or functionality or innovation and we want to be able to benefit the business that way. So with continuous delivery approaches, we need a new way of conducting our performance tests. We want to test source code. We want to test the new code that we commit, preferably right in our desktop without having to wait to submit a ticket for some specialist team to create scripts for us to test that code. We want to be able to do it in a familiar environment. We also want to test our CI builds. We want to be able to see that every build that we push through works and performs as expected, especially when it involves multiple code commits. And of course, we want to test our deployments. We want to test staging, rehearsal environments, QA environments, and even production environments. So we'll talk a little bit about some of the specific objectives for those different kinds of stages that we're going through here. I want to introduce Taurus. And this is the game-changing tool that allows us to do the kind of performance testing we need at each important stage of delivery. You remember the old tools, Load Runner or Silk Performer and so forth. These tools don't really work in the kind of process that I just introduced, that I just went through. They're heavy tools that have to be installed on a Windows desktop. They generally have a lot of complexity to them. And there's big, rich, gooey, and it takes a long to learn them. We want to introduce a technology that's familiar and makes it super simple for developers and other team members to get tests up and running fast. And so what we did is we created a YAML-based approach. And for those of you who work with configuration or pretty much any kind of delivery or deployment mechanisms these days, you'll find YAML's pretty familiar. With a YAML-based test in Taurus, the first thing we want to do is set the run options. And this defines what the test is going to do. And just a quick refresher, load testing is all about sending traffic to the application for some period of time to see how it responds. And so here what we're defining is the concurrency. We want 100 concurrent users doing the things we're going to tell them to do, getting the home page or getting a shopping cart, whatever it might be. And we want to run those 100 concurrent users for five minutes. I'm going to ramp up one minute too. And that's kind of a gradual buildup of that load from 0 to 100 over 60 seconds. Sometimes we do that just not to overwhelm the server. The next thing you want to do is describe the actual requests. These are the pages or the code paths, the URIs that are triggered by the test execution. And you can see here pretty standard HTTP stuff. We have a couple of URLs. If it's a GET request, you don't even need to say anything about it. You see that index.php? It's just going to know that TARS will know that that's a GET request. The second one is a POST request. And we wanted to add a header because there's going to be a JSON payload here. And we also have a body that describes two values. What this test represents is a sample app that I'll show you in a minute that allows a user to choose cities to fly to and from and then proceed through the process of buying a plane ticket. So that's the second section. We have our execution and we have our request. And then when we're working with CI and some other environments like that, we typically want to have a criteria that fails that test if necessary. And so here, for example, you can see I'm saying that if the average response time is 150 milliseconds or greater for 10 seconds straight, then I want to stop this test to fail that build. So these are just some general descriptions of what that needs to go in this YAML to make this test work. It's pretty straightforward. It's not a very complex test. TARS is great because you can have very simple tests. You can have very complex tests. And you can actually piece them out and merge them together at runtime. I'll show you how to do that. So those are the main modules that I just wanted to introduce to give you a feel for how TARS works. When you're running these tests, you run them from the command line. The command line is BZT. The test names are YAML. They could also be JSON. And you can run two or three or four together. And TARS will know how to manage those different sections in each of those YAMLs and run them as an intelligent test. When you do execute the test, you get a nice console-based dashboard that shows you all the metrics that matter and shows you the response times, even the percentiles of response times, as well as any errors that are happening and odd response codes and things that you want to keep track of when you run this test from your desktop. Very specifically, you can see the number of hits that are getting to each endpoint, the number of failures for that endpoint, and the average response time. So already as a developer, I have a generally pretty good sense of how my new code is performing as I've just built it. Then, of course, we want to run in Jenkins, or our builds are right now in OpenShift. Jenkins is built right in. And so you probably want to run your builds in Jenkins and have these tests run as part of that build or as a post-build job. And there, we can do it just as a shelf. And here, what you're looking at is just an example of a few tests that I'll run together. And note the modules console disable equals true in the upper right. That's a command telling TARS not to launch that dashboard, that ASCII dashboard, because it will fill up the console output with lots of text that you don't want and it kind of makes it a mess. So that's a good thing to remember as well. And then finally, of course, that threshold command. We want to set those response time thresholds or those failure thresholds. And that can be response time. That can be standard deviation. It can be error rates, lots of different possibilities there for whatever actually does fail your build or in your criteria should fail the build. And those are really easy to set up. Of course, they will show up in Jenkins and you can see build over build, what kinds of failures or what kinds of problems you're having with response time as you commit new code, as you run those tests. Ultimately, what we love to see is teams getting familiar with the behavior of their applications and then kind of keeping an eye out for what I call millisecond creep. The build over build over build, you might say to yourself, well, 500 milliseconds is great. And maybe the next build is 550 milliseconds and you think, you know what, that's not so bad either. But all of a sudden a week later, you've done 20 releases and your response time has quadrupled. So those are the kinds of things you want to start to keep an eye on that you couldn't do before with the old low testing tools. So with that, let's go into a demo and I'll show you how Taurus works here. Okay, so first things first, I set up a PHP app here on Overshift. Michael, we've just lost your screen. So if you want to share your screen again. Oh, sure, we'll do. I hit escape and that seems to have. There we go, great. All right, I'll be back. Give it a second and there we go. We're watching OpenShift online now. Great, great. So I had a really easy time setting up a PHP app on OpenShift. I actually set up a Python app as well and I was playing with Mongo and I kind of got into a little bit. I did end up deciding to use my own Jenkins server because there were some dependencies and things that I wanted to install and I needed to have root access and so forth. So what I did is I just basically set up a PHP cartridge and it's a simple app. As you saw in my Taurus example, the nature of this demo app is it will allow a user to choose some cities here. Let's walk through it. We can choose a couple of cities just from the list and find a flight. Go ahead and choose that flight. You can see the URL is changing here. Fill in the credit card form if you need to. I've made this a super simple lightweight app that doesn't require much of anything. But that's it, right? Just a few endpoints, a few pages, a few interactions. And so the idea with Taurus is to represent that in a YAML file. Represent what we wanna do there as this YAML. So now you can see my text file here and it's those same kinds of sections, YAML blocks that we saw in the demo. Let me go ahead. I'm gonna actually remove that for now. So you can see execution and scenarios. I have 200 concurrent users. This time I'm running for 10 minutes, I'm sorry, I'm running for 30 minutes and I'm ramping up for 10. And I have one scenario in here. It's the main scenario and it's got some attributes. You can have multiple scenarios in one file. Excuse me, then you can just toggle them using the descriptor here. So that's another easy way to kind of keep things organized. One thing here that wasn't in the slide is think time. For those of you who haven't done much load testing, represents those gaps that are natural kind of pauses for users. When you go to, for example, the credit card page and the app I just showed you, you may want to account for the time it takes for someone to type. So you may want a 30 second or a one minute think time at that page so that the load, when it starts to really ramp up and you get hundreds or hundreds of thousands of users connecting to the app, those pauses start to matter. They're realistic fluctuations in the load in the way that it hits your servers. So I like to add think time. In this case, it's just one second. Just as an example. If we need to add things like custom headers, very easy to do. And then of course the request and it's very similar to what we saw in the example in the slide. But here what I've done is I've filled it out a little further. I have my first URL, just go into the main page there. And then at this time I'm describing again and I'm giving it a label. I want in my reports, I want this to show up as homepage. So that I remember what it is, I can logically relate to it as the kind of part of the app or the page in the app where the load is being sent. The second one is the flight chooser where I call choose cities. And here are my arguments, London and Rome. Now you can actually draw these values from a CSV file and that's easy to do as well. So when you get into Taurus, you start working with it, you can parameterize values and you can really kind of get sophisticated in exactly the same way that you would have done in sort of the older style tools. So that said, I'm gonna go ahead and run a Taurus test here. Let's go ahead and let's move to a different shell. All right, so Taurus, I have my Taurus test here. You can see, I'm gonna go ahead and run one called complex blaze, which is the one that we just saw. Actually, I'll run the all.yml, that's the exact one we were just working with. So it's very simple, bztall.yml and Taurus will launch. And it looks like I did not save that with my change. Let's go ahead and do that and go back to try it one more time. All right, so Taurus is telling me there's a new version. It's always nice to be reminded to upgrade. We just introduced some really cool selenium support. For those of you who've done functional testing or work with selenium, Taurus can also execute those kinds of tests. Selenium, Gatling, Grinder. Really, our aim is to let Taurus run any kind of open source tool that's used for testing. Right now it's pretty focused on Jmeter. Most of what Taurus does is Jmeter specific and when you don't have any tool at all, as soon as you start a Taurus test, it will automatically download the Jmeter dependency in the background. You don't really have to think about it, but it will do that. And so now you can see I have my test running. I have my kind of eight video game graphs turning over here on the left. And here I can see all of the different values, the percentiles and response times that are coming back from the application. And these are the two pages that I'm working with on the OpenShift hosted application there. So pretty good. Under a second each, I'm pretty happy with those results. It's a long test. I put some long times in there, so I'm gonna go ahead and just stop it. Now that's one full test, but really what we wanna think about is rapid iterations, right? So you're a developer, you're working just on your code and maybe that's the only test that you really care about and other folks are managing their test cases or maybe even metafiles like execution or reporting options. And so what you can do with Taurus is split those up. Excuse me. For example, I've put a separate module here just for reporting and in fact Blazemeter hosts Taurus reporting for free. So if you want to see some pretty graphs and such, you're welcome to use our service to do that even anonymously. Actually, you don't even need to sign up for an account. If you do have an account, you can use it with your account and you can save those results and see them over time. I can also separate out my requests. So here you can see my scenario section that was in the all file here. I've just broken out some of that into one request section. Again, this is where if I'm the developer and I'm staying in my environment, I'm already working in this editor, writing code for this app. Maybe I've changed some really important underlying code for this page. All I have to do is change my test in real time in the same environment and then I can execute it. And that's really different, right? That's that context switching has become a big deal in continuous delivery. The whole premise of continuous integration and continuous delivery is that we want to find defects quickly and the best way to solve them is if developers are still in the mindset of when they wrote that code because they're still thinking about that problem and its parameters and specific challenges. So we want to keep the context the same and Taurus, of course, can do that right in the same editor. And so I've broken out this request file separate from my execution command. And it's super simple to work with. I can just type, clear this again, BZT execution and what was it again? It was requests and then also reporting. So let's do that too. If I can run those three together, Taurus knows how to merge them and now it's opening up the blaze meter because I put that command in their chat to report from blaze meter. So it looks like we have some problems starting to happen here. Oh, I forgot to change my execution. I made it localhost and that's not running. That doesn't work. So we'll change our request. I don't know what it's called. In any case, you get to see how that works. I'll show you what a blaze meter report looks like here. And so you can see Taurus is immediately sending its stats including 100% errors up to blaze meter. But here's what blaze meter looks like. In fact, I will log into my account to show you a better example of this. Here's a Taurus test that I ran earlier in the week. And what it gives us is all the stats that matter for troubleshooting the response time or the errors or the other things that are going on with the performance test we just ran. Taurus will send its data up in real time to our API and then just graph those out for you. And again, that can be done anonymously without even a blaze meter API key. And here you can see the high level stats like concurrent virtual users, how many hits per second the application was handling, any errors that are happening there, and of course response time still sub second, which makes me happy, but some transactions were slow. And you can go through the reports here and begin to investigate what's happening. You can drill into certain spikes if you want to investigate a little bit further. And you can of course turn on or off different values like the hits per second just for the cities page or the latency time is one that's not there about default, we can look at the latency time average for cities as well. Whatever metrics you want to investigate or gather for the performance troubleshooting that you need to do. Okay. So that's a basic demo of Taurus. Again, it's YAML based, it's also JSON and I'll show you here in just a moment one cool thing that you can do with the JSON approach. But I think this might be a good time to stop for a question or two if there are any. I'm looking and let me just see if I can get someone to there's any in the Q&A. I think you've done a pretty awesome job of covering it. I love the 1970s little bit graphics in the terminal mode, but that was good. Yeah, it's sort of the Donkey Kong reporting tool. Yes. Mario Brothers. Great. So I see the chat as well and I don't see any questions. So we'll proceed. One of the things I really like about Taurus is it opens up some, by the way, this is the website for it. GetTaurus.org, you'll find all sorts of cool stuff here including the manual, how to install it. It is a Python app. It's just pip install BZT. And then some great sample tests that you can start with as well as all the extensive documentation for the different things that I've covered. But one of the cool things about Taurus that I really like, let me know if my slides continue to share here. They're working fine. So one of the things that I really like about the JSON approach is that we live in an API world. All these different tools out there, cloud providers or SaaS solutions, all sorts of different things are now API driven. And we can integrate where it makes sense. And that opens up new possibilities for automation, I think. And one of the things that we can do is test in production. So you're looking at a New Relic screenshot here. We use New Relic here. I love New Relic. I used to be on the New Relic team. And it does such an amazing job of gauging performance really deep down and especially in production. So you're looking here at the New Relic browser tool, which is it shows us the different endpoints or URLs that are being hit most frequently or most slowly by users out there. And here you can, this is actual blaze meter data. So about 5 seconds, 5.3 seconds on our app homepage, which might be something we want to improve. You can see some other URLs in there as well. Well, because Taurus can work with JSON data, we could actually create a script in this little Python script I wrote to query New Relic for these URLs and then to write out a Taurus test automatically to run performance test against them. It's really just a proof of concept kind of script that I wrote. But you can see there we're hitting the URL, the New Relic URL. I have some arguments for my API key and some other things. But basically what I'm doing is I'm grabbing those values and then I'm writing out that JSON. You can see there the concurrency and the duration and the scenario and everything. And that's going to turn it into an actual Taurus test that I can then run automatically. So that whole process, query your hottest URLs, the slowest ones or the busiest ones, turn it into a test and then run it. And that takes all of the human effort out of it. Now, a developer may still want to run tests purposefully. And certainly for the staging events or these big events like Valentine's Day or March Madness or the Super Bowl that are coming up, you'll want to do a big group test. But day to day, when you're doing lots of commits in every release, it's much cooler to just kind of have Taurus or have a script that figures out what should I test based on real-world stats, whether they're coming from New Relic or Mixpanel or KISS Metrics or Google Analytics, automatically create a test and then go over on that test. So that to me is kind of the future where we're headed a little bit of, wouldn't quite call it AI, but a little bit of automation magic to make those tests a little bit less hands-on. And that's my presentation for today. Any questions or any further thoughts? There's one question in here from Judnit about if there's any OpenShift specific integration like the ability to change resource limits. That would be beyond me. I actually got into OpenShift a little bit. I did not have a time to kind of become an OpenShift master. I did learn that there's a lot you can do with it. But I think that Taurus' flexibility, given that it's text files and CLI, it should be pretty adaptable and flexible to pretty much anything you want to do with it. One of the challenges I ran into actually was I wanted to install Taurus on an OpenShift Jenkins server, but I couldn't use YOM, and then I sort of didn't know what to do next. So that was my own limitation there. Okay, well, I'm going to unmute, if Judd un-mutes himself, ask a further question on that, and there's a couple others popping up. Jonathan's asking, can you run this with a farm of workers so you aren't running this from the Jenkins server directly? For sure, yeah, absolutely. Taurus can fold itself into any kind of workflow logic that you have. In fact, I work with the CloudBees guys a lot, and they're working on very complex workflow drivers that kind of branch off based on certain logic and so forth, and because Taurus can piece together those YAML snippets, you can dynamically change things like thresholds or execution conditions or reporting conditions and just kind of assemble the right pieces for the logic of how your workflow needs to run, but of course, slaves or workers are easily supported. So, Judd was asking again, can I checkpoint times during a test where I might make infrastructure changes and note them in the reports? Yes, I mean, you can do, if I understand what you're asking, it sounds like you're asking if you can kind of, what I'd call thresholds where you are looking at the average response time over a period of time and then pause the test or stop the test. You can absolutely do that. And you can also use, if you get into Jmeter a little bit more deeply, there's an upcoming controller that will allow you to pause ramp up. It's called the Concurrency Controller. I don't think it's out yet to plug in, but we're going to be able to do that to all this, driven by the open source contributions. But yeah, I mean, you can do that and whether you do it with the thresholds to pause based on that criteria, make some changes and then see how it proceeds or to stop or whatever, absolutely. The reporting, just go ahead. So I have a quick question for you too. You were demoing an OpenShift Online, which is the V2 architecture, which is our old gears and cartridges and it's awesome and that's still in production, so it's all good stuff and usable. But as we've changed over to a Kubernetes container-based using Docker, does that change anything in terms of how we would use Taurus? Have you been doing any testing with new architectures like that? Yeah, for sure. I mean, actually I think you could leverage that pretty heavily and Taurus is a great use case, given its dependencies and given some of the things that you kind of need to do with Python and PIP and such in advance of getting it on the build server or on the deployment mechanism. Containers are a great example for that, a great use case for that. So you can just add some Taurus dependencies to your Docker file, have that spin up, Taurus, and be your execution engine and sometimes if you're running a large test, you'll need multiple execution engines because there's a kind of upper limit for how many threads or virtual users each engine can run. So you may want to dynamically spin up 10 load generators. If you're running a very large test, that would be a really cool use case for that. Well, we just announced moving OpenShift dedicated over to GCP and I'm here to tell you that being running on OpenShift on Microsoft Azure, so there'll be a lot of different places where we'll be able to test those theories out soon. Yeah, for sure. OpenShift on Azure, I love it, it's worlds colliding. Well, yeah, it's just, it's crazy making. I mean, it's really OpenShift will pretty much run anywhere including on bare metal. So we've got lots of different infrastructures to deal with. So it's pretty cool to be doing some of this testing and I really love seeing the New Relic integrations. We all do love our New Relic dashboards. For everything, they sort of up the bar for making things that were normally very boring and tedious and beautiful as you have with Blazemeter and your dashboard too. So that's pretty cool stuff. So I'm looking to see if you can go ahead. Also, I just want to point out we are on the marketplace too. Right? So if you do, if you do want to just use Blazemeter, which you can use Taurus tests and other tests. And if you already, you know, no Jmeter, you can simply upload those tests. You can actually find Blazemeter right in the, in the OpenShift marketplace there too. And that's a kind of a, you know, one click or two click way to get started. So we're interested in probably in a follow-up sometime soon is creating a Docker container that has the YAML in it. And once we have our OpenShift online migrated to V3, which is coming soon to a theater near you. That will be a fun time to do another round of this presentation. I'd love to. Yeah. And I think that would be fun for me to prep. Exactly. Like we said, kind of create a little bit more of a complex, low generation farm and have it a little, be a little bit more dynamic based on what the teams want to do. And in terms of that workflow, that would be a lot of fun. I'd love to do that. So there's one last question, which is kind of Docker related because everything in the world is Docker related these days. It is now. Is it not? Yes. Jonathan's asking for those of us who prefer Docker over installing on our local computers, do you have a Docker container available to run VZT or a Docker image? I don't think we do. I'll have to check. That might have to be a follow up. The chief committer, Andre, he's actually coming tomorrow from Russia. He lives in Russia and he's coming to visit us tomorrow. So I'll have a better answer for you then. I don't think he's published anything up to Docker hub yet, but that's a good idea. We definitely need to do that. So what we usually do with the, with the sessions, the common three things is we will publish it probably on Monday with a link to the recording. If he tells you something different, drop me a note and I will put a link to it. Or if he creates one over the weekend, because you bought him a lot of beers vodka vodka. Okay. That would be down it, but I'll add that into the blog post that we put up on Monday. So that's the great thing. So I think that's all the questions that I'm seeing. This has been a very informative session. I really love seeing YAML on my screen again. It's near and dear to my old heart. So we'll definitely have you back to do, you know, the Docker eyes version of all this is very, very helpful. So thanks very much, Mike. For sure. Thank you. All right. Take care. Okay. Bye bye.