 All right, welcome, everybody. Hopefully, bellies aren't too full. After having some lunch, this is kind of a warm room. So hopefully it doesn't just put you to sleep with a full belly in a warm room. I would say that if you guys start feeling kind of warm in the back, just open some windows. Don't be shy, because we were kind of in a sauna in here earlier. My name is Dustin Black. This is my colleague Marco Karn. It's hard to tell us apart. I know. We are members of kind of a special team within Red Hat's storage and now storage and hyperconverged business unit. We're actually a technical architecture team. And we do a lot of performance analysis of full workload stacks more and more. So the big disclaimer here is that we're going to tell you that this session is designed to tell you about our experiences. So I'm not here to dictate to you the right or wrong ways of doing something. We're trying to give you what we've learned from experiences. So the tools that we're talking about are things that we have used that have been effective for us. And we're going to tell you the stories about how well those things have worked for us, what we did and didn't do right, and kind of the fundamentals you should think of along the way. There are somewhat specific examples in here of tools and the way we use them and the way we test. But hopefully you get out of this as kind of like something a bit more broadly applicable. It's more about the methodologies and how you think through these things. So these are very simple slides, too. We had fun, just we're just going to throw them up here and no templates, so why? So why performance tests? Why does this matter in the first place? Think about what you always start with when you're building a system. And in our case, in particular, a distributed system. That's something to note is that our performance testing was in how an application runs on its server. It's how does a pretty distributed application with lots of different clients run on a distributed storage system. So there are a lot of pieces going on here. So when you start trying to architect an environment, you have hardware specifications, you know what CPU and memory and network performance specs are and to some extent what to expect. But what you never really have is a full idea of how these things are going to operate holistically. You can come in with some educated guesses, but without actually doing the legwork to say what does an application or a particular workload do on this hardware. And again, particularly a distributed hardware set. You're not really going to know what to expect and knowing what to expect can be extremely important in building out an architecture. You may find that I can kind of balance things to one piece of the stack, the CPU or whatever, but if I don't fully understand how it uses everything together, I'm not going to be as effective. And you also have to consider what different workloads you may have. It's, you know, you can't just go in saying that I'm going to build a system for one particular workload most of the time. There's usually a variety of activities that are going to be happening across the system. So you can't just come in with specs that are written on paper. It doesn't solve all of the problem. When we do performance testing, most people come to the question of performance with the idea of what's the maximum that I can get out of something. So this is just a simple example and we've done numerous things kind of like this. But what I wanted to show you here is that across a couple of different hardware configurations, you may do some analysis and say, oh, well, this configuration looks like it may be my best option. And in this case, this happens to be a difference between whether a system has 12 drives, 12 physical hard drives in it versus 24 physical hard drives in it. So you can see, okay, 24 physical hard drives, I can actually push more IO through the system and therefore I get greater performance. And if I just kind of glance at this and say, so that's the better way to go. I get better performance for this workload out of a 24 drive system. Let's get a 24 drive system. The trouble with this is that it doesn't always take into account the value of the dollar that you spend on this. So we often actually take a look at our data in a different way. This is the same data plotted now actually considering instead, what are we getting performance per dollar spent? So how efficient is our dollar going to give us this performance goal that we're aiming for? So in this case, you actually see it kind of inverts things. It inverts the story. If I go back, the differentiators here are 12 disks versus 24 disks, but it's also how we're actually building, in this case, the Gluster volume that was behind it, whether it's a distributed replicated volume or a distributed dispersed volume, or ratio coding on the right side. And not only does the 24 drive system appear to be better, but it also appears that using a distributed replicated on RAID 6 is better. When we go here and actually look at it from a price performance standpoint, you see that we now move to both a 12 drive instead of a 24 drive system and a distributed dispersed volume instead of a replicated volume. So understand what it is you're targeting, what you need out of this. Maybe it is pure performance. Maybe you're trying to eke out the last IOP you can get out of whatever system. Fine, but understand what you're going for because it's going to change your testing methodologies in your data collection. This is kind of drilling down into that example a bit. So once we say, okay, our 12 drive system is our best building block. But now there are a lot of different ways I can leverage that 12 drive system, different protocols I can access through, different ways that I can arrange the hardware and RAID configurations, different ways that I can arrange a software stack on top of it. So the example here is that these are all the exact same hardware. Nothing has changed about the hardware. The only thing that's changed is how we're leveraging that hardware, how we're actually interacting with it. And you can see very dramatic differences based on your choices. This is one particular workload that I pulled out as an example and what we're looking at is a four gigabyte file throughput. So this is kind of a, for Gluster in this case is kind of a sweet spot for performance. Large file throughput it does quite well with, but under what circumstances. So if you take a look here, our best overall performing for both RAIDs both RAIDs and Creates is again that 12 drive J-Bot distributed dispersed system. So no RAID, no replication, we're using erasure coding and we actually get our overall, and this is price performance again. So our overall best value of performance out of that configuration. And if you look down here just to compare it to the worst configuration of that, in this case I have to use the NFS client and I'm doing a dispersed volume on top of RAID six. And you see that configuration, I get horrible performance by comparison. These are important things to understand in your testing so that you know which direction you may need to send the architecture or it may be an inverted thing. What if the solution comes to you and says no, no, we can't use Gluster's native client, we have to use NFS. Well now my choice becomes different. If I say okay, that's fine, but we're not gonna get as great a performance out of NFS. That's okay, we understand that, just get the best performance you can out of NFS where required to use NFS. Well now my selection becomes different. If I were doing J-Bot distributed dispersed with the Gluster fuse method of accessing the files, you can see that this design is clearly the best choice. But if I look at the same design, which is J-Bot distributed dispersed and I do the same thing with NFS, this is not my best choice. My best choice within NFS is actually to do a distributed replicated volume. So the protocol, if I have forced, if I'm forced to use one protocol or another, it may change my architecture choices as well. So we always like to say start with the workload. Don't go in architecting a system until you fully understand the workload or you're going to make mistakes. Another thing drilling down even a little bit lower into the layers is that there are many things you will find along the way that are the small knobs that you can turn and often the small knobs have small effect, the low level things. It's really the architectural decisions that have the biggest effect on the performance of a distributed environment. But there are certain things at a low level that you may find that you can tweak that may have a huge difference. Some of the big ones we've seen are RAID controllers. You know, in this case, make sure you design your RAID controllers with a battery-backed cache and put it into write-back mode. We're going to get all of the benefits of that caching applied to all of our IO and we see it makes a huge difference. But it's really easy to order a system with a RAID card that doesn't have a battery-backed cache and then you can't leverage that. And it could be a big detriment to you. So understand these things. On the network, jumbo frames make a huge difference. Tuning your VM-30 ratios and background ratios whether you're using Direct or Buffered IO, some of these things, again, might be dictated to you by the application and how you're using it, but some of these may be choices that you can make. Just understand those things as you're going in. So how do we perform this testing? Yeah, the question is, how, what do we use? First of all, it's important to understand that you're not using or not testing a single system. You're testing distributed systems, but you still need to make sure that every component of these systems work as expected. So make sure you check your network, make sure you get the actual expected throughput, make sure your devices like HDDs, STDs live up to the expectations you have to them. Make sure the CPU delivers as expected, although we found out that is pretty tricky. So if you have an idea how to do that, probably let me know. Yeah, do some smoke tests, make sure that works. Other fundamentals we learned over time, make sure you run enough iterations of your tests. A single test is not exactly giving you the correct results and make sure you run them long enough. We've seen interesting caching effects happening once we went from five minute tests to 50 minute tests and to 30 minute tests. Calculate your standard deviation. If one test gives you totally different results, make sure you have covered your bases in that regard by doing the deviation checks. And we learned that the halfway to check one variable at a time, do not change anything beyond that. Because I remember pretty well when we did that, results changed dramatically and we didn't understand why. Collect submetrics of all the affected systems. Not only the systems you're testing, but also the test generators, the load generators. We ran tests where our servers didn't go beyond the load of 30% and we didn't understand why. So in the end, it turns out our load drivers were not beefy enough to actually stress our machine. So yeah, make sure you monitor that. Find out which is the system resource that gets saturated first. This will be your showstopper basically. That's what it is. In our case, it's typically storage, but that of course heavily depends on the workload you're running. Unless we want it to be storage. Well, that's... It exists. We're building storage system. We want to eke out as many IOs as possible. If our bottleneck turns out to be on the network, that's a problem for us. That shouldn't be happening. Right, but also make sure not to overdo. I'll explain why. Monitor as much as you need, but keep it to the bare minimum so that you don't have to take into account monitoring effects because monitoring doesn't come without a cost. So if you monitor every little bit and piece, you will run into problems because the monitoring agents or whatever you run will eat CPU. You will end up with horrible amount of monitoring data and you need to correlate those data in the end. So for example, is it really necessary to do a monitoring within short-lived VMs? You're testing, you're firing up. Make sure you capture whatever you need, but not what you don't need. And also spend time on bringing up those metrics in a meaningful way so that you see resources you want to see at one glance and not spread all over the place. So what are the fundamental metrics to monitor? Of course, it's CPU usage, memory saturation, or paging effects, storage device loads. You want to know when your storage device goes bonkers or hits 100%. You want to see, is it network throughput that's limiting my tests? There is one point to all of these. They are local tests only. They have not been correlated overall, the systems that are affecting these tests. And these local metrics are of course pretty difficult to correlate if you have them on each system separately. So spend time on that. Think about how you can do that. And one very important point we learned is make sure you know what you're looking at. Is looking at averages good enough for you? Do you have percentiles? You have to be answered within a certain amount of time so the maximum latency comes into play. Do you need per driver or per load driver metrics? Do you need them in a correlated way? Most of the metrics you can collect can be reported in either way. So make sure you know what you're aiming for. And as I said, monitoring averages can be very deceiving because, and we'll see that later, if you only look at the average, your tests may be pretty fine because you're doing a lot of simultaneous IO ops, for example, but is that good enough for all of them? And yeah, the max within a percentile may be a better way to look at it. So typical tools we use for local metrics and you probably know all of them. That's SAR, that's VM, that's iOS, that's top. These are all fine and good tools, but they have one common problem. They give you the view on the system for a point in time, not how the load develops or monitoring or trending. So you have to find a way to do that. Typically when you only look at one system, you do it manually, but what if you do it for more than one system, Dustin? So systems generate all of this data and when you're using tools like we're on the last screen or a lot of similar tools, you can dig into it on a per host basis, but again, we're testing distributed systems which could have a lot of nodes, a lot of VMs, a lot of clients involved in this and correlation of that data is really important. So definitely look into tools that allow you to aggregate data into a central place. These are a couple of the common things you'll find out there that we have used in the past. We used to largely use CollectD. If you're not familiar with CollectD, it's basically an actual service that's running on each node that's being monitored and it has kind of its own format for generating all of this data. It can store it in a local file, but what's beneficial about it is the demon's ability to actually push that data up to a remote data store. So all of the clients running CollectD, you can tune what they're collecting and what they're pushing up and so there's this constant push from the nodes to a central collection point. Now if I have 20 nodes in my test environment, I have all of that data in one place that I can leverage. Prometheus is newer, Prometheus you'll find, Red Hat is integrating into a lot of products nowadays. It works in a similar way to CollectD except that instead of running a demon to actually go and collect data into its own format that it's pushing up, Prometheus is actually leveraging kernel level information that's already being reported by the nodes. So it doesn't actually have to do that kind of active data collection. It's more of a passive data collection. It's not even really data collection, it's data presentation. So effectively Prometheus takes that local information that the nodes already reporting just via normal system tools and it hosts that information up through a standard presentation method via a port on the node. Now the difference here is that you end up in a pull versus push. So CollectD is using a push mechanism, running a demon, collecting this data, pushing it out to a node that's then aggregating all of that for you. Prometheus on the other hand is just hosting it. It's just advertising it. Hey, I have this information about this node and therefore we have to have something that comes in and actually scrapes that off of the node. So it's definitely a flip in how you think about doing things. I think in the end Prometheus is kind of the way to go. I'll talk a little bit on this slide about some of the limitations you have to consider when you're going with one or the other. When we were using CollectD, largely our receiving in was GraphEye. That's kind of a lie. GraphEye itself is really a kind of the graphing tool. It's the thing that's creating the time series visualizations for us. But it has to work in conjunction with the receiver. So there's sort of this modular plug-in system that acts as a receiver for CollectD. So all that data we're pushing with CollectD goes to the receiver. You also have to separately manage a database system that's going to be holding all of that data. You have to be a little bit cautious that whatever database and collection system you're using doesn't roll up your data too often and lose resolution. This has been a problem for us in the past is we built a system, we were monitoring things, we kind of stepped away from that environment for a little while and we came back to it. And all the time our database had been running and it basically got to a point where a time limit expired and it starts compacting your data. What that means in visualizing your data is instead of one second resolution and being able to really understand what was happening during a workload, I've now rolled up to 10 second resolution. If I wait too long, it rolls up to 30 second resolution or one minute or 10 minute resolution. And so all of your historical data is a lot less valuable. That kind of roll up is probably fine if you're using it to actively monitor systems that are operating. If you're using it to monitor tests that you're running, you probably won't maintain that resolution. So you need an efficient database system that can actually hold the data without rolling it up like that. So keep that in mind if you're going to build one of these. Graphite does a great job of graphing time series data, a lot of flexibility of how you can build these graphs and charts. It's not particularly user friendly for creating dashboards though. I wanna look a lot of this data together and do a lot of comparisons. You can build the charts and graphs but it gets a little bit cumbersome to do that kind of effective analysis, especially if you wanna share it out and collaborate with other people. Prometheus, you can think of it as the Prometheus server side of things is actually doing the work of going and scraping the Prometheus client, right? And again, the Prometheus client's not really a client. They call it a node exporter, passively hosting that information, right? So the server now has to go out and scrape that information from each of the nodes and collect it up. So the active part of that work is now happening on the server side. And we get the same type of time series graph generation. It's almost a bit more cumbersome to use than graphite when it comes to creating those individual charts. But it's largely because it's a lot more flexible. It's a lot more powerful in the query language that it uses to build those charts. And again, directly using the charts that it generates is not very user friendly. So what you really need in the end to visualize everything is a tool like Grafana or Kibana. There were some other ones out there. We've largely used Grafana. And it can basically take either graphite or Prometheus data and then turn that into this really customizable dashboard posted on a web interface. And you can get down to fine tuning all of these charts and plotting them against each other. This is just a small snapshot of what we've done with some of our graphite dashboards or Grafana dashboards. This is like, you can actually see the highlights. This is a series of pre-runs. We talked about running three or more. So imagine we ran three iterations of it and we wanted to understand what was happening across them. And you can see it's a very high resolution just by how small the lines are in that time slice. This is only like a 10 minute time slice. So we're getting very high resolution plotting these things and I can easily look down and make these comparisons. I can drill down into things. I can decide whether I want to plot these things in averages or centiles or whatever flexibility I need. And this is a small snapshot. We have many metrics on one dashboard that we make a pair against each other. I think this is all disk related stuff but we have CPU and memory and network related things that we're recording all at the same time. So very, very useful. As we said, you need to understand the subsystems so that if you just run your workload against it, you don't fully understand where those bottlenecks may be and how to tweak the architecture or configuration unless you have this data. So now we'll get into the tools we actually use for generating workload. And we're gonna start at the low level and kind of work our way up through getting more and more sophisticated in your testing. Hopefully some of you guys are familiar with FIO, pretty standard anymore. I actually have a hand in making sure that the FIO package is in RHEL now. So if you're a RHEL user of RHEL 74, I think, and beyond, you can just yell install FIO and it's there. Very flexible, very versatile. You can run a lot of different workloads. Obviously we use it a lot for our storage related tests if there's a ton that you can do with this. And as with any Unix-type tool that's very flexible, it's also very complicated. There's a long man page with a lot of configuration options and you kind of need to know what it is you're doing. It's one of those things where maybe you go out and find somewhere on the internet somebody else's test the way they run it and hopefully use their config as a starting point because you may not wanna write your own from scratch. But we have throughput latency information that a lot of hits the brand type stuff that it reports is very good. Small file is a really great tool. It's pretty simple to use, it's written in Python, actually by one of our performance engineers at Red Hat in England. This one allows us to do aggregated tests across clients and it is focused on generating small file type workloads against a storage system, which is really important for us when we're testing a NAS-type system. We wanna know how things that are highly transaction-based and there's a lot of metadata type activity we wanna know how that performs and small file reports that in a really nicely consumable way. So we've used this one a lot. And prior to getting into using FIO where we used to use IO zone a lot, this is again, this is like a classic, very, very flexible performance testing tool. Still highly recommended in some ways I like IO zone more than FIO, but as a team we've kind of standardized on FIO. But I've got a lot of historical data here. It's still a very great tool and it does some of these, you can generate some of these types of graphs itself. So if you just need to do kind of visualizing your own information, it's great. So, again, this is just repeating and excuse me, digging in a little more to what we had mentioned about distributed clients. If I have a big distributed system that I'm testing, I need to make sure that I'm generating enough workload to do that. So what we're talking about here is kind of related to how do I do that in relation to these tools I just described. Thankfully they all do it pretty well these days. FIO didn't used to do distributed testing very well, but now it's got a pretty good system for it. But you need to know how each of these tools does that. It's not the same and sometimes the terminology changes between the tools. For instance, what one tool calls stonewalling and other tool also uses the term stonewalling, but it means something different. So you gotta watch out for that a little bit, understand what it's doing. The example here is if I want to know, and this is commonly the question we're asking, if I want to know what the performance of a system is under a maximum load over a period of time, I need to generate that maximum load. If I need to do that with 12 or 24 clients or oftentimes we ramp up the number of workers depending on the workload, I may have 100 workers or more actually pushing data to the system. Well, if I just hit go on that and start recording, what happens? Well, the clients don't all instantaneously start generating IO at the same time. All of these processes have to compete for their client resources to actually get their IO processes up and running. I don't want to start measuring the load that it's generating at the time that I hit go. I want to make sure they're all actually running before I start measuring. Same thing on the tail end. As soon as the first client stops doing workload, I want to stop measuring. So I only want to measure load during the timeframe that all of my client's operations that I've started are doing the work at the same time so that I actually have a good view of what's happening during that timeframe. That's true for our case most of the time. You may actually have different needs but know that these settings are there that they have to be used appropriately or you could end up with measurements that aren't quite accurate. This is kind of just a visualization of how you'll handle that. You usually have some controller nodes. Sometimes a controller node is one of the clients and in a lot of cases there's no reason I can kick off the job from one of my client's systems but I may use a remote node to actually manage all of these things but I need to actually aggregate, I need to push out the workload and divide it up among all of my clients and all of my clients then need to generate the workload up to the system that I'm actually testing. And so that's all very important for getting that just right and honestly you're likely to get it wrong the first time so play around with this a little bit. It gets to be a little bit tricky. And finally as we mentioned that you have all of these tools that are all able to do these distributed tests that they all have different types of user interfaces, little bit different syntax and how you operate with them. We are performing hundreds if not thousands of these test cycles among a project that we're doing and we may be flipping back and forth between using small file or FIO or IO zone or different tools. Make your life easier. Wrap some kind of scripting around this that's going to simplify things and is going to do the tests in the standardized ways that you expect to repeat. If you're interested in getting started on that there are things like P-Binch which are actually wrapper tools that begin to do some of this for you. And that can be really effective. We kind of started rolling our own before P-Binch was really maturing. So we have a bunch of tools that are effectively just batch scripts that are wrapped around these tools that I was mentioning. But our batch script user interface is consistent. So how I run an FIO job looks the same way that I run a small file job and it looks the same way that I run an IO zone job. So it makes it a lot more user friendly and portable. These are rough around the edges. They were designed for our internal use. So you're not going to find that they're perfect but take a look at them. They could be a good starting point for you. And we have, those are kind of like workload generators. They're pretty small. They generate this kind of IO, right? Do four gigabyte throughput for X amount of time or whatever and see how that is. But you can see that's very synthetic. So what we want to do is keep moving up the stack of maturity. And the next thing to do is to do something like SysBinch. This is a tool we've used a bit. You'll commonly find that this is used out in the wild for database testing but it's actually a pretty flexible tool and can test a lot of different things. And it generates workloads that are a little bit more real world. And you have some flexibility in the kinds of things that it tests. You can combine these things together. So you get a lot of good information out of SysBinch. SpecSFS really takes us up. And I'm talking about SpecSFS but the spec committee has a lot of great tests. Downside to these is you have to actually purchase them or be a member of the spec committee to have access to these. But they are really, really well-designed workload tests that really simulate very realistic scenarios. In order to use this data, especially if you're going to publish any results of this, you have to have actually approved results summaries by the spec committee under most conditions. So a lot of restrictions, if you're using it internally, it's a lot more flexible. But if you're ever gonna publish data, this can be a little bit cumbersome even though the way that it generates workloads is actually very, very nice. We've largely used the streaming video capture which they call video data acquisition which does simulations of like high definition streaming video of CCTV scenarios and also runs like an interference read workload. And you look for how many camera streams can your storage solution support before you hit a latency threshold. Really nice test, but again, a little bit cumbersome to use in the end. I'll pass on to Marco for taking this next step up. Yeah, exactly. Now that we laid out the foundation for all of our tests, we want to take it to the next step, meaning we're not taking a single resource. We want to test, we want to test a combination of all of them. Typical that's network CPU storage, memory, think of any combination of them. These tests do require more than a simple script to run. Most of the time they consist of a web server application server database server. And of course the client systems are also there and you need to take care of them. So we went for a very popular testing stack called dvdstore2, which as the name already suggests, simulates a dvdstore. The dvdstore2 version only consists of a database of any flavor you can use. MySQL, Postgres, dvd2, I think anything, yeah, mostly anything works. And the clients are Windows systems, but we managed to get them up and running in containers on the client side, so that works as well. Customers typically log into that dvdstore browse, check out some dvds, pay for them. And the configurables of these tests are typically the number of customers. This is a number of threats within the client system. The database size, the larger the database, the more dvds are available in the store and the more transactions you get. But as you can clearly see, this is mostly network and storage intensive, so it doesn't burn too much CPU cycles. So a graph we get out of this looks like this. So what we've done here is we've tested a product that's called RIV, basically that's hyperconverged storage and virtualization solution, meaning we run ref on Gluster on just three nodes. And we wanted to know how many VMs of a certain type can we spin up on that system. So we started working out should the database sort of look like, how large should the database be? How many clients can we run versus that database? And we did some baselining running just with one client and that was the number of OPMs we can get from a single client. And then we scale up to, as you can see, a hundred and more stacks. A stack is a client and the database. And what we found out rather rapidly was that we are running into one system constraint hitting in probably about here, somewhere around here. From there on, we saw that the number of total OPMs, meaning the aggregated number of OPMs, basically leveled out at around 7,000, no, 750. 7,500, 750 is the OPM per thread. But what this graph also shows is that right from the beginning, the number of OPM per individual thread or per individual stack, dramatically goes down all the time. So if you want to look at it from that single stack perspective, run one because it doesn't get any better. If you're looking at the overall performance, yeah, you should probably stop at 45, 46, maybe somewhere around there. But you can always push it out further, doesn't get worse. So it's pretty clear that at this point, one of the resources was saturated. Turns out it was storage. We didn't see any pressure on CPU or memory or network. Yeah, it turns out this curve too, matches very closely a lot of storage tests we've done in the past. This is a full stack test, the full application, but we clearly hit a bottleneck and plateaued out. We don't see another cliff at any given point, but this curve very well matches what we usually see when we're trying to push more and more IO workload with a larger number of clients through a system. We eventually hit a maximum storage that we can do, and if we ask it to do more, it will try. I can keep scaling out, but I just plateau out the total that I can get out of the system. So yeah, adding more VMs doesn't get you anything more, but it also doesn't break the system. Right, so we were looking for something that uses the resources in a more balanced way and makes use of more resources than just storage in network IO. So we came up with DVD Store 3, which is similar to DVD Store 2, but implements a web server in between the client and the database. So more like a real-world scenario web shop, typical web shop. It adds some features like reviews and premium membership, but that doesn't mean it's that much different. And as I said, it stresses CPU memory and IO more equally than DVD Store 2 does. So with that test, we wanted to see how the latencies work. So again, this shows the average OPM versus the latency per OPM. And if you look at this, blue bar, these are the OPMs, they grow to a certain point. This is not on the slides anymore. And the red bar is the average latency and this is what I mentioned before. If you want to stay within the average latency, it doesn't grow that much. I mean, it's going from around or it's not on there. Doesn't matter. Right here, it's not that much different to with a single stack. But if you need to stay within a certain maximum latency, then you see that from 12 stacks onwards, the maximum latency goes through the roof. So if you need to have, let's say 99% of your requests answered within the average latency, that's fine. If you need to have them within the max latency, well, it doesn't even align, you should probably stick to five or less because latency from there on, the maximum latency you can get, doesn't mean every request gets that much of latency, but there are requests. So think of a trading system. You want to be your requests answered within a certain time. You don't care about average latency, you care about the max latency. And this is what is important to look at, depending on your workload again. If we made the mistake of plotting only the blue and red lines, then we would believe that we could scale out to a larger maximum number of operations because our average latency wasn't changing much. So what we learned by seeing the max latency is that clearly we shouldn't have, we really hit a latency problem before we hit a throughput problem. And we need to account for that. Yeah. And last but not least, we learned that you should automate whatever you do. You want to rerun those tests in half a year's time, in a year's time. Make sure you use whatever you like, be it Ansible, be it Puppet, be it BashScript, make it repeatable. We had to create hundreds, maybe thousands of VMs over time. You don't want to do that manually. You want to script that. You want to make sure all of your containers get the same configuration every time. You want to make sure that the caches get cleaned up on the hypervisors, maybe in the VMs. So make sure you have a consistent and reproducible way to do that as well. And I guess that brings us to the end. Questions? We have a 10 minute warning, so we have 10 minutes for the questions. Come on. Sure. I can have one. So when you do your benchmarking, and you have the results, and you have the numbers, and you see that the numbers are maybe not matching what you expect, but you're also seeking for the kernel bugs, like you know that it should not behave that, so you try different channels and see that if the kernel A was better than the kernel B, do you do something with that? That's something we do not do, but we are in very close contact with engineering, and we have peer reviews with them. We have peer reviews basically every other day with them while we did that. During that project, we also did throughput tests, and those throughput tests were like, oh man, this is so far off from anything we would like to see, and we threw it back at engineering, and it turns out that we were doing something wrong, and there were bugs, and yeah, in the end, it aligned pretty well, but we do that, yes. I'm gonna go back to another slide, too, to answer that. This one. We kind of just touched this real quickly, but this is just an example. We're running through a lot of different test scenarios, and we were sort of under pressure to complete our tests under a certain amount of time. We really, really want to be as scientific as possible, so each yellow box indicates a change that was made to the environment, right? So this was our starting point. It doesn't really matter what they are, but this was a set environment, and we tested that, and we got certain results. Then we said, okay, let's see what happens if we change this variable, and we got results. So let's see what happens if we change that variable, we get results. Well, here we kind of got under pressure to accelerate things, and you look at what we did, we changed like five things in one line. Guess what happened? We got pretty different results. What caused those results? And we had to go back and kind of unwind all of this to figure out what had changed, because it just raised more questions than it was worth, so sure, we kind of had a shortcut to better performance, but now we need to know why we were getting better performance so that we could use that information. So to the end of kind of like AB type testing, at a higher level in the stack, yes, we as a team are constantly doing that. If it gets to a lower level in the kernel type stuff, that's usually outside, because we're more of like a post product team. If it gets down to that level, it's more of a reference back to, you know, engineering and performance engineering. Yes, there's another question. Yeah. So do you also conduct a kind of continuous benchmarking to catch progressions in software or kind of like a set of benchmarks we run every couple of days or whenever something changes? No, that's not within the scope of these project nature things. We get a certain task, we fulfill it hopefully with the expected results, but then we are off to the next project. Yeah, we have a broad set of themes doing performance within Red Hat though, so we actually have a performance engineering team whose job it is to figure out at the low level what needs to change in the product before it's released to make sure that we're getting the best performance results. We also have QE teams that run their regression type of stuff that you're talking about. A change happens, we run that, see if we get different results. So there's a lot of that work going on, but it's a little outside our scope. Yeah, well, I have the same questions as just one edition. So if you do this set of tests, do you, for example, possibly do engineering or some kind of maintenance team who then, for example, can do regression testing with the same set? Like, let's say you have the architecture done, your project is done, but then they do some software of this, like new kernel stuff. Yeah, that happens a lot. What is your setup to them and they use it? Actually, we share a set up or a number of setups with engineering directly in Bangalore, for example. And they do further regression. They do. Yeah, depending on the scope of the test, resources may get directly passed around, including the lab as it exists physically. Now QV, go use this lab and do some work. Sometimes those things are scoped for a shorter period of time and they learn what they need from it and maybe start implementing that in their CICD workload for continuing to test these things. But yeah, there's a ton of sharing that's happening, which is great for us. I mean, again, the pressure is often on us to produce these results as quickly as possible because it's tied to some software product release or some document, some collateral we want to generate of architecture document or something like that. So when we hit some of those rabbit holes that could take you really deep into some strange areas of troubleshooting, it's really important for us to be able to sort of pass those things on because we just don't have the time to do them ourselves. So knowing that we have other resources to help us out. And I always encourage everybody to know in your own organization, whoever those other resources they can help you. Don't try to do it all your own. There was another question over there. Yeah, so you actually just do benchmarking. You don't do optimizations and you don't change the optimization algorithms or anything else. No. Just do benchmarking, just do fixed variables like these. There's no underlying science. No, there's definitely optimization we're doing, we're working at a higher layer in the stack. So the optimizations we are doing is more about tuning the kinds of things that you as a sysadmin or an operator of the environment would change. So again, everything that we do is post product. The software is already out there released. It's in front of the customer. We want to make sure that we give you the best guidance for how to run a workload under those conditions. So we're not down at the low level tweaking kernel parameters most of the time. Sometimes we do some tunables if it makes sense, but we are doing that. We're just doing it higher layers in the stack. We're not touching code if that is a question. Yeah. Well, yeah, I mean, there is lots of kinds of optimization algorithms. So benchmarking only on fixed parameters like these, they only give a set of possible results. Yep, right. And you can go way, way deeper. Yeah, but. And you can make a science out of it. That's fine. Yeah, absolutely. Any more questions? That's it. Thank you all. Thanks, everybody.