 way through. So my name is Mark Wagner as you could probably guess. I'm in the performance team at Red Hat. I've been there for over six years now. One of the nice things about being at Red Hat is you get to work on everything. And I mean, you know, one day I'm working on RL7 release, the next day I'm working on OpenStack. I currently run most of the cloud performance work at Red Hat. So you get to see a lot of things. And it's good because you get to go through and influence the whole, you know, if we see something in the kernel, we go talk to the kernel guys. You know, so it's good in that regard. I've done a lot of the KVM work underneath. So that's more or less it on me. I guess a quick question before we start here into this is how many people here are familiar with OpenStack, okay, which is good. So we won't spend a lot of time going through that. So if we jump down into test strategy, you know, with the OpenStack, can you guys hang on for one second? Try this one. Go a little better. All right. You guys are all familiar with this, right? It's the problem with editing your slides just beforehand, leaving the other one up. We'll get into some diagrams as we go through this. But there's a lot of interaction between these. And so as we get into the test strategy, the main part is how do you test this? We care about guest performance, but it's an elastic cloud. You're supposed to just add compute as you need to go. So the goal here is to make sure that this will scale. So now when we get into test strategies, you know, we really, we focus on what we call the control plane right now, which is Keystone, MySQL, the messaging. People here familiar with Oslo? Oslo is an attempt to go and make common set of libraries across OpenStack components so that if Keystone's calling something and Nova's calling something, there's one set of code to do that. So if we can focus energy there and make that perform very well, or at least fix any bottlenecks, things like that, you get a win across multiple components. And then there's new things like salameter coming in. So we'll start focusing on that soon. That's the monitoring and metrics metering. So a lot of that will be important for things, people trying to do charge backs, SLAs, things of that nature. The next area we focus on too is the data plane, more how do the guests perform. We've poked at the Nova scheduler recently and found that interesting. We're getting into Neutron now, particularly the open vSwitch work, which is again interesting and we'll tie into that. Another part of our team has been doing a lot of work with Swift, particularly for most of you you probably know that Red Hat and Gluster are fairly integrated now. So there's been a lot of work there to get Swift integrated with Gluster. And then we'll spend a little bit of time at the end talking about the working with the community as well in the industry. So when you're trying to evaluate the control plane, it's really, you know, what we've been going through at Red Hat is, you know, what to test. And we can look at the architecture and say okay, we think, you know, database is going to be a bottleneck, whatever. And we'll get into some of the specifics. Then how do you test it? Do you test that scale? Do you test smaller? And we sit down and try to identify areas we need to look at. So first tackling what to test. You know, we go back and we look and we say, what's working in the real world? We see Swift is deployed everywhere. So we probably don't need to spend a lot of time on Swift outside of the Gluster integration. You know, we know that works. We try to focus on things that are Red Hat specific. So for messaging, we use AMQP with QPID versus RabbitMQ. My team in particular is very familiar with AMQP and QPID. We use that as one of our basic network tests. It's in heavy use in stock exchanges all over the world. So we just have to make sure that the characteristics of OpenStack will match up with those or figure out the differences and know what we need to do there. We work with the developers to get their input on what they think the bottlenecks are going to be. We've gotten some interesting feedback from them that sometimes turns out true, sometimes not. We look at the architecture and, you know, have a see for ourselves based on our experience. And so, you know, let's look at the next diagram here, which is a fairly high level view of, you know, the older components and how they interact. So, you know, ask yourself how would you try to test this as a, you know, your boss comes and says, okay, go test this today. What's not shown here is each component has its own database. They all use MySQL. So you have a MySQL server, but they all have separate databases. You know, how do you go through, tune that box? How do you, do you make it a box? Do you make it stand alone? How well are people at writing queries? Things of that nature. And this, again, is a high level view. Each of those components has sub-components that talk to each other and things. So this next page that I'll show you, I stole, I forget where I stole it from. I'm not sure it's currently active and accurate. And don't really try to read too much into it. But there's a little more detailed flow. So within each of the major components, you have all these other things that talk to each other. So this is part of the challenge for us, is how we're going to go through and test this. So you take a deep breath and we get in on how to test the control plane. You look at things like what can be tested stand alone. So we can take and stand up a keystone instance and hit it with API calls. We don't need to go through and, you know, hit horizons and go start six NOVA instances, see how it works. So we can limit, and, you know, folks can do this on their own, you can limit the scale of your test environment by doing things like these. It's great also for isolating performance issues because, you know, it's just here. And you can make sure that your test, you just need to make sure you're testing the right thing. So all the keystone calls we're doing, you know, is there a certain timing of them or whatever that's going to be important. Then you, configuration to be used. One way to force performance bottlenecks, nice thing about virtualization is, you know, as the boxes get bigger and bigger, you get two, you know, common two-node Sandy Bridge with 256 gig of memory. It takes a while to run out of memory on it. So you can go through, you can move your MySQL server into a smaller VM. And then there you can put constraints on it for memory network, CPU, so you can force issues to happen at a smaller scale. So we've been doing a lot of those. You need to be careful when you do that that you understand what you're doing and the impact of doing that. So questions so far? I feel like I'm blowing through these. We might be done by. So again, you know, how to scale is one of the questions we keep asking. We have a very limited number of machines available in the performance lab. I think we're probably like 35 now. Which for some of you might seem really big, for some might be small. But when you're trying to simulate something that's going to be tens of thousands, it's small. So one way we're doing it is, in the initial work, we used our rev product. We'll probably switch to use OpenStack now and do triple O OpenStack on OpenStack. But basically you go and you create a bunch of images, instances under virtualization. You turn around, you tell Pax that when you bring those up that those are all hosts that you're feeding for you over compute nodes. You don't really try to go test guest performance in them, but it's great for testing the management platform. I need to go off and investigate if there's a way to have a simulated VM under OpenStack. Nobody knows that, do they? Save me some time? No. Okay. I know there was a way when we did some rev testing, we had a little widget that we used that we tended it was a guest for answering a null driver. Okay. Yeah. Well, excuse me. Fake vert? Okay. Yeah, I'll go check that out. The challenge is I have not personally run it yet, so I cannot speak to it. I know Red Hat has a team of people working on it. So my son is actually working on it. So that's how old I am by the way. My son is an engineer at Red Hat as well. So I can't really talk to that right now, probably in two weeks once I get back. Do you have any experience? Are you just interested in doing it or? Okay. Okay. Yeah, I don't have any personal experience there yet. It's often the case that you find a performance problem at scale that developers can't reproduce in the unit test kind of environment. Do you have a way that you hand side over to the developers so that they can use your environment for pinpointing the issues? We will share the environment with developers. Lots of times in our work, we go back and we try to find a reproducer that happens not at scale. I mean, once we can understand what the problem is, it's sometimes fairly trivial to go find it. Sometimes you can't. It needs to happen at scale. Going from a 10-gig network to a 1-gig network, if it's a network problem, that is going to put a problem there or whatever. But we've shared our environment within Red Hat. There's also some bigger environments coming into Red Hat that we're setting some continuous integration work up for things to test more at scale. The good thing is Red Hat is now convinced that scale matters, which is pretty good for what you consider an OS company. So I think I have covered everything on this one. I've gone off. I've created hundreds of instances under our Rev product and turned around and fed them into pack stats so we can go and look at things. When you do that, you need to be careful of the hypervisor running on is going to introduce some of our official bottlenecks that won't happen in the real world. You need to understand that. But in general, it's a good way to cheat and at least get things up so you can see what happens when you have hundreds or thousands of things hitting your keystone or whatever. Yes, I'm sorry. Could you just talk a little bit more about the artifacts or what kinds of things to watch for that the virtualization infrastructure introduces that doesn't happen when it's physical? Sure. One thing would be your network configuration, where each hypervisor is going to have its own access to the raw network. If you're in a guest, you're going to have to share, depending on how you configure it, you could use SRIOV, you could use a bridge. The simple case is everything's going through the same bridge, so there's going to be some artificial bottlenecks there. But that may actually do more to mimic a top of Rx switch as well. So you really just need to make sure of that. You don't want to oversubscribe the hypervisors that are holding what I call fake hosts because then there's a swappiness as you swap a hypervisor in or a guest in or out. Fake hosts in or out. Let's put it that way. It's too confusing otherwise. So you don't want to greatly oversubscribe those, at least in this case. There's other cases where I have, but those are more or less, right now I don't run any true guests or instance critical things outside of using to start something up that's real minimal and goes away as soon as it starts up. But that's again, if we could do this fake guest thing, that would be great. So because then you can spin them up and they just respond to salameter or whatever you're using underneath to see if they're up. We good? Did I answer your question? I had 10 hosts out there and I think I did 10 fake hosts on each one to make sure that I didn't oversubscribe. It depends on your boxes. I mean, I had Westmere's with two socket Westmere's with 48 giga memory, so I didn't want to greatly oversubscribe the memory in particular. But you need to make the guests big enough and you'll see when we get into an overscheduler has some interesting behavior, so you need to make sure your guests have enough memory. As we've discussed, there's in the control plane, I think I already mentioned on the messaging side that it's an area of concern, but we know what we can do with AMQP and Cupid. We use it as one of our standard network tests. For the database performance, we have a lot of people on the performance team that have been DBAs before, so we can dig in and understand what's going on with MySQL. We're actually starting some work on that this week on how to go in, basically see all the queries that come as you do things. Go create an instance and go try to spin up 100 of them. What are the calls, hitting the database to do that from all the different components because it involves Keystone, it involves Nova, Plants, so if we can quantify what those are. Yes, question? What kind of performance work we've done there in regards to the database for MySQL versus Postgres or anything, we haven't really done a lot yet. In the initial testing we did, it wasn't a bottleneck, so we focused on things that were bottlenecks. Keystone was the main one and we'll get into some data on that. As I said, we're spinning up a guy to start that today. I talked to him Friday before I left. He was going to start that work today where he's going to go through. The other thing we're looking at is when we have to test with high availability versus standalone. Again, it's a scale issue. If you have tens of thousands of machines, you're going to need some high availability out there. You need to understand the differences with a high availability version of your database versus standalone instance of it. Hopefully, in a couple of weeks, we'll have gone through the database. As a team, know the differences between MySQL and Postgres and how to optimize for each of them. As we get into this, part of it is going to be how do we optimize for that because out of the box is usually not the best for any of the components. We'll probably do a best practice or performance brief. I think the reference architecture team has just done a paper that got released on high availability, but I'm not sure it's going to address what you want. It's more just how to configure for high availability for the database. I think we basically covered everything on that slide. Question up here? He's going to run up with a microphone. He's got to lose 10 pounds this week. We use both depending on the tests we're doing. All my machines have 10 gig, so we did not see saturation on one gig for most of the work we've done yet. We look at the stats on each hypervisor of the actual physical machines to use IOS stat and SAR and we can see what we're doing next out there. Did everyone hear that question? Whether we use one gig, 10 gig, so pretty soon hopefully I'll have some 40 gig I can run with, but I have to show 10 gig is a bottleneck first. So we'll scale up. The two new areas to poke at, we are just talking about how we're going to do this now. One would be Oslo, which again is the more common library. I want to make sure I have a Python developer I can put on that, not just someone to go test because we need someone who knows Python in and out that can go in and say, okay, this is the bottleneck because you're using this kind of thread, not that kind of thread and fix everything across the board. Then with Solometer again, it's waiting for that to gel enough that we can actually start testing with it. Part of my background is from the network management side of the world many, many moons ago when my engineer son was just a baby. That's how long ago it was, but with Solometer hopefully that's going to be critical I think for things and that needs to perform well and not be intrusive. So finally after just a lot of talk, we can start getting to some test results if people are ready for it. So the first thing we looked at was Keystone, one of the initial concerns, and this is back on our Folsom release, was how chatty was it. We knew it, got called a lot. The default test environment didn't have memcache or any cache in front. And so what we found, several issues, new tokens, nothing ever went back and prune the database. So all these tokens you could never use again, just sat in your database. And some inefficiencies with curl versus CLI. So we did the initial tests as we're all getting our legs under us with this. We did Horizon, we measured the calls. So when you were using Horizon, it took three tokens just to go off and log in and get an answer or something. If you went to the image page, it took two tokens. And then when you're doing the CLI for an over image list, again, that was two tokens. One of the main things was there was no cleanup in the database. So we were able to measure as we just kept inserting records that for every 250k rows, response time went up a tenth of a second. So with nothing pruning the database after a while, it's going to take, you know, one, you're going to run out of disk space, your indexing is going to be horrendous. And so I believe we filed the BZ to go fix that. I don't know if it's been fixed yet. I should make sure that it has. The other thing I had mentioned is the CLI versus curl calls. If you ran Nova image show, it executed in 2.6 seconds, roughly. If you did the curl corresponding curl call for that, it took half a second. So, you know, performance guy's like, wow, hey, something we can go dig at. So we went back, we traced it, it showed that, you know, Python was making a call to the HTTP lib library. And it was a known issue, and it's one of those things that it's so generic that it's going to take a while to get fixed. The next thing up for testing here is Havana. We've moved to the PKI tokens. So probably about two weeks we're going to be spinning up a new round of tests with Keystone to see how things are going. And by the way, these slides, I haven't posted them yet because they weren't really done yesterday. They were mostly done. But I will post them this afternoon so people should be able to get them. So as we move sort of into the gray between control plane and the data plane, recently a guy just finished up some work with an overscheduler. And it had some results that were interesting to us, maybe because we didn't dig into it enough ahead of time. But basically, the way the default and overscheduler works as it goes, based on the amount of free memory. Question? Sorry. For this test, this is false, I'm right. But you're doing this particular test? This is grizzly. We're up to grizzly for this test. Okay, so you're using the filter scheduler? Whatever the default scheduler was. I didn't do these tests. I'm leading a team and I'm sort of presenting the team's results here in addition to mine. But basically, I told the guy, oh you're already using these two machines, go grab these two new P-Cloud machines. And I forgot that I had recently upgraded the memory in those from 48 to 96 gig. So what he found was this interesting characteristic that the nova schedule basically determines how busy a machine is based on the amount of free memory it has. So when he tried to spin up 100 instances across all four, we were thinking we're going to get, you know, 25 per machine. But what we found is we got 49 on the two big ones and one on each of the others. Which is great if you're doing a memory intensive thing and you want the load fairly distributed, they're not going to interfere with each other. But if you're doing other things and your guests that might involve CPU, probably wouldn't work too good. So he dug into this. He found there were some ways to change it. And so, you know, I'll mention it here. Just understand, you know, if you have a rack of machines, make sure they're all the same. Or understand what's going to happen. Yeah, there's a different scheduler. He found some ways to change it. Schedule a host subset size equals four. Does that mean anything to you? Okay. You're not shaking like you're happy with it, though. All right. So when we did that, this graph, each color represents a test run. So you can see how you got a little better distribution. Again, it's still somewhat random in how it does it. It looks at what's available and then randomly assigns things. And you can see, you know, I get to use my laser planer. So over here, you know, one run, it was seven. The next one, it was two. So it's still somewhat random, but it's not 49 and one. So, you know, that, again, would be something to be careful of. We're going to dig more into that as we go forward and figure out, you know, how to keep that level. One of the things I'll get into is we're working on a sizing calculator that I don't know if that will be released or if it's more just going to be for our sales force, but how to help sell this and people can figure out what they need for machines, licenses, whatever. So, you know, and I'll get into that in a few slides. So as far as the guest performance, and again, keep in mind the lineage that my team comes from is we were, you know, rel kernel. You know, how do we get the best out of that? I started at Red Hat right when they were doing, just before they released their Zen product. So I worked on Zen to start, and then I led the team doing KVM. And even KVM, we didn't treat it that often as multiple guests. It was just individual guest performance. Then we'd do multiple. Wow, doesn't work. So there's been a mind chef going on the performance team on we're not all just testing a single standalone kernel anymore. But the good news is as we get into the guest performance is it's really boils down to a real KVM issue the way the Red Hat product is. And our team is really good at that. Couple little marketing slides I'll blow through. I don't know if people familiar with spec for it, but it goes through it's basically six VMs per tile and then they go by the best mark. And these are like my marketing slide. So if you could bear with me they wanted them in. Red Hat more or less owns the spec for results. So that's any questions on that or we could each of these little circles shows different class machines basically. And there was a while where Red Hat and VMware were leapfrogging. But most of them are owned by Red Hat. And the other interesting thing is all of the hypervisors regardless of the vendor using it use well underneath the covers for their guest OS. Yes. I think there's an LXC driver for this as well. Have you tested that? For spec for it? No, for open stack. Instead of using KVM can you use LXC? No, the Linux containers. No, we've only tested with KVM that right now that's the Red Hat product. So we're, you know, special sauce, I call it or whatever, which is sort of disgusting if you think of it. So our group focus is mostly on the product. We'll do some competitive analysis but the containers isn't there yet. And that's I think slotted for REL7 at this point. I don't believe there's any AMD results in there. Apologies to any AMD folks in the room. So again, you know, we expect the same REL KVM. One of the things we added, you know, we made sure we added to our open stack product is REL has a concept of these 2D profiles where it goes through and it will configure based on enterprise storage or virtual host or virtual guest. It changes some of the parameters by default. So you don't have to go do it for each system. So this data here will show Ross is the Red Hat open stack product. This would be Grizzly. I'm sorry, this would be early Havana. But the red is our KVM that's not tuned without the tune D in there because it doesn't ship the tune D on by default. So this is a Java workload probably looks a lot like JBB, but I probably can't say that. And so you can see that the open stack slightly out performs the basic KVM. And then we did some work early on with the overcommit ratios. Nova was pretty aggressive with that. The CPU had an overcommit of 16. The memory was much lower at 1.5. We dug into those a little bit. And again, with a Java workload, you could see that as you overcommitted. You know, you peaked out at 8. Once you started there, your overall performance started going down. So again, it's all going to feed back into our size and calculator, whatever. But you need to be careful. You can't just keep loading up. How long do I have since we started late? 20 minutes? Okay, thank you. I'll slow down a little. How's everybody doing today? One of the things we spent a bunch of time on is the thermal storage. So basically, when you go to boot instances, you go off to Glance. You say, I want to boot this, you know, you're told you're booting this instance, so you copy it over from Glance, store it, and boot it locally. If you boot that image again, or multiple copies of it, it only copies it over once from Glance. But by default, it stores it on your system disk, which when most people set up their system disk, it might be, you know, rated for protection, but it's a single disk performance or worse. So we looked at ways, you know, can we improve boot performance by putting different storage underneath there? It is a tunable again and how we configured things, so we're able to do that. So we went through, you know, as a user, you're going to have to trade off of the cost versus, you know, okay, you want eight disks in your box instead of two. And then, you know, more important for your data systems will be power and cooling. But how does it do overall is, you know, what we looked at. We did some network-based tests. Again, you're going to hit bandwidth issues if you try booting too many different things at once. So for the test data we're going to go through now in a minute, we show with a single system disk. Then we take a seven disk array. These would probably be like a 380 class machine with the built-in internal arrays. We populate them. Don't want to give too much vendor-specific information or upset other vendors. But we, for the NFS tests, had, you know, an identical machine that had the seven disks, we just exported that for NFS. We did some fiber channel SSD drives and then we did the direct mount and then did them again, NFS. We did those two, sort of by accident, because the guy thought those were the drives he was supposed to use. But since we got good data, we did it again. And all of this used a 10 gigabit network underneath. So a couple things, as you can see, the impact of tune D, we'll show some slides that show that. And as we did this, we didn't scale everything out. When the boot times became what we'd consider unacceptable, we'd stop the test on that size. And don't be misled by the NFS data, because we were booting, you know, we weren't loading the network down, we're just booting on a single hypervisor, not a whole rack of them at once. So this shows basically boot times as you scale up the number of VMs underneath. The red is the system disk, you can see, you know, that was fairly linear, but it picked out, you know, nice, steep angle to it, it peaked out pretty quick. So if you were trying to boot 10 VMs off your, off the default install, it, you know, took on average 74, 75 seconds to do it. If you drop down to the array at that same point, you were at about 40. So when you were doing these boots, did you just issue like Nova Boot? So when you were, when you were booting these, did you just use the Nova CLI to just call Nova Boot 20 times and start the, or sorry, Nova Stop and Start 20 times, or did you do something else? We did Nova Boot, but we didn't measure, this doesn't include the copy from Glance, this just includes from when the start was issued on the box till when the system was actually up. Okay. So the reason I asked this is obviously there's a delay when you issue the commands, not booting the Glance. So there's just a delay in how fast it's actually sending those messages out to. Right, and we cheated and changed that, so it's something as quick as it could. Okay. Just one check. Yep, no, thank you. So the blue line would be with a 7 disk stripe internal to the box. The orange, I guess, was that a RAID 10 or just RAID 0? It was RAID 0, just, it's a thermal storage, so you're saying you don't care what happens to the data there, so let's make it as quick as we can. We did the same thing with the SSDs that were mounted, you can see they're again faster. And then at the end, we had, rather than show all the data, the last two on the end here, the NFS of the array and the SSD you can see from the first to the end, those are pretty much flat across the board, so we didn't plot them all out. Are these all cal images or are they LVMs? They're QCal images. And this again was, I'm sorry, I missed that up front, this is multiple images, so this isn't the same image 20 times, this is pulling in, for each of these tests it's pulling in 20 different images and booting those, where if you're doing the same image it actually boots quicker because it's cached. And then basically, there's this one back for a second here. So this one was without the tune D, which I talked about before, you can see, you know, with the system disk you're up at about 75 seconds to boot. When we add the tune D profile, which adds the vert host profile which changes the whole disk characteristics, you can see that we stay pretty flat across the board. And we didn't show each thing here, we went up to 16 VMs. The system time still went up, but at 16 it was only at 62 versus 75, but the array was still flat for the most part. So the impact we made there by just putting, you know, something in the config to say go put vert host on all the things made a big impact for boot times. This is some updated data with Grizzly, we were around the test a couple of weeks ago. And here the red is the Red Hat storage 2.1, which is our Gluster product versus the 7Disk array set. So it's pulling it in over the network. We have some interesting data coming up with a Gluster bit. Questions on any of this? Yep. How big is the margin of error in these measurements where it's only off by 21 versus 22? This is the average time, or was this the max? I think these were actually the max time. So there was margin of errors probably, you know, four or five seconds. So anything that looks more or less equal is probably in the noise. Another thing that's going to be a big concern is snapshot times when you want a snapshot. So for the OpenStack snapshot, this is you know, how we ran the test as we did it, the QME Virgin image convert mechanism. The data gets written to a temporary space. And again, this was back on Folsom. The destination's tunable, so we're able again to go in and change where the snapshots get written. Similar kind of test you can see. This would all at this point be tuned, tuned data. So you can see the system disk again has the same curves, but going with the NFS of the array or just the array itself is much better. So you know, I think the main takeaway from a lot of this is if you can afford the cost and you need gets to boot fast for a thermal storage you should probably look at using an array. We never really went back to see, you know, is a three disk array going to, you know, where's that going to fall. Again, it's going to be on your pain points. The more disk you have, the more power you're going to use, more cooling you're going to need. Some snapshot data with Grizzly. You can see that the Gluster actually, you know, here it's time, so smaller is better. Keep in mind that Gluster, our configuration is replicated, so you need to do two writes instead of one, so it actually scales fairly well. Another thing we tested was with PG bench, the Postgres bench mark tool, if you will. Again, the seven disk set versus Ross. This is the load times for the database. You can see that Red Hat storage actually came out a lot faster, especially considering it's doing replicated writes. The interesting thing was when we went back the transactions per minute were also a lot lower. Not sure why I have to dig into that. As I mentioned, we're doing a sizing calculator. So it's, you know, a fairly simple capacity planning tool. Really a spreadsheet that you go and you plug stuff in, you see how it does. More or less using the Amazon style, you know, M1 large, M1 medium, kind of characteristics. This is very much work in progress, so these slides will show data that hopefully will get a little better as we go. And it's, you know, giving us some extra tests we can do but extra extra ways to characterize things and find things. We're using the YCSB benchmark with Mongo and Cassandra, which we've actually found some interesting data with those as well that I'll get to. So from a latency point of view, you can see, you know, the baseline, all right, so let me back up for a minute. So the M1 testing we use 21 VMs is what the calculator says it can do. So any, anytime you see the blue bar here, this is the average of 21 VMs. For the M1 large it's 10 VMs and for the M1 extra large it's five. Baseline would be just a single VM. So you can see as you add more VMs, you know, your average latency is going to go up. The average throughput is going to go down. The next slide though will sort of show this is the aggregate throughput of each of the sizes. And I added the green line last night which I like a better thing and maybe it was late, I called efficiency. So you can see with M1 large we're right at 50%. You know, we go up a little higher on either end. But ideally we'd want that flat and for the sizing calculator, you know, I'm looking at talking with the author to say let's go back and add this so the end user, you know, sysop can say okay, I don't mind going down to 70% or whatever. You know, because it's performance you're going to lose if you go too low then it hurts you. So we got to figure that out. So for Cassandra scaling with the number of nodes, one of the interesting things we found as we did this and the next two pieces of data will show it is the MongoDB has a right lock. So as you increase shards, you still have the same right lock that's going to limit you. Cassandra doesn't have that. So you can see here that we scale well with Cassandra. The interesting thing with Cassandra though is that the numbers overall numbers are much lower than they are with Mongo for the same test, same load. So here you can see, you know, one versus two nodes for the database. Workload A, workload C, which is 100% reads. You can see that you go up actually 2.5%. Not sure why it was 2.5 versus 2. I cut that one last night and so I sent the guy mail. He's going to look into it. So I can't tell you why today, maybe in a week. And then Mongo sharing was when we added shards. We went from 2 to 4 shards. I don't know why I made that 3D, but you can see performance actually went down a little bit. And again, it's probably lost to the noise, but there's a common right lock that's going to keep you from going beyond that as you add more. But at the same point, you're up to between 20 and 25, which is about double where you were with Cassandra. Neutron performance, a lot of new work in the REL products to support OVS. I guess one of the drawbacks of having a stable REL product is that a lot of new features don't get added to it right away. But they're backporting GRE and some of the other stuff needed for OVS namespace, things like that, into REL 6.5. So we've been testing on some of that. There's still some issues. We'll be open here. There's still some issues with the MTU sizes, issues for performance perspective with GRE and VLANs that are all being actively worked on. There's a page here for people who want to go off and look at how to use GRE tenant networks. There's a page up on the OpenStack Red Hat page, the RDO page. So we're not really comfortable putting numbers in right now, just because we know it's going to change. Most of you probably wouldn't want to share numbers of something in the middle of development either. Let's just say they need some work. Swift performance. There's been a lot of work going on on Swift and some of this is more or less older data. But the changes are now being accepted upstream. We're able to focus on tuning. Tune off the directory listings for Gluster. We're adding new pluggable backends, things of that nature. A lot of work was done with the XFS file system layout. Even though this started as a Gluster project, found a lot of issues with XFS, with directory block size, the iNode size, Rack Space. I don't know if there's anyone here from Rack Space that they've gone with the suggestions we had and agreed that they sort of working independently came to the same conclusion, which is always good. So again, a lot of this is new parameters introduced for max clients. Tweaking work is as needed. So this shows he was doing some tests where he was doing clients that were writing 3K, 30K, 300K, and 3MG all in parallel. And without tuning you can sort of tell when each one falls off that's the blue line. When the 3K clients finish everything's running in parallel but the 3K will finish first because they're a lot smaller. They fall off and you can see. But when he went back and did the tunings, you can see that basically it was a flat line across for a seconds per object, which is good. And you know, down here it's a little higher, but overall I think it's a good trade-off. As far as working with the industry, outside of all the work we're doing, question over here? Back on the other one? Yeah, on the previous one. Okay. Sorry, what is that in a traditional kind of megabytes per second? So half a second for 30 megabytes? So 60 megabytes a second? Is that what the red line is? That's how many seconds it took on average per object. So I'd have to go back and look. This was again someone on my team had done the work. It says 150, so then I multiply by 150. Oh, so he's doing 150 megabyte, I mean 150, 30 megabyte objects. In half a second. So 150 times. Well, that was the average per object. So yeah. Alright, thank you. So we've got about three minutes left. Working with the industry, we've, you know, Red Hat's always been involved with the spec TPC things. We're helping drive. I personally have been on the spec for it in the spec cloud committees. Currently not, but we don't publish a lot of benchmarks ourselves but we work with partners. The other thing, are people familiar with the RDO efforts that Red Hat's done in the room? If not, you probably want to check it out. It's community distribution of OpenStack. It's packaged for any of the, you know, EL6 kind of stuff, free available without registration. OpenStack.RedHat.com. It's a vanilla distribution, doesn't include a lot of our special sauce, if you will. Release with the upstream cadence. There's a thing that describes the differences. You know, it's the latest code where on the OpenStack side, it's going to be enterprise hardened. The tougher part for a team like ours is the Red Hat OpenStack product is still going to be released with a six-month cadence where we're used to a three-year cadence for an OS. You know, we've been working on roll seven on our team for probably a year and a half now. It's not going to be out for a while still. So I guess for a wrap-up, you know, as most of you know, it's what I like to call the, instead of the bleeding edge, it's the bleeping edge when you're on it. Out-of-box performance is pretty good. We need to focus on the infrastructure for the out-of-the-box stuff or things like my SQL. We haven't even touched on memcache yet when that should get used or not. So like I say here, we're still scratching the surface. Need to refine the scale methodologies. And part of it, you know, just the mind shift, especially on my team, is, you know, we're talking about tens of thousands of these, not one. So how do we, you know, guess performance is important, but it's not the end-all. You know, we're not spending four people trying to get another half a percent out of KVM performance. Not going to matter at this scale. You just spin up more. So you've got 15 seconds for questions. I'll stick around. Are you posting these results or like as you test anywhere? I thought, or did you say it was like a lot of the testing was proprietary so you're not posting anything out? We're still figuring out how my team's going to work with the upstream community. So these slides will be posted. A lot of time, the performance work we do will either get out in a tech brief or reference architecture or the developers themselves will share our data. But at this point, you know, on the Swift side, you know, that's been pretty much upstream. Where would I be looking for this information on a regular basis? I will. Right now, you'd probably have to go to the Red Hat site and look in the reference architectures. But if you're not a Red Hat customer, you won't have access to them. So if you give me your business card, we'll figure that out. Right. Right. So if you stop up afterwards, we'll figure that out. Anybody else? Well, thank you. And I'll stick around for questions. Thank you, Mark.