 Oh, there it goes. Turned over to right on the dot, the time o'clock. All right, welcome. We're going to talk about Cinder Performance, which it's kind of a misnomer, but performance of storage in OpenStack here. I'm Ed Balboff with SolidFire slash NetApp. My two colleagues here, David Black out of our North Carolina lab. He's more NetApp than SolidFire, but he's coming to the dark side. And John Griffith, who everybody knows. He's infamous. I'm sorry. So the agenda we're going to go through here, we're going to talk. John's going to talk a little bit about LVM performance, and he's been doing some work there. I've been doing a lot of kind of investigation around the hypervisor, the VM, those kinds of things. David's got some stuff he's been doing on storage networking, iSCSI sessions, and QoS. And then we'll come back up with a conclusions and maybe take a couple questions. So David and I are going to step off and let John go through his cup bits. All right. So I wanted to kind of talk a little bit about LVM, mostly because as some of you know, LVM is the reference implementation. Gets a lot of flak. People talk about how it's no good, it doesn't perform, and I need raw disks, or I need that, whatever. So what I did is I put together a little bit of information to talk about some of that. So the first thing is, as I said, everybody always says, nobody in the real world actually uses LVM, or why do we bother with LVM? Nobody uses in real life. And my response now is don't make me flip this table. That's a new Cinder thing. We flip tables when we get angry or upset, so. The point that I always like to make to people though is this is the last user survey. If you take a look, you will notice that the second most popular backend in use is LVM. Not only that, it's actually growing cycle over cycle. So to sit there and say that it's not something that's used or gonna collect it is probably not really a wise choice on our part. So that being said, there are a few misconceptions out there about LVM. A lot of people seem to have the impression that LVM adds some overhead and slows down your performance. The other one is that LVM does not have an HA solution. And then my favorite one that has been kind of purported around for a while now that drives me absolutely crazy is that I must have raw disks in order to get performance to do things like Hadoop. Completely untrue. So the facts are, LVM and even with an LIO target on top of it, they really don't have much of an impact on disk performance for standard workloads. As far as HA solution, there's actually a number of options. Cinder has had a mirrored LVM option inside of that driver for a number of years now actually. On top of that, there's DRDB now is in tree and you can use that. There's still some work to go before I would probably put it in production and fully trust it. But they've got some performance data out there and things like that. And it's actually coming along really well and it's a pretty good option. And then back to the last one, I just don't even know what to say about that anymore. So Pishposh, I took that from Simon. Now, I'm not gonna just go crazy here and say LVM is the greatest thing in the world and everybody should use it and stuff like that because that's not true and we all know that, right? I'm not a zealot much but it is pretty good and it should be something that we continue to consider, I think it's cheap, it's free, it's easy. So I think it's a perfect fit, especially for a reference implementation. It also behaves more like most of the backends in Cinder. By the way, there's like 85 of them. So again, not saying everybody should beat the LVM drum and go down that path. It's not right for everybody. It doesn't fit all use cases, absolutely that's true. But I am saying we should continue to nurture it. We shouldn't just dismiss it as a thing that's only good for the gate, right? When you get into large scales, production environments, service providers, things like that, it may not have a place for you but then again, it might. It's a fantastic solution for TestDev. It's a fantastic solution to augment something that may be higher performance, higher reliability storage inside of your private cloud. So those are the types of environments that's really good to look at. And then of course, you can't beat it, right? If you wanna just do a POC of OpenStack and do some evaluation, it's cheap, easy and free, right? What more could you ask for? So it's kind of a good deal. And in that respect, I say if that's gonna be a potential OpenStack deployers or users first taste of Cinder or first taste of OpenStack, we should probably try and make it as good as we can, right? It's kind of important. So what I did is I put together some basic numbers. Nothing really like crazy or anything like that or intensive. So rather than just sit up here and try and tell everybody, hey, it really does work. I know it, I use it all the time and it's great. You should just believe me. I put together a little bit of data because everybody likes to say things without any data to back it up. So I took a machine with some raw disks. There's some Intel SSD 320s in them. I've got LVM version 2.02 130. It's running on CentOS 7. I've got a 3.10 kernel on there. You can see all the information at FIO 2.2. And I used RDO and PacStack in this particular example. I've duplicated this on a identical machine with Ubuntu 16.04 and I get almost the same results. I mean, it is within 5% for everything. So that's, by the way, using DevStack, which you're never supposed to do, right? So the FIO settings I used for these tests, I used a block size of 4k just because that's a common thing, right? And that's what most people do. LibAIO is what I use in there. An IO depth of 16. I didn't want to go crazy and I didn't want to go stupid so I went right in the middle. I used 50 gig test data sizes and I used direct. I didn't use any buffering or anything like that. And I only used a single job. And the reason I did that was, particularly in OpenStack, you're hopefully booting up a lot of instances that are fairly small. One vCPU, two vCPUs, doing small things. So I want to make this somewhat realistic. And then I run each one of these for two minutes. The device, as I said, it's a raw disk. What I do is I start with that. I run this test suite on it. I run it 10 times. And then I go ahead and I put LVM on it and I run it directly to LVM 10 times. And then I do a striped LVM across three of those disks and I run that one 10 times. Finally, I put an LIO target on top of it, run it 10 times. And then finally, at the very end, attach it to an instance, go inside, run FIO inside there to the dev VDB, run it 10 times, get the average. You get the idea, right? So this test that I'm running is a group of tests. I do a sequential reads test, sequential writes, random reads, random writes, and then a 60-40 mix. The thing that's kind of cool is if you look, I was, actually, I didn't think it would be this good. I thought it would be a little bit worse than this. But if you look, compared to the raw disk, I'm not taking a hit at all. And in fact, if you're smart enough to do things like stripe your LVM volume, or stripe your volume across multiple disks, you can see the impact that that has, right? It's fantastic. And it's really cool. You sit there and you watch DSTAT and you see it just pegging all the drives in the setup. Unfortunately, I didn't have a bigger server available to me at the time. I'd love to have done something like this with, say, 10 drives, 12 drives, something like that. That'd be really cool to kind of check out. But anyway, you can see the difference, and you can see all the way down the chain to the VIRT IO, that's when we're going through the instance of BDB. There's not that big of an impact, so that's good. It's good stuff. The 4K random reads, again, pretty much the same story here. This one, you know, there is a little bit bigger of a difference that you can see on the striping. So that's cool. I'll give you a second to take a picture. Hurry up. There we go. All these slides will be on SlideShare. So if you want to get them, you're able to get them up there. Again, 4K sequential writes, same type of deal. And the random writes. I have a thin column in there. So FYI, I forgot to take that out. I had thin data in here as well because I thought, well, maybe that makes a difference. In terms of these tests, it really didn't have any impact or make any difference at all. Where it did make a difference is I started playing around with what happens if I have a whole bunch of snapshots of volumes while I'm doing read-write activity. And that actually started to make a significant difference. But that's for another talk. And then the mixed workloads. The mixed workloads, we, you know, this one was a little bit interesting. I expected to get better numbers out of this than I did, honestly. So I ran the suite, I ran my script like three times. And it just, for some reason, there's some really weird latencies that came up in there. I don't know if it's an FIO thing or my disk or, you know, I'm not really a performance engineer, so I did the best I could and sort of came up with pretty consistently. So at the end of the day, I just want to say, you know, you look at that and it's really, it's not bad, it's not bad at all. This little handful of examples I gave is definitely small scale, no doubt about it. But you can scale it out from there. And the other thing that, you know, that I've noticed over the years playing with some of this stuff is it's a fairly linear scale. So the more drives you add, you know, you can extrapolate it pretty easily. Striping is a huge thing. We don't actually do that in Cinder by default. You would have to set that up on your own. There's a really easy way to do it, set up your thin pool and use thin provisioning and have that thin pool striped. And I'll take care of it. You can also go ahead and mirror that thin pool or that striped volume group, whichever one, or striped volume, you can do both. So there you have some redundancy. Since this is iSCSI, 10 gig network, dedicated data network is essential to this. You start doing this on one gig and you'll be really disappointed. The other thing is, keep in mind in terms of scale, if you don't have a large number of disks and you don't spread it over multiple disks, you're gonna get completely hosed if you have more than a couple of users inside of your cloud. So it's kind of key to play with that and factor that into your equation. So everybody calls LVM the pig. So as far as I'm concerned, that'll do pig. All right, so John talked a lot about kind of the smaller scale stuff with the LVM and the reference implementation. I'm actually gonna take a look at some stuff I did with a much bigger setup here. So this is the quick version of the diagram. I actually had eight solid fire nodes, so I had 80 SSDs in there and a full, fully redundant 10 gig network going out to the compute nodes, the control nodes. But I did have them going through top of rack switches and a core switch. So you can see the diagram there. We had 40 gigs up through the core. The goal being not to limit ourselves with the network. So we're trying to make that as kind of production type as possible. A little more about the details here. I used, they were all Dell R620 servers. You can see the specs on the core count and the memory utilization there. I used a Fuel 9 OpenStack deployment. So the hosts were Ubuntu 14.04. The only change I really did with that was change the force config drive, driver typo in there. So I could vMotion stuff around so we can kind of test this stuff on, put it all on one hypervisor and put it all on three hypervisors. The image I was using is CentOS 7.2. David will talk a little bit. He used the CentOS 6.7. So there's some different tuning parameters. I'm gonna try to point those out here. So we've got some differences in versions. So that's kind of important. I'll talk to you about some conclusions there. And so I have a heat template to deploy the instances and the volumes. I can build a eight gig root volume, 20 gig test volume, and I can do this automatically to turn up how many instances I want. None instances and we can build as many as we got storage for. And then I automatically pre-fill all the volumes. So I've got a random non-compressable, non-dedupable workload that I fill the volume with before I start running my stuff. So I looked at hypervisor tuning, VM tuning, other things like that. I guess I'll tell you the conclusion to begin with really here pretty quickly is that a lot of this stuff doesn't matter a lot. It's in the, again, the older versions, there was more tuning as we've built the newer versions of OpenStack and the new deployment tools would kind of do a lot of this automatically. But you can see here, we changed, I changed the scheduler. I added the number of requests like rotational gets set like it should be. You can put all this in Udev rules. I actually had a little script so I could set these as I wanted to back and forth. So we'll show you some results here in a second. And then we look at the KVM and the VM itself for some tuning issues here. Interestingly enough, in the VM with CentOS 7.2, you can't, for instance, change the scheduler. So the scheduler is none. And that's the way it should be, right? In the older versions, you had to set that to no op, but now it's set to none. You can see the green things here are highlighting the things I changed. I meant to put that on the previous slide. Within the VM. And then for KVM tuning, all the best practices are around using Vert.io, using AIO for the driver mechanism or the AIO mechanism there and turning off the page cache so you're not buffering there. Now you may want that for some other reasons, but we set that all up, made sure everything was set correct. And then I started making a lot of graphs and I have a ton of data. So I'm trying to present the stuff that's most relevant here. So I'm gonna, in this one, I'm gonna go through all of the four of these and then for the subsequent sequential read or sequential stuff and random writes, we'll put a little less data up there. So you can see the basic conclusion here is this, you know, I've got tuned hypervisor, I've got the defaults and then I've got the tuned VM and the default VM. And really, not a lot changes. You tend to get a little bit better latency when you tune things and you push the knee out. So if you're looking at these curves, these are all latency on the vertical scale, IOPS on the bottom, and this is all coming out of the VMs, right? And this is running, in this case, it's running one VM. So I got some tests where I ran lots of VMs, but this is just like focused on one VM to try to see what happens as we tune these things. And again, you'll see the needs on these curves move a little bit. We set these all up. If you're familiar with SolidFire on the back end, we can do quality of service. I set some minimums here to keep these things sane, but their maximums are turned off. So they're all free to fly as much as they want. So when we look at this, kind of the summary on those slides, right? The three millisecond crossing, which is a value where you would start to see some issues potentially in your application, was at roughly 18K IOPS on all those graphs. It didn't change a lot. So the nominal one millisecond for the hypervisor didn't seem to change much, but when I started tuning things in the VM, we see that our nominal 1.75 milliseconds, we could cut that down to somewhere in the 1.25 to 1.5 milliseconds. So there's some tuning differences you can make here, but in your overall performance, it doesn't really affect it a lot. So if you look at the random writes, now I've got a lump in my defaults here. John probably ran something on a cluster while I was in the middle of running a benchmark. So we're gonna ignore the little lump because that should be a standard curve. And you can see the changes here. So we've got the three millisecond crossing at about 12K IOPS for everything, nominal 1.25 for the hypervisor. The VM adds three quarters of a millisecond of latency going through there overall. Again, these are all single VM performance. So we've got them here for sequential reads. This changes a little more. So you can see the curve in the VM is very, we just, there's no knee, it just kind of goes up, right? As the IOPS, and of course the bandwidth goes up, the latency affects that pretty quickly. And that's the conclusion there, right? The VM really adds a bunch of latency and you start to see that with these sequential things. And it cuts our performance down from about 275 megabytes per second to 195 megabytes per second, just that added latency. You can see on the, it's my right, but I guess it's also your right. So the hypervisor doesn't quite have that. It pushes the knee out a little bit. So those tests were actually run right on the hypervisor on the same volumes. I'd shut the VM down and then I actually jump into the stream there and run the tests from the hypervisor. And similarly for sequential writes, again we got another lump in here. I don't know, somebody else will blame that on Chad who probably ran something in there, but a little less so, right? The writes are, the latency is buffered a little better here somehow through the stack. We queue it up and therefore we're not so dependent on getting the stuff from storage and through the network. But again, you see that the knee, and in this one there is a bit of a knee here on the VM, basically I have the data for the tuned VM but it didn't change that much. So I found a place to put my conclusions in the slides there. So that's my look at what happens with the hypervisor in the VM when we tune it. This was not messing with the network and some of the iSCSI session stuff, so that's where I'm gonna turn it over to David here because he looked at a little bit of that so I guess I'll let you bring up the next slide and then I'll come back up and talk about what happens when we scale this out to a lot of VMs. So, David. So we've seen a couple different environments here for some LVM testing with John and then some testing in his lab that Ed did. As Ed mentioned, I work in the customer proof and concept labs in North Carolina for NetApp. We set up multiple environments for testing for demonstration, that sort of thing. This was conducted on a certain environment that is actually our first one with OpenStack. So this is something my group's getting to know and I'm kind of spearheading that. But anyway, we use that for demoing, benchmarking. It's running OpenStack Liberty and it is using Mirantis Fuel 8.0. So I think I'm down a level from what you were doing there. That's because it's stable and that's what I wanted to stick with for now. We're using nodes Ubuntu 14.04 and notably IO equals thread. So some of the previous testing, especially with new versions of Sentos and Ubuntu and so forth, we're using native IO. But this is using threads just to let you know that. So this configuration is one controller and seven compute and the networking on each servers, we used one one gigi interface for Ethernet for management and two 10 gigi trunk using LACP per server. So this is how we got data in and out of the servers. Sender storage is configured as a multi backend and I got a list there of all the different storage backends configured, notably solidifiers. So that's what we were testing. Hardware looks like this diagram, mainly for solid fire testing for nodes and you can see some of the other gear there all connected to a Cisco switch. And we have eight IBM X3550M3 servers. So a little bit older servers and those will become relevant here in a minute. So what I was interested in finding out here with our test scenarios is to find out what happens when I increase the default of one for iSCSI sessions for open iSCSI on each of the initiators upwards. So my reason for this is I wanted to cause storage contention. So I wanted to see what would happen. I want to push the storage as hard as I can of course for a demonstration reasons. So the configuration has 30 VMs total and each one has a 10 gigabyte solid fire volume attached and on those compute nodes you might notice there were seven, we had one out of service right now. So I had six to use and five VMs on each of six. So they're evenly spread out. And the default host and VM storage settings applied for this test because I just wanted to find out what happened when I increased iSCSI sessions. The benchmark used here is VDBench. I used the benchmark load generator tool and call it four corners. We saw some results in ads where we have sequential reads, sequential writes and then random reads, random writes. And then the block sizes that accompany those. So my test here went from one to four iSCSI sessions. And then scenario two which I'll also show you was not for storage contention but I wanted to contrast it with one that did have contention. So I just had one VM and one volume on a single host. So the results I got for those one to four iSCSI sessions are graphed here. And as you can see, they're not super different but something to keep in mind if you can see those numbers and yeah, I think you might be able to see them there. Pretty high numbers. So the random read is the one I want to point out because we were seeing about 200,000 random read IOPS from the system at this point but with that increase to four sessions that gained us about 15% which you might think that's not a lot but that took us up to about 230,000 IOPS with random reads. Little bit less deterministic results with the other three but it definitely did have an effect. So this is something that I may investigate further increasing the sessions for instance two as many as eight on it. And then test scenario two this was with the single VM where we weren't continuing for storage bandwidth. So really not much difference there and things kind of bouncing around for sequential writes but got a little improvement for random read and a little decrease for random write. So in a situation where there is contention the storage is being heavily loaded I would tend to want to use more iSCSI sessions maybe four or more where I don't have that kind of contention it doesn't really seem to make a lot of difference. Okay part two of this got a couple other test scenarios. The first one is a back in QOS test and this is where I wanted to find out how can I keep one volume up in its IOPS how I kind of keep the response time down how I can get the IOPS up while I have some contending volumes. So here I used a nine to one ratio so total of 10 VMs and each one had a 10 gigabyte solid fire volume attached and all those VMs are on one compute host so this is actually over provisioned somewhat not a huge server. And here this time now I use the optimized settings that Ed mentioned a few slides back so using all these different settings trying to get the most I can out of the host and the VMs and so the scenario I used was to leave what we call the slave one VM unlimited so it could run as fast as it wanted and this could be say an important application a database for instance the other slaves two through 10 were throttled to 4,000 IOPS this 4,000 4K IOPS so I wanted to find out what was that one VM doing I think the pig we call it sometimes and the other one's piglets when all of this was under heavy load so same test run exact same four corner test at the mass rate and here I vary the I SCSI session counts and I say averaged here because I want to tell you this up front that in this case when I was on one host I didn't actually see a difference between using one to eight I SCSI session counts fact they're almost identical so I use that as additional test data and averaged all those for the numbers I'll show you here. Second scenario was using front end QOS so I'm going to save the interesting part here for a couple slides but I switched to using the front end sender QOS type okay so just FYI the type specs I use for solid fire if you've ever worked with solid fire on the back end these are what the QOS specs look like and here I set minimum maximum and burst IOPS so you'll notice the max and the burst are the same number actually and this is so I could try to get an apples to apples comparison with front end QOS so this means I had a really pretty much a hard cap at 4,000 IOPS similarly on the front end I set total IOPS to 4,000 per second and total bytes per second to about 16 megabytes which reflects a 4k block size okay so it's kind of one thing that was kind of unexpected and someone might know more about this than I do for sure but something that surprised me because I worked mostly with solid fire in the back end is that when I used front end QOS and I wanted to do volume retyping like to set up for another part of the test I got this error message so I couldn't change the front end QOS settings either to actually change the numbers or to reset it, remove it or add it so what I had to do simply is to detach the volume set them and reattach the volume which was fine for what I was doing because I was testing raw block devices but for instance if this would have been a mounted file system that would have been somewhat more complicated so something to keep in mind it's not required for back end QOS I can set those on the fly so that was kind of cool but I thought I'd mention it okay so under the results this is random read IOPS versus QOS type so looking at front end versus back end and as I mentioned I had a total of 10 VMs running and I just included that on the left hand side there for your information so we were seeing about just under 45,000 IOPS with the front end type and a little higher than that with the back end but what I want to focus on is the middle graph there with slave one so this is the IOPS figure and I can't really see the number there exactly but that was about 35% more so we're seeing maybe 8,500, 8,600 with front end and more than 11,000 with the back end so I got higher performance there of that one VM and volume that I wanted to perform well slave two and 10 were even as you would expect the throttling worked as it should so those were sitting right at 4,000 IOPS for those okay so you remember the IOPS we had more performance on that one high priority VM and volume now we look at latency in the same scenario I'll say the best for the latency of the slaves there's about 17 milliseconds almost exactly the same so they were being throttled their latency was higher because they were queuing but again we see the QoS type here for front end I had about 35% higher latency than I did for the back end and I think that they were both 35% just a coincidence but so I enjoyed low latency for random reads as well as higher IOPS on that high priority one okay and last to try to get these all came in one graph but the sequential IO latency looking in the same test was even more markedly different for the slave one for both reads and writes it was very even and very low for both of them so that's a good news but you remember it was a different scenario for random IO on that but when we look at the slaves it looks like there was a lot more queuing on the host so a big difference there and for sequential reads I think that was about 67% higher latency using the front end QoS and actually about the same for the right so very different scenario so the net of this is what the back end I was able to enjoy lower latency especially from the throttled instances there and so hope this is useful to you and I'll turn this back to Ed. Okay cool so I got one slide left here and then we'll talk to some conclusions but this is kind of a pretty interesting slide so what I did was I set up my scenario I set up 32 VMs tuned them all and pushed them all the one hypervisor so I had four I pushed them all the one then I ran my tests, all those tests lots of graphs and then I started moving the VMs apart so I took the same 32 VMs and live migrated half of them over to the next hypervisor and then I ran the test again and then I ran, I took half of off of each and put them to four so I'm gonna throw this up here as the one example of this this actually happens to be random reads but they all look fairly similar to this so I didn't wanna death by too many graphs and slides but interestingly enough we look at this, the one hypervisor with 32 VMs and you can see that the latency goes up pretty quickly so this is kind of as we would expect as we spread this out on the hypervisors our latency knee goes out the interesting part is the maximum performance of all of these and these are all scaled the same so if you're looking at these I made sure that they were all scaled the same our maximum performance when we get to two and four is the same but our latency is much higher so that contention on the hypervisor that's its effect is to really add latency so the moral of the story is spread your stuff out and that's kind of, I guess I got two slides here Nova VM layouts are important and that's what it's going for there Nova does some weird things so spread the stuff out is the moral of the story but when you do things like I put five I have five hypervisors and five VMs you would think that Nova would put one on each hypervisor nah, it puts two on one hypervisor and none on another and oh by the way I had one off VM and it put these two on the same node with the one that was off so one hypervisor had nothing and one hypervisor had three VMs one of which was turned off when I do the 32 VMs on five hyper I had five hypervisors and then somebody stole one, John which is why I did it with four because that actually made the math easier but I was looking at this kind of the layout stuff and so 32 VMs on five hyper and I got one hypervisor had two VMs one hypervisor had 12 VMs and yeah, so that's why I started off I'm like oh this is just gonna work it's just gonna put these 32 VMs out there they're gonna be evenly divided not the case so I wound up doing a lot of scripting to move these back and forth live migration so I think if there's Nova guys in the audience we might wanna look at kind of why this is the case I didn't mess with the schedulers I didn't do anything all my hypervisors had exactly the same RAM and CPU in them so something to kind of normalize that the other thing that from those pre yeah, okay, see he's suggesting the RAM waiter changes how that if I hear and you're correct from here they all should have been set the same right, these are the other identical machines with one controller oh right, okay so you do group them yeah, okay so there's some more investigation to be done there's a lot of investigation I think that's what all three of us have we spent a lot of time working on this stuff and there's more investigation but also we might wanna look at how do we look at what the volume type is is part of the how do we do the Nova schedule I don't know, that's tough right because we're cross-projecting but how do you get take if this is a high performance volume and take that into consideration and sort of where you put that with when you lay out the VMs something to think about and then there was a Veritas v. Brown bag about residual IOPS scheduler I didn't happen to go to what David did and he said we should reference that because I guess they had a pretty interesting way to kind of get some feedback about what was actually happening in each hypervisor and then feed that back into the scheduler right, get that right so and with that, we're gonna hit some conclusions here we'll take some more questions they didn't want us to ask to go to the mic so yeah, I'm gonna tell you that but so let me talk about the conclusions and bring these guys up here a little bit basically we think OpenStack works pretty well a lot of the stuff that's happened here in the recent versions has made it the defaults are pretty good there's some tuning to do if you're trying to eke out every bit and do the tuning but it doesn't help that much and especially if you're using a lot of little workloads the multi-session I SCSI stuff is interesting and I think John and I have talked about how do we maybe incorporate that into there back in QoS is more fair to mix workloads and spread out your VMs as best you can so I think that was it and you guys got anything else you wanna add into there? No, I steal your stuff and I'll keep doing that okay, all right, with that can I ask you to just tell if we can hear you, we'll repeat it and we'll get it on video so well, so that was kind of a misnomer I think we should have said storage performance yes, we did we did look at some of that and John and I are gonna have a conversation in the book that part's great we're gonna have a conversation over a beer tonight if we're still awake after we spent so I've actually been working on that a little bit lately especially the past couple of weeks because there's something not right something really not right I don't know what it is yet I haven't figured out but something particularly at the API and it looks like it's something going out to Keystone but I'm not sure but I am aware of I think I'm aware of exactly what you're talking about but I don't know what it is yes, I think we all saw that and that's why we started kind of looking through it a little bit here so that'll be next summer we'll talk about that sure, yes yeah, that was one of the tuning things that we kind of looked at there's a lot of information out on the web like if you Google this stuff and you start like you wind up reading a ton of documents and some of it's older stuff that's not that's one of the big things I was finding was that oh, I read this recommendation okay, I gotta go check that and make sure I'm doing that and a lot of the modern the Senthos 7.2 really the guest's image really fixed a lot of that fixed it in the sense that it made it what the recommendations out there were and so you don't have to go mess with it to get it right 6.7 was a little different on that note so the Vert.io SCSI is the default in OpenStack and that's what it uses for pretty much everything except for Ceph which uses the RBD stuff and then of course NFS one thing that has come up in the past is people have talked about using the Libvert and KVM iSCSI stuff and going direct that way and using Libvert as the iSCSI Attacher or the initiator there's a lot of foot out there folks under the impression that that gives you better performance and stuff like that it actually is almost exactly the same there is almost zero difference between the two it's easier to manage and easier to maintain because it's built into your Libvert version and you don't have to worry about it doesn't work the same on Ubuntu or it doesn't work on you know, Senthos or Red Hat or whatever the same way and the Nova guys have to have all this code to make sure that it works right so that's the only advantage it's not a performance thing it's a maintenance thing and if you're gonna do your favorite fiber channel then you gotta have yet a different way to do it fiber channel is dead to me I could add to that too there was a session yesterday I went to that was for QMU and KVM IO did you did that one? I know I recognize you so there we go so you had some great data there okay so did you actually get you set that up and ran the whole iSCSI Libvert stuff so that was one of the things we thought we were gonna talk more about that too but I didn't put the data in there because it wasn't very clean I had a really really difficult time getting it to run it turns out you can't you cannot run it in open stack right now period so what I ended up doing was my own KVM box on my workstation and I did it all manually and set it up and everything and I got to work that way but I saw absolutely no difference whatsoever right I think we're at our time we'll be around here if you wanna if you wanna come up and beat us up some more that's cool we're good thank you