 All right, I think it's a little bit past 5.30. Let's get started Welcome folks to block stories consideration for open stack. We're gonna share some tips and tricks Thanks for attending. I believe this is the last session of the day. So we are officially between you and happy hour So we'll try to do our very best to entertain you for the next 40 minutes or so And if you have any questions, I want to share some thoughts your experiences in the field dealing with open stack Feel free to ask and interrupt us anytime. We welcome questions feedbacks keep it interactive And since this is the last session of the day, we're gonna do something a little bit different So instead instead of having us present, we're gonna have you introduce yourself each and every one of you starting from the back So once we get done is about midnight so we can all go home We're not gonna do that or joking aside, but I do want to understand who we are presenting to So just a simple race of hand how many of you in the audience are responsible for building a cloud back by open stack Or helping others to build a cloud Well quite a lot of you good and then how many of you work for technology vendors doesn't have to be storage that Build technologies around open stack Okay, so we have a pretty good half-and-half Ratio, so that's great and essentially the reason why we're here today myself and Jay It's really to share our experiences dealing with customers and prospects and reseller partners out in the field There's rising interest as you all know on using open stack and fit open stack into the future IT Vision and strategy so we get a lot of questions from our customers and partners about how to go about Leveraging what they have in their infrastructure and build a solid cloud for other people to consume Let it be internal deaf and test organizations or external customers meaning service providers Okay, so a little bit about ourselves. My name is when you I work as a solutions architect and nimble storage And don't worry. This is not a marketing presentation about nimble storage We will not talk about nimble storage at all just referring to it by the name and the methods and practices that we give out here By the most part vendor neutral and we'll definitely call out the ones that are specific to us Okay, and I work out in the field dealing with customers on the almost day-to-day basis So I hear a lot of challenges that they come across Specifically dealing with open stack. So I'm going to share a lot of that as well as some Practical solutions that we have come up with that have helped customers Okay, and with me today is my partner in crime Jay Wang from our engineering team And Jay you want to quickly introduce yourself? I work at nimble for as a software engineer and Usually with quaint with when on the cloud area and storage are to get our decay in the claw area Yep, so thanks Jay and my job is to figure out the what and a why what problems we can solve and Jay is the smart guy behind the Scene working on the how okay, so three things we want to talk about the first thing is What is happening out in a field with customers thinking about or deploying open stack? Okay, so we split that into two personas that we deal with on the day-to-day basis and then go over the problems and top of mind Things for each of these personas followed by practical solutions leveraging Block storage for open stack and lastly we do want to share some ideas About what we can do better in the future to make open stack sender a better project overall in the longer term, okay? Now when we talk to customers day in and day out there are two types of people we deal with the first type is The most common and these are VMware customers They have when we're ESX we send a server and other switch of product in their data center serving Mostly puppies pet type workload. Do you all know the analogy between cattle and puppy? Most of you okay So these are mostly puppies that they serve in the organization let it be exchange SQL SharePoint file serving and whatnot and other custom applications that support their business critical applications, okay? And a lot of these admins, they know a lot about VMware. They can operate ESX. They can install ESX They can patch ESX you name it they know a lot about windows because most of their workloads are windows and these are not your typical DevOps guys that know chef and puppet or a coplar. They know a thing or two about PowerShell, but To make themselves more competitive in this cloud landscape. They are thinking about open stack. So this is not really your normal top-down approach where the CIO or CTO gave direction to these folks to think about open stack These are folks that are thinking about open stack Themselves to see if they could add value to the organization by having a self-service portal like horizon for developer and test team to orchestrate the provisioning of virtual machines, okay? So we get a lot of questions from these folks about oh, so how do I get started with open stack? We try that stack is great. It's not really doing much of real workload. What can we do next and Keep in mind these most of these folks don't really have an IT budget to go and acquire some commodity servers To kickstart their open stack deployment No, do they have the budget to allocate for system integrators to come in and help them customize their cloud like other larger companies? Okay, so we actually Propose a solution for them to leverage what they have in their environment and get started with open stack So I'll go with that in a little bit and the second set of persona And so the top of my things for these folks are can I deploy open stack with a easy button and Can I roll back to a specific point in time if I screw up? All right, keep in mind these folks may not have a lot of experience with CLIs and config files in the comms No.com right, so they have to muck around with stuff day in and day out And they do want to capture a specific point in time where everything is working fine and have the ability to go back Should they have to okay, and second set of persona is the more experienced folks All right, these are more like architects. They come from DevOps background. They know chef public cobbler All right, they use cobbler in conjunction with puppet for example doing instantiate environments puts out applications And they support a lot of the cattle type workloads in conjunction with the puppies right the applications that are born in the cloud and More or less stateless in nature. Okay, and these folks have both the JBOTS locals JBOTS attached to service that they acquire as well as Santa rays So we do get a lot of questions. Oh, should I junk the local JBOTS and not use them? Absolutely not Why not you have them? Why not use them and have a tier model? Okay, so we'll go over some of the key questions that they have in mind and of course these folks The top of mind thing for them is To prevent people from going to AWS right In other words offer better service than elastic block storage Okay, so let's talk about how these folks go about architecting the environment to compete with Amazon All right starting with the easy one, right? These are the easy guys to deal with right these guys know vSphere day in and day out and they have a sand that they purchase to run ESX with VMBuffers volume. All right, a lot of you know, how many of you know ESX? All right, so I'm not speaking to an audience that don't know ESX. That's good. All right So what we propose to them is instead of starting with a workstation dev stack VM that doesn't really run anything useful Why not just leverage your deaf and test environment your ESX cluster that is serving deaf and test workload? Back by sheer storage a lot of times when we propose this People said oh, yeah I do have some resource pools that I can create in my ESX cluster to house a few VMs that run as controllers As you might know if you don't have like thousands of virtual machine incidences You don't need a whole lot of CPU or memory resources right controllers are pretty lightweight in open stack And also they have some headroom as well as capacity in their sheer storage in this case nimble And what we present to them is really you can carve out a sub resource pool under your ESX cluster And then use you can deploy a virtual machine a lot of these guys know how to Click and create a new VM and place the VM DK on the VMF as volumes it's pretty straightforward Install send OS and what we typically recommend is Red Hat OpenStack audio you guys use audio before Okay, it's pretty good tool to get started and just a three command You get the RPM for open stack you get pack stack and then you do a pack stack all in one So installs all the services that are needed for your open stack cloud Meaning horizon Nova sender volume sender scheduler API Solometer you name it everything in one VM. Okay, and the next question you might ask is oh, how the hell do I run VMs? Within ESX VM, right? So it's nested virtualization and as you might know the popular choice for open stack hypervisors KVM And a lot of questions come up is oh, so I have the centos VM great How do I run KVM inside of it so other instances can run on top of it? So it's pretty straightforward very easy to do ESX.conf file. You just add the command Option call VHV equals enable. Don't worry about taking notes We have it in the notes for this slide if you can't wait to get it We can send you a copy Okay, and then in the virtual machine itself the centos You can just add couple of command options to control the behavior of the virtual machine monitor and off you go So you can have a nova service running inside of a virtual machine inside ESX It's like the inception of virtualization All right, and if you have a beefy enough box this actually works pretty well You can have you know tens or 20s of virtual machines running now you get open stack running You you're ready to show your manager the horizon view of things working where people can use self service orchestration The next thing to do is you know play around with settings and deploy additional nodes nova nodes or send a volume nodes And if you happen to screw up Let's be honest here I screw up many times when I install open stack when I play with advanced parameters I cause nova and sender to have segmentation faults where I can't log into horizon So what do we do then? It's pretty straightforward if you have some capacity in your santa ray Make sure these controller nodes and the mysql database runs inside a VMFS volume It could be a same volume or separate volumes The key thing to keep in mind is you want a point of consistency across both the controllers as well as the Mysql database so different sand vendors call it differently So for us we call the volume collection some of them call it a consistency group What that really means is when you initiate a snapshot from the sand it captures all of the volumes should you have separate volumes? Okay, so take snapshots every point of the way right snapshot when you after you install the centos vm or Ubuntu, okay, take a snapshot after you did the pack stack installation of all in one All right, take a snapshot before you mug around with your cylinder calm very helpful and to recover very straightforward What do you do you power off the controller vm? And then you go into the sand a lot of sand vendors nowadays have these in the plug-in in vSphere already so you simply go into the plug-in or the sand management interface work to a snapshot usually this takes the volume offline And then you bring the volume back online you go into ESX you re-scan and you power on your controller again You're good to go And you can even try to a rm-rf on the controller and see what happens and revert back to operational state So this has helped a lot of our customers tremendously when they want to think about open stack So a question to you all is is this relevant to you and your end users? Do you think they might find it useful? Okay, cool, and really if you get more advanced in this setup You can easily expand out to run KVM on physical boxes by changing the answer dot text file and Deploy another Nova node so you don't have to have this nested virtualization thing that is totally unsupported Okay Any questions so far? Okay, cool. Let's move on. So this is the easier one to deal with when we come across these type of questions now Let's get into the advanced users. These are namely service providers These are bigger organizations that have some DevOps presence So they do have some architects that understand DevOps. So a lot of times what these guys think about is how do I have? The best of both worlds using both local JBots as well as my share storage that I pay a bit more money for compared to The JBots on the servers, how do I add differentiated value for my consumers? Let it be internal depth and test environments or external consumers of my cloud that is backed by open stack and Of course go silver bronze service tearing in these environments What we often get asked is number one. How do I leverage both? local JBot as well as share storage Answer is very simple from a high level, but we'll dive into the weeds Okay, don't worry is to leverage the multi back-end sender feature that was introduced back in grizzly release And the second one is traffic isolation for bigger jobs They have the storage array dissected in a few ways with a to that one QV land trunking So they do want to dictate the deaf team going to this specific interface so they can have a higher level quality of service as well as metering and measurement of who uses what when and how much and They do want to have the ability to separate out Who has access to what interface on the storage controller going from Nova? Okay, and the next thing is can one add additional services in addition to a number of IOPS I can have Some capacity some performance. Can I do more at the instance level? So people feel good about paying for premium highest level of service All right, this is something that Amazon doesn't have EBS doesn't have any sort of multi-packing. That could be wrong They might have added it already But this is what people can do and we'll show you how and last but not least is this is a storage off-load operation Are you guys familiar with the restores API for a rate integration VAI? Okay, some of you so the ability to offload Certain operations that are pretty efficient from storage perspective, but as a storage meaning share storage Okay, and were you all in the previous session where the folk folks from eBay and PayPal? So we'll compliment that they talk a lot about the benefits as well as the what so we're going to show you the how okay So the first three ask could be easily complimented where they multi-sinder back in With the next one enable through glance booth from wall in clone from snapshot We'll show you a screenshot of what to do when you have this implementation now with a reference model like this, this is Exactly what one of our service provider customers has done in their open stack cloud Unfortunately, I don't have permission to share the name once I do you can expect a blueprint from us on how exactly they did it Okay, so these folks they have a Model in which they order servers the x86 servers with local JBOTS not a whole lot of them but enough capacity not to put to waste and They do have a share storage ray from us So just the two types of storage here and you could certainly create three tiers of offering here One of which is to have LVM on top of a VG Created based on the local JBOT. You don't have to have ray configuration if that's your lowest tier So with people that use this type of storage day most of them times have chef and puppet to instantiate the VM When they go down it the server die You lose the local JBOT then you could certainly instantiate it in automated fashion So snapshots and replication those are not very important in this in this tier of service at least Okay, and it's pretty native built in you could easily configure it and we'll show you example as well and the second tier up is We often get asked so I can I can use JBOT to back a LVM through center Can I use a share storage volume from the array and have LVM back in it? Absolutely very simple to do and you'll see in the example that we'll show you and last but not least the highest tier of offering is the per Instance level mapping each instance could boot from a center volume from the array and then these snapshots and cloning operations are all Offloaded to the array so no more copying like Amazon EBS You take a snapshot you copy to s3 and God knows when you can restore that thing over the network that has no guarantee So this is very different is offloaded to the array some storage vendors have pretty efficient algorithms to do redirect on write for example instead of Copying the data block by block like some vendors do We can copy the data just by manipulating the pointer of the metadata for these volumes. Okay, and Some benefits of this model is the lowest here is cost effective You can have the lowest tier of offering to attract customers to come in the next tier up is you have higher performance nowadays with most storage vendors Legacy or not they have some flash optimization Okay, so typically you get better performance out of the sheer storage here And you can configure multi-packing at this layer So it's transparent to the instances if you want to provide more value add to the cloud Users then you could certainly advertise this as oh and in addition to more capacity Faster performance you get multi-packing for your instances if you use this tier And last but not least the highest tier offering you have multi-packing at the instance level You have traffic isolation like I mentioned based on what subnet of what VLAN is Orchestrated by each and every instance and last but not least with this you can have snapshot and clone offload through glance Having said all of that this is an example of how you would go about doing it So this is what the sinner doc coms look like and Jay reminded me earlier. I actually exposed the Common password that we use in our environment So I changed it before the presentation. Thanks Jay. What's oh, yeah, I'll take out that little gray box I'll move it out so you can copy it Good guess You're right Okay, so so at the top is it's pretty straightforward The the option enable backends is already built-in in sin comm so you don't have to write it You know just uncomment that out and then define the different Backends that you have and whatever back and you put up here. It better match with the bracket down here Otherwise, you're gonna blow up, right? You won't be able to get anything done. So it within each bracket. You just define for LVM local JBOT LVM Just match up the name of the volume group that you created based on your JBOT attached to servers Okay, and the driver of course is the LVM ice-cazi driver that's built into open stack And the backend name is you want to give it a meaningful name that matches what you have up there as back ends okay, and this is the configuration parameter for our sinner driver if you work with a sandmander that Integrates with open stack through sender. You're gonna get this information from them. Okay, each one is unique and Then make sure you point to the driver All right, then you should match and then last but not least. So this is what we did with a VG backed by a shear storage volume you just present the volume to your Cinder volume note and Then you create a meaningful name so you know what it is and then you just put the name up here That's it and then define it back in for it. Okay So far so good Good question. So the question was what if I want to do service level offering based on the storage back end meaning How many IOPS I can guarantee from the storage back end that is one of the methods in which I've seen people implement and For us we are building the foundation in which we could apply services in the storage layer So that is one model that could work meaning the simple fashion of so what's my number of shares similar to DRS in vSphere You can have this number of IOPS. This is your ceiling. This is the minimum This is the guarantee and this is the burst so that that model works as well So yeah by all means if that's what you want to do certainly you can do that But this is really to leverage the best of both worlds to have local tree bots as well as shares storage in the back end Great question. Any other questions? Okay, so you know all know how to do this, right and then feel free to copy and paste this thing one thing I ran into is I'm not sure if folks in the audience have run into this before is I started out with the open stack deployment Deploy my controller knows got my sender volume server up and running. I just Didn't bother with types. Alright, so I just started adding my my nimble driver under the default Tap and it started going wild and creating all these instances from volume and stuff like that And then later on I figure oh might as well start with the tier model. Let me try some LVM stuff and guess what? It didn't work So sender service cannot start. I'm not sure if that's a bug So my recommendation to you is always start with a type even though you only have one type so in the future You want to add type you can just expand this Back in and add some bracket that represents the other type that you want to define for the end users and off you go Okay, all right So that's the example. So before we get into the multi-packing advanced stuff at the KVM layer We will talk about the snapshot and clone offload through glance So if you attended the previous session, this is essentially how you do it. So you have your cloud image This could be an image from Ubuntu or Red Hat you download from the repository This could be VMDK or ISO file or VMDK that you upload into glance They it does like a block by block copy and you have the choice as the end consumer of OpenStack to Deploy the instance and boot off the instance from a sand volume. So what you have to do is in the Launch instance within horizon you basically select Instance boot source you can boot from an image Which creates a new volume? So it does a block by block copying of the image into the sand volume in which you can boot in the future okay, and Then if you have a multi back-end sender approach Implementation no problem. You could still have dictation of who deploys instances over what type of store? to do that the trick is you want to create the volume first before you allow the people to go into Instance to launch the instance because in within this tab there's no way for you to tell OpenStack what type of volume to create so that's something we would like to propose in the summit So people have flexibility without having to go into here to create the volume first and then have the Instance boot from this volume. Okay, so if you take a snapshot of this thing So the beauty of the sender driver implementation is Notice if you do a sender volume and you want to take a snapshot of the volume from the volumes tab Guess what you can't it's attached to something so you can't take a snapshot But with this approach you can you're going to the instance and you take great snapshot if you it doesn't matter whether this instance has External volumes attached or not it will stick it will take a snapshot Let's say you you have an instance booting off a sender volume from share storage You could attach another volume to the VM when you hit that snapshot create snapshot button You'll create an array based snapshot for both the volumes that are attached to the instance While the virtual machine is running Okay, so that's one advantage of using this approach and give people woman fuzzy about using your most expensive here of storage Okay, and of course the cloning of this is offloaded in our implementation We don't do block by block copying should you want to take a snapshot of an instance and deploy hundreds if not thousands of clones off of it Instead of copying the volume block by block We simply manipulate the pointer in our file system which represents that volume and give you writeable snapshots. Okay, so you can Have no data to do. Yeah, no data duplication. There's no need for D to a 400 gig volume you clone it it takes a few bytes of metadata and that's it So this is perfect use case for a different test environment where you have a two terabyte or petabyte database that you want your Developers to write code against all right when you test you want to test production level data So this makes it easier without having to copy those things Which brings another challenge that we run into and we're hoping to implement this in the future is Right now we have these volumes on the array side serving Let's say physical databases or virtualized SQL databases running on the mware There's no easy way to import that Big volume into OpenStack and have OpenStack recognize that as a Cinder volume what we want to do there's a blueprint exists in the Brute launch pad is the import export a volume meaning you take this piece of volume and change the metadata So OpenStack understands it so you don't have to do a Copy into glands and then from glands into a volume which is to exit copy which ultimately very redundant and inefficient Question to you always would that be useful if we implement this and share with the community? Okay, your hands. Okay, cool. All right, and that's pretty much it in terms of snapshot offload clones etc and now moving on to The more fun stuff which is multi-packing and How many of you want to hear about the multi-packing at the ESX layer? Okay, so we spend a little bit of time. I saw a few hands back there In the ESX layer should you use the Nova driver for vCenter and you have a dedicated ESX cluster Very straightforward if you use a storage array that is certified with VMware. Hopefully you are Chances are it works with the native multi-packing Plug-in within the VM kernel storage stack NMP So they have two plugins one is called SAPE storage array type plug-in and the other is called PSP Path selection plug-in So both work hand-in-hand one is responsible for monitoring the health of all the available paths you have for your environment The other is really making sure to pick the best available path to issue IO So if you use a Nova driver for vCenter with ESX you essentially dedicate your ESX cluster as the Nova Note meaning your virtual machine instances will run on ESX So you inherit the benefits of HA, DRS, VMotion all of those things will work and the multi-packing is underneath the cover Okay So that's pretty much it you want to make sure you have a storage array that supports ESX And if you use iSCSI and you want to make sure you bind the software iSCSI initiator with all of the available interfaces that you dedicate for VM kernel iSCSI traffic Pretty easy to do and now on the KVM side and we'll have my smarter half J to explain the behind-the-scenes on how to do multi-packing and OpenStack with KVM Take it away J so When briefly mentioned about how open how ESX is done with multi-packing So the question is actually why we need multi-packing at all First is performance, right? So let's say you have One gig link on your host or on your array and basically you saturate the link then multi-packing actually can provide you Better performance because you can create another path to another link on the storage or on the host So definitely will go up by performance and in addition to that if you have let's say you have 10 gig link I never saturated the link. Why any multi-packing and then there's a iSCSI If you're using open iSCSI the default iSCSI session has 32 Q-debt limits. So if you create create a modeling actually you increase your performance and let's say you don't have Application that need that much performance why I need multi-packing But maybe you don't have it now, but there may be a burst or I will request burst Or a peer of time you need high I'll and that actually can help you a lot and another main reason Is also low balancing right? multi-packing in SCSI layer you actually have low-backing algorithm per lawn you have you can use run-off or Q-debt different kind of low-balancing algorithm and not a Obvious reason is redundancy, right? If you only have one path to your array and that path is down then basically you lost connection to array So multi-packing is very useful in this case So when mentioned about how we get done yet on ESX is basically you have to enable There's a default multi-packing Depends on your type your SAP type and then you assign a default PSP for you and you can change to default from default to the type or the parking that you want to use and for LVM if you configure LVM back-end then basically your multi-packing is configured with your Multi-packing device. Let's say you're using Linux or a centOS then you have to install multi-packing package and when you Create do PV create then you have to make sure that you create the device on the multi-pack device and If you're using single share storage and you have to enable Nova, so there's a multi-path Configuration in Nova.com we have to make it to true So Nova when you attach a volume to a VM Nova will automatically create session to the target Depends on how many target that back-end return. So there are different storage has different Design some storage return all their data IPs when it return all their data IPs Nova actually will make individual connection to all the target IPs So you have multi-packing, but if vendor only return one data IP We should virtual data IP and then they redirect in the back Then you have to do a little bit tweaking either you set the NR sessions to create default sessions for the target or we have to manually manually create some sessions and This is how to we have more details steps on how to enable multi-packing, but This is just Broad high-level steps first you have to install device mapper multi-path on centos red hat and for Ubuntu You install multi-path tools after that each vendor it has its own recommendation on how you configure multi-packing then you have to copy and paste that recommendation to multi-path.com and In Nova you have to enable this lip or SCSI, SCSI use multi-path to true and After that when you attach a volume from cinder to a VN or an instance you'll see multi-path actually get created and and by default you use all the path and Actually depends on how you set up multi-path.com if you use wrong Robin then or this QDAP or Algorithm and then you'll start using all the path and Based on the algorithm We have the step-by-step down with the notes So if you get a copy of the slide feel free to try it out and the J mentioned the NL sessions that's useful if you have a An ice-cazi array that returns a single ice-cazi discovery target Yes, so you can modify the number of sessions and connections you want as opposed to the other Implementations where they return every possible discovery target Something to keep in mind depending on what ice-cazi solution you have be sure to check with your vendor of choice on that one Yeah Good next topic we want to talk about is backup and restore and today One options you've wanted to do backup is from cinder and you back up to Swift or s3 and then it's basically a whole volume backup and when you back up you have to consider compression Ddupe encryption and maybe incremental backup and That probably will be controlled by if you have a backup software and they can do incremental for you and then maybe with two compression Ddupe encryption for you or you're using s3 and actually they have some Capability build-in, but Definitely is one way to do it But restore can take time and maybe costly if you're using s3 and if you also you have to consider the cost of backup software So one way using your existing share storage is that you can simply use snapshot and replication for your backup So you can take as many sketch snapshots as you want and you can actually set up schedules on your snapshots or or your volume right or your consistency grouping away and Once you set up a schedule that actually do the backup for you And if you're back in storage provide replication and you can also set up replication so not only you have your backup strategy or you also have your dr and That's also helpful if let's say you have multi-side development and up right and then you do 24 hours development you develop in sales a In the morning and then you can transfer the data by replication to Hong Kong for example far for the rest of the time for development. So that's another use case to use to take advantage of your share share storage and For restore and scale. It's also pretty easy in a share storage. If there's because there's still they're already Technology building for restore is simply from a snapshot. You make a clone and you can install or you can simply revert You're volume from your snapshot. So recovery is also very fast and straightforward That's how you can also use your back-end storage technology to do your backup and restore I'll have when to address some ideas for the future. Okay. All right. Thanks Jay Well before we get into future ideas, I want to make sure we can address all your questions And I know we are right at the dot 6 10 want to make sure we can address any questions burning questions You might have feel free to ask Mm-hmm. Yeah, I'll make sure I understand your question So you wanted us to address the use case where you have VMA fast data store and the usage of rdm in this Yeah, right Yeah, that's an interesting question now in regards to rdm. I don't believe correct me if I'm wrong I don't believe the current VM DK driver from VMware works with rdm I think it's mainly VM DK's that they create so for rdm's. I I'm not sure how you would manipulate that Okay, oh, okay, so that works Okay, that's good to know so if you present an rdm directly to an instance so apparently that works as well So that's something we should give it a try and another similar use case is a lot of people do a direct ice-caze Mounting within the virtual machine instance so that works in open stack as well You just want to make sure you have the right software ice-caze initiator You present the target and you want to add that IQ end to your storage array So it knows that instance wants to directly mount your volume So that works as well, but it's really the puppet puppy approach I'm sorry puppy approach where you have to take extra care of this guy to make sure you mount the right volume And it could be useful if you know how to automate at the virtual machine layer Great question. Any other ones? Right Okay, good question. The question was when we talk about the Cine driver with volume in the storage back end can multiple instances mount the exact same volume like ESX does with BMFS answers. No is really one-to-one mapping I believe there's a blueprint talking about multi attach Where you can have multiple instances attaching the same on the line physical volume. That's something to look out for Good question. So do we support live migration? We So let me phrase it that way. So if you have a way to have the Shand stores volume share across Instances within KVM that yes, but we haven't tried that myself Honestly, the people that we deal with their SLA is really not five-nines at this point for deaf and test So they could suffer an outage on the Cinder volume note and leverage cobbler and puppet to instantiate another Cinder volume instance Present the Cinder LVM behind the scene and off they go great question and I think you had a follow-on question No, no, okay. Yeah, go ahead Go ahead. No, it's a king. What was the main use case to have share Storage, I know some clustering environment you may have a requirement for the share Okay, so the use case if I understand correctly is you have a database instance where you want multiple machines to have different databases on the same instance Is that correct? Oh, okay, very much like the BMF has tomorrow Okay, yeah, good point, I think right now the landscape is to to get HA for share storage, let's say through LVM or Through this model with the Cinder driver You would have to do like pacemaker and also some configuration on so the other know in the head for to know configuration Is it bit complicated so if we can inherit this with then Cinder it would make life a lot easier Would you agree? All right, any other questions Well, hopefully this is useful and the future ideas we prior presented in the Summit the designs of it. So if you have questions that you want to talk to us about feel free to come up. Thank you