 All right, he's just oh, there we go. We're live all right Welcome Pax crowd. There's a couple seats up here if you guys want to sit What happened there? All right, you're gonna fix that cool So I'm at bald off with solid fire now net app This session is how stuff works. We're gonna talk about live migration in the context of storage and and then replication for Cinder so I'm gonna start by introducing Alex and he's gonna talk about the live migration piece and then I'll come along and talk about replication Sure, so I'm Alex Mead. I've been working on opensack for about five years I'm at net app now focusing mostly on Cinder and Manila And I wanted to talk about live migration specifically live migration of instances that are backed by Cinder volumes And what that means is if I have an instance running on a compute node I want to get that instance onto a different compute node with no or little downtime Ideally the guest operating system would have no idea. I even did a migration So there are three types of migrations There's block migration Which would be the situation where you have a compute node and the compute node has all the bits for the instance stored on the compute node And if you want to migrate that then obviously you have to migrate all of that information to the other compute node So you have to send all the disk block by block over the network. That can be very expensive time-consuming and the VM well the VM is paused on both Compute nodes that hold time. So it's not really true live migration The other type shared storage So this is where you have maybe an NFS share or some distributed file system mounted on both your compute nodes and the compute nodes are storing the instance information in that share So if you want to migrate the instance between you just have to move the memory You don't have to move any of the disk and then there's volume base where you've spun up your instance based off of Cinder volume and You don't have to Move those bits either because the volume can be attached to both computers at the same time So there's some false documentation out there that says that you can't do a live migration Unless you have shared storage. That's not true as I just discussed and we'll prove that wrong and I demo shortly John's clapping because that's been a myth for quite a while Here's a compatibility matrix about Win-and-win certain migrations work. We're gonna focus here on the Cinder volumes section and the bottom right there so you can see that There's a lot of red around when you have an attached read only device So one example of that is the config drive. Let me explain what that is and then I'll explain how that's a problem So the config drive is one of two ways that you can inject configuration information into a VM So let's go to the metadata service first. So the metadata service is basically here's a special IP that cloud in it or any other Agent on the VM can talk to you to know how to set up networking and the hostname, etc Config drive is just another way to do that except it is an attached CD ROM disk onto the VM and Some the reason you would use a config drive over a metadata service is maybe you're using an older version of cloud in it or some other Agent that doesn't understand the metadata service So the problem with the config drive is that it is attached to the VM by Libvert as a CD ROM type and Libvert can't migrate Instances that have CD ROMs attached to them unless you're on shared storage And I'll get into that more into the demo part So let's talk about the flow of live migration a little bit colors didn't show up, but that's all right The first two steps are owned by OpenStack. That's Nova and Cinder The three four and five are owned by the hypervisor and then it goes back to Nova and Cinder to do some cleanup And we'll go more in depth right now. So Let's say that we have two compute nodes some Storage and then some sort of storage protocol that we're using maybe ice-cozy and FS barb channel, whatever And let's say we've booted up a VM off of Off of a volume on that storage and it's connected to the compute node and the keep you mode passes it through to the VM So we have a volume backing the VM now Let's say we want to migrate that VM off of compute node a and onto some other compute node So the admin can either say I would like to migrate this on to compute node B Or it can say I would just like to migrate this and the Nova scheduler will then say Oh, here's the node you can use or it'll double-check the node you provided saying hey This node has all the memory you need the right instruction set And you know everything else that is involved in provisioning of the end So after the scheduler decides that it's okay to migrate to the host you'd like to migrate to It reserves those resources on the compute node So that no other provisioning requests come in and steal that ran that you need to do the migration And then Nova will come in and say oh, I need to connect the volume to that compute node So that both compute nodes are connected to the volume So it sets up maybe another ice-cozy session to the compute node And then the next step is it tells the hypervisor to go ahead and start a live migration And what that'll do is create the VM on compute node B In a pause state it's still running on compute a it'll then copy all the memory This is a situation where if we were doing a block migration It would have to pause both VMs and then copy all of the disk as well And it would be disruptive and very slow, but luckily we are doing a volume backed instance and It can already see the volume on both nodes, but since it's paused on compute B We don't have to worry about any IO interactions or Concurrent rights to the volume after it does that the hypervisor will pause both VMs and Copy all the dirty memory and the CPU state dirty memory being anything that changed while we were doing the initial copy You may say well, this is this looks disruptive Briefly it might be There's a configuration setting Live migration downtime and Nova it defaults to 500 milliseconds, and that's how long it'll allow The guests to be down Hopefully this is quick enough where The guest doesn't really care and you don't lose any you don't bounce any sessions or losing a package or anything Hopefully So then the hypervisor says okay, I've got everything copied over to compute B Now we can start the VM there and I can delete it off of compute a but we can see there's still Maybe an ice-cazi session to the compute a that's where Nova comes back in and says, okay, let me clean that up So it cleans it up Let's go over to the demo where I can actually show you this in action Is this bigger All right, cool, so let's start by doing a simple live migration with an instance that's backed by a sender So oh, I can't read that. Okay, so we'll create a volume We'll go ahead and do it on our solid fire back in and we'll create it from an image so that we can boot it into an instance And we'll wait for our volume to come up and once it's created We'll go ahead and boot an instance from that volume Do all the usual things you have to do with provisioning an instance And the key thing is we have config drive the force config drive turned off in here So that I didn't in the demo go to the config option, but it's not checked I'll show you later in the demo where you actually can check it and how it Can affect it so yep, and we're what we just did was we had an IP to the instance We're gonna start pinging the instance and wait for it to finish provisioning we got to go ahead and Make it so that it responds to ping traffic and we'll go ahead and look at the console to make sure it's booting up There's nothing special here. It's just booting an instance from a volume All right, so booted we can see it's pinging just fine Now let's go to the admin panel where we can start a migration And we can see that it's on our second compute node if you saw that really quickly We'll go ahead and list all of our SCSI sessions We can see that there is one on the second compute node We'll go ahead and start a live migration To move it to the first compute node Still pinging now. It's got a session on both compute nodes Now it's probably doing that copy And I'll cut over here in a second. All right, so live migration is done. We can see that there was one little miss On the ping there. I don't know if that was related, but might be but it was just one And then we finished that migration we can do this with any SCSI storage NetApp SolidFire. We're doing it here with our on tab back in over in our SCSI right now We'll do all the same things that it requires Spin up an instance from that volume and we'll do the same thing. We'll ping it We will remove that security rule so it will respond All right, and we'll wait for it to boot up. We skipped ahead a little bit now It's booted and we'll go ahead and list the connections on our Storage on the bottom left there. We can see there's one and our node For our VM is on compute node one here. We started my creation. We can see it's got two connections We can see that both on the back in there and on both compute nodes and it's done and we can see the Hosts had changed so now I'll show why config drives are bad and how that doesn't work And that counts for CD-ROMs, too So the config drives mounted as a CD, but if you had an ISO CD mounted you'd have the same problem So what kind of go through this? Oh sure reports to apparently. Yeah All right, so we're creating the instance, but this time we'll go ahead and Check the config drive option should have sped it up more I'd made the demo everybody Try to speed up the parts that aren't relevant, but I guess I could have sped it all up more anyways You need that downtime so you can collect your thoughts So we can see it booting up here. It should boot up just fine Fig drive shouldn't hurt any of that Okay, so it's booted up now. Let's go try it to migrate it put it on our first compute node And we see we get a failure, but it doesn't really say anything other than failed We'll get more into that here in a minute This is a problem even on NFS storage. It doesn't matter who you're using liver Just can't support migrating those read only devices unless we set up shared storage So we'll go ahead now and configure shared storage for our VMs or for our compute nodes So we'll create an NFS share on our own tab back in here and we've got our NFS share So what we'll do is we'll hop on our compute node. We stopped the Nova service We'll mount our NFS share And what we're doing initially here is we're going to mount it to a temporary directory Copy all the existing instance information that we already have on the compute node into the share So that it has all the information about stuff. We have already done Now unmount it So keep a key point here is that this shared stores that we're setting up right now is only for the config drive So if you don't have config drive or attach CD-ROMs, you don't need this But it's kind of to show the hybrid Approach here, so we've kind of gone through the first step was just migrating instances that use the metadata service and had volume backed Storage so now we're we're gonna Do a hybrid here, and this is also a cool trick if you're trying to test it out with DevStack You just spin it up normal, and then you go ahead and do the copy thing Yeah, you don't have to worry about weird local comp stuff or anything if you're familiar with DevStack So now we've mounted it on our first compute node And we're just mounted on the second compute node And we're just mounting it over top of the instances directory that Nova's already using We'll start our Nova services back up And that's all you have to do is set up shared storage So why don't we go ahead and prove that that works now? migrating an instance with a config drive That is backed by a sender volume. All right, so we'll launch an instance instance just like we did before Off of our solid fire volume this time. We'll check the config drive box All right So we've got our instance now, we'll go ahead and let's just take a look at our NFS share here and see that it has all the information for the instance on both compute nodes Now that'll prove that we have our shared source correctly. Yes, it exists on both You can see the config drive there is the second line. Yeah And let's go ahead and find out which We can see that it's on our second host here So let's go ahead and migrate it over So we can see our connections just like before migrate it It said green stuff this time. That's better than red stuff And now we can see it's connected on both Looks like it did the migration and it's on the new host I think that's all for the oh no, there's a little bit more Oh yes, we're going to do it again on NFS to prove that it works no matter what the Or you can kill it Yeah, this word. Yeah, there's actually LVM in this setup We we videoed it, but then we decided to take it out in the essence of time. So This part sped up. So you know something like the whole whole thing. So we got our instance. Let's migrate it And we'll wait for it to complete you can see it's migrating Oh, I did it, okay It might have you never know I don't trust him was it all anymore cool So there's a few gotchas you may have noticed When we tried to do the migration with the config drive without shared storage it aired, but all I really said was I Failed didn't say why There's a few other situations that can happen where you don't even get that message Such as if your hypervisors can't communicate Then what will happen is the migration will fail It'll go back into active state for your instance and it'll still be on the same host So there's no notification that it failed unless you're paying really close attention also In mataka, this has gotten a little bit better with upfront checks to make sure that The it thinks it can do the migration and won't even have to fail later on Now this was even worse in older versions where it just it was really frustrating because it would not move And you couldn't figure out why so There's been a lot of cleanup the more modern open-stack you're on the much better. You are yeah about new you can always cone the logs But nobody wants to have to do that right if it's an obvious issue you'd rather be told about it in Newton I've been working on a feature called user messages. It's a working title Here's an example of an issue that might occur Let's say I couldn't do the multiple attaching to both compute nodes So you could get an error like this saying hey, I couldn't I couldn't map it to both And this is more helpful than not knowing fail at all and then I can use this information such as a request ID Volume ID, etc. It's a look in the logs find more information easier. There's some really good resources now for live migration The upstream documentation has become excellent. I should have looked at who did that but great job It describes Everything about live migration all the different types how to set it up all the libvert flags You can do to speed things up and throttle things. It's crazy There's also like a blog post for every release of a sec about live migration So there's really good resources out there and in Vancouver There was an excellent video that goes a much deeper into what liver is doing and how to configure that and when you would use live migration, etc So with that, I think I'll turn it over to Ed to talk about replication All right, so I'm back up here again if you guys were in Tokyo a few of us got up on stage and talked about replication and cinder We thought it was all fixed then and it turns out it wasn't quite fixed to work back again to talk about the plan and this time We got a big this cycle got a little farther. So a little bit of history here This is design number four In the replication saga the first the first design and we'll show you a couple of those was really about Vendors doing it themselves. So both solid fire net app and some of the other vendors out there had the ability through extra specs to tag A volume and say that it needs to be replicated. However, the back end was set up So in in net app it was a snap mirror With solid fire was a replication and that was once it's set up It got that tag and it put the volume there. I'll show you how to do that in a second version one In Juno was was built it was put out there and everybody expected it to work the only people that got it to work were IBM and Then there were a lot of issues around that. So that's where we led to design number three Which is what we got up and talked about in in Tokyo Which is we're supposed to go into Liberty and the plan and it did go into Liberty The core pieces of it what didn't happen was and this was by design was we didn't allow any drivers Into the into the release Because they were worried that the core, you know, just like one if it got into the core and it didn't work for all the vendors Then you'd have like one or two and we'd have the same scenario again so that the drivers were kept out on purpose and The reality happened so that we got to what we're now calling 2.1 or we've given them dessert names I think whoever I wasn't at the mid-cycle when you decided on the dessert names, but cheesecake as we're calling it Really found some issues with the way it was designed in Liberty and redesigned it again Did it much earlier in the cycle? So we we were able to get some drivers in there. So we actually have a fully functional piece of replication in Cinder in mataka now I say fully functional and that it was Functional to the design The design there's a lot of discussion around how much to design how flexible it is What users get to control what admins get to control and so we'll we'll talk actually it's at the bottom of this slide And I've got another one about it for cheesecake. It was really going for simple Disaster recovery only so this is not oh I might have a disaster. I want to do a lot of testing. This is Site a is a smoking hole. How do I move over to be and and get that effect that to happen? And it was decided that this is an admin only Occurrence because the admin says oh, yeah, there's like a water leak in the data center and the Storage is flooded. I'm moving it over. The tenants don't actually have control this because The challenge starts to get like if the tenants got some stuff failed over and some not and you hit the failover button Do you like do this or you know how there's a lot of intelligence that needs to happen and trying to get that in One cycle wasn't gonna occur. So we'll talk a little more about that here in a second But let's go back to you know, what was wrong with the first the first Design of this right and so here's some examples just two out of you know for solid fire. It was in Essex you put in their SF colon replication into the extra specs and it in some other variables and it just happened right and so the Key thing there was a tenant didn't know anything about this The admin really had open stack didn't know anything about it, right? It was replicated and that's good because it's on both sites But if there's a failure or anything else outside of open stack We got to go flip over do all the the stuff on the the array and then reconfigure open stack to the secondary side And you can see the net app example there. It's net app mirrored Was the the keyword that that's in the driver the solid fire one wasn't in tree But you could go get it the net app one actually is in tree and you can go look at it. So Kind of back to cheesecake Unlike when we were in Tokyo and you're talking about replication and you had the option if you had a managed secondary or a non-managed Secondary and that became Challenging whoa brightness All right That was interesting So we've gone to again talking about the very specific very simple in case for cheesecake is There is no management of the secondary system. It's it's not seen initially by open stack It's not that open stack it could be configured to be putting volumes over there, but it doesn't realize there's a relationship So when you see the demo that we go through we don't have that secondary in open stack It's just not there but open but the drivers know about it enough to set up replication and auto fail over when you type that special command When it fails over non-replicated volumes are offline So we will show that they do so if you your tenants set this up and they pick something You'll see I got the storage catalog in there and they pick a volume that's not replicated when I caused the failover to happen That that volume is offline because that storage system has burned Or as a smoking hole or flooded or the water balloon fight as they talked about in the in the spec There's no fail back at this point in time in the design. There isn't a methodology to fail back remember that first system is On fire right that was the design concept and we've implemented the design concept Is that right long-term? Maybe not and that's where we'll talk a little bit about what's gonna happen in Newton with the tiramisu design and Again, no concept of managed secondary So a big part of this with all these different design changes that the terminology has changed So this time we're down to three terms. That's all you'll see Failover is pretty obvious that means to tell the driver that is to use the information and I'll talk to show you the line here To pass this through to the secondary so that volume that was amounted from the primary storage now gets Auto redirected through the driver to the secondary It doesn't account for what happens with it's attached So there's a whole piece of and this is the thing for you guys to think about as you're designing this what happens with Nova what happens to that instance if that side goes down Can I still live migrated? How do those things pick up? But the storage will do as best it can as quick as it can The other concepts that we we introduced here were The freeze and thaw concept or unfreeze and those were really about okay If I do fail over and things are you know? I always going to the secondary do I want snapshots in all kinds of other things to be to be occurring on that secondary? Or is that something when I do fail over I'm gonna try to limit some of that stuff So if I do fail have to fail back or in the future want to fail back I can limit some of that so that those are these are three separate commands. You'll see them in cinder now Some of the doc strings didn't get in there So it's important that we talk about them here and what they do so you can kind of figure it out The help is there but not the doc strings But as part of this there's the three previous designs There's a whole bunch of terms that were used and so I'm not gonna go through these but they're listed up here Although we may reuse one of these right now We've been pretty good in all these designs about changing terms every time But you when you do a cinder help you will see some of these old terms like promote and re-enable So be careful careful of that if as you're going about trying to play with this so How it works so the driver first off the driver needs to be enabled for this So those are I've got a list of the drivers that are in tree here the driver reports up to the scheduler that has Replication enabled and it's true and that's something that you can't control as a user that comes from the driver because it's got it in there And then you you put a stanza in there You've got your stanza in your config file You actually put this last line in there and that's the key word there is replication device equals and then the backend ID is required So there's one key word that's needed in here and that's back end ID equals and then the rest of the stuff on that line You see here in blue is all vendor specific Should have had these slides switched around so there's the replication enabled disabled and then there's other vendors specific like for example HP Has a sync and and a periodic option You'll notice our stuff in the the solid-fire piece here was just An MVP which is our management IP of the secondary and then the login and password So there's a bunch of every vendors got a few different things. You got to look at the vendor specific documentation for that and Here's the list of drivers that are currently in tree the solid-fire one and Our rock star was busy working on the core of this so we're actually we weren't in tree But it's there so you can get it and pull it down and put it in metaka. That's what the demo is done on and The net app stuff is gonna come along in Newton. So Sometimes there's you know, there's lags in things and that's what's going on these other ones I don't know. I'm not a vendor for these other guys, but they're all have the replication enabled flag in the tree for metaka. So So The one kind of ugly in the room is failback or fail over as we move along So the scenario that I always like to think about is a failed I moved to be everything's working now It's time to like clean up the mess build see make be primary and start the the replication to see And so that was not built as part of metaka. I'd say that's probably one of the things. It's a little bit lacking The way to do it and I'll show you in the demo here as you go in and fix the database Alex has put in a blueprint for Newton to add this functionality. So you don't have to do this by hand The commands are here. I'm gonna make these slides available I'll post it on Twitter. So it's a go follow me on Twitter I don't post much, but I'll post the link to where I put these slides You can go grab this if you want and so once you failed a to be you've got this kind of redeflection to be This clean up here moves B to be the primary and then as you see at the last line here It's go back to the beginning and set up B to C And so that's as I go through the demo. That's what I'm gonna do is I'm gonna have a and I've got a Not doing anything then I create its replication capabilities to be Go through and then at the end of the demo when I'm cleaned up I'm running fully on B and if you wanted to set up C go back to the beginning of demo. So here's this so again kind of we video these because Murphy lives in demos and so Yeah, we don't want to try to do it live. I do it sometimes. So here. I'm gonna do some It's a basic sender information stuff. I go through do a listing. You see there's no volumes And then we're gonna look at this command service list It shows all the services and there's a new option here called with replication Which will show us the extra replication columns in the display if you're not doing replication you can turn that off You can see it's all disabled And then what I'm gonna do here is actually create a volume That's that's this is my example case of not picking replication. So we're gonna create one We're making it on a volume type of gold, which happens to be redirected to the solid fire And so we'll list it you'll see it's there and it's active And that's again, that's not replicated. So I kind of go back and forth between command line and GUI GUI is a little bit better demo and you'll notice here. I've got two tabs for the two different solid fire arrays I'm going through here. These are pretty busy arrays. So we're gonna sort the account list by The heading for the different open stack that's running on here. I've got about six open stacks running on this solid fire You'll see it's there same ID is at both sides. It's on one of them. We go to the other one here. We're gonna like look at the cluster settings It's not replicating and Then did I actually go I didn't go pull up that there's no volume there. It's not there. So at this point in time We're gonna edit the config file. So we're going here if you haven't seen the cinder.com there it is We're gonna take our stands at the end and add our one line cut and paste into there again you can pull that out of the presentation, but Want to show that and then here I go through and got to restart all the processes So we're going through this is all dev stack. So we'll kill all the processes and start them again You're really only need to restart cinder volume, but I'm paranoid old guy. So I start them all And we sped that up to make it better. So now you can see when I do this Replication is enabled For the one so I've got LVM in here and it's still disabled. There is no replication there So here's looking at my storage catalog and this is where I might pause it for a second So you can see I've got stuff set up here. I've got LVM in there I've got gold silver bronze those are all redirected to the solid fire You'll see the extra specs here in a minute and you can see I've got this gold replicated volume type here I always put in the description some charge dollars like a public cloud may do and then Here's the extra specs, right? So we've got this gold replicated and you'll see that its replication is enabled and the volume back in name is solid fire Goes through So we're going to now create the volume using that volume type, right? So this is where we're going to go Set it up one of the things the different drivers have some different requirements The solid fire one happens to note that the replication the pairing of the two nodes is not there yet And so it actually creates that on the fly And I'll show that when we go back to the GUI's So here's our here's our cinder list you can see the gold replicated volumes there So now when we refresh this actually I'm going to go over here and look at the replication pairing so you can see it's paired in there and then We're going to go Look at that account back to the other one. I'm kind of just this functional going back But there you can see it's a replicated volume. That's the source Here's on this other window is the target again We got a filter because you got a lot of stuff going on in these different arrays and you can see there It's the target of the replication. So that all got set up the volume got set up on both sides You'll notice the UIDs are the same on both sides. So when it does fail over it just can pick that up And here's the command sender failover host I ran it once if you only have one back end You don't need an option if you have multiple back ends, which I do you got to go find your host here from this service list Plop that into here, and then it'll run fairly quick no response, but if you run the The service list again with the with replication flag you'll see that it's it's failed over So at this point in time you'll notice Also in there in the the active What's it called active backed ID that actually is the IP address that it's using now So the driver redeflects that over to that That that storage unit So now when we do a sender list, this is where it's interesting because you'll see that we have our Replicated volume is is still available. So as a client you would see this You'll notice the other one is in error. That was the one that were created that wasn't replicated the secondary And so it's offline It's the way the design was that's the way it was designed. It's doing as design So be aware of that if you're building something on this, that's how it was supposed to be and so I Go through here and just attach this Of course, I script the typing a little bit, but just to show that I can't attach it right I've got a nova instance running here So I'm gonna actually attach it to it if you attach the error one it's gonna error out But just just so that See it work And then we can see it also from the array over here. You'll notice it's not replicated anymore I Didn't go back to the primary. It's still on the primary. So it's still there, but the replication is broken sender lists will show that it's attached and I think I go back here in the gooey and show that it So you can look at the the ice because the admin command there I that went a little too fast, but it shows that it's coming from the secondary So how do we fix this a is dead and we want to be want B to be the primary now This is the the go-through and move the cleanup over. So we're gonna stop all the sender processes here Spend that up again to make it less painful So now I'm gonna go edit the center I edit the center.com and take out the replication line, right? So I'm gonna take out the replication line. I'm gonna move my management IP to be I Forget exactly how I did this if I made another yeah, I comment out this So I'm gonna leave that stands in there, but I'm gonna comment it out put another stanza in that points to be but then If you bring it up at that point It's still got the database pointing to the wrong thing So that's why we're have to gonna have to go in and fix so you can see I changed the IP address there Where the line came up and now fix the database. So my sequel It's pretty straightforward command, but I know some people are hesitant, but that's why we document it in In the slide deck and we'll have the API and some kind of replication event This is where we may use that reuse the command. So you see here I pulled up a listing of that the table I actually do it a second time here with only the fields you need it becomes a little more cleaner a little more clean And so again, we'll make this we may re-enable reuse the re-enable command Or promote actually is probably what we're thinking if you got a better word If you got a better word, we need another word run another word So there you can see I changed that the tables changed and then we go restart the the cinder processes here What Jerry Okay, we'll put a cherry on top of the cheesecake. Okay, that's a good point So a couple more commands here to kind of show you what's going on We've restarted the process and do a cinder list. You'll see that other one's still in error We didn't have a there anymore. So this is the way it works in cheesecake And then we're gonna go detach it and clean up you got to go delete them on a And this is this is a complicated problem, right? If you think about this if I want to auto fail back Then I've got to do a resync and there's a lot of differences in the different array manufacturers and how they do a Resync back. So if I had a and I failed over to be and now I got to flip back In open stack and cinder the the philosophy is we want that to be kind of abstracted So it's very simple and it's cinder commands and trying to accommodate all the different The different array types and how they do that resync is really tough. And so that's why There's a lot of challenge going on. I would encourage y'all to come to the design session and give your input And so we'll talk about that here in a second What's gonna happen with the Newton release and I'm gonna go I go here and reattach it and detached it reattached it all that stuff Okay, so John Griffith up in the front here saying it all seemed fine when he when he merged it and now he's listening to us go And maybe we need some more stuff So this is yeah, again, this is the way the design process goes So that's where I wanted to I'm gonna kind of this is my last slide. We're gonna wrap up here a little bit So tiramisu is the next dessert Design 3.0 or whatever and the goal here is to in Austin and actually I see shing here in the in the audience So she's actually kind of working through trying to figure this out The goal was to figure out how to do much more granule. There's kind of two things much more granular failover So I can fail over one volume, but then it came into some of the arrays have this grouping construct if you're familiar with net app Volumes from solid or from open stack go into flex volumes and flex volumes are replicated So you may have two cinder volumes that wind up in the same flex volume They both need to be replicated. Maybe you want them replicated. I know EMC has a consistency grouping Capability that they I think there's some bounds on replication in that and so that's the second part is how do I deal with these groupings? and Then as out of all this came okay if I'm a tenant and I want to fail stuff over like I want to test disaster recovery those kinds of things What happens if I say fail over a cinder volume and there's more in the group that need to be failed over So these are the kind of problems that are being worked through in the in these design sessions They're very complex. How do I have a one tenant that's failed over some volumes? Maybe grouped maybe not grouped and he wants to fail them back. How do I resync? What happens when the admin declares a disaster? What happens if there is a perceived disaster and the tenants fail stuff a few of their volumes over and then it's not a real Disaster or it is a real disaster and the admin then fails everything over What do we do there? What's the right thing to do with non-replicated volumes? You'll see currently they just errored out What do you do with those do you know as an operator you can force everything to be replicated if you want to that's fine if you're in an enterprise or anything else, so There's a lot of work going on if you've got an opinion, you know You can come up and give it to the folks here that that care or show up at the design session You know, we're trying to accommodate everything as much as we can can't accommodate everything So that's that's the moral of the story and with that I've got to put up the advertisement. I guess is what I'm told So got to put the advertisement back up any questions There's mics over here the guys in the back wanted me to encourage you to come up to the microphones or I need to repeat them So any There so the question is about the cleanup and going to my SQL. It's it's ugly There is a proposal to put that into Newton as an admin command to sender So if you fail over today, and then you can run in that I'm failed over mode until Newton Then you'll have the commands in there The question is is the replicated volume Addressable so I could snapshot it not through it is once it's failed over When it's replicating you would have to use the array tools to do a snapshot. Yeah, you'd need to use the array tool Yeah, you didn't need to use the array tool So you're you see there's a lot of switching back and forth between the array tools and sender That could be the boot volume. So the way we set that up. I just I just did a simple I didn't attach it and but if if it's so the question was can I replicate a boot volume? Yes, I can set it up that way as a replicated volume and then boot from it the question back to the audience and the operator is I've got this booted VM off of this volume That storage goes away for some reason it fails over you're going to need to clean up Nova and bring it up again over there So I got a couple at the microphone over here. I'm going to take a little preference to those guys Yeah, just rewinding back to the live migration stuff. Yep You mentioned that with Non-shared stores or blocked live migrate the m2 has to be paused That's not true anymore with the latest versions of live vert and q mu. You don't need to do that anymore It can actually be live block migrated. Yeah, we still have the downside of Transferring all the blocks. Yeah, it takes longer is more complex and I tried to set it up and I couldn't get it to work So, okay, that's why we that's why we that's why we didn't show it That's why we didn't show It just requires a I think we couldn't get to work because it requires a later version of liver that's right Yeah, yeah, and we're we weren't at those. Thank you for the input so that he was on the mic So I don't repeat that good Yeah question by the replication. So you were very clear about the scenario you're considering which the smoking hole disaster Yeah, but the command that you showed to invoke the failover. Yeah from what could see it seems to be run in the same Site as the primary which is now presumably a smoking hole. I hope you have yet, so Perfect question. I would assume and we're assuming you have some kind of ha in your control Plane infrastructure so that you have Cinder running You know on multiple controllers. So the target for the replication is the assumption. That's in a different zone region Entirely different open-stack deployment. Yeah, it's it's not it. There's no Facility or requirement to manage it within the open stack that you were right Okay, so I could I could have two open-stack to plot separate deployments replication between them and then after a failover Or after disaster other I could run the I could invoke the failover from the secondary site You could do that with some fancy Stuff around controller zones, but you could also have like a pacemaker cluster that's Separated by the sites for the the initial open-stack instance That's what we see a lot of right where people set up some kind of a clustering and you can have Cinder volume run on both of them Yeah, you can John's up here and you can use Cinder manage to pull these sites in and To pull in another array So if you have a volume that's outside of Cinder and you want to like bring it into Cinder management So there's a few things you can do there too No, this that lot so the question you again you were sort of at the mic there on the question is do I have to Distribute the Cinder comp when I when I do the failover and no you'll notice I just ran the failover command that line that I added to the Cinder comp you would need to replicate and to put That everywhere when you did it but then all the sudden when you declare the failover it basically is telling the driver Hey, look through the main part of the the configuration stanza get to that Replication line and do whatever you need to do there So you don't have to like go about editing the Cinder comp to do the failover You just have to run that Cinder failover and which back-end you want to fail over and it just does it It'll be the failure to happen now my C has not come up yet Correct there was no provisions for bringing up C That's why I was editing the database right and so that database edit was really a failed over to be How do I make be the prime once I've realized that that's happened and you know? I'm buying a new array and a is really dead How do I make be the primary and again once you follow that demo you're back to you're running with a single Array and then you can go buy C and go back to the beginning of the demo and do the do to add the line so Do I have people lined up for the next session or questions all right with that? I think we're supposed to be done, so thank you I will tweet out the location of the slides and the demos here shortly