 All right. Welcome to Storch Capabilities in Cinder. I'm Sean McGinnis. I'm the current Cinder PTL. Joining me are Jay Bryant and Zheng Ying, some of the longtime cores of the project. Just to make sure everyone knows why you're here or what we're going to be talking about, Cinder is the block storage service in an OpenStack Cloud that is a control plane abstraction to be able to manage LVM, CEF, Lenovo, Dell EMC, whatever storage arrays you have in your environment, being able to pull those in and use those storage resources in your cloud. We currently support over 100 volume drivers as of the Ocata release. So it's pretty flexible to be able to use whatever your storage of choice. So the whole reason for this presentation is there are some features within Cinder beyond just creating volumes and exposing them to guests. We wanted to be able to just highlight a few of the extra features that are available and make sure people are aware of those. So there are certainly more beyond this, but we picked a few things that might be interesting. And I'm going to highlight how those work and hopefully show you it actually working. We'll have a, we're recorded, so at least we'll have one good Cinder demo today. All right. So that. We weren't feeling as adventuresome as others. We went for recording it, which that's a process in and of itself. But hopefully you'll enjoy the output once we get there. So volume migration and retype was one of the features I wanted to talk about. Because it's, especially as an administrator, it's something important to understand how it works and what it is within Cinder. But it's not always clear. So one of the things that I've got a number of questions about while we've been here is, oh, migration. So that's moving with your VM if you do an instance migration. No. This is just talking about the data migration and where that data is held at and stored from a volume perspective. If you do an instance migration, we handle disconnecting, reconnecting, but your storage in that case stays in the same location. So volume migration you can see from the graphic here is taking that data that's running on one physical backend and moving it via Cinder, via the Cinder APIs, working with the host backends to another location. So why would you do this? Well, OK, the case where you have maybe you've brought a new storage backend into your environment and you want to rebalance data and move things so that you have certain storage in another location, that's a case where you do migration. Migration is different from retype, which is the other one I'm going to talk about here, and the fact that from a Cinder perspective, you have multiple volume types. So that's how you label basically how you classify where that volume is stored in your environment to backend storage. The simplest example, which I'll be showing later, is basically I've got one backend named LVM1 and one backend named LVM2. They're two separate volume instances that you can migrate the data between because they report the same capabilities. So I'll show an example of that to clarify it. But in the case that you are changing the type, that's where you do a retype. You change how that volume is labeled as far as the capabilities that it requires on the backend storage behind it. When you do a migration, your data is going to move from one location to another. When you do a retype, it may not necessarily move. So I'm hoping my demo that I do a little bit later will make that clear. But you then change the type that is associated with that particular volume, and it's only going to work in the case that the type that you're switching into satisfies all of the needs in that capability. If not, it will actually perform a migration, and we'll show that later. So the two commands that enable doing this are cinder migrate. And the couple of interesting points here to note are whether you lock the volume or not. If you don't lock the volume, you can do migration on a volume that is currently attached. The lock is for whether you can cancel it or not in the middle. That's right, whether you can cancel the move. And then you indicate the name of the volume you want to move and the target host that you want to migrate to. In the case of the retype, the migration policy option here is important, because if you set never for migration policy and you retype to a type that is not satisfied by the back end that is currently on, it will not retype. But if you have on demand, it will take it and move it to back end storage that satisfies that type. Of course, then you need the volume name and the target volume type. Another fun feature that we added a couple of releases ago is the ability to do generic image caching. So what we do here is we have a number of larger installments that we're having issues where they're trying to do boot from volume. And when you do boot from volume, you mount up the image from glance and you pull it through the control node over to the, or wherever the volume service is running, by default, the control node. You pull it through that and you do some QMU possibly on it and write it into a volume. And if you're doing a lot of that, that's obviously quite, you know, puts a load on your system. So this aims to reduce that by actually creating a image volume cache so that you've got a cache of that image in your volume service that short circuits that path of having to mount it up from glance and having to write it through every time. So the first time you do it, it downloads the image, caches it, and then subsequent boot from volumes will perform much better because you're not having to repeat that copy process from the image store. To do that, there are some things that you need to set up in the cinder.com file. Your tenant project ID and user ID, and then setting things like the maximum size that you want to use for your image cache. Obviously, you don't want it to just grow forever. And the maximum number of images that you're going to allow in your environment. And obviously, you have to enable that if you're going to use it. Other than that, that's pretty much all you have to do to get this set up. And then there are a couple examples here of the kind of commands you would use to do that first volume create where you're going to pull the image in and get the cache primed for future use. Another way to do this is to just work with a volume backed image. So you can set up the, in this case, you need to actually get Glance ready to be able to use cinder as the back end for that image. And so you need to set up your authentication into cinder here. And also, obviously, cinder needs to know how to talk back to Glance. So you need to set those options up there. But then you can use the OpenStack client and do an image create and give it a volume. And then it will create a volume for that image. And it will be backed by volume storage instead of having to copy it each time that you do a boot from volume. So with that, we'll talk about replication with Shane. So replication API version 2.1 was introduced in Mitaka to solve one particular problem. That is, if the storage back end is hit by a disaster, we provide a way for the admin to fill over the entire back end to prevent the loss of data. Admin needs to make configuration changes in cinder.conf and to create a volume type to support replication in cinder.conf, there is a replication device option that needs to be configured. There could be multiple keys in the replication device option. Back end ID is a required key. There could be one or more replication targets configured. Some drivers only support one replication target where others could support multiples. In the volume type extra specs, you need to set replication-enabled to is true. Also, drivers need to report replication-enabled and a list of replication targets. We have three replication comments that fill over, freeze, and thaw. The fill over command is used to fill over all the replication-enabled volumes to the replication target, or the volumes that were not replication-enabled before the fill over will become unavailable after that. The freeze command is used to disable the cinder volume back end. This needs to protect all the existing resources so that no changes will be allowed before you fix the setup and until you run the thaw command. The thaw command is used to enable the volume back end again. So after filling over from back end A to back end B, if you want to promote back end B to be the new primary device and replicating to back end C, some menu steps need to be performed. So first, you need a freeze host, stop cinder volume service, go to cinder.com, and manually replace back end A with B and B with C that reconfigures the replication relationship. And after that, you go to the database to manually change some database entries so that the back end is no longer in the fill over state. After that, you can start the volume service and thaw the host. So this process is, of course, the menu and the error problem. Unfortunately, right now, we do not have an automatic way to do it yet. There is a spec trying to make this more automatic, but it has not been implemented yet. Also, we do not have a command to do a fill back, but some drivers do support fill back by using fill over command and specify default as the destination back end. So after filling over to the replication target, admin can run fill over again, specifying default as the destination back end. And this is after you fix your setup, of course, and that will bring the system up and running again as before. The generic volume groups feature was introduced in Newton to provide a way to manage a group of volumes together. One motivation for introducing this feature was that the existing consistency groups feature can only be supported by a small number of drivers. This generic volume groups feature provides a default way that works for every driver. The default implementation simply loops around and create a group of volumes and a group of snapshots corresponding to the original volumes. Drivers supporting consistency groups can add that capability to generic volume groups. By using different group types, a driver can create a group that supports consistent group snapshot or a group that does not support it. This is similar to how volume type works. By using different volume types, you can create different volumes with different capabilities. The generic volume groups feature can also be easily extended to support other functionalities in the future, such as the replication group. We have commands to create a group type and set group specs in the group type. These are admin commands. This is similar to volume types and extra specs. We also have commands to create, delete, update, show, and list groups. To create a group, you must specify group type and all the volume types supported by the group. This is to make sure that the scheduler will pick a backend that can support the group type as well as all the volume types in the group. To delete a group that is not empty, the delete volumes flag needs to be set. This will delete the group as well as all the volumes in the group. Also using a group update, we can add a existing volume to the group or a deleted volume front, a group that do not delete the volume itself. We also have commands to create, delete, show, and list group snapshots. And we also have support for creating a group from a source group and create a group from a source group snapshot. In order to upgrade to Pyke, operators need to run the data migration scripts to migrate data from the existing consistency groups tables to the generic volume groups tables. And all drivers supporting CGs need to add this capability to generic volume groups in Pyke. CG APIs still work in Pyke, but they will be deprecated in Queens. To create a group that supports consistent group snapshot, you need to set consistent group snapshot enabled to is true in group type specs and volume type extra specs. If you don't set it, then a group will be created using the default implementation, which does not ensure consistency. Also, do not set the consistency group support key in group types or volume types because now drivers report consistent group snapshot enabled in capabilities. So, Xia will be talking about backup and restore. So, we have backup and restore capabilities in Cinder, and this gives away, obviously, taking a volume and backing up that data to somewhere else. And then if something happened to your data, being able to restore that back to its original location. This, we do support for incremental backup. So, if you're doing this over time, you don't need to copy the entire data out. You can do a one full backup incremental through the week. Just the thing to keep in mind with any backup solution if you've ever worked with. If you just keep doing incremental, when you need to restore that data, you need to start with that full backup and then work your way out. So, it's not something you want to just do full once and then just keep it incremental after that. There's non-disruptive backup. So, if you have a volume attached to an instance and you're doing IO, obviously we can't backup that data because IO is in flight, so we may copy off some data that doesn't have information that later on has information that was written out. So, the way that works is it'll create a temporary snapshot of the volume and then with that frozen point in time, we're able to take the data off of there. So, we know it's in at least a crash consistent state if you have a running instance. We can also backup snapshots themselves. So, one thing, non-integrated with Cinder, if you're backing up your data from a volume, you may lose all of that snapshot information if you need to blow that away and restore the data, but we have a way to actually pull that out of there so that we can actually get back to that state. And we support several different backends or destinations for these backups. Default, I think, usually is Swift. You can do it out to an NFS share. A few releases ago, we had someone add a driver for Google Cloud. So, there's a lot of different options of where that goes. Now, I should say, and it's not really called out in here, we're not trying to be a backup product. This is not something that would replace some of the big vendors out there where their main focus is to do backup and recovery of data. We provide a mechanism through the Cinder APIs so that regardless of what type of storage you're using on the backend, you can get that data pulled off of there and written back, but we do not do things like scheduling when these happen. There's no way through Cinder to say, I want this volume backed up nightly at 2 a.m. So, we provide the APIs, make it that nice storage API abstraction so that other applications, you could write scripts that make calls into our API to pull, integrate this backup into some other system process that you have. There's also other storage systems, storage products that could write to the Cinder backup API to be able to leverage our capabilities and then add their value on top of that. So, we provide the mechanism to get the data off. We do not try to be a full service data protection product. So, there's a few commands. You see there's a mix here, create a backup, open stack volume backup create, open stack volume backup restore, pretty basic. If you haven't heard the open stack line, why we have some command lines here that are open stack volume and then some that are Cinder is the, we're trying to standardize all of the open stack products around having a consistent API. That's the open stack client. That's the open stack volume, open stack server, open stack, whatever. We don't have every functionality of the Cinder client moved over. So, there's a couple of commands on the bottom that are less commonly used maybe. The Cinder backup export, where you can take a backup that's been done, export the metadata about that so that you know out on the Swift object store, here's all the data about backup for volume and then you can later import that. So, we're gonna walk through a few demos here. Hopefully some of this makes a little more sense when you can actually see it in action. We'll start off with showing the volume migration and retype. Yeah, so hopefully this goes smoothly. So, this is gonna demo the migration and retype process and what I did here to demo this, it's kind of a funny setup, but it's a simple example to give you an idea of how it works out. We had a controller with two different LVM driver backends and then I had a separate NFS host that I had access to to provide a third backend. The reason I have the three is I wanted to show the way that you could retype from LVM driver one. So, I had two types that were just named based on the back end name LVM driver one and LVM driver two and then I had a type that was for iSCSI protocol which is satisfied by both of those backends. Obviously then the NFS one was named NFS one and it's not going to report an iSCSI capability because it's an NFS is the protocol for it. So, what the demo will do is I will create a volume that's of type LVM driver one then I will retype it to be iSCSI and you'll see the type change but it's not gonna move because the storage it's currently in satisfies that need. Then at that point I'll do a migrate which I can do here. I'm gonna move it from the first back end to the second back end which works because the iSCSI type is compatible with both LVM driver one and LVM driver two. They both report that same capability. So, it lets me move it and then finally it's like well I wanna get it off of my LVM space. I'm gonna move it over to NFS and so at that point I do a retype and set it to the migration policy is true so that it will actually move over into the NFS space. Go along with what Sean mentioned a little bit earlier. I use Cinder Client in this case. So, the Cinder commands to do this because OpenStack Client has migration commands but it doesn't yet have the retype commands. So, just for consistency and to make things less confusing in the demo I just use Cinder Client for all of this. We're working on getting parity. We'll get there. Click on it. Okay, everybody cross their fingers. Ta-da! All right. So, all of these require to be admin user so that's why I wanted to show actually sourcing that there. Starting with a clean slate. Don't worry, you don't have to watch me type all of the commands. This shows here we've got our back ends. We've got a NFS back end and the two LVM driver like I mentioned. These are running in VMs on my laptop, real exciting. There are types, the four that I mentioned. And I'm gonna show you the extra specs that go with those. So, we've got three that use volume back end name and then there's our storage protocol one that just says, hey, I'm gonna go out and look for that type. So, I'm creating my volume now. Look for that capability I should say when I'm using that type. So, I created my volume in LVM driver one. You can see there, showing you the details about it. And I also, I can go and show you that it's currently in the LVM driver one space running under DevStack. That was what that LVS was. I just did my retype and moved it to iSCSI and now if I show you the details again you can see that it hasn't physically moved. The type has changed but it's still on the LVM driver one. So, I'm going to now migrate it. And it says, hey, I can do that for you. And we give it a little time here. As the data you can see it's in the migrating status and that is the last action that it completed. We can see now it's in the LVM two, when you do the LVS and now we can see that that has also changed from a host perspective and we have successfully migrated. So next we're going to retype it over to NFS with on-demand migration. Do a list and we can see there's a brief period of time where you see it in both locations where one is available and one is deleting or retyping or then deleting as it goes through the process. It's no longer in LVM. I jump over to my NFS server. I must type LS. There's my volume and there's my ID and if I jump back, do a sender list. There's the same ID. It's available. Name's the same and we're now in NFS one volume type. Ta-da. So that's how you move your data all around, round and round it goes. Where we stopped was on NFS. So we'll probably go back to LVM eventually. All right. Next, generic volume groups, follow that up. So I'm going to do a demo on generic volume groups. My demo setup has two sender volume backends. One is connected to a unity and the other one is connected to a VNX. The VNX driver supports consistent group snapshot. So I'm going to create a group on VNX that supports consistent group snapshot. The unity driver does not support it yet. So I'm going to create a group unity using the default implementation. Just click, yeah. So first create group types for unity and VNX. Now set consistent group snapshot enabled in the VNX group type. Next create volume types for unity and VNX. Set consistent group snapshot enabled in the VNX volume type. Create group on unity. Next create a new volume in the unity group. Add an existing volume to the unity group. The volume is not in the group and first use a group update to add it to the group. Now it's added successfully to the group. Now create a group on VNX that supports consistent group snapshot. Create a new volume in the VNX group. Add an existing volume to the VNX group. The volume is not in the group. Now use group update to add it to the group. Now it's added successfully. Now create a group snapshot on unity. Now delete the group snapshot. Create a consistent group snapshot on VNX. Create a group from a consistent group snapshot on VNX. Now create a group from a source group on unity. Finally delete the group and its volumes. We need to set delete volumes flag in to delete a group that is not empty. Okay, so that's the demo for generic volume groups. All right, aren't these console demos exciting? All right, for the next scenario showing backup recovery, I wanted something a little more realistic. So I have an instance in my Nova compute of a DB server database running there that has a volume from sender mounted as slash data. So the database is running, it's got data written to this volume under slash data and I'm going to back that up to my Swift object store to kind of make this easy to see the multiple pieces. Just to explain what I've got here, the top panel is going to be my connection to the database server so I can show you that actual data that's running there. Center part here, I'm gonna just run some command lines so you can see that happen. I've got my server named DB server and I've got a volume attached to that DB vol as you can see is attached to DB server. And then on the bottom here, I'm just gonna tail the sender backup log just so you can see some of the activity that's going on in the back behind the scenes. So on the database server, we'll just take a look at the database. We have a table there called inventory. Don't have any data right now. So I will just add some data, the inventory truck comes in, unload it, add the inventory to stock. And now we can see got a bunch of data, really important critical data there. So I wanna back that up. I will run the open stack volume backup command. I've already done a backup, a full backup once. So I am going to do incremental. I'm also giving that force flag. I had mentioned we can create a backup of a volume that's attached to a running instance. That needs that dash dash force to tell us, okay, yeah, it's attached. Create a snapshot backup from there. You can see here that kicks off some stuff on the backend and eventually we'll see create backup finished. So I've got a backup, I'm a new DBA. I go out there, I'm messing around in the database, trying to figure out how's this Mongo command line work. What, I saw this on an example on the web somewhere. What if I do delete many and, oh, wait. Okay, my inventory is gone. What do I do? Oh, no. Quit, get out. So I'm just gonna power off that machine because things are bad. So go back over to my command line, open stack volume backup restore. You just tell the DB backup. That was the name I gave my backup and DB volume. But wait, it's available. That means it needs to be available. Our volume, even though I've shut down that instance, is still attached to the DB server. On the sender side, we don't know if any IO is going to come down. We can't restore data onto a volume where possibly IO could come and crop that data. So I'm just gonna shut down my instance, remove that volume from the server while it's shut down. Now I can do that restore. That's gonna copy the data back, and then through the magic of video editing and hand waving, that data gets back pretty quick here and restore backup finished. Your time might take a little bit longer. So now, you know, okay, I saved the company. I need, I can go back. I need to reattach that volume back to my instance. Just tell it DB server, DB vol, and I'm giving it a device. Make sure it shows up in the same place that I'm expecting it. And I can start that instance back up and running. That's really the only tricky part, is that you need to detach the volume in order to do this. So now that I've started up my database server again, I can go back over to the command line. We'll take a look, reconnect to DB server, go in there once it boots up. And now I can go back in, connect to the database, and take a look at my inventory table. And wait for it. Ta-da, my inventory's back. So, whoo-hoo, saved it. All right, so for all of these, there's a lot of other features in Cinder, but if you're interested in any more information on these, I have a few links here, the migration docs. You'll notice there is a mix here. A lot of this is in the admin guide. Unfortunately, some of this is actually in, you'll notice debref in our API, in the URL here. So that's usually our developer reference, but right now we're kind of sorting out where some of the documentation is and how that gets generated and how it gets published. So some of this, we have better user information in our developer reference. So you'll see a couple of links, several points used to that, but then we do have hopefully most of this in the admin guide. So you can go there, find out more about these and find out more about other functionality. So that, I think we have a few minutes left. If you have any questions, if you could come up to the microphones, that would really help. We'll try to answer whatever we can in the next couple of minutes. Yes. I got two questions. All right. The migration and retype, can that be done while they're attached? No. If you don't need to migrate, you can retype, I believe, I don't think we block that. Well, right, if you're retyping without migration, that's fine. Because a typical use case of the retype without the migration is just you want to change a setting on the volume. And for most settings, you can do that. For some backends, that might be changing caching policies or things like that. So it's something that can be done on the fly. If you need to actually move that data, then you're looking at more like a no one live migrate type of thing where we can't really. Okay. We can't do that from Cinder. Okay. Yeah. A second question. I have one of those backup managers you're talking about, and I make use of the Cinder backup, which is fantastic. Thank you, by the way. Good. The backup import and export records, I have a need to unmanage the abandoned adopt abandoned. I need, there's no way now to remove control of that imported backup from Cinder, except for deleting the backup. But I don't want to delete the actual data, just get it out of the database. So is that coming? I haven't heard the use case yet, but I can totally see it now that you pointed out. So you want to be able to backup the data and have that information out there and export it out of Cinder and have Cinder forget about it, but you still want your data out there so you can re-import and pull it back if you need it. Yeah, that is probably something we should look at doing and that is a gap that we have right now. Okay. Yeah. Good question. Thanks for the feedback. Things like that, any kind of feedback of something you need, let us know. Yeah, thank you. For, I've got a question concerning the backup. So what you get if you back it up and you have a file system on top, what you get is a crash consistent snapshot, is that right? So you rely on the file system and the journal may need to be replayed. There's nothing like a freeze guest FSH and whatever there was in order to actually title the file system, okay, now you need to flush everything. So versus if you have like, okay. Yeah, there's no. It's not your list that you have X3, but if you have something that's not journaled, you may end up with something that. Yeah, there's no guest integration in this backup. So the only way that you can get to guarantee that you have a totally consistent backup is actually shutting down that instance. If you do the backup by taking a temporary snapshot, that is just, there could be an application that has IO in flight. So when we take that snapshot, there is a small chance, well, depending on your application, hopefully a small chance that there will be something there and they'll have to replay some data. Yep. I believe, are we 40 minutes? I think we're... Okay, any other, yep. The cinder migrate. So when you create a cinder volume like you just did, it actually activates the trunking command. So even though it's like a 10 gig, it's only on disk is like 10 meg or something like that. So the question is, using cinder migrate to migrate to another volume backend, what will be the actual size on the backend? Will it be like one meg or 10 gig? I believe it'll be 10 gig. So... Yeah. Because the issue is most thin provision volumes, there's no data. If you try to read anything that hasn't been written, you get zeros back. Yeah. So I guess if you're migrating that out somewhere else, some storage types actually need those zeros written out. So depending on your backend storage, I know some arrays are smart enough if they see a bunch of writes of zero in their thin provision, they'll just throw that away, kind of make it a no op. But I believe... You can write the full size that is. I believe we got to write out that full size. Right. Thank you. Yep. Good question. Yeah. Very good. Any other questions? All right. Great. Well, thank you very much. Thank you. And again, any other feedback, let us know. Thank you.