 All right, let's get this going, guys. One does not simply create multiple backends in Cinder. You have to listen to our nonsense first. All right, so I'm going to do some introductions here. My name is Walt Boring. I work for Hewlett Packard Enterprise. I've been working on Cinder for several years now, primarily responsible for attachments between Nova and Cinder, a library called OS Brick. And here's Kendall. Hi, I'm Kendall Nelson. I work for IBM. I've only been there for about a year, not even a year yet. I started working on Cinder right away. And then over the last couple of months, I've been getting involved with Brick as well. I'm Jay Bryant. I'm also with IBM, serve as the Cinder subject matter expert for IBM. I've been a core member on Cinder and working on Cinder since the Grizzly release. And I'm Sean McGinnis. I work for Dell. I'm the current Cinder PTO. Been working on Cinder since ISOCE. All right, so this is a beginner session on Cinder and multi-backends. As many of you may know, multi-backend has been supported in Cinder for quite some time now. So this is good for beginners. So this is the agenda. We're going to go over what is Cinder, right? So and how does Cinder fit within the OpenStack ecosystem? Where does it belong? And we'll go over some examples of how the process of attaching a volume works. A little bit of overview of the architecture, some of the services. Then we'll get into how to configure Cinder.com for multi-backend and look at some of the drivers and then talk about some of the more advanced features that is tied directly to multi-backend, which is volume types, and discuss a little bit of the differences between retype and migration. It can be confusing for some people. And some of our future plans. All right, what is Cinder? Cinder was created in the Folsom release about four years ago. It used to be part of Nova Volume. We got the team back then, John and Vish, decided to break it off into a separate project because it seemed to make a lot of sense. And its primary use is for managing block storage, provisioning volumes, attaching volumes to VMs. It is not a file-level storage. It is not object storage. That's Manila and Swift. So as the slide says, Cinder focuses on attaching volumes to VMs. But we're also looking at attaching to ironic instances as well, booting from volumes. And volumes definitely have life cycles independent of VM instances. So you can attach a volume, do some work, detach it, attach it somewhere else, and so on. All right, so where does Cinder fit? Cinder provides an API and an abstraction layer to multiple vendors' backends to allow you to do block storage provisioning and attachments to Nova instances and bare metal. As the slide says, we expose the vendors' backends, the storage hardware to the cloud, basically. And this is what Cinder's job primarily is, is to allow it to be an abstraction layer to all of the different vendors' backends. HP, Dell, IBM, Pure Storage, SolidFire. We actually have a slide at the end of the presentation here that goes over about it shows how many vendors that we actually support, and it's quite a few now. All right, so this enables users to manage the storage, create, snapshot, backup, attach, detach, all the important things that you really want to do with volumes with respect to your VMs. So this goes over a little bit of a diagram, a workflow, or architecture, if you will, of the interaction between Nova and Cinder, and how volume attachments actually work. So over here on the left, you have the Nova compute host with the Nova compute instance VMs talking, driving Cinder to do attachments. And you can see on the slide that this is an LVM back end for Cinder, which is the LVM component as part of the actual Cinder controller at this point. And it exposes a nice, fuzzy target. Nova gets that, looks for it to show up on the Nova compute host instance, the volume showing up, and then it passes that up into the VM. So if you look at another example here of an example of, let's say, a vendor's back end, HPE, or Dell, whatnot, it's basically the same exact process. There's really not a lot of difference here. The only difference here is that the I SCSI target actually lives on the storage controller itself, not on the Cinder controller, but the process is the same. Nova drives Cinder to say, hey, I want to do an attachment. Cinder creates the target, the export. Nova gets that information to look for that connection or that volume showing up. Discovers it, hands it off to the VM, and away you go. All right, so this is a little bit of an overview of the Cinder architecture itself. It's built up of three main components, really. The Cinder API, the scheduler, and Cinder volume. As a side note, there's the backup service there and the client as well. What we actually mean by multi-back end is that you may have one or more Cinder volume instances that may be pointing to one or more storage arrays or back ends. It could be the same back end, but multiple instances for different configuration reasons. And so you'll see, where's the laser pointer on this guy? So this is the multi-back end component here of Cinder. We have one instance of the API, one scheduler, but multiple Cinder volume instances over there talking to one or more storage back ends. So here's an overview of some of the services. Again, there's the API service, which handles all the rest, calls into Cinder to make requests to provision of volume, to create a volume, to delete a volume, to do an attachment of volume. Then there's the scheduler. The scheduler is a component of Cinder that helps decide which back end, or which Cinder volume instance, will receive that provisioning request, depending upon how much is available on that particular back end or not, or however it's configured. So the Cinder volume service is really what is the crux about what our talk about is today, which is how to create multiple volume instances that point to multiple back ends. So it's interaction. This is where the vendor plugin actually comes into play here. The Cinder volume service and the plugin are tied directly at stand-up time when Cinder comes up. And the plugin is what actually talks to the array to do the provisioning work that Cinder's directing it to. The volume manager is, in fact, the Cinder volume instance that talks to the driver that talks to the array. To do the creation, to do the target creation for exporting a volume for doing attachments, for detaching, removing the volumes, doing snapshots, et cetera, et cetera. And the same with backup. The backup is a different service that knows how to backup your volumes to different other back ends, such as Swift, et cetera. So how do you actually talk to the Cinder API? And that is what Kendall's gonna talk about. So basically we have this Cinder client, the Python Cinder client CLI. And it is the command interface, like Walt mentioned, to Cinder. All the commands are formatted, Cinder, and then the command cell list, create, delete. So it uses rest, like we said before, to communicate with the Cinder API service. And currently we are working on, oops, I didn't mean to go, oh no, what is happening? I don't know what's going on. Okay, let's get forward, forward. Wait, oh, Lord, okay. Sean, help. Okay, well, that's getting fixed. So there's an effort right now to make the OpenStack client achieve parity with the Cinder client, because it's lagging pretty far behind. And we're trying to work with them through the fish bowls and various other discussions this week to achieve that parity there. So maybe I'll just use the laptop. So, all right. So the commands that are currently in the CLI are creating volumes, delete, list, show, all of those. The attach, detach actually works through Nova commands. We're working on getting those separated out so that you can do it without Nova. I don't think that's done yet. And then you can obviously create, delete, list, show, snapshots, backups, and create, update, and delete volume types, which we'll get into more here shortly. We also have really basic QoS commands, and then quotas is another ongoing effort. Nested quotas has been kind of a problem child, so we've been working on that in Mitaka, and we'll do so more in Newton. So getting into the CinderConf, this is used by all of Cinder's services it's like the setup file. Multiple backend support is enabled there by a specific section, the enabled backend configuration option, which I'll show an example of here in sec. Basically, after you get them set up further down in the CinderConf file, you have to add them to the enabled backend section, otherwise they don't exist. So some other things you can do in the CinderConf file are adding more debug for logging, setting verbosity to true to get more detail. Just a note on the logs is that by default, Cinder logs go to the var log Cinder, but when you're using DevStack, they're actually in a different place. They go to Obstack logs, which can be kind of confusing. When using iSCSI, you need to make sure that you have the right iSCSI helper. It's different for Ubuntu versus Role. That's something to look out for in your comp file. And then any changes made to the CinderConf file require restarting all the services. Spec that I have out there right now is actually to do a dynamic reload. It's in progress and we'll probably talk more about that later in the week if you're interested in that discussion. So Cinder has tons of drivers. I think the last count was over 70. This list may not be complete, but it just kind of goes to show that like there are tons of different architectures out there and you can actually see the three bolded ones, the LVM, the NFS and RBD are reference architectures. So there are even more different architectures to be used than those. And then so this is our example Cinder comp file. At the top in default, normally there's a lot more stuff. We cut it all out because it wasn't important for the multiple backend part. But you can see that we have the default LVM driver in there and then we added a store-wise SVC driver. And so it's a SAM-based driver that IBM has. We just had all the information there for setting it up. And then once you do that, you gotta go restart the Cinder services like I mentioned. And then when you do a Cinder managed service list, you end up seeing both of your drivers there. So we have the LVM driver second from the bottom and then our store-wise one and they're both in happy state. So yeah, and moving on to volume types. Thank you, Kendall. So okay, so we've walked through the comp file. We've shown how you can add additional backends to your system. And now how do you make use of those? So that's where volume types come into play. And so what you do is it's an admin interface. Generally the way that this is seen is the admin knows what storage is back there, knows how he wants it to be used. And so he then configures his volume types so that the tenants that he wants to get whatever levels of service have access to those backends. So the example that for storage people you think of you may have gold, silver and bronze levels of service and you would configure volume types that either point to your backends or have the appropriate extra specs which we'll talk about here in a minute to access that storage. So the admin creates volume types and then the users when they wanna go and create their volume will see a list of types that they have access to and they can select which type they want at the time of creation of the volume. And then on the back end it goes through the scheduler and the scheduler finds the appropriate place for that storage to be created and provisioned. So I'm gonna, I'm a chicken. We're not doing a live demo. I do do this as a live demo when we do this internally at IBM. But for this we did some screenshots so that you'll be able to see them in the slides online just was afraid of having demoitis and everything going wrong. So we'll walk through Horizon so you can see how you do it from the Horizon interface and then also what it looks like when you're creating volume types and using extra specs from the Cinder client. But there's good parity here as far as the functionality between Horizon or using the Cinder client command line so whichever you're more comfortable with is you can use to work with your volume types. So the first example here is just showing, if you're in the admin view for volumes there'll be a tab up at the type up at the top that says volume types and you go in and you click on that and it'll show any types that you currently have configured for your system. There's the option to click and create a new volume type or you can click next to an existing type to be able to edit it to either change the extra specs associated with it or to change the name of that type if you wanted to do so. So here's the two key windows that you get coming off of that volume type screen. There's the create volume type interface where you give it a name and then optionally you can give it a description if you wanna describe what it is about that volume type that why you created it. In this case, we're just using, we've got the default LVM backend and we're wanting to add the store-wise SVC backend so I just named it store-wise SVC and then put horizon on there because I was creating it from the horizon interface. Once you've done that, then you go and you edit your volume type and this was one kind of tricky thing. It just has the option to show extra specs and that's the single interface to extra specs. So you go in, you show them and there's the option in there to create extra specs as well. So there isn't a separate tab off of the volume type screen for creating them, it's all under show extra specs. So that's one that always seems, kind of throws me off every time I'm playing around with the types. So what is an extra spec? In this example, it's the simplest example. It's a key that says volume backend name and I give it the name that's configured in that cinder.comp file and what that does is it says, if I'm using this volume type, I just wanna go to that backend. So when I go and I create a volume, it says, okay, can I provision the scheduler checks and sees can I provision from the store-wise backend and if it can it creates your storage or otherwise you get an error. That's a simple example of an extra spec. The reason we call them extra specs is because it's extra data that the vendors are able to choose to expose to users to control how they're using their backend storage. So each vendor has, and one thing we're working on fixing in the future is being able to have this interface query the backend and drop down a list of all the different specs you may want to be able to use for your backends. We don't have that yet so you have to go to the vendor documentation to see what extra specs are exposed up for that backend storage. But hopefully that'll be an area for improvement here to make a more user-friendly experience to the admin setting up the extra specs for interfacing with the different backends. So once you've done that, you end up, this is the interface that you see when listing the extra specs for a volume type. Now you can see I created that and we've got the volume backend name set to store-wise SVC. In a real-world example, I would have a number of other specs set in there that would then go to a scheduler and the scheduler would check and find which of the configured backends satisfies that requirement. And it may be, if you're not specifying it to a single backend, you may have multiple backends that satisfy the requirement at which point it'll balance between them or depending on which scheduler you're using, balance the allocation of the storage. So this is kind of the same process as seen from the CLI interface. Use the type create command and give it the name of the type you wanna create. In this case, instead of store-wise SVC horizon, I just created store-wise SVC. Use the type list command to then show what types you have created. So again, this was on a default dev stack installation where it came with LVM driver one initially and I added in the SVC type. From there, okay, so I got my type list. I wanna set up a type key. So you use the sender extra specs list. You can see initially my store-wise SVC type had nothing configured. The LVM driver one option comes with the backend name set by default. And then from there, I was able to use the type key and the set keyword to set that key value pair that I wanted configured on my type and I do a list again after that. You can see that I successfully set that key into the extra specs for that type. One thing I don't mention here, but to be aware of is there is a default type field in sender.conf that configures if you do not use a volume type when creating your volume, it will default to using that type. That was one thing that kinda threw me off once when I was initially playing with this. So if you use a type other than the LVM driver and you're using DevStack, you want it to default to something else. You need to go change that in your sender.conf file as well. So I have my type created. I have the backend configured with my extra spec and now it's time to actually create a volume using that. So I do the sender create. I call it sender volume and then I specify the volume type that I want to use. Obviously in Horizon it's easier. You got the name of it. Here you gotta use the key and give it a size and then it creates the volume and sends it off to the appropriate storage for the volume type that you choose. So now we're gonna talk about how you use your multiple backends with retype and migration. Thanks, Jay. All right, so there's been some confusion on retype versus migration. So we wanna make sure point that out when you would use one versus the other. So retype is really if you need to change an attribute of a volume where it's on the same backend but you just need to change a setting, probably one of the extra specs that Jay was talking about. So for example, my storage we have something called a replay profile say I create it with one replay profile initially, decide I need it set to a different one. The way to get that to happen is to do a retype. On the other hand, if I have a volume that I created say on LVM that currently doesn't support any type of replication and I want to be able to replicate that volume to my backup data center, in this case I actually need to move that volume off of LVM onto something that supports that functionality. In this case, I would wanna do a migrate to actually move all the data from where it is at the beginning over to a different backend. So hopefully it's a little clearer definition. One note while I'm thinking about it and Jay talked about setting extra specs. I just wanna point out the caveat that we've had this confusion before too. If you want to add an extra spec that wasn't there initially to set a value on a volume, you can't really just edit that volume type and change that extra spec. There's no hooks that go in and change all of your volumes. There's actually a lot of caveats with doing something like that. So really the recommended practice then would be to create a new volume type, set that extra spec that you want in the new volume type and then do a retype to change your volumes over. If it's something like just changing a setting on the array, typically it's a very quick operation. There's no data of migration, no data movement. So if you need to change something like that, do a retype, don't edit what's already out there. So to give an example of that retype, I kind of pulled out some pseudo extra specs here. In this case, I've got, it's called a replay profile. It's just something on the array operation that takes a snapshot. And it can be done nightly, it can be done hourly. So initially on the left hand side, I created a volume type that has nightly and I actually call out in here volume backend name. You can explicitly say what backend name you want a volume created on or you can just set attributes and then let the scheduler decide. So just for the purposes here, I'm explicitly calling out, it's on my array one, two, three, four, five. So both of them are on the same array. So I don't need to move anything. The only difference then is that second line under the extra specs that says storage type. Left hand side is nightly, right hand side is hourly. So in order to get that hourly setting, I, you can see I originally created a volume and I said volume type, del se one nightly. To get it over to be hourly, it's just sender retype volume one, del se one hourly. And that'll go out and change that type of the volume. On the other hand, if you look closely here, hopefully you can see that, but in the extra specs now, I've explicitly set that volume back in name to be a different array. So one, two, three, four, five on the left, five, four, three, two, one on the right. These are two separate storage arrays. In this case, if I would do a retype, same command, sender retype volume one, del se two hourly, that will fail because it has to go do a different array. It can't just send that change in the setting down. And this could be partially what causes some confusion is you can then do a retype and set migration policy on demand. So this tells sender, I want to retype my volume and if you need to migrate it to get that new type, go ahead and do that. You need to explicitly say this, we won't move any data around without being told to otherwise, because depending on the size of your volume, where it's moving to, this could be a big operation and take some time. On the other hand, you're not retyping and you just want to migrate data from one place to another. So you started with LBM, you got some new fancy backend from vendor X and you want to migrate your data off of the old storage or you're decommissioning something, you want to move it over. In this case, I've created the volume again, then sender migrate and you just tell, I want volume one, moved to whatever the name of that backend is. So we have the full syntax here, it's sender two at vendor X, pool one. This is some backend support, multiple pools. So this is sender two is the name of the host, vendor X is the name of the backend and then telling it pool one. So if you don't have multiple pools, it's a little more simple. So to give you some of those examples, again, hopefully you can see this, if not, maybe you can look at the recording later and get out the magnifying glass, but here I have sender list, you can see my volume, it's on store wise. I take a look at the sender type list, you can see I have a few different types out there, store wise SVC, store wise SVC horizon. It's the same backend, but one has slightly different attributes to it. So then I do sender retype, now in this case, being explicit, giving the entire ID of that volume for both the volume and the volume type, that makes sure there's no naming conflicts and it's very clear which ones you're doing. And then when I do the sender list, you can see what had previously been the volume type of store wise SVC, now shows up store wise SVC horizon. Same thing can be done from the horizon UI, changing the volume type, the name, the type, and you can see there there's the drop down migration policy. So that was the case where I was saying where you need to retype or you want to retype, but you have to migrate your data in order to get those new settings. This is where default is never, because we don't want to move that data around if we don't by mistake, but this is where you can say on demand and then we'll know that we can actually go ahead and move the data. And here again, now I have a volume, it's a volume type store wise SVC horizon. I do a migrate telling it to move it over to the LVM driver backend. And in this case, maps up both volumes, copies the data over. There's a lot of IO movements, that's a little more expensive operation, but it gets that volume from one backend over to the other. And again, the horizon. So wanted to talk a little bit about future functionality. Some of these aren't really directly related to multi backend, but there are some impact with multi backend or just things that you want to be aware of with them. We've started in Mitaka with working on active active high availability. It's in progress. So we have some stuff that landed in Mitaka. We're continuing that work in Newton. Hopefully we can get that all implemented, just being able to, you know, one of the reasons you might have multiple backends, just for redundancy, active active high availability, kind of brings that up into the center services so that you can build out a really robust infrastructure. Multi attach, being able to attach a volume to multiple instances, that's another kind of ongoing one. Hopefully we get a lot of progress in Newton, but there's a lot of little details there that aren't very obvious when we first start looking at it. Scalable backup was something that came in Mitaka before the backup service had to run on the sender service node and be able to mount it up, use the same volume drivers. We've actually done some work to be able to break that out and then potentially have multiple backup nodes. So you're not taking that hit on the same node. You can break out your backup onto a different node and have it attached there, copied over to backup. Looking into containerizing sender services, you know, there's containers in general. There's a lot of benefits of having everything kind of packaged in one. Might make it a little easier to deploy, upgrade, things like that. And we're looking at capability and reportings like was mentioned of those extra specs right now. It's really going to docs.opensac.org and looking at for vendor X, what are the extra specs? What can I do here? Hopefully with the capability reporting, we can kind of streamline in that user experience and be able to know, okay, if I pick vendor X and then I go into edit my extra specs, I can see what my different options are and I can hopefully, you know, somewhat user-intuitive and I don't need to go out and read a bunch of docs to do what I want to do. So with that, I guess everyone come back up on stage. We've got a couple of mics here and got some time for questions. So step on up and there's four lights. So hopefully someone gets that reference. So can you talk a little bit about access control? If you've got multiple volume types, do all tenants have access to all of those types or is there a way to control that? No, it's just attached to specific instances. So depending on the back end you use, some of them have distinct targets that each individual one attaches to. Some have a single target but then have different kind of access control to only allow access to those lines to the hosts you use specified. Not thinking about volumes but the actual types. So it sounds like you're saying that every tenant will be able to create volumes of every type. Per tenant, right? Yeah, yeah. There's also, there's, sorry. So yes, it is per tenant. There's also the ability to make them public too, I believe. So that way all tenants may see one specifically when you create it. Okay, any more? Go ahead. Yeah, so I have two questions. One, I'm still a little bit confused on the retype stuff. So when you did the retype from the replay, something setting, extra spec, the driver has to have some kind of a hook, right? Correct, yes. So a way that I think about this is that on a back end you could say, hey, I want, when you originally create the volume type, this is like the low end volume type where it's not even thin provision, right? It's full provisioned. So you have one type for that. The next type is thin provisioned, right? So both of those entries are extra specs in the volume type itself, right? And the driver that's associated with that volume back end has to understand what those key value pairs are and know what to do with it on its particular array. So for example, when you get a retype call that comes in, it lands on the driver that owns that volume, right? Where the volume is originally created. The driver then looks at the extra specs that are coming into the retype call itself and decides what it has to do to change this volume type from full to thin or thinner full, whichever one you're, which direction you're going. So there are hooks in the driver that allow it to actually see what those, all of those extra specs are. Some of the extra specs are vendor-specific too. So there are vendor-specific extra specs for three-par that are different than store-wise or Dell, and those are all documented on docs. Not all the drivers might actually implement the hooks. Correct. I've seen a bunch of drivers that don't. Yep, exactly. Okay. Last question, when you did the migrate from one back end to the other, when you had the volume on LVM and then you moved it over to whatever fancy storage, when you, to do a migrate like that, they have to be of the same type, right? So you actually change the back end for the default type where you put the two different back ends in the same type, right? Actually, I believe you specify the volume type that you want on the migrate command. They could be different volume types. Okay, but then what's the difference between that and retype and the migrate on demand? That's where it gets confusing. There really isn't at that point. And that's why it is confusing. And this is one of the reasons why some people hate that option that you can do migrate on demand with retype because it makes it, well, why am I not just doing migrate? You know, it doesn't make a lot of sense. So it's very confusing, but it's basically the same thing. Okay, thank you. Two questions. One is on migration, you've got the capability to migrate specific volumes. What about I want to evacuate an array of storage migration? We're five years into this. I know it's a long time to talk about an open stack, but we're five years into this, the hardware's in the warranty, or heaven forbid, my stuff cluster is just too slow now and I want to move to the store ice. Right, as it is right now, you would have to migrate each volume, retype each volume with migrate on demand or migrate over to the new array to evacuate everything off of the old array to get onto the new array. We don't have any single push button evacuate this array or decommission array. We're having a lot of conversations around replication. That's actually one of the things that I'm kind of interested if that's a use case that we need to do to support of replicating all data from one array to another and then basically clearing out that array. We've got some rudimentary pieces in to be able to do something like that, but as it is right now, it would take a lot of admin work so you're probably better off just doing the retype with migrate or migrate. Okay, and then can you speak to the guest impacts during migration, is there a VM pause and some device replugging going on behind the scenes or what? Great, thanks. So as far as I know right now, you can't do that with attached volumes. We don't have a way of having, so this is driven by Cinder, a migrate is driven by Cinder, right? So it's a call from Cinder client into Cinder to say, hey, I want you to migrate from this array to the next. Currently, there's no back channel talk from Cinder back to Nova to say, hey, if this volume is attached, I want you to queers IO right now so that way I can do the migration. It doesn't happen right now, so. So these are all dead migrations then. So live migration is a different thing. When you issue that, that is a Nova command where you're doing a migration of the VM instance itself from one compute host to another and that really involves with respect to the volume, you're just reattaching that same volume from one compute host to the next in the process and the VM instance itself kind of pauses at certain parts of that migration process itself. Is there a concept then between Nova and Cinder? Is there a concept of live migration from this array to that array, not to another compute host? No, Nova doesn't have, you need a guest agent to do that. Any more questions? Yeah, feel free. I don't, it's on. Can you guys talk about Cinder quota with multi backend types? So quota with multiple backends, okay. So that's obviously when you move from one backend to another, the quota that is being used on the one backend is decremented and it's increased on the backend that you move to and follows the location of the volume. Is there a specific question around that? So maybe I'm missing something Mataka. Do you guys actually support quota per backend type now? It's pertinent. Pertinent, yeah, but not per backend type pertinent. Right. So is there plans to address that? Maybe there's no other talk, but that's a... You would like per backend? Yeah, so Ceph and XYZ vendor, right, and you build set strict quotas so I don't go to read only, right? Okay, yeah, that's a good thought. Right, okay, so you could potentially do that with volume types per backend than in setting quotas on your volume type. Would be the current kind of hack to get that functionality. So yeah, that's actually a really good question. Quotas, at least nested quotas were not working well before Mataka, we've greatly improved that with Mataka. Basic non-nested quotas, that should have be working, going back for quite a few releases, so. When you do migrate, I would imagine you lose all your snaps and how about clones? Same goes for retype. Right, so. I'll test it. I think we error out if there's snaps because there's dependencies there. I honestly haven't tried that, so. Okay. It lets you migrate, okay. We'll fix that. Yeah, and there's, I think ideally you'd want the snapshots to migrate with the volume. I think with some backends, we probably can't do that, so probably the best we can do is just prevent it from happening in the first place. How about for retype? Does the sender provider has to provide the feature set to you? If you go to retype, obviously you're not within that range. Yeah, if it's retype without migration, the backend hopefully should be able to change whatever setting the retype is looking to change, and that shouldn't have any impact on existing snapshots that are out there. All right. All right, okay. Thank you. Thanks, guys.