 Well, it looks like we're right at about 11 o'clock. So all right, thanks everybody for coming. So today I want to talk about Cinder a little bit. Hopefully that's why you're here. The key here is we want to talk about how it's not just for secondary attached storage anymore. So a little introduction about me. My name is John Griffith. I'm a principal engineer at NetApp. I've been working on Cinder since about 2011. I work on mostly open stack upstream development and then also just all kinds of other things that are what I call cloudy. So whether that's containers or whatever. There's my contact info. If you want to get a hold of me, feel free, reach out. If you have any questions, you want to try and make some of this stuff work on your own, happy to help out. So IRC is usually the best way to get to me. Email for me is kind of a black hole. I get too many emails, so I tend to just ignore. So a couple of questions. So show of hands, how many people saw the keynote not demonstration yesterday? All right, cool, epic. So I am going to, if we have some time, I'll talk about that and maybe try and show you what was supposed to happen. So that could be kind of interesting. But first, let's talk a little bit about Cinder in kind of the general use cases that people see and use with Cinder. So predominantly, when people think of Cinder, the whole idea is to think of it in terms of secondary attached storage. And that context is usually always in the context of Nova instances. And that's great, because that's actually what we designed it for. And that was the intent initially when we started it. So what that means is the expectation is you boot up an instance in Nova, and then you create a volume in Cinder, you attach it, you mount it, put a file system on it, and you use it as your database backing store or some other form of data repository, whether that's logs or source code or whatever it might be. So that's cool. And that's kind of how we started out, and we were pretty static with that for a year or so. And then we started moving into this concept of boot from volume. So things like ephemeral instances and stuff like that, that's all well and good in theory and in Amazon. But what we found is most people in private cloud and running stuff in their data centers, they didn't like the whole concept of ephemerality. So they had a really hard time getting their heads grasped around it. So, and they wanted to run crazy things like Oracle and stuff like that. So we worked on some things around boot from volume. So the idea there is now you can take a cinder volume and you can create it, put a glance image on it, and then tell Nova to boot off of that volume. So that way anything you do on that instance or anything while it exists is gonna persist on that volume. So your instance goes away, you delete it, whatever. You can always bring it back and get all of that data back, right? So that ended up being really, really popular for a lot of people because what they found was they could do these really cool test harnesses. So they would have these test harnesses of instances and they would save them on these volumes so that they could spin them up on demand, run their test suite, run their CI or whatever, and then spin them down and put them away and bring them back up the next morning or the next commit or whatever it was. So that was kind of cool. The thing is, what a lot of people don't realize is inside of an open stack setup there's actually a lot of data being consumed in various places, right? So Glance is a big one obviously because you're putting all of your open stack images up there. So if you have a Ubuntu server image that's custom and it has a gig worth of data or whatever, you're actually using up all that storage for that image somewhere. In most cases, people are using Object Store for that and that's great. Object Store is a great fit for this type of thing. Works fantastic. It's cheap, it's deep, it's a good way to go. Other folks like me are commonly using local file system. That's not so good because there's no efficiencies or anything else to be gained there and it's kind of expensive, relatively speaking. So Glance is a big data hog, but the other thing is, to keep in mind, is Nova is actually using up a lot of data too. So anytime you put up an instance in Nova and you're doing things on the root disk and everything, you're actually creating a Cal file on Nova and you're using that space. Now, Nova does a good job in terms of, if you select to use Cal files, it's gonna go ahead and do copy on right and everything and save you and help you out and that's great. But you're still using a lot of space and you're using that up. So the other thing is, just kind of keep in mind, is we talk about block storage, but there's a world outside of open stack. So there's all these services in open stack and they're great and everything, but most of them actually work pretty well outside of open stack as well. So just something to keep in mind, a little preview. One of the things that, on boot from volume, I had mentioned that it was something that became a popular use case and everything else. One of the problems that a lot of people didn't really realize is the way it's implemented. It's kind of ugly and very inefficient. So the way it works in a traditional or an older deployment is you actually have your images stored out in Glance, whether that's on an object store or file or whatever, it doesn't matter. But when you go to Cinder and you say, hey, create a volume from an image, what it actually does is it goes out and sends an HTTP request to the Glance server and says, hey, get that image and pull it down and copy it onto the Cinder volume node. So it's gonna go through and it's gonna pull that whole image down. And once that's done, the fund's not over, the next thing it has to do is it has to do a QMU convert on it. So it has to convert it from the QCal format or whatever it is into a raw format. Even if the format you have in Glance is raw, it's still gonna do that conversion because it's a security issue, right? Because who knows what somebody may have injected into that image. So we go through this whole big process and we do a conversion on it. And then finally after that, we mount the volume to the Cinder volume node, the newly created volume, and we DD that image over onto the volume, right? So it's pretty, it works, and it works fairly well, but it's pretty awful when you dig into it and look at the details. So this is kind of a depiction of how a standard traditional open stack deployment works with volumes and stuff. So you've got your, in this case, you have your image store, you have Glance, you have your compute node. What we're doing here is we're actually pulling that image over from the image store, putting it on the compute node, booting off of it, and then attaching a volume from Cinder. Now, how many people are using Ceph in a deployment? Okay, so a good number of people. So the thing about Ceph that's kind of interesting is it can be an object, a block, and a file system store all at the same time, right? So that's kind of cool, right? And that's neat. The thing is, since it can be all of those things, it can also be the backing for all of the services in an open stack all at the same time. So you get a lot of efficiencies in terms of utilization and moving things around and stuff like that, and I'll show you a little bit what I mean about that in a second. But the problem is it's good at a lot of things, but it's not great at one thing, right? So, and it's also, you know, it's just one option out of 80, right? So there's 80 different backends that you can use in Cinder. So the way it works with Ceph is you've got, for example, Nova, Glance and Cinder. They talk through the RBD layer to the Ceph cluster. So that example I gave of taking a image from Glance, downloading it to Cinder, copying it to a volume, it works significantly different because what we do now is Cinder just talks to RBD and says, hey, I want this image. And then the cluster, the Ceph cluster, does all of that work for you internally, right? And it does it really fast, really efficiently, without all the overhead, all the conversion nonsense, everything else. So it's a pretty big deal. So a number of us were looking at this over the years and came up with this idea of, hey, we can do the same thing with other devices. Because a lot of devices have the intelligence to do this. The only difference is we're gonna do everything as a block device, right? So we came up with this idea of an all block model for an open stack deployment. So caveat, this doesn't work for every back end in Cinder right now. So the reason why is because you need some special techniques, right? You need things like fast cloning, you're probably gonna want deduplication, stuff like that, right? So that makes a big difference. If you don't have those things, for example, the LVM driver does not have those, this type of a deployment may not be a really good idea for you. Anyway, the idea is we just turn everything into block and we just deal with block the whole way across the stack. And that actually works really well because keep in mind, whether you're an object store or a file shareer or whatever you are, you're all built on block anyway. So you're using the same underlying thing. The difference here is a lot of us have optimized block devices that we can use and use really well. So we're gonna go ahead and exploit those features. So using Cinder as a backing for glance, right? And that's kind of the key of this talk. The key here is I've got this warning. If you're not doing boot from volume and that's not something that you're interested in, don't even bother, right? Because there's absolutely no reason why you would wanna do this. You should use the file system, you should use an object store, whatever, it doesn't matter. On the other hand, if you're not doing boot from volume, I would say, why aren't you doing boot from volume? Because quite frankly, it's a much better model for most people. So it's definitely something you should consider. So it's important that you're gonna utilize the capability here that you do boot from volume. It's also super important that you're gonna use a backend device that has things like fast and efficient cloning. Because if you don't, a lot of this is lost. You're gonna end up back in the same boat doing copies of files and DDing and everything else, right? So you don't want that. I am working on trying to make this work with LVM. I don't know if there's any LVM experts in the room. I've played around with a couple of ideas with mirroring as well as mountable snapshots. I haven't come up with a good way to make it fit inside of Cinder's architecture, but if anybody wants to chat about that or has any ideas, I would love to talk to you. So, but stay tuned, because I am going to make it work. I don't know when, but I'm gonna make it work. Okay, so, making this work in Glance. This actually turns out to be super, super simple. All you have to do is make a few modifications to your Glance API Conf, right? So we'll go into our Glance API Conf, and you can look here. I've got this, oops, don't do that. I've got this stores line, right? So there's a number of things you can put in here. You can put file, HTTP, Cinder, et cetera, Swift. The key is you can actually put multiple things in there. So you can have Glance actually go an attempt in order of how you put them in, which location you go to, right? We're gonna go ahead and we're gonna change this and we're just gonna say use Cinder. And then we're gonna make our default store Cinder as well. There's some things you need in here, state transition timeout, things like that. These are things to deal with transfer times, things like that so you don't get all kinds of funky RPC errors and stuff like that. So those are good. And then you're gonna need your credentials, right? So you're gonna need your Cinder credentials and you're gonna need your credentials to talk to Keystone and how to get the endpoint and everything from Keystone to do all the login. So it's actually pretty simple. Those are the things we have there. And then after that, you just restart the Glance API server. And then you can simply just do like you've always done as far as an image create, right? So download your QCAL file, do the image create, upload it and you're good to go. So there's an example of creating a trusty image. Like I said, nothing any different. But the key is what's actually happening when you do this, right? If you've got a back end that has the cloning and everything set up, what's gonna happen is the next time you come out and you call create volume from image, the Cinder driver for that back end is gonna look and it's gonna actually query its cache. So Patrick put in a global cache mechanism inside of Cinder a while back. It's gonna look at that cache and it's gonna say, hey, do I already have this image with this metadata stored on this back end somewhere? And if it does, then it's gonna go and it's just gonna clone that volume and create a new one from it and give you that volume. So the difference here is, it used to be if you take a standard Ubuntu or standard CentOS image and you create a volume from it, you could be to looking at up to a minute, right? Maybe two minutes depending on the size. The difference with this is you're looking at maybe a second or two, right? So that's a pretty significant improvement. And the other thing is you get rid of all of this back and forth, download, convert, DD. One of my favorite bugs that we got a while back was people reporting all these errors in the logs that turned out they were running out of space, no disk space. And it turned out what was happening was you download that image, you run the conversion on it and when you can run the conversion on it, it blows up and they run out of disk space and it fails, right? And we didn't log it, so it took forever to find. But so anyway, all of that goes away. The other thing is if you've got a back end device that supports deduplication and does deduplication, then you get all that for free, right? So you get some efficiencies there. Images are one of those things that deduplication is actually extremely important because what happens there is most people have 15 or 20 different Linux images that they use and they may have hundreds of them booted up. But if you've got deduplication, those are all gonna dedu bound to almost nothing. So you're really, really gaining a lot of efficiencies there. All right, so from there, we can now just do standard boot from volume stuff. So there's no difference here. It doesn't behave any different. It's all seamless to the end user. So the first example I have here is a multi-step process. What you do here is you actually create the volume first, right, and you say create from image, and then you go ahead and you give that spec to Nova and tell Nova to boot that. So that's cool, but what's even better is the second version is you just do that all at once. And what this will do is it will send the, you send the Nova volume attach command and it will actually have Nova go out to sender and create the volume, put the image on it, do all those things and everything and then bring it back and then boot it up. Now, the problem with this in the old model, if you're not doing something that has some kind of cloning or some kind of efficiencies or anything like that, the problem with this was it would typically take too long and you would usually get RPC timeouts and errors. So it was actually a feature that was put in a long time ago that we've actually talked about trying to take out because it actually caused more problems than it solved. But if you get to this model where you have a cache or if you have a block device as the back end and you can do this stuff, it solves all this problem for you so you don't have this issue anymore. So there's a couple of things that we've been talking about for the last several months to kind of make this better, right? So the standard workflow for creating an instance would be open stack server create, flavor, image ID, name, right? Key pair, whatever. That's cool and it's simple. It's easy. The thing with the volume is that command's not so simple anymore, right? You've got all these different variables you have to put in and all these things you have to keep in mind and stuff. The other thing is, is what we found out is there's a lot of people that are actually trying to run open stack compute nodes or Nova compute nodes on smaller diskless servers or servers with like a 20 gig flash drive and stuff like that. It's actually getting to be fairly common. So the idea was we should give people a way to actually configure their cloud to just say always boot from a volume, right? So always go through this process of creating the volume and everything else and using that. And honestly, really when you put it all together what we're trying to do is just kind of be more like how Ceph works, right? Because it does things really efficiently and we want to kind of match that. So for everybody. So we've got some proposals out there as far as how to do this. This way the user won't need to deal with it, won't need to know about it, doesn't even have to know about the volumes, none of that. So even ephemeral, you can configure it so that an ephemeral device is now a cinder block device, a cinder volume. The only difference is on termination of the instance, you delete that block device, right? So we've got a patch that we proposed. The patches, the nice way somebody put it was a hack. So it was my patch. So basically all I did was I allowed you to put a block device mapping inside of the extra specs of a flavor. So you could select that flavor, it would go out and Nova would detect that and then go out to cinder and get the volume and do everything for you. It was pretty cool, but it's not gonna work and it's not gonna get merged. So what we're looking at now is actually adding a flag to the flavors that says force boot from volume or something like that. That will kind of do the same sort of thing. And then long term, what we're gonna do is take the entire image backend driver inside of Nova and we're gonna convert it into a cinder image backend driver. So one big driver that handles anything that cinder supports. So that includes Seth, Pure, NetApp, SolidFire, Hitachi, EMC, everything. So that's gonna actually give us a ton of improvements and a ton of efficiency benefits. But unfortunately it's also gonna be a lot of work and take a long time before we get it done. So as far as the using block for everything in OpenStack, I mean, that's kind of it. It's really a pretty simple idea. It works very well actually. The only trick is if you're running CentOS or REL, you're gonna need to change some of your security settings to allow this to go through. We've got those documented so we can help you with that. So that's about it on that side. The next thing is, I alluded to the idea of using cinder as something outside of OpenStack. So ideally, you've got this investment in your cloud or in OpenStack cloud and you've got cinder, you can use it for other things. So what I thought I would do, since we have a little time, unless there's some questions, is I can show a screencast of what was supposed to happen in the keynote yesterday, if anybody wants to see that. First I guess I'll ask, does anybody have any questions? Yeah, I'll just repeat what you, yep. And I, you know, eventually I directly pull the glance image from, you know, because if you're slow on the camera, it'll be exposing you, right? But it's easier, you get cinder, you configure it, you get this driver, you configure it to know the path to your glance images and it just makes a sparse clone. And it just goes, I can move, you know, a one gigabyte and a one terabyte to be able to the same amount of time. Yep, yeah. Yeah, it's, you know, most people, once they start using it, especially service providers, they never look back, right? And keep in mind, so he had mentioned, you know, GPFS, you can do this with NFS, you can do this with block, you can do, there's all kinds of, the other thing that's interesting to me and what actually started a lot of this a long time ago was people have discovered all kinds of little tricks to try and make everything a little bit more efficient and better to do something sort of like this. So that's kind of why we decided to just make this a feature and make it something you can configure and do, right? So, you know, if you've got a test cloud or, you know, a development cloud, I'd highly recommend checking this out and looking into it. There's a lot of advantages that you'll realize from it. And the other thing is there was some misconception for a while that if you do things like this, you can't do live migration. And that's actually not true. So live migration does work with attached sender devices. There were some bugs back in the Juno release and I think the Kilo release, but I think all those fixes were not, have not only been fixed, but I also think they've all been back ported. If they haven't, if your distro hasn't updated those, you need to tell them that they need to update them. So, because that's unacceptable. Does anybody else have a question? Yeah, multi-attach feature. Yeah, so that's a fun one. Multi-attach has been an interesting battle in OpenStack, right? So you may have noticed there's an inside of glance when you use this model. We do have the ability to set multi-attach on volumes, right? And that kind of works. And the idea there was you could do a multi-attach and you could do read only so that you could use different volumes at the same time, or the same volume for different instances and everything else and do the deltas to a QCal file or whatever. What we've kind of, where we've kind of ended up on that though is, we need like a real global option to do multi-attach with Cinder, period, right? Not just with images, not just with glance backing and stuff like that. So we're actually in the process of rewriting our attach APIs to enable multi-attach. And that will significantly change a lot of this and make it a lot better, a lot easier. So you could do things conceivably, you could do things like boot multiple instances off of the same volume. You wouldn't even need to do the clone anymore, right? Yeah, so the question he had was, is it a feature that he can use right now or if you have, for example, 10 instances and you want them all to be read only to that volume? You can actually make that work today. It's not tested. So I would say, when I say you can make it work today, you can set it up, configure it and try and run it. You're probably gonna have to dig through the code and do some things to make sure it works. But it's not something that we test in the open stack space, but interestingly enough, there are folks using Cinder in the Docker space, as a standalone service using Docker. And they use that explicitly almost everywhere, right? And it works fine for them. So there may be some hiccups, there may be some bumps along the way. I would recommend you test it out, try it. And if it doesn't work, file a bug, we can get it fixed. Or ideally, if you can wait a few months, hopefully we'll get the next release out and we'll actually have real multi-attach crossing fingers. Well, there's the other thing. So this is Ildiko and she's actually been spearheading the whole multi-attach effort. So that's another thing for folks that are developers in the room or even people that aren't developers but have a use case for multi-attach and would like to see it. We would love anybody to get involved and help out. And whether that be through code, whether that be through use cases, testing, reviews, documentation, whatever, we can use all the help we can get. So that would be fantastic. Yeah, so let me see if I can paraphrase a little bit, right? So your point was on the multi-attach going forward. If there's a plan for limiting access, who has access? So we're not there yet, but kind of the idea of where we are going with that is similar. You would have the ability, for example, to say, for example, you have A, B, and C sharing that volume, A is configured to be read-write, the rest are read-only, right? We are looking at the ability to flip that. So change A to read-only and give read-write to B. There are also, people do want to actually just do read-write across all of them, right? And that's something that I think will probably be configurable and you'll be allowed to do. But the problem is, is we take no responsibility in terms of whether you use the share file system or not. So we don't have any knowledge, we can't control that, so if you do that, it's gonna be at your own risk. Anyone else have a question? Is there any, so how many people are running OpenStack? Okay, so quite a few. How many people are using, I already asked Seth, how many people are using LVM, anyone? Hey, all right, there's not that many less than Seth. So then everybody else is running, let's see, how many of you are using NetApp? All the NetApp guys. How many are using Pure? Any SolidFire? Woo! Hey, hey, all right. IBM, HP. All right, what are you guys using? How many people use Boot from Volume in their cloud? Okay, a few. The rest of you, why not? Does anybody have anything against Boot from Volume? What's that? Yeah? So it's gonna convert it to raw, no matter what. So yeah, you can't actually just, you can upload KUKAO and say you're using KUKAO, but behind the scenes, it's converting it and making it raw. And no, we don't have a way to get around that right now. The reason why is we actually did submit a patch a couple of years ago that did bypass some of that, right? So if you could do the file system translation or everything and use a KUKAO, it would do that. The other thing we did was as we said, oh, well, okay, that's really dicey because most devices can't handle it. So what we'll do instead is we'll throw it up there as raw and then you just pull down the raw and then we won't convert it and that'll be more efficient. The problem is that got flagged as a security violation because again, users have the ability to upload images. They could put whatever they want in that raw file and there could be an exploit and they get access to the host machine, blah, blah, blah. The KUKAO thing, you can't do that right now unless you're using something that's a file share, right? So I believe you can do that sort of thing with an NFS backend and with GPFS, things like that, but not with like a true Cinder block device. It would be interesting to look at that and see if there are some tricks that we could do in terms of not just doing raw devices only, but actually mounting the device with a file system on it and use it in reading a Cal file. And I think that's possible. The only thing that has kind of kept anybody from looking at that, I think, is it's a lot of management and state transitions inside of Cinder that I don't think anybody wants to deal with. I think we've got, we only have five more minutes. No, so the nice thing about most of this is depending on what version of OpenStack you're on, you can get all of this for free from the manager and most of the stuff inside of Cinder. There isn't really anything specific for the back ends. There's a couple of calls in terms of creating the image and stuff like that, but those are all things that they're supposed to have implemented, so it shouldn't be a problem. You shouldn't need anything special. So if you run into something you should file a bug because there's somebody that somehow slipped under the radar. Thanks everyone, appreciate it.