 Cool. All right, I think it's about time. So, late Friday, last day of the conference. So, thanks for being here, instead of napping or going to the airport or getting out of here. So, we're going to go through some things and what we want to talk about today is using Cinder as a storage, block storage provider for Kubernetes. So, we'll go through and do some introductions here and get rolling. First off, my name is John Griffith. I've been working on OpenStack, the Cinder project. I helped start that about six years ago when it first came to be a thing. So, I've been working on that. I've been doing some things in the container space as well and all kinds of anything cloudy, basically, these days I kind of work on a little bit and have various things. All my contacts info is up here. If you ever want to reach out, ask me any questions, anything follow-up on today, give me a holler. I love to talk. So, sometimes that's the only thing I'm really good at. My name is Juan Michien. I work for RedHass. And just at the time, Kubernetes took off. I wrote some bunch of proteins. So, if you are using one of those, iSCSI 5 channel, RBD staff, whatever. And also, I'm the one to play. And also, my GitHub ID is on my Twitter. Feel free to follow me. All right. So, what we kind of want to do today is, Juan and I have worked on a number of things with Cinder together over the past four, yeah, it's been a while. So, we're going to kind of walk through. And for those of you who may not be familiar with Cinder, I'm going to talk a little bit about that and go through some of that. And then, Juan's going to talk about what the volume and the persistence model in Kubernetes looks like today. And then, we're going to kind of tag team what we've put together to use Cinder to solve some of those problems. And then, talk about where we're going to go, what the future looks like. And then, hopefully, get some dialogue going with you folks, maybe get some input, kind of go from there. So, first off, in terms of Cinder and what it is, so the idea of Cinder, way back when, was to create a block storage as a service, right? So, the whole idea was to give an abstraction, to give something that could be used by a cloud, particularly OpenStack, that would let you plug in various backends and seamlessly consume them as a user. The whole idea is, you can scale this out, you can scale this in, you can dynamically change things behind the scenes without your users ever knowing, right? So, that was the whole idea. One of the other things that some people aren't really aware of is, we did have all along this idea of having it be a standalone service. So, if you're familiar with OpenStack, you may know of a project called Swift. They had sort of the same model, it's an object store. It can be used in OpenStack or outside of OpenStack, right? And so, that's kind of the same thing we want to do with Cinder. We want to make it something that is useful for OpenStack and that may be its primary use case. But it's extremely valuable, as you can see these days, because there's at least eight or nine that I count right now, projects out there that are trying to abstract storage devices into an interface, right? So, we've already got one. It's been out there for almost seven years. It's got over 70 or 80 backends that are supported. And it's been used in production by really, really large clouds for a long time. So, it's battle-tested, it's got a ton of features. We've had thousands and thousands of bug fixes over the years. So, it's solid, it's rock solid. So, it's a really good choice. And best of all, we've got a great community with over 500 contributors. And that's just code contributors. So, it's a pretty huge deal. As I mentioned, it's not something that is just for OpenStack. So, I've been doing this crazy demo thing for a couple of years now on using Cinder as a standalone thing in Docker, as well as Kubernetes and Swarm and everything else. And for anybody that's seen my demos, you probably know why I'm not going to do a live demo today. So, I used to have good odds. And right now, these days, I'm at like a two-in-ten shot. So, I just don't do them anymore. But that's okay. I put the stuff on my blog and you can check it out there. But the cool thing is, one of the things that we did recently is we went ahead and we put stuff in the Cinder repository in the master tree to actually containerize Cinder and run it as a service using a Docker compose file. So, for all those people out there that you hear, you know, OpenStack projects are hard to deploy and they're difficult to upgrade and blah, blah, blah. The whole idea behind this was just to say, hey, you know, a lot of that is not true. So, inside of the contrib directory, inside of the Cinder repo, there's a compose file. You can go in and you can do a make to build the images, the Docker images, and then you can do a compose up and actually have a full-blown running Cinder service on your system. Now, I say, you know, oh, that's it. You're done, right? Okay. It's not really that simple if you want to do something in production. But if you want to just try it out, it's extremely simple. So, you can take that. You can modify it and everything. So, check it out. It's on GitHub. It's pretty cool. So, of course, it's cool, right? So, these are the steps. You clone the repo. You CD into the contrib directory. You run a make. What the make does is it uses a project called Loki that builds all of the images that you need. So, you're going to get a Cinder API container, a Cinder volume container, a controller container, a database container, and a RabbitMQ container. It's going to build those up. You're going to modify your compose file if you want to, run compose up, and that's it. You're ready to go. So, if you want to modify it and use a back-end that you have, maybe you have a Seth back-end or a NetApp back-end or a Hitachi back-end, some other back-end that's supported in Cinder that you want to plug into it, you can just modify the conf file, just like you normally do, and you now have support with the standalone thing. Now, you can use this for things like local storage usage and storage consumption, or you can get into really cool things and plug it into Kubernetes. So, I'll let Waman talk a little bit about that. So, Kubernetes, that's why I'm bad at all. That is something we want to make Cinder to happen and to manage the storage that you previously have orchestrated by Cinder. If you look at the Kubernetes volumes, there are two types, the ones that works for hypervisors and the ones that works on environmental. The ones that work for hypervisors, what you have is volume ID, like AWS, EBS, you get a volume ID, and on Cinder, you get another volume ID. It's an abstraction of the reference in a database that's only the cloud and the hypervisor knows. But on the other hand, when you're working with environmental, you get all kinds of PV types that gives you detailed information on how to connect the host to the backend storage. Just take an instance of the NFS. You have the server's IP address, you have the exports, and from that point, you can make a month. If you go through the ice-cloth and the clusters, things like that, you know the endpoints and you know the path where the volumes can be attached. And that is something we really want to have from Cinder and how this magic works. It's from the abstraction, the volume ID, to the endpoints that's associated with each of the basic PVs, and that's the thing we want to make that happen in this project. And why do you want to make that happen? We want to take advantage of all the features that Cinder already provided to us, and that's our last present to Kubernetes yet. If you have one for volumes, backends, and that have all the features already worked pretty well on Cinder, but you don't have these features working on Kubernetes yet, then you can just leverage Cinder to make all this happen to your one for storage. And so far, we have a lot of things in Kubernetes, the dynamic provisioning, the attach and detach, meaning that you can do a third-part attach and detach, and we also recently added this, a volume expand, so-called resize feature, and that just happens in the 1.9. And a snapshot, this works out of three, but it's almost functioning ready. It works on limited storage backends. Cinder happens to be one of those, and thanks for the wonderful contributors. Before we go into details, let's look at how the volumes, Cinder volume works for in Kubernetes. So you have the Kubernetes controller master, a bunch of controllers running over there, and things are interesting to us is the provisioners and attach and detach controllers. So if you create the volumes, you call the Cinder SDK, provision creates essentially equivalence to what Cinder creates, commands gives to you, and you get the volume, it's a volume ID, and from that point, you call the SDK to attach this volume to a certain Nova instance, and under the hood, Nova, as well as the OS Brick, details, I guess, detailed information about the connection, and then the touch happens. Just like that. Oh, yeah. You know, I just shaved my beard this morning. Yeah. So this is the model we are used to, if you run Kubernetes and OpenShift on OpenStack, this is how it works. Now, we're going to change the flow a little bit just to make sure the bare models, the host machines can get the detailed connection info, so the host can make direct connections without OS Brick and without the Nova. So how it works, and the picture looks pretty much the same. We don't add a lot of stuff, except there's some magic happens over there. Before we go to details, here is the workflow. So we still have the stock Elemental Kubernetes cluster, nothing changed, and we still use the cinder volume and everything, the stock cinder, and it could be containerized or it could be a standing OpenStack setup, and we still use the existing storage types, iSCSI Fab channel and RVD, nothing changed over there. At the high level, the only thing we add to the picture is pretty much standard. We have an external cinder volume provisioner, that's our whole trick for us, and the workflow is, we request for PVC, obviously we have to define storage class, and once the storage class is defined and your PVC use that storage class, then it's going to our provisioner, and the provisioner talk to cinder and use the cinder API and increase the volume, just still use the cinder API and then get the connection info, and still that is also cinder API, so nothing inclusive, pretty much standard. And from that point, we do not return the cinder volume ID, but we return the underlying connection info, if that is iSCSI, we will convert the cinder volume to iSCSI volume, if that's RVD, we get the RVD volume, so everything happens over there. From the user's endpoints and from the Kubernetes perspective, pretty much nothing changed. And you don't need a hypervisor anymore, and you don't need an OS brick anymore. It's not inclusive. Cool. So what we're going to do here is we're going to kind of walk through the process a little bit and kind of click these slides. So I said we raise questions and stuff at the end. Also, as we're going through these, if you have questions, we've got plenty of time, we'll shout out and we'll kind of dig into it, right? So it's actually really simple. So we've got this external provisioner. It's sitting out there and it's listening and it's monitoring, and it's basically looking to see for PVC requests on the Kubernetes side, right? Most of the external plugins these days, this is how they all work. This isn't anything revolutionary or new or anything like that. Well, it was, yeah, when we first did it. So let me take that back. Now everybody copied us. All right, so the first thing that comes in, the provisioner, it sees a request for a PVC, capacity 100 gig, access mode read-write, storage class is sender, and a volume type. So one of the things you can do is, so sender has the ability to specify volume types and that can give you things like replication, quality of service, compression features, deduplication features, all that stuff, any custom feature, that's where that goes, right? So it'll come in your pod file and expose that this way and that'll go through. The next thing that'll happen is the provisioner will go and just call out to sender, just like anything else. It'll use the sender API, it'll do a sender create request, give it all the parameters that you fed in, everything that's needed, do that, and it'll come back and it'll just give the sender volume ID. So now the provisioner will have a sender volume ID, so it's got a handle, knows what to do, and needs it with the volume handle that you put in your pod file. So that's the dynamic provisioning piece, which is fairly simple. The next part is the attachment. Attachment is pretty simple. All right, so we do kind of the same thing. The provisioner gets the request to do an attach, so what happens is it goes ahead and it calls out to sender and it does what sender expects, which is called a reserve, and the volume is something that's flagged, is going to be used and attached. It'll then go and it'll call an initialized connection, and this is actually the important call. So this call is what includes all of the things like for iSCSI, for example, it includes the initiator IQN, the host IP address, all that sort of thing. So that's where all the data actually is. And then it responds, and you can see over here on the left, it goes ahead and responds back to the provisioner and back to Kubernetes with all that, and then we call the Kubernetes iSCSI mounter, or RBD mounter. So then we call that and we go ahead and we attach that volume to the bare metal node. So now you have a sender volume attached to bare metal. Connection's all done. We notify sender that we're finished, which then in turn calls an attachment completion, which on the sender side marks the volume as in use. Now the thing that's cool about this is as Kubernetes progresses, as the Kubernetes APIs for storage progress and things like that, there's a lot of things that people want to do, and they can't do yet because it's not in the API. But the thing that's cool is now you have this other piece, you have the sender piece. So you can go in from here and you can do things like cloning and replication and all of those things that may or may not be in the Kubernetes API anytime soon. Yeah, Lewis? We're going to address that for the in-slice. Good call, though. Anything else on this one before we... Okay, you want to talk about this one? Yeah, sure. And before we go there, I will have to make a big thank for Adam and Adam to make all this happen. Otherwise, we are just standing here and just talking on paper. The co-design is actually in the Kubernetes external storage and the new incubator has been heavily contributed by community helpers. It works pretty well on the number of plugins on iXXC and RBD. These are the first because iXC obviously gets the best adoption so far. Back to Lewis's question, the QEMU and the QCAL that potentially could be the bridger for things like Gloucester and NFS that does not have the native block device support because they essentially provisions the block device and the block device on the NFS or Gloucester is just a file emulation. So the TCM runner could be the bridging technology for that to happen and also QEMU. Alright. As John mentioned, we leverage Cinder a lot not just by the drivers' repositories but also by Cinder's rich set of features and for things that happen to Kubernetes yet or haven't happened to the drivers yet and if it works for Cinder, it's going to work out of the box for your backend. If you want for iXXC backend, that is already working in Cinder and you really have the snapshot controller working on the backend. Just use Cinder for that. Because right now, just take for another example, the volumes resized in Kubernetes does not work on individual drivers yet but it works for Cinder. If you need this feature and you don't have that on iXXC plugins, use Cinder. And replication, that hasn't happened to Kubernetes yet, already have it so that's another place. Right. CSI, I know that's a hot topic and where we can go from here. Just before this presentation, we just merged the Cinder CSI driver for the convenience. It doesn't have all these cool features but it's going to be a good starting point and from that point, from there on, we have enablements for lots of drivers without having to have individual drivers sitting underneath. And so for those, well, I know there's already so many CSI presentations, just for those who haven't gone to those presentations, CSI so far, in leaving the repositories, they are provisionals and that's why just create and release volumes. Attachers, that just manage the attachments and the drivers, that manage the node mounts and unmounts. So as you see, Cinder, the Cinder CSI driver already handles all these three. All we need to do is to expose the connecting info and then hopefully everything will work from there. So, you know, a couple of things to point out. One of the reasons why this is kind of valuable right now is sort of part of a transition process, right? So one of the things that I didn't really touch on about Cinder and the way it works as far as the drivers, on the Cinder side, to submit a driver into that code base, you have to actually fulfill a set of behaviors. And so we actually run a continuous integration test on every patch that gets submitted and it has to run against every one of the drivers in the tree. So that gives you a little bit more in terms of being able to support it and make sure everything works and stuff like that. And then best of all, what it means is you can swap devices in and out from behind this without your users ever knowing, without your applications ever knowing and not impacting any of your workflow, right? So that's where the advantage is. Granted, it's not for everybody. You know, there's some people that definitely are going to want to go and write their own CSI driver or whatever plugin and stuff like that. And that's great. This is just another option for a lot of people that I think has a lot of benefit, especially when it comes to the support and testing model, right? Because this has been tested pretty thoroughly. So I think with that... Oh, you had some... I kind of just touched that. So anyway, I think that's about pretty much what we got. Hopefully you guys got some questions. Maybe you're interested or feedback, crickets. All right. You can reach out to either one of us, IRC, Slack, email, carrier pigeon, GitHub. Yeah. Not if they're iSCSI or RBD. So as long as there are those two protocols, they work. Other protocols like NFS and things like that... You know, that's... Yep. iSCSI and RBD, Fiber Channel. We haven't done anything with Fiber Channel yet. But you can write a patch. You know how to do it. You know how to do it. You're already familiar with this. I know you. So the question was, have we done anything with AZ Awareness and Cinder? So if you're familiar with Cinder, it has availability zone type model. We haven't really put any of that into this provisioner yet. Because quite frankly, right now, I don't know... Yeah, Kubernetes doesn't have any way to interpret that, right? Now, there might be something interesting you could do in your pod file to try and distribute your storage a little bit. There might be something you could do, but that's only half of it, right? But again, pull request. It would be awesome. Yes, sir. Actually, I forgot to talk about that. So I started working last night a little bit, but it was a really late night, so I didn't get very far, on converting this to run-in pods and possibly be dumped into a home chart. It's not there right now, but it certainly can be. So you and I are going to talk here in a few minutes, and we're going to look at doing that. Because that would be pretty cool, right? Because you could say, hey, run this script to launch a Cinder service inside of my Kubernetes deployment, and then use that and consume it from my Kubernetes deployment. So it would be pretty cool. Just some piggyback for this question. So there's actually charts for the informative, external storage, and there's PR going on from Mr. Sergei. Oh, excellent. Totally not support, that's fine. But something along those lines. Cool. Anyone else? Cool. Well, thanks, everybody. Again, especially on a Friday last day. Thank you.