 Please welcome our next speaker, Bradley Charles from Red Hat, who will be talking about features in Kubernetes storage. Good afternoon, you guys. So have we enjoyed very much? So today I'm going to talk about the Kubernetes storage features. My laptop doesn't go to sleep. I kind of struggled with what to talk about here. We have three different talks about storage in Kubernetes. Steve White yesterday gave a pretty good, high-level overview of some of the features that we support. And then we have Vian Sanfranek, who's going to give a talk later, going deep into the ins and outs of each of the OpenShift and Kubernetes storage features. So I thought today I'd probably, you know, give some overview of containers and some of the confusion that comes along with persistent storage in containers. Because containers are a pretty cool concept. It really enables these microservice architectures. But one of the key features that allows these architectures to run is the ephemeral states nature of the containers. So you'll have your container, you have your image. You don't want to keep state in the container itself, but you still want to persist data someplace. Right? You want your containers to be able to run portably across different nodes. And so you can't change the state or the container image itself. You can't store any data there, but you still have to persist it someplace. So there's always some confusion around, like, how do I save data when I'm running these microservice architectures using these container patterns? And, you know, the stateless nature of the containers makes them fault tolerant. It makes them easy to scale, makes them lightly importable. But persisting the data is still a, you know, a common ongoing problem. So we've been working to add features to Kubernetes to allow you to persist data. And it's, I don't know, been about a two-year journey. We started out with some pretty simple volume architecture where you could very specifically define what storage your pods would use. Then we added different file system types. We had support for iSCSI, Cinder, Gluster, Ceph. Then we started, you know, adding stuff like dynamic provisioning. We started enhancing security. And then we started adding, you know, some abstractions. So it would allow reportability, you know, of your application from, you know, different environments. Now go into that kind of stuff later. So our first feature really was the straight-up volume plugin framework. And over time we added support for, you know, we started off with just host path. We added NFS and iSCSI as some of our first enablements. And we had Fiber Channel, Ceph, Gluster. And now we have stuff like Flex, which allows users or administrators to write their own volume plugins without actually having a volume, you know, compiled entry. But the volume plugin framework itself was primitive in that the end user had to know the details of the storage when they were, you know, authoring their pods. So you have a lot of supportability with this. If you have to code in your IP address for your NFS server in your pod and you want to move it to a different platform, you're kind of stuck with changing your pod definition, you know, to match whatever environment you're shooting for. So then we added this persistent volume framework. And this was really an abstraction that allowed a separation of roles. So now the administrator can come in and create persistent volumes. They can, you know, fill in the specific details of the volume. And the end user doesn't have to know about these details. They can anonymously claim, you know, the storage. And the way they do this is with a persistent volume claim. And they reference the claim by name in their application. Just curious, who here has used Kubernetes? Are you guys familiar with persistent volumes and claims? Okay, cool. So we'll go a little bit into the details, but not too much. So the persistent volume really gives us portability, right? Like, this is probably the key feature that allows applications to move from different environments. Users don't have to know the details of the backend storage they're requesting. They don't have to know if their application is running on Amazon. They don't have to know if it's running on Azure. All they need to know is, like, I need so many gigs of storage, and maybe there's some other characteristics that they might use in selectorism storage class, and I'll get to that in a second. But the real thing is, like, it decouples the application from the actual implementation of the backend storage. That's why persistent volumes are very important. If you don't have persistent volumes, if you don't have that abstraction, then the user is very tightly coupled to whatever the backend storage is. It also allows for dynamic provisioning, which is a super cool feature, and I'll show some details on that in a minute. Labels and selectors. Okay, so our first matching criteria that we started with was size. So the user would request volumes based on size and access modes. We wanted to expand the capabilities of there. So if you're familiar with, like, node labels, node selectors, we added the same thing to persistent volumes and persistent volume claims. It's pretty open-ended. I mean, there's some interesting use cases, but it's really up to the administrator to define what kind of labels and what kind of volumes and how they want to, you know, use this to let the user specify more characteristics of the backend storage than just size or just access modes. Some stuff they may want to use labels and selectors for, backup, if you want to, you know, have a set time on how often a persistent volume is backed up. Maybe the environment, production, staging, GID ranges, security. You might look at encryption. There's all sorts of stuff you might want to label your volumes and then have the users be able to select, you know, against those different labels. But it's really up to the administrator to define what these, you know, labels are and then communicate it to the user. Like, here's the labels. Here's the selector values. And it does support fairly complicated, you know, logical expressions. You have equality. You have set based. You have, you can combine the logic. So you can have some, you know, fairly complicated selectors and, you know, really narrow down specific details of the persistent volumes that, you know, the user needs. So kind of the basic workflow about how this stuff works. The administrator will create a persistent volume. They'll specify their labels as part of the markup. And then it goes into your, you know, your big pool of persistent volumes. Over here, number four, we have a controller that runs. This is part of the, pretty much the master node. The controller sits there looking at the persistent volume pool and waiting for claims to be submitted by the user. The user submits claims. The controller comes in and binds the claims to the volumes. And let me tell you, this controller was actually pretty complicated. We spent, I don't know, two or three iterations with Google working out the specifics of it. And it's pretty solid right now. It doesn't support active-active HA. We have an active-passive HA support, though. That means anything to you guys. The more recent feature we've added is storage classes. And this was something that really enabled dynamic provisioning, which is probably the more exciting feature that we've had for storage recently. Storage classes is a set of properties that all persistent volumes belong to that storage class possess. And so there's no hard definition of what belongs in a storage class or what those properties are. But persistent volumes belong to a storage class and users request persistent volumes based on a storage class. So as a user, I will create a persistent volume claim and I'll specify a storage class name in it. And the persistent volume that I end up being bound to will have a set of properties that the administrator defines. And this is most important in dynamic provisioning, but it's also applicable to statically provisioned volumes. So I may create statically provisioned volumes that have performance characteristics that I specify in the storage class and I want users to claim those volumes or be bound to those volumes based on those properties. So here's the markup for it. Since this is a DEF CON, I figured I'd show you guys some YAML. Pretty straightforward, you have a name. You have a provisioner. So this will come in important when we look at dynamic provisioning. And then when you actually create a volume, you specify an annotation that matches the same name as the storage class. So storage class and label selectors have some cross functionality, right? You may want to have backup policy specified as part of the storage class. There may be performance characteristics that are part of the storage class. There is some overlap in function, but probably the key differentiator with storage class is the hook into dynamic provisioning. So when you have dynamic provisioners, they'll have properties that they require or that they need to actually provision the backing storage and the way that you define each... It's just not going to let me stay off the internet. So you have properties that the provisioner needs to actually create the backing storage. Let me show you the flow of dynamic provisioning. So dynamic provisioning is the creation on demand of the persistent volume. And it's not just the persistent volume object that's created on demand, it's actually the physical storage backing that persistent volume that's created. So for dynamic provisioning on EBS, you'll have an API call out to AWS to create an EBS disk, and then you'll create a persistent volume object that matches to that EBS disk. Similar for Cinder, it could be... You could have actual physical storage that you're dynamically provisioning and the dynamic provisioning part of it is sticking the disk into the drive. So with dynamic provisioning, really the key part is like the backing storage is created on demand. And then a persistent volume object is created to represent the physical storage. We have both entry and out-of-tree provisioners. For all the volume types, there's... I have a list, I'll show you in a minute, of what's supported entry. With entry, it's mostly the cloud types who also have Gluster and RBD, but it's pretty standard just regular provisioning. The real power comes with external provisioners, because if you have external provisioners, you get to basically do whatever you want. You can maybe send off an email after the storage is provisioned, maybe you copy files onto the volume after you've created it. It really opens the door with what users can do or what administrators can do with the backing storage setups that we didn't have before. So external provisioning. Here's the flow for external provisioners, and this would be simplified a little bit if it was entry, but... The entry stuff. Once you understand this, the entry stuff just kind of follows. So first step is the administrator creates the actual storage class definition. And this will have a set of properties that the provisioner is going to need to actually create the backing storage asset. So this depends, of course, on the implementation, what kind of parameters it needs. Maybe they want... For a fast storage class, you want to specify the IOPS, or if it's an SSD disk that's actually using the storage. As a second step, the administrator will actually deploy a standalone executable that will listen to the Kubernetes API, it watches for new claim events to come in, and then it'll act. And we've actually created libraries for this, so it's pretty simple to write your own external provisioners. You fill in a provision and a delete method, and you're good to go. We have an example, NFS Provisioner, and that works quite well. We have NetApp, I believe, just created an external provisioner. And then we have, I don't know, I have an S3, which is not a great external provisioner, but it was a proof of concept. But we have multiple external provisioners right now that you can go look at and see as examples. So after the external provisioner is deployed, it can run as standalone on a host, or it could actually run as a pod on Kubernetes. So running as a pod, you get the benefit of discovering the API server itself, like you don't have to specify what the API server is, it can just figure it out. But it sits there listening to the API server for new claim events to come in. So when your external provisioner sees the new claim event, it's going to go and actually create the physical storage, and then it's also going to create the persistent volume object, which ends up getting bound to the user's claim. So if this was entry, the only difference here is like the administrator wouldn't have to go and deploy a provisioner pod or deploy like a provisioner executable on a host. Some of the entry stuff we support, here's the list. You notice it's mostly cloud types because those are really easy to provision. If you want to move these out of tree, we plan on moving these out of tree eventually, just for the maintenance aspects of it. But if you were to want to like add extra features, maybe you're, after you provision with Cinder, you want to copy files or set permissions on it, it wouldn't be hard to copy the code and, you know, write your own out of tree provisioner. There's something very important on the internet that it wanted me to find. So I talked about out of tree provisioners. Hyperconverged storage. So I'm going to gloss over this. We've had a couple talks about it. But hyperconverged storage is running your storage platform on top of Kubernetes itself. So we have Gluster is a pretty good example of this. And it really makes it easy to deploy, you know, a storage cluster when you're using the Kubernetes framework. So it's pretty cool. Like you don't have to go and set up your own special Gluster cluster anymore. You just have your Kubernetes cluster, deploy Gluster on top of it, and you're good to go. And it ties into the provisioning. It'll tie into the storage class and the provisioning, so you pretty much get automated Gluster volume creation. And we've seen some other hyperconvergence. Our NFS external provisioner actually runs as hyperconverged as well. And I've heard talks of SFRBD and SF file hyperconvergence coming along. So tie it all together. The automatic creation of storage is probably about, like, the coolest feature you can think of. You don't have to mess with it. It's a pain in the butt set up storage, and it's a lot easier to create it on demand and then also delete it on demand. It makes it easier to, like, play with the application. If you wanted to develop something and test it out, now you don't have to worry about going and setting up an NFS server, going and setting up a Gluster server. Have it dynamically provisioned and dynamically deleted. The indirection of the persistent volumes allows for cloud portability. So I strongly believe in this. We don't want to be tied to any particular cloud provider, right? Like, our data is very important. You don't want anybody owning your data and holding it hostage. You really want to use abstract concepts when you're coding your application. So if you need to move your data from one entity that may become expensive or hostile to some other place, it's easy because you've abstracted. You've used an abstract framework. There's some things that you can't get around. You have to have selectors. You have to have, you know, some special casing for certain scenarios. And you also would need some way to classify storage, some way to group persistent volumes. And storage classes is a feature set for that. Looks like I'm out of time. Do you guys have questions? We have, was it like coffee, hot coffee to talk? Lean coffee, where me and some other storage guys will be, if you guys want to talk specifics about this. 430. I know this is kind of high level, but there's a lot of, like, pretty deep stuff. Yeah? We are talking about replication. We're talking about snapshotting. That's stuff that's in the future. So depending on your backing file storage, you may get some of those currently, but it's not built into the framework. So, like, Gluster supports geo-replication. You can use that, you know, for some high-availability stuff. But we're trying to trickle some of those features into the core framework. Other questions? Standardized storage classes. Well, there's, I mean, there's properties of provisioners that's difficult to standardize, right? Like, the provisioning properties you'd need for CIF, are a lot different than the properties you would need for Cinder, right? So it's, it is a little bit difficult. We've, right now the properties are string-string. We may move away to, you know, to something more generic, but no, I don't think there will ever be, like, one standard set of properties for all backing file system types. Is that it? Any other questions? I think that doesn't know. Thanks, guys. I'm going to turn the controller. It's probably six months out. Is that a scenario you're looking at? Let's talk. We've been working on it. I mean, we've been thinking about it. We haven't actually started working on it.