 Well, everybody again, this is the OpenShift common briefing. We do this every week as often as we can on Thursday mornings, and we are trying to address most of the things that are of interest to folks who are in the OpenShift community. And this one today is top of mind for me, has always been, as we say, a persistent problem, persistent storage for containerized applications. Steve Watt, who is our Chief Architect in the Inverting Technologies Group at OpenShift, but at Red Hat itself, is going to walk us through how to do this. And I'm going to move over to Steve and let him introduce himself a little more and get going. And then we'll have Q&A through the chat and then at the end of his presentation and demo. Go for it, Steve. Thanks, Diane. So as Diane said, my name is Steve Watt. I work in the Inverting Technologies Group at Red Hat, and I lead the Container Storage Engineering Program at Red Hat. And so that means that I work with OpenShift to enable the storage features within OpenShift. And then I also work with our storage business units and our engineering teams there to basically prepare our storage portfolio to work with containers optimally. So in this presentation, I'm going to first catch you all up on the current state-of-the-art around storage enablement within OpenShift, just to sort of set a platform. And then I'm going to talk a little bit about where we're going. And so I'm going to talk about containers connecting to storage to start off with. And then I'm going to actually talk about running storage platforms in Kubernetes in OpenShift. OK. So I first want to jump to start on this slide, which is sort of the road to storage as a service. And so what you see here are sort of five dimensions around enterprise IT and how they've sort of progressed downwards towards the current state-of-the-art, so that the top row is sort of where we started and the bottom row is where we ended. I'm sure many of you being OpenShift users are quite familiar with this, but the main thing is we're sort of in a world now where we're using sort of DevOps software engineering processes where the development roles and the ops roles are often co-located within the same team with people starting to share roles that are both dev and a bit of ops, incorporating agile, doing small scope, very rapid iteration, and sometimes iterating quite often with releases in production as opposed to very long periods of time before changing things in operations. And then we also have a sort of a shift to microservices where applications are becoming much more granular, the contract is very service-oriented where they communicate with each other based on restful interfaces and they don't care so much about what's running behind the interface. And so it's very service-oriented and we're seeing broad adoption of containers as a unit of deployment, often packaging the entire microservice itself, and these containers are running sort of on cloud platforms. We've seen that the data center specifically changed from having the cloud and the data center and everything in the enterprise running in the data center and just start-ups using the cloud to the enterprise running some stuff in the data center and some stuff in the cloud and now where our enterprise data centers are starting to look much more like the cloud. And so this is pretty interesting from a storage dimension because if you look at where storage started, you sort of had a single box with a certain amount of storage that would get carved up and that had various amounts of constraints, they're very expensive, you need to buy a better sort of storage appliance if you're starting to run out of storage capacity. And so we got many kinds of economic benefits by moving to scale out software-defined storage platforms like the Hadoop Distributed File System and Gluster Affairs and SEV, things like this which took advantage of the economic benefits of commodity x86 infrastructure. And that was awesome, but also not very cloud-like in that you have to go tap a storage administrator on the shoulder and say, hey, would you go make me a block device or a distributed file system volume and tell me what the connection information is? And so your productivity was sort of gated on how quick storage administrators could turn around those requests. And so eventually we're moving to the enterprise data center being able to consume storage as a service in the same way that we consume storage in the cloud. So we'll have microservices running in containers, running in a cloud-like enterprise data center that can submit restful calls to a storage service that says, hey, go make me a block device or go make me a volume, provide the information and I can attach it, mount it into my container and use it right away. So that's sort of broadly speaking where we are. And so in OpenShift, and I just wanted just to double-check, Diane, are the slides changing? Yes, indeed. They are changing. OK, all right. So just double-checking. So the one you're seeing now is what is in full-screen, the storage innovation for containerized applications. Yep, that's correct. OK. So what you're seeing here on this slide is what you get with OpenShift 3.1, which is being released this week. So and the way this works now is that your containerized applications that run in OpenShift and this feature is obviously available in Origin as well is that you have your OpenShift cluster and then you have independent storage clusters, storage assets. And you require a sort of a storage administrator to go and create these. They may have done them in the past. And so you have a lot of pre-existing storage assets. These might be block devices like CIF, RBDs, iSCSI, block devices, Amazon EBS disks, Google Persistent disks or fiber channel block devices. Or these may be shared file systems like NFS or GlustroFace. And what you can do is that you can basically register them within OpenShift for consumption as a persistent volume. And then you can submit claims that get bound to these and then you can use them. And this is great in that you have sort of a single control plane for being able to at least take these pointers to your storage and allocate them to applications and then use them. And you have a much wider choice of persistent storage than you used to, which was like the available storage capacity on the Docker host. And so obviously these are the additions. We've always had host path and empty door within OpenShift, so which are local storage available within Kubernetes. And the other big thing that we've added recently is volume security. So before when you were accessing NFS or iSCSI, everything was done as root-root. And so essentially the storage asset had to be configured to be accessible wide open. And so this is a huge thing that we've added recently, which is to be able to add support for Docker supplemental groups. And so the way we've incorporated that within the OpenShift security context is that for block devices, your OpenShift user will be assigned a particular FS group, a particular type of supplemental group. And that supplemental group will be used to perform what we call a takeover of the block device, which means completely own it. So it sets the AC Linux context, sets all the appropriate AC Linux labels so that that group and that container can access that block device. And then it does the relevant churn and chomods to that device so that the file system on that device is set for access only for that particular group. And so now we've gone from essentially wide open security to at least group level access control. And same with shared storage. Shared storage works a little differently. So we don't have the same level of automated access control that we can do on block devices. There are a little difference in that the other aspect with shared storage is that the container won't necessarily own the shared storage, right? The container might be using just a part of it since the accounting group might be running Python applications that use the same shared storage as well, right? So you've got a shared asset. You've got to be a little more considerate about how things work. And so the way that works is that basically you configure it. If it's an NFS share, you configure the directory access permissions for access by a particular group. If it's a cluster of FES volume, you do the same thing. And then you basically provide that group with your pod in a security context as a supplemental group so that all IO that accesses those NFS shares or cluster of volume directories will have full access. But any other groups that attempt to access it will not, right? So this is what we have in 3.1. And here is sort of an example, a visual example. Diane, just double checking. Are you seeing a diagram? OK, great. So on the left, what we have here is an OpenShift cluster using shared file storage. So this is read at GlusterFS, read at GlusterStorage. And an example of way in how you might use this is that you have a scale out web server form. And you really don't want to copy your web content onto every single server within your web server form and have the pods mount it and use it directly using something like a host path volume plugin. Instead, it makes a lot more sense to be able to store all that information centrally in some sort of shared file system. So at any time your content team wants to make an update, they just change it in one place, and they don't have to update all 50 servers in your web server form. And so what you would do here is configure your nginx pod to use a claim against a GlusterFS volume and store your content inside that. And then basically gives you kind of a unique set of controls from a control plane perspective and that you can have your development team, your content team copy data directly into your Gluster storage just using file system mounts. But if you want to scale out your web server, you just change the amount of replica instances that you're running for that. And as the new pods come up, they will automatically bind to the same cluster. So there's literally no work you have to do to scale out to make sure that those new pods connect to the same cluster of storage. Another example, sort of the same cluster where that example is running, but your web application has a database back end as well, right? And so that's running my SQL. And so the issue is that part of the benefits of running your containers in OpenShift is that you get container orchestration so that if there is some sort of disk failure in your cluster that you can use OpenShift to ensure that your container gets started up somewhere else, right? Sort of like a singleton pattern, right? And so the problem is, you know, if you're storing your database on the local file system, when that container gets moved somewhere else, your database files aren't there. And so that's a problem. And so when you want to basically be able to store your database on some sort of network disk to get around that problem, that's a classic example when you choose something like several RBDs, right? Which is basically a file on block device. And so you would use a persistent volume claim against a CIF RBD or a direct volume against an RBD and in your pod definition for your MySQL container, and you would store my SQL into wherever you're mounting the CIF file system. And so what would happen is you would create this database and then if the container needs to get moved when it starts up, it will connect back to exactly the same CIF RBD and automatically and you won't have to do anything. And your database, as it starts up, your database will come back right back up with all of its files and keep working. So this is what we have today. And so this is great, but if you recall what I spoke about, storage is a platform versus storage as a service, this model requires the storage to be managed by a separate storage administrator. There needs to be communication between the OpenShift users, administrators and the storage administrator users to be able to consume the storage. So we would like in the theme of DevOps, right? And agility to basically empower our development community. We'd like to make this a whole lot easier. So what we're moving to now is a hyperconverged model as an additional option. So we'll keep offering the others. Obviously you have storage investments that you've paid for that you want to use. So we will keep offering all of that inside Origin and OAC. But what we're looking to do for the roadmap is basically be able to run storage in OpenShift itself. And also to be able to add dynamic provision. So let me start with dynamic provision, right? So dynamic provisioning, the concept here is that for storage providers that can have servicing interfaces. So examples of this might be if you're running Origin in Amazon AWS or GCE in Google Cloud, or in OpenStack. You know, AWS has EBS disks, GCE has persistent disks. OpenStack has Cinder. All of these have RESTful interfaces which you can communicate with. We're planning on adding provisioners so that you're able to basically submit claims for a particular class of storage. So say you would like a AWS GP SSD instead of an AWS Magnetic SSD, right? You could basically create tiers of storage and use arbitrary labels to describe those. So I would like, you know, the word gold, the label gold map to EBS GP SSDs. I would like silver map to EBS IOPS SSDs. And I would like bronze map to EBS Magnetic, right? And so your developers will be able to submit claims against one of those labels. And then the idea is that we will have a provisioner for each different kind of label built into the volume, the overall volume plugin for that storage type. And it would, on demand, go provision the storage asset that you want, register the persistent volume and bind it to your claim so that you can use it. And now, obviously, if you're an ops person, you know, you might get nervous about the accounting dimensions of this where your developer community could run up a very high bill without you being able to control it. So what we've set the ability to control claim quotas within the projects themselves, right? So you can actually sort of empower your developers but sort of set boundaries as to exactly how empowered they are, right? So that's dynamic provision, right? So regardless of which cloud provider OpenShift is running in, you should be able to dynamically provision storage within that. Now, these are all block storage. What we also wanted to do is be able to offer something cool around shared file storage. And so what we've done is started the Aplo project in the Gloucester FES community, Kubernetes community. And Aplo, which is a cling on for a container was suggested by one of the members of our development team and it's sort of a simple term that sort of stuck. But what Aplo is is actually running Gloucester FES as containers inside OpenShift. And so later on after this, I'll actually show you a demo of how this works in Kubernetes. And the idea is if you look at this picture that I'm showing, if you look at the dotted orange line at the bottom, right? Each one of those sort of cylinders is a storage container, right? And so the idea is that we can containerize Gloucester FES. We can actually deploy Gloucester FES as pods within Kubernetes and OpenShift. And we can constitute a Gloucester FES cluster that runs inside an OpenShift cluster or a Kubernetes cluster. And then from that, we can carve out storage. Now, there's obviously a prerequisite to this which is the nodes that those containers land on, the hosts, the OpenShift hosts, Kubernetes nodes have to have some sort of direct attached storage available for Gloucester to use, right? So this could be an individual JBOT disk. So I mean, I would say the smaller unit of server would be one new x86 server which they typically ship with around four to five disks, four to six disks. You really just need a single whole disk spare to be consumed in this model. But if you have more, that's great and we support both JBOT and RAID 5 and RAID 6. And basically that storage would sort of get aggregated across the cluster, federated and presented inside the Gloucester FES cluster and then you could carve out Gloucester FES volumes from that. And so then, the next aspect of that was to add a dynamic provisioner which we're still working on. So I'm not gonna be demoing that, but so you would create a provisioner for Aplo volumes. So you could create a label called Aplo that is mapped to the Aplo volume plugin so that when someone puts a claim it can actually go and dynamically provision storage out of your Aplo cluster that's running inside your Openshift cluster. So there are many benefits of this. Mainly unified orchestration which sort of means ease of use greater control. So single sort of control plane. And then obviously the convergence benefits lower total cost of ownership. So you're running your storage cluster inside your Openshift cluster. So you've got one set of hardware and a single control plane for sort of your software to find network, software to find storage and your software to find compute and applications. So one stop shop from Openshift Origin. And this is actually quite valuable because even though you may have storage in your data center, it may not always be accessible. There may be physical impediments to accessing it but there may also be sort of organizational impediments to access it in that it's being used in production. And any additional IEO traffic could degrade the performance which would make it, the other workloads that are using it suffer to a point where the business wouldn't tolerate it or something to that effect. Very similar to database administrator not allowing any additional connections in the database server. So you run into those sort of issues with storage platforms too. And so with this aspect, you don't have to wait for that and if you have to procure infrastructure for a storage platform, that's lengthy. I mean, if you spend a bit of time talking to an infrastructure vendor to buy X86 servers and then you have to go order those and wait for them to arrive and then you have to wait for somebody to install the software on them, the storage platform software on them and only then can you make it available to your OpenShift cluster. So there's sort of time to solution advantages in this model as well, right? And all right, so with that said, I just want to sort of clarify that all this work is sort of not downstream not product related yet. We're still building this out in the community but it's not architectural vapor whereas I'll show you I have a sort of a demo working already. So we're kind of focusing the initial integration within Origin, you know, providing storages of microservice with dynamic provisioning and it's not just cluster effects, right? We're looking at containerizing Ceph as well. There's a pretty well-known project out there called Ceph-Docker, which is Docker containers. So we're looking at working with that project to sort of bolster it, build out some readout Ceph clusters from that sort of deciding on the granularity of the images there. It's a little Ceph is a sort of kernel space storage platform where cluster is sort of a user space. It's a little different working with Ceph but you know, the intent is to hopefully be able to do the same thing with Ceph that we're doing with cluster and bringing it into OpenShift so that you'll be able to not any provision shared file but also, you know, file on block. So with that said, I'm going to show you a demo. So Diane, are you able to see the demo? Yes, indeed. Okay, all right. So let me just find a little bit here. So what we're seeing here is a Kubernetes cluster. I've got a couple of files here. So basically my GlusterFS, whoops, my GlusterFS cluster is really just two files, a Gluster-1.yaml and a Gluster-2.yaml. Each one of these is an individual pod. So I'm running a two node GlusterFS cluster. Once I run that I will, and the first part of the demo is actually getting GlusterFS running inside Kubernetes. So it's able to sort of, I'm able to create volumes from what's running in Kube. And then the second aspect is, you know, the pre-existing bits that already existed. So we've always had a GlusterFS volume plug-in in Kubernetes for a long time. Kube hasn't been around that long but forever how long it's been around, in that for quite a bit of time. And so we're gonna use that GlusterFS volume plug-in for those of you that aren't familiar with volume plug-ins, they're just storage adapters. It's just a way to connect your application, which is your pod in OpenShift and Kubernetes, to some sort of external storage, right? And so I have an application which is this nginx-gluster and that's actually the web server that's gonna mount and use Gluster. And there's a GlusterFS-endpoints file, which is just an array of IP addresses that the GlusterFS cluster is running on. That's the volume plug-in requires that. Okay, so let's just move along here. And so you can see here, this is an example of one of the pods that GlusterFS is running. So there's some design decisions that are reflected in here. The first is the label, right? It's got a, each pod has got a unique label. The second aspect is, if you go look at the host network, right? This is a pretty new parameter to origin and Kubernetes, which is that you don't have to use an overlay network or the Docker network. You can actually use the host network. So I'm running this demo on three virtual machines, running Fedora 22. They, you know, they're 192.168.58.something, each one of them, right? It's themasters.20, worker1s.21, and worker2 is .22. So when the containers come up, they actually share the IP of the host that they're running on. And this is required and this is also an element of quality of service that we're looking to do in that, you know, it means the storage traffic isn't mixed up into the overlay network traffic that we're able to segment it. And in some cases, you know, if there's a separate network available, so there's two nicks on the server, for example, we might be able to take advantage of that. So it offers some sort of quality of service for a performance fan. The second aspect is overlay networks are kind of a new thing for storage platforms. And you run into a lot of trouble when you're trying to use them. So we're trying to iterate up from a working solution and host network was something that would just work out of the box with Gloucester. So the next aspect is the node selector, right? So this is something that works in conjunction with sort of another design decision, right? So a node selector is something that you can use to pin a pod to always be scheduled on a particular host, right? So worker one is one of my host names. The cube node has that same label. And so the pod is always gonna get scheduled on that server. And if that server's down, I don't want that pod to come back up because I want Gloucester Fester node, it's lost the node because Gloucester Fester is designed to run on commodity infrastructure. And it basically rebalances data, right? So you can detect when nodes or peers in Gloucester Fester have gone offline and you can rebalance your data to basically have additional replicas of your data created elsewhere in your cluster, right? And so that's the one aspect. But the other aspect is, when the Gloucester Fester container on worker one starts getting used, what it's gonna do is it's going to mount basically a directory on the host called MountBrickOne, right? And so MountBrickOne we're using with a host path volume plugin. So host path means, hey, take this directory from the origin host and basically mount it into this particular place within the cluster, right? So the place I'm mounting it into in the container is the same place that's on the host. Then basically make it, so I'm providing a local directory on the host available inside the container. And I'm using this Gloucester Fester image to actually run the Gloucester Fester processes. Now, here's the thing, right? This is why I'm pinning the pod onto the same worker because after it's being used, there's gonna be data stored on that local host directory on the host path volume. And so if you go and move that brick to some other server within the cluster, sorry, if you go and move this Gloucester container to say server 15 instead of worker one, when it comes up, it's gonna try and it's gonna need access to the data things that needs to be storing and it's not gonna be there on that host, which is another reason why we're putting it on worker one. Now, we could use something like CFRBD here, right? As an alternative, we could use block storage and that wouldn't be a problem. But the performance would be, it wouldn't be as good, right? I mean, obviously you're talking about IOs that are going to local disk as opposed to going across the network every time you're making an IO request. So this is a design decision that we're currently making for performance reasons. Okay, so if we essentially go look at the second pod, the second pod, it's identical, it's identical. The only thing that's different with the second pod is that basically, it's got a different name, so instead of cluster one, it's cluster two, and it's the node selector set to worker two, right? So, but beyond that, it's exactly the same. So let's take a look at my cluster so that you can get a sense of what I've got. So I've got a single master and then I've got two Kubernetes workers server one is worker one, server two is worker two. You can see that I'm not running any pods right now if I do cube control get points, get pods. And so what I'm going to do is I'm going to go off and I'm going to submit both of these cluster affairs pods, right? So I've submitted the first one and now I'm going to go submit the second one and then I'm going to do cube control get pods. And if you do the dash O wide, this works same with OC, right? As well in origin, you can actually see the nodes that the pods got scheduled on, which is useful in our example, right? You can see they're getting pinned to the right thing, right? So ready one out of one, the cluster is running. So at this point, what we have is a cluster affairs cluster running inside Kubernetes, right? Now I don't actually have any volumes to use the right and I've actually got to be able to set the cluster up as well before I can properly use the volumes, right? So the first thing I do is cube control exec into one of the pods. This is the same as doing like a Docker exec dash IT, you know, bash or SH, I reset the UID. This is just a bug in the image that community image that I'm using. So, you know, it's a step I have to do. And then the second step is to sort of tell Gluster what the, how to constitute the cluster. Now, in our ultimate design, we're planning on automating this incorporating information in HCD, but I'm just doing it manually now. So I'm from the container, Gluster container on worker one, I'm probing the container, second container on worker two, and then basically when I run Gluster peer status, it looks at worker one and says, how many additional peers do you have? I've got one additional peer. So a total quorum of two. So I've got my cluster in effect. So now I can like go and create a volume, right? So now I can go create a volume called new vol. I specify the IP address of the first container, which I happen to be in right now and pass in a brick path. That was the host path volume that I mounted into the container that I spoke about earlier. And I pass in the second one as well for the second container and its brick path, right? So that's great. The command worked. So now I have a volume, I need to start it, make sure that it started and then double check that it's running. So if I do a Gluster volume status, you can see that everything's running and you can see the two bricks. I'm gonna highlight these that I created within that, right? So you see the two containers and the host path directories that they're using on each one of them. So now we have a Gluster FS volume that's ready for use. And we're able to actually use a Kubernetes application and configure it to mount and use this Gluster FS volume to serve some data. So the first thing that I'm showing is I run a mount command on the host and while I have a Gluster FS volume, you can see that the volume's not there. In fact, that was a bit quick. So I'm just gonna go back a little bit. So there we go. All right, so when you run mount, it'll show you all the available mounts on the host, right? I'm on the actual host right now. And you can see that there are no mounts on the host. So what I'm gonna do is I'm going to mount the volume that's running inside Kubernetes and I'm just gonna create a file inside of it. Now the Nginx application will automatically do this, but I want to, it'll mount the volume, but the volume needs something inside of it for me to actually do a demo. So I'm going to just mount the volume and then create a file inside of it, a little hello world.html. And then, so that will be there that when my Nginx web server uses the volume, I can basically test that it can serve that file, right? So this is how you do a GlusterFace mount manually. It's pretty simplistic. So when I run mount, you can see that the volume is then mounted at the bottom. So it's mounted one of the containers that gives it access to the entire volume. And now I'm going to just create a little hello world.html using vi onto the mount path. Okay, so now that's done. So now I can just unmount the volume from the master. And so it's not mounted at anywhere at this point after this command runs. If I run mount again, you'll see that it's no longer there on the host, right? So now it's just living inside the containers with it not connected to anything. So now I'm going to, let's see, I'm going to show you the Nginx. Well, actually first, I'm gonna show you the endpoints file, right? So you can see if I run cube control, get endpoints, I'm not running any endpoints right now for GlusterFace. So what I'm gonna do is show you the GlusterFace endpoints file. So this is something that you'd need to create to use a GlusterFace volume directly. And it's pretty simplistic. All this thing is an array of IP addresses of some of the addresses that are in the cluster. So now that that's done, I'm gonna create the endpoints file. Just to submit it, I'll do get endpoints and you can see that it's there. GlusterFace-cluster is the label associated with the endpoints file. You can see the two IPs in there. Now I'm gonna show you the actual web application. So it's a Fedora Nginx web image. I'm using the GlusterFace volume plug-in. I'm using the GlusterFace-cluster endpoint that I used. I'm mentioning the volume name that I created and then in the read-only semantics, I say false, which means it supports full read-write. And then I'm mounting this into user share Nginx HTML inside the container, right? So at this point, what will happen is it will take the GlusterFace volume I created, mount it inside the Nginx web server at user share HTML test. This is the document root of the web server. So this is the place in the file system of the web server where it serves content from, right? So I'm gonna submit the pod at this point. So the application pod, the Nginx pod. And you can see it's still starting up. It's ready zero dash one. You can see the other two GlusterFace pods still running and now it's ready one out of one. So we're good to go. Now I'm going to, I have to figure out what IP address this Nginx GlusterFace pod is running at so that I know what URL to curl. So I do a describe pods and I find the IP that's been assigned for the containers as running on worker two. So I'm just gonna shell into worker two now and run a curl command against that IP and then the path of the document root. And let's see if it serves it from Gluster and it does, right? So what we're seeing here is an Nginx web server mounting a GlusterFace volume and serving web content from it as demonstrated by this curl request. And that volume itself is running inside containers as pods inside Kubernetes. So that is the end of the demo and the end of the sort of overview. So at this point, I think we can take some questions. Wow, that was awesome and very fast. And I'm gonna be watching that recording again and again, I think so I can figure out all the steps there. Maybe we need to turn it into a step by step thing so we can walk through it. There actually is a step by step thing and I will post that link in a GitHub repo. I'll post it in the BlueJeans chat window. Okay, perfect. That's a lot of, it was very interesting there. There was one question that popped up from Lee Caldcott of course, and he was asking how this all compared to Flocker if he wanted your opinion on that. So to the best of my knowledge, I'm not a Flocker experts. I've chatted to them a few times at different conferences. The Flocker is a lot more similar to the first part, which is the first part, remember, I spoke about having existing storage and being able to automate and manage connection. So as your containers move around your cluster, keeping that connection information to your storage around, so that's one thing they do. And then they also have something to do with ZFS, I think where you can sort of use some storage that appears to be direct attached like host path that is able to move around the cluster with you. I don't know if they support dynamic provisioning at all. So having projects be able to say, hey, I would like an EBS volume, go make me one and that happened automatically. And I also do not believe they're doing anything in the hyperconverged model. So where they're running the storage itself within containers. So I haven't seen anything pop up in the Q&A section here. So I'm gonna put it up again to see if there's folks. There's a lot of folks on the call. I think they may be stunned and amazed at how forward-thinking some of this work is. But let's see if they've got anybody here. On you coming in. I think they just basically, they basically want your presentation so they can go all run out and try it now. I think that's really, it really was a very detailed end. The step-by-step was probably one of the best that I've seen in a long time. And I'm not just doing this so that I can get you on another common pre-thing in the future, but I really appreciate the detail that you went into. It was very clear and concise. So thanks very much, Steve. Sure, no problem. Popped in one more question. Just to be clear, you're using persistent volumes here to connect to pods, to clusters. Yes. So Nicholas, just to clarify. So this is one of the things that I think we could describe a little better in the Kubernetes project, but there are two things that refer to storage in Kubernetes. There's volumes and persistent volumes. A volume is a storage asset that you have intimate knowledge of. In other words, you know what IP that thing's running on, what the volume is called. You basically are able to provide all the connection information to a storage adapter to connect to that thing directly. And that volume is an independent storage. It's not running inside OpenShift itself, right? Usually. Then persistent volumes are similar, but they're abstracted. So the idea is that developers don't really care that, hey, I want to connect to this thing directly. What they care about is, hey, I need 300 gigabytes of storage capacity on a shared file system or a non-shared file system. I want other people to be able to access this file system or just me. And I wanted to have this sort of IO performance, right? They don't really care what product is serving them. And so that's essentially the idea behind persistent volumes in that, as an OpenShift administrator, you can create a ton of these persistent volumes. You can register them inside OpenShift and you can specify whether they are, what subantics they support. So whether they read-write many, which basically means they shared storage or read-write once, which means that they're block storage. And then when the developer makes their claim, they can sort of claim what they want and they'll just get something out of the pool that uses fuzzy logic to best match what they want. So what do I mean by fuzzy logic? I mean that, if you have 100, one terabyte persistent volumes in the pool and then you have say 1,500 gigabyte volume in the pool and a developer asks for 400 gigabytes, the developers claim will be bound to the 500 gigabyte volume. So it does its best to match what's available with what's requested. Now, these things, what we're changing or hope to change within OpenShift and Kubernetes is the ability to provide for these persistent volumes and volumes to point to external storage clusters and add the ability to be able to run storage within OpenShift itself. And so that you can basically share the same infrastructure for your storage, compute and network management. Well, when that happens, we'll have you back on and doing another session on all of that. That's pretty good. We're getting the advanced warning here that things are gonna get even more fun. So thanks again, Steve. Oh. Yeah, so there's another question here. So can one grow the storage once this has been claimed? That is in the plan, right? Only certain, so for dynamic provisioning, right? It sure would be annoying if you could provision storage but then, so say you provisioned an EBS volume or something to that effect and you put your database on it and then you're like, oh man, I didn't ask for enough. Storage capacity and then you run out of this space and your database won't restart, right? And so for the volume types that we supported, the current thinking in the design is that what you can do is set a flag on each storage type associated with the particular label that says, hey, I'll allow the developer to manage this, right? And so say Diane has some budget flexibility. The OpenShift administrator can say, hey, Diane, you're building this cool application. I'm gonna allow you to increase or decrease your storage capacity as you see fit. It's not gonna be a financial problem for us. And so they can basically set the capacity management flag to develop as opposed to administrator. So that means developers aren't powered. Now, obviously that means that when you go and look at your claims and not all claims are connected to things that will allow you to dynamically increase your storage capacity, most of the cloud options do. And the Aplo one, the GlusterFS one that we're running co-located within OpenShift, we intend to offer that as well, right? So that's the scale out storage. It's not too hard to scale out once you've got something. You just throw more servers out or more containers in this concept. So the long answer is yes, the intent is to allow developers themselves as well as administrators to increase their storage capacity when they need it. So Nicholas, I've just taken you off mute. Is there anything else you'd like to ask rather than trying to do it through chat? And can I ask you, are you already using Gluster there where you're at? No, we're not using Gluster. One question I do have, it's somewhat on topic, but today, if one gets a persistent volume and the persistent volume, let's say, is defined as 50 gigs and right now the only supported sort of persistent volume until three one was NFS. If that NFS volume was to grow and whomever claimed that persistent volume needed it, could that extra storage be taken advantage of even though it was only defined as 50 gigs? I'm just trying to make sure I understand. So you're saying in the initial claim, you asked for 50 gigs, but really the NFS share is capacity for 100 gigs. Can you keep, could you just keep growing to fill the available capacity? Yes. Yeah, so right now for shared file storage, NFS and Gluster, that is a conversation you can have with your storage administrator. So the only way to constrain you to 50 gigs if the storage administrator puts a quota on your directory in NFS or Gluster. So if they don't do that, you can technically grow a cube or OpenShift will not, 3.0 and 3.1 will not currently limit you from growing beyond what you originally claimed if the administrator has not put a quota on you. Good to know, thank you. No problem. Well, I'm looking, going once, going twice. I think that's got everybody covered. Thanks Nicholas for those questions. And if there's other topics that everyone would like to hear, I will post it to the commons mailing list. Oops, here comes in one more. See, we can never give up on these questions. How do we handle things like snapshots? So we're designing Aplo to include snapshots. There are trade-offs for things like snapshots and that they require a much more complex brick set up. So you have to thin provision an LVM and provide that to Aplo if you want it to support snapshots. But so Aplo is designed or it's intent when we, if we can get to a point where we're shipping it, that it will support snapshots. The other volume types, if they support snapshots, the intent is in future OpenShift releases that we will expose that feature to the developer in the same way that we expose the ability to increase your storage capacity. But again, not all storage platforms offer snapshotting, but for the ones that we do, we plan to build that into the volume plug-in for that particular storage provider and so that we can expose it to both developers and administrators. Julien, I've taken you off mute, but you have to unmute yourself if you have a follow-up question to that. No, I'm okay. I was curious about snapshots, but I can probably start asking DR tech questions, but I think it's probably way out of scope at this point. Thank you. That's okay. All right. There we go. Going one, two, three. Well, it's the top of the hour almost. That's an hour-long session, Steve. You deserve much kudos for a great session. So thank you very much everybody for joining us and we'll be back next week. Last week's session is now up, which was storage as a service or security as a service, rather. The cryptarian, that's now up there. If you'd like to review that as well. So check it out commons.openship.org slash briefings for the upcoming events and past recordings. And Steve will be up there shortly. Thanks Steve. Thanks everyone. Bye.