 Well, hello and welcome again to another OpenShift Commons briefing. This time we're getting a tech preview of Gluster FS and OpenShift. So I've invited two of the folks from the Red Hat Storage Group, Sian Zaha and Michael Adam, to give us an update and an overview of Gluster and OpenShift. And I'm gonna let them introduce themselves and kick it off because I hear they have a lot of content. So let's get going. Michael, Sian? Yeah, thanks. Thank you. So, yeah, so we are here to talk about the Gluster and OpenShift story. Sian, a few words about yourself, maybe? Yeah, so can you guys hear me? Yes. Yeah, so I run product management for Red Hat Gluster Storage in Red Hat and then I'm also responsible for the container native storage solution which runs containerized Gluster inside OpenShift in a converged manner, basically providing a solution where you can run storage containers side by side with application containers. This is actually generally available today in form of a solution from Red Hat. And I have other responsibilities in Red Hat regarding storage product management. I'll not go into those details. Michael, go ahead. All right, thank you. So, yes, I'm part of the open of the storage engineering team. And I'm working, I'm running the team that actually works on that so-called container native storage solution, which is the solution that ties together Gluster and OpenShift, which we are going to talk about today. And yeah, let's proceed and we'll explain what this all is about. Say, if you could advance this slide, please. Right, yeah. Can you see it? Yes. So this is the rough agenda, what we're going to cover. I'm going to talk about what actually is Gluster. I'm going to talk briefly about the storage use cases in OpenShift. Then I'm going to describe the way to what we call container native storage and what it actually is. And then I'm going into the details how it all works. Finally, there's the roadmap thing. If we have time inside the how it all works sections, we have two demos, but if we don't have time, they're recorded, they're available on YouTube even. So if we don't have the time, then we're going to skip those. And yes, of course, questions are encouraged whenever something is unclear or people want more details. So let's go on. Next slide, please. So we're gone about Gluster. What actually is Gluster? Next slide, please. Elizabeth? Yes, thanks. So what is Gluster traditionally called Gluster-FS? Nowadays, usually Gluster is a software-defined storage system, meaning we have a stack of software that implements this storage solution. It's not tied to any hardware. It runs on most any commodity hardware out there. It is providing a scale-off file storage, scale-off in the sense it's Gluster. It's running on multiple storage nodes at the same time. But that is also highly available so that it is not required for each and every single host to be up. So certain down times of individual hosts are well acceptable as long as enough of the other hosts are up. And for such a scale-out highly available storage system, it's fairly easy to set up and fairly easy to administer. So it is not as complex as many other scale-off storage systems. It is also, in all its simplicity, very flexible, offers a lot of additional nice features and it's very flexible in this configuration. What does it offer by way of access methods? So there are the native client is a fuse-based POSIX file system mount that is available on Linux or your next clients. We have NAS heads like NFS and SMB, so Samba offering the SMB protocol for typically Windows clients. And a built-in NFS server and a new extra NFS server with the Ganesha NFSv4 newer system. So this is sitting on top of Gluster basically, but we also as new features have or are about to have in a very new release an iSCSI front-end based on RUPEC file sitting inside Gluster storage. And we also have a object access by so-called Gluster Swift front-end, which also features a plugin for offering Amazon S3 object access. So as you see, file is the native access, like really offering a scale-out file system, but block and object access is added on top of that. So that's roughly the overview of what Gluster is and what Gluster offers. And on the next slide, I have a few more details about the community. So Gluster is an open-source project. You can visit its home on Gluster.org. You can visit the GitHub space with a lot of repositories with a code and stuff surrounding it on github.com slash Gluster. And it's what is not written here, Red Hat ships this as part of the Red Hat Gluster storage product, RHGS. But this is the upstream, the community-facing thing. Some more technical details about Gluster on the next slide. Thank you. So the basic concept of a volume is inside Gluster. So the volume is the unit of storage that Gluster offers. These volumes are composed of local file system directories. So it's really very simple. You have local file system like typically XFS. And these directories composing volume are called BRICS. These volumes can be composed of BRICS in different types. They're volume types, typically different characteristics of durability. We have replicate and distribute. Those can be combined to have a volume that is distributed, distributing its storage over various local replica sets. Replicar set is a group of BRICS typically located on different nodes where you store something. It is replicated to all lists. For instance, say you have three replica, then one storing a file will replicate the contents to three BRICS. Another characteristic is the dispersed type of volume, which is an erasure coded way of doing volumes. And there are more, but these are for our purposes the most important ones. I alluded to the flexibility and features. This is mostly due to the architecture of a stack of translators. So these translators are basically pluggable modules that can be and a volume definition consists of a list of these translators. These translators add some features like, I mean, some more of the interesting high-level features are encrypted volumes. There's a snapshot feature. Glastro even has the so-called user serviceable snapshots where the snapshots are visible inside the actual mounted directory. And you can very conveniently access your old versions of the files. We have something called geo-replication as a set. Gluster is a clustered file system. But what if you have multiple clusters on different sites where with a link that has too high latency in order to form a single cluster, then there is something called geo-replication where data is replicated between these clusters in a asynchronous fashion. Quota is there, et cetera, et cetera. Many of these things also more posix file system features like posix echoes, et cetera, et cetera. So access control. All these things are available through translators and also additional daemons that are running. So this is not a monolithic software. It is quite the whole Gluster system consists of many daemons that are running on each of the storage nodes. We have a GlusterD, which is kind of the central cluster membership and volume management daemon. We have brick servers for each brick. We have daemons for Quota and snapshots, et cetera. So this is also very flexible and very robust in the sense when something bad happens, a disk dies or something, then there is you can repair a volume, you can re-add new disks. There is something called cell field, which will lead to automatic repairing of a failing disk when you add a substitute, et cetera, et cetera. So this is very feature-rich. You can visit Gluster.org and navigate to the documentation section there to read more about the details. And with that, of course, any time when there are more questions, we can come to the story, how this can go into the world of OpenShift and Kubernetes to be of great help there. Next slide, please. Next slide, please, Simon. Yeah, I'm Donnie. Thank you. Okay, sure. Thanks. So this was about storage. So what are the use cases for storage in OpenShift? There are basically two main use cases. There are application containers and there is the registry. So for both points, I have a slide explaining a little bit what this means. So persistent storage for application containers is next. Next slide, please. Yes, thanks. So the containers in itself have the paradigm of being stateless ephemeral animals basically. So they can be brought up and down in an OpenShift cluster on different nodes and this will actually be just instances of the same application container. But if they're brought down and up again, any data they store on their local disk has been removed. It's reset to the original state. So the apps, usually they want to store their data somewhere persistently. So the state of the application, the user's data, the configuration of the application, all that needs to be stored and persistently. And for that, something like Gluster, which is highly available, robust, scalable, seems like a natural fit. So this is the first use case. Second, here's a question. Do you count the OSCP-owned parts by the modeling metrics as applications? This is currently not the main focus, the owned parts. I'll take that question. So the answer to that is, yes, I mean, the way to look at it is we are chipping away at the problem one by one. So we understand that OpenShift has infrastructure which needs storage too. Infrastructure includes things like registry metrics with Cassandra and Hocular and logging with Elasticsearch. So yes, those need storage too. We will do the registry first and then we'll look at Cassandra and Elasticsearch in a future release to see what we can do there. Both Cassandra and Elasticsearch recommend that local storage be used with SSDs, you know, that's their recommendation because they need really low latency local storage. But we are making some improvements in Gluster to see if we can take those workloads too. And that would be great, but we're starting with the registry. Right, exactly. So that seems to have answered the question. There was a thing from Alex on the chat. So, right, the second natural use case would be the registry. Registry is also a place where data is stored persistently for access by many processes in the OpenShift cluster, namely the container images. It's the repository of the container images. OpenShift uses NFS as a registry storage backend by default. And this seems to be a very good fit for Gluster as well because how is NFS usually done? You have local disks that are then exported as NFS exports. And so the kind of flexibility and high availability is not granted in the same level as with Gluster FS. So I'll have some more details about how we're approaching this in one of the later slides. There is another question here from Jonathan. Are you running NFS on top of Gluster, transparent to OpenShift? Or does OpenShift know about Gluster as for this? This is a good question. I will cover the details in later slides. We are not running on top of NFS. We are doing it natively. And yes. On somebody comments, they're using Cinder instead of NFS. Yes, of course. You could use all those abstractions, but we are using natively Gluster in this solution because that's the fastest and the most performant rather than going through abstractions. Sure. And if you have, of course, an OpenStack environment, Cinder may be a good and natural fit for the registry, of course, as well. That's fine. So we'll cover a little bit more about the registry, but since registry is currently under development, and we are talking about what we've done in the past first and what we currently ship in, and then we go into the development. So this is what I call two awards in that container native storage. The term will also be container native storage will also be defined as move along. So next slide, please. You'd see it. Yes, there it is. So we have had several steps of bringing Gluster and OpenShift closer together. The first term that was coined was called container native storage. What did that mean? That basically meant when container orchestration was added to OpenShift in 3.1, there was the opportunity to have a Gluster storage cluster outside of the OpenShift cluster. For instance, installed in bare metal or wherever. I mean, with this somehow your screen is showing some calendar. Okay. Sorry about that. It's just a calendar, but I killed my calendar instances. Okay. Thank you. So the reddit, in this case, the Gluster storage cluster runs outside of the OpenShift cluster somewhere on bare metal or wherever, since it's a software defined storage system, it basically can run anywhere. It can be in a virtual environment or bare metal. And the OpenShift containers make use of the volumes that are provided by that external Gluster storage. So there is some link from these containers to the outside storage world. And you have all the flexibility, but it's like a separate setup. That was step number one. Now please, for step number two in the next slide, called that containerized storage. Thank you. So we started to containerize Gluster into, so putting the Gluster storage node into containers. That was not done before. As I explained earlier, Gluster itself is a relatively complicated set of daemons. They need a lot of special privileges since they act on local storage devices, et cetera, relatively low-level system daemons. And so it was not entirely trivial to containerize those. But it was done. And so in the second intermediate step, called containerized storage, Gluster was running in containers, but still outside of the OpenShift cluster. But this already indicates where we are going. And please, next slide. This is container native storage. In this setup, Gluster runs inside containers in the OpenShift cluster itself. That means the containers that were previously somewhere outside on some atomic host setup were now running inside OpenShift and managed by OpenShift, basically. And so it is using local disks from the hosts that are consumed inside the OpenShift, inside the Gluster storage containers. And the cluster is really running, as we say, hyperconverged inside the OpenShift node. And the application parts that are consuming those Gluster volumes, they can either run on separate OpenShift nodes or they can run on the same nodes. They can, but they don't have to be co-located. This is also demonstrated in this picture here. You see there's an OpenShift node 1, which only runs application containers. OpenShift node 2 and 3, they run both storage and application containers. And OpenShift node 4 exclusively runs storage containers. So all three combinations are possible and they're possible on the same cluster. So it's entirely up to the administrator how to set this all up. Sometimes resource-wise, it may make sense to have it entirely separate. Thank you. Yes, so this is the general what we are doing. It's fine. We can move on and explain in a little more detail how this all works, because there are many nitty-gritty details about that. Okay, next slide. Thank you. So we have a couple of components coming into the picture here. I already mentioned, of course, I talked about OpenShift or Kubernetes in the back end. I talked about Gluster or GlusterFS clusters that are providing the storage, but that's basically not all. So how does OpenShift actually talk to Gluster? How does it make the connection? It does so by some components that we added to OpenShift or to Kubernetes, which is on one side the dynamic or at all the GlusterFS provisioner, which is a piece that creates the so-called persistent volumes to be used by the application containers. And we have the mount plugin that is responsible for actually doing the mounting so that the application container can use the storage that the provisioner created for it. GlusterFS, as I said, yes, one or more GlusterFS clusters running inside OpenShift as containers, but OpenShift does not directly talk to them, but we have in the middle between the two, we have a service interface that is called Hackety. This is also running as a container and it is just a very high-level interface to hide a lot of the low-level details of Gluster and makes it very easy. It offers a restful API so that Gluster Kubernetes or OpenShift can naturally talk to it. It then turns around and looks for available space in Gluster and creates volumes according to the requested size, etc. The fourth component that is also in the mix is the deployment. So this will be the last part of the overview of what we currently have. I mean the first three components describe an installed system and how this all works. The last thing is how do we install something like that? And so there's a deployment tool which is basically one command and it will install everything for you. This will be touched last. So let's first of all, moving from OpenShift Kubernetes down into the Gluster world through the components and explain a little more in detail how this all works. So how does persistent storage in OpenShift today work? This is just a brief recap assuming that most of the audience will be familiar with that. So when I'm talking about a pod, this is the group of one or more containers that phone entity, basically the smallest entity of applications that we have. A persistent volume is the concept that provides persistent storage and can be mounted by an application pod. A provisioner, as already mentioned, provides persistent volumes upon request. And the plugin, the mount plugin, is the mechanism that mounts the persistent volumes and it is referenced in the PV. So the PV, the persistent volume carries the knowledge how it can be mounted. A persistent volume claim, or PVC, is the mechanism by which a user can request a persistent volume to be created. It is also the link by which the user can later use that volume in an application container to be mounted. There are various access types for volumes. I have listed the acronyms and also the spellings out. So R-W-O read wide ones, R-W-X read wide many, R-O-X read only many. So this is very important. So the typical thing we are currently targeting for or concentrating on is the R-W-X read wide many. So there is one volume and a lot of application pods on multiple nodes can access that volume simultaneously. Imagine a web application and you have the data with the persistently stored data for the websites and the web server has these pods. They are accessing the same data and this can scale out. You can spawn more of them of these web server pods. They all access the same data and serve it out to the clients. For read wide ones, I have some more details. It's also supported, but there is some special thing on the roadmap for that. And yes, of that provisioning, there is actually static and dynamic provisioning. All right. So now in order to be able to appreciate the dynamic provisioning, here is what static provisioning is. This was the only way to provision persistent volumes before OpenShift 304. So the admin had to pre-create a set of persistent volumes and the static provisioner gets a request for a certain size and then chooses from the existing persistent volumes based on the type and size from the claim and it just chooses and tries to best approximate from above the size requested. So that means if the admin has pre-created like a couple of 100 gigabyte volumes and then it gets request for one gigabyte volumes, a lot of space is wasted. And so it has more work that has to be done and it is likely wasting space. So on the next slides, I have just a few screenshots to see, okay, what is the workflow of commands for static provisioning. First, you have to create some endpoints and service. I don't go into the details here for application parts to be able to reach your storage provider, Gluster in this case. And then in the case of Gluster, I used the CLI, the Hackity command line interface in order to pre-create, for instance, 100 gigabyte persistent volumes. These would be a lot of those. And on top of that, we'd need to create OpenShift level persistent volumes with OC Create, with some kind of JSON file describing the persistent volume here. And an application part could then, or a user could create a persistent volume claim. So from here, it's easy. It would get the persistent volume and could use it in an application. But the tedious part is the manual pre-creation. So you would go to the storage back end and create the volumes. And then you would create a lot of these persistent volumes manually. So here is the question, as I understood in the past, you will need one Gluster per persistent volume. Is this still the case? Yes, this is also in the later, in the dynamic provision right. This is very much true. We'll see that a little later. So for the static provision, here's an example how the PVC would look like. We see that, yeah, we have a name, and you just specify a request here, and then it can be satisfied. So this, from here, for the end user, this is very simple. Nothing complicated here. So please proceed and can talk about how dynamic provisioning works. So it was added in 3.4, and how this works in general is a storage class concept is added, and the administrator creates the storage class. This describes the storage basically in a high level, and it references a dynamic provisioner. You'll see that in the example in one of the next slides. And so the difference now is the persistent volume claim that is done by the user, it references the storage class. And the provisioner from that storage class is automatically called. It creates the persistent volume of requested size and type, etc. dynamically on demand. So there's no pre-creation. That whole thing is going away, and the user can just mount the PVC, a PV, or reference the PVC after it's bound an application just like before. The main point is no pre-creation required, and the requested size will be fulfilled exactly and not approximated, maybe poorly. So please proceed. How does that work with a dynamic provisioner? So I don't only have it in the flow of bullets from left to right and back again. Apologies for not having a real diagram, but this is how it goes. As I said, the persistent volume claim references the, yeah, basically the storage class and implicitly, thereby, the Gloucester provisioner. It is called, the Gloucester provisioner extracts the details of the request from the persistent volume claim. It tells, it goes out to Hackety and tells it to create a certain volume of a given size and type. Hackety looks for Gloucester cluster that can satisfy the request. So it looks for a cluster that has enough storage available. If it finds one, so there can be multiple of them, then Hackety tells the cluster instance to create the volume. Once that is successfully done, Gloucester reports back and Hackety hands the volume that was created back to the provisioner. The provisioner creates a persistent volume of that and puts the Gloucester volume details into it, so how to reach it, et cetera. It also reference the Gloucester FS mount plugin to be used for mounting that. And the provisioner then returns the persistent volume to the caller. Upon success, the persistent volume claim is bound to this newly created persistent volume and then can then be used in a part by the user as before. So this is how it works from left to right. PVC to the provision and set communities to Hackety. Hackety finds a cluster, cluster that is fitting. It calls that, creates the volume there and goes back all the way. So this is how all the components work together. So here is an example for storage class. And you see the provisioner colon, Kubernetes.io slash Gloucester FS. This is the reference that tells it that this is to be used. So, and also in the parameters, there are the details how to reach the Hackety. In this case, oh, this is basically, if that's an error here, this shouldn't be 127.001. It should be a real resolvable name that leads to the Hackety pod wherever it's running currently because it can move in the cluster. So, but in general, this is the main point here is we have the provisioner named in the storage class. And we have how to reach the Hackety inside the parameters. And you need notice the name of the storage class. In this case, it's called Gloucester container. I mean, you can give any name. It should just be unique. And so on the next slide, you'll see how this is used in a PVC persistent volume claim. So here's the PVC. It again has a name and the annotations in the first block you see storage class Gloucester container. So that Gloucester container in this case was the name of the storage class. Whatever you have given there, if you reference it here, this will let the OC create when you run it with this claim. It will let it know that it has to go out with the Gloucester FS provisioner with the data down here. For instance, we have accessible read-write ones. We have storage for Gigabyte. And it will try and create exactly a volume of this size. Next slide, please. Another example of how this thing is used in an application definition. So we had the name of claim one and right down we have the persistent volume claim claim one. So creating this app will only work after the PVC has successfully been processed and has been bound to the creative persistent volume. So this is not changed. There's a question. How is resize working? Increased by 5G, reduced by 2G, etc. So technically, Hakedi can be used to do that. Expansion at least. We are working on adding more features like shrinking, etc. Expansion is available in Hakedi whether it's fully supported at least in like Red Hat product is another thing. But technically it is there. Yes, Hakedi CLI has a lot of these things. I think to my knowledge is not able to trigger that with OC commands currently. But you can still work on the volume with the Hakedi command line. Okay, let's move on. Thanks. So again, to not be able to compare to the previous static provisioner, I have here the flow of commands for the dynamic provisioner. Of course, you first have to create the storage class. And then also you need the secret here for accessing the Hakedi. But then all the intermediate steps they are missing, like you don't need to pre-create the Hakedi, all the volumes with Hakedi and the persistent volumes with OC create. But then you run just as before the OC create on the persistent volume claim. But this time referencing our storage class and then it works exactly the same way just from there. And I don't have all the screenshots here for that now, but you'll see that this dynamic provisioner would always for each persistent volume claim persistent volume that is successfully created, it will create endpoint and service. Because this is important, so even though maybe each time the same cluster cluster is referenced, these are also tied to the namespaces of a request. And hence we need additional endpoint and service here. Right, Cheyenne comments that Kubernetes does not support resize operation yet. Hence you have to use the Hakedi in a future version, this will likely be integrated. So what does the mount plugin do? The OpenShift host has Glastrof as client installed. So it is not imposing any request or any requirement on the application parts. The host mounts the Glastrof as volume and then bind mounts it into the application container. So for the container it will just see some directory which presents a file system, a POSIX file system to it. And all the magic is happening on the host. This is how the Glastrof as mount plugin works in simple words. All right, so next slide please. Let me see. We are 20 minutes before the end of the meeting. I think we should move on and if we have time at the end, we will have a demo. I think we have probably sufficiently explained how it works in principle. And the slides which can be shared later are contained the link for the demo which can be watched later. Is that okay? Yeah, that's perfect. There's one question there. I don't think that Cheyenne has answered. Jonathan is asking, what does the secret look like? This is just credential for your Gluster FS servers? It's actually credential also. I mean is this for accessing the Hakedi service basically? Did that answer your question, Jonathan? And it can be of various forms. I mean, it's the concept of OpenShift or Kubernetes Secrets here. So it can be an password or something else. Okay, then let's move on and defer the demos to the end until we have time. So a little bit more detail about Hakedi. I mentioned a little bit of it. So it is a high-level service interface for managing lifecycle of cluster volumes. In the sense it can create it, it can expand it, and it can delete the volumes. It is also providing this RESTful API and a command for interface to access these features. So Kubernetes will, when creating a persistent volume, the Gluster FS provision will talk to Hakedi via the RESTful API. And for any additional stuff that is not exposed via Kubernetes or OpenShift, you can use the Hakedi CLI from some, for instance, if on the master node you install the Hakedi CLI package, then you will be able to use that. The main point is it manages one or several cluster clusters. Yeah, what it can do, I've already said, more is coming. That's right, shrinking, replacing disks, etc., like that is in the working. One of the main points is that really high is the nitty-gritty details of volume creation and cluster selection from the caller. So usually you have to specify each and every single brick, i.e. disks, etc., and the volume type, etc., so the creation in Gluster is some amount of command line. And for Hakedi, you just say, hey, I need a five gigabyte volume. By default, it uses replica three volumes, and it could also, upon request, do other types of volumes. Currently, Kubernetes always requests replica three. In the future, we are going to offer other types as well, exposed through the storage class, but this is how it currently works. As I said, just take size and durability type, and it finds a cluster with enough disk space to satisfy the requests, goes there, creates it, and in order to do all that, it has its own state starting database. That is one thing to be aware of. This database maintains data about the nodes and all the clusters, and also about the volumes and the disks. So this is very important to bear in mind. I also put in the link here. As the other components, this is all open source software. Here's its home on github.com slash Hakedi. By the way, Hakedi, you may wonder what the heck that name means. It is the word for one in the native Puerto Rican language, because the original author of the software was originating from Puerto Rico. So this is the origin, like a fun fact about this software. Let me see. There's a question or two already answered. What I think I understand is that you might run a cluster cluster outside of OpenShift, then the hosts will map remote volumes locally. Is that recommended approach here? Yeah, you can. You can do it. You can run it outside, and you can run it inside. In both cases, Hakedi will run as a container. Right. So the Hakedi container is much less special than the cluster containers. I told you earlier that cluster was not trivial to containerize. There's a lot of things. It's even running today as a privilege container, accessing the disks and network from the host, et cetera. Hakedi is much less special. It is a single container. It's not clustered. It's not highly available. It can move in the cluster. The main point is that it's database needs to be persisted in order that the state is not lost. So this is currently stored in the cluster volume. In the next releases, there will be another storage place. But currently, this is how we're doing it. There is this kind of chicken neck thing where we already create a cluster volume and then put our database inside that. And the database is also the reason why you have to be very careful. You should only use Hakedi to modify your volumes, and you should not fiddle with cluster directly. More questions. To run inside your hosts, need Big Drives is one comment by Jonathan. Cheyenne says, yes and no. You could load up some of the hosts with Big Drives and run cluster container only on those nodes. That's right. And how big the drives need to be, of course, depends very much on how much storage you are expecting to need. So for many use cases, a few terabytes may even be sufficient. So here is the warning sign again. Don't mess with the volumes manually. Manually meaning going into the cluster containers or cluster nodes and running the cluster commands to modify the volumes. That should really not be done. Use Hakedi or Hakedi's own state may get disturbed. So this warning sign, so we are working on in the future being able to remove that warning sign to more tightly integrate Hakedi. Hakedi's state with cluster itself. But currently, this is what it is. So the warning needs to be done. So we can move on, please. So the next point is now that we've described how this all works, how to get to such a system, be it with cluster outside or cluster inside OpenShift. So for these purposes, we created a tool. The upstream version of this, like the upstream project on GitHub is under the cluster organization currently. It's called cluster Kubernetes. And the tool that does the deployment for us is called GK deploy, like cluster Kubernetes deploy. But it's also capable of running in an OpenShift world using OC commands instead of Qt control. And in the Red Hat products, we ship it as CNS deploy. CNS being the acronym of container native storage, which we kind of hook upon the OpenShift releases. So you see that I'm talking about CNS 3.4, which is more a synonym for OpenShift.4, in this respect. So that's what to do, right? I mean, we need to install the cluster or more of them. We need to install Hakedi. We need to tell Hakedi about all the disks and nodes. And all these needs to be deployed and appropriately configured. So this is what is happening. This tool takes a topology file, so-called topology file as an input. The topology file has the details about the node IP addresses, the names, the disk devices in each node, et cetera, et cetera. It takes that and it creates an Hakedi pod. It creates, if told so, it is optional. It also creates the cluster pods. If it does that, it uses a Damian set to set up the cluster clusters. And then it creates and deploys the Hakedi pod and then has Hakedi load the topology file into all the necessary steps of the setting up of the actual cluster. So the deployment of the cluster nodes is just the nodes. The software is there. The disks are there. And then Hakedi goes and joins these cluster nodes into clusters. It has so-called cluster storage pools. And it also teaches or initializes its own database with the knowledge of all these disks associated there. And after this script has run or this tool has run successfully, you will immediately be able to create appropriate storage classes and to PVCs. There are a couple of prerequisites to fulfill regarding firewalls and stuff like that. I have not put all the nitty-gritty details in here, but the deployment tool will tell you about them. In the future, the next version will even be able to check these prerequisites and also potentially satisfy some of them automatically. There are more questions, but Humble is already answering them. That's good. So let me just read the questions for the audience. Where will the cluster endpoints be set up in the persistent volume claim namespace exactly? How does the user develop to get the endpoint for this PVC? Oh, there's a lot of stuff going on. Maybe let's continue with a few slides because time is running short. And maybe I can read some selected questions later on because the discussion seems to be referring to some earlier points. Like the demo of the dynamic provisioning, let's defer the deployment demo to the end if there is more time left. So let's advance. We can also answer these questions in the blog post that I'll put up. Just so everybody knows. Sounds good. Thank you. So as the next slide, we have a little bit about the roadmap. So can you please? I've told you what is available in OpenShift 3.4, which is Gustav as a provision and mount plugin satisfying persistent volume claim and dynamically creating the recording volumes. It is possible to use that for all types of access. And you can, if you know how to do it, in theory, also use Gustav as a registry backend, but in 3.5 for the first time document that officially and have the reference process for doing that. We are also working towards improved what we call day-to-day maintenance operations like shrinking a volume by removing disk, replacing a disk with a bigger disk or different disks and without a downtime. And this is what we're working towards in the very, very soon release of 3.5. For 3.6, there is some bigger change coming on, namely, as I said, all access types are access nodes are supported, including read-write ones. So the read-write ones access type is actually defined as not as one might expect access from a single container or pod, but access only from pods from a single node. Typically, this is in many cases, though, used as access from a single container like exclusive access. And these applications requesting exclusive access for some persistent volume, they usually expect a certain performance guarantees. Like, for instance, the things alluded to earlier in some question like logging and so on, Shayan mentioned the Cassandra, et cetera, all stuff like that. And I mean, just playing databases where there are some application is exclusively accessing this, they usually have very high performance demands. And the Gloucester of S file mount, they usually have some difficulty satisfying those. You have to know that, I mean, and this is true for each and every single scale-out distributed file system. This is, of course, never as fast as a local file system, because many of the operations, especially the metadata operations, have to be coordinated in the whole cluster. And this is much more expensive than just going to a local disk. So creating files, locking parts of a file and stuff like that, all this is very expensive and the text time. And if some application has needs for some latency here, they may not be fulfilled. I mean, you can always try, it's supported, but it may not be satisfactory. So for 3.6, we have on the roadmap improved support for these special read-write-ones scenarios with the so-called Gloucester block provisioner. So what's going on here is this. We are going to offer an ice guzzly type of volume from Gloucester, and this is going to be mounted on the host. And then again, I mean, formatted and mounted on the host, on the open shift host, and then mounted into the container with bind-wound just like Gloucester volume. So the interesting point is that this will also be hosted by Gloucester, and it will also be able to be dynamically provisioned. So we'll have dynamically provisioned volumes, which in the back end are implemented as some kind of block transport. But this is completely transparent for the application. The application will request read-write-ones volume from a certain provisioner, and it will get this. And this is going to be, this is planned for the summer release of 3.6. And for future versions, 3.7 and later, we have some more things on the roadmap here. One thing is object access. I mentioned that S3-based object access on top of Gloucester volumes, file volumes in its case is available. And we are planning to implement that so that applications who don't want to use file mounts, but want to have some object storage for some put and get operations, or even, as the second bullet point says, even the registry may be a perfect fit for some object interface, improve back end support for the registry, is the second bullet point. So these are the very high level objectives for the future. And so 3.7 is roughly planned for autumn. So from autumn until later, you can expect those features. Then generally, I mean, the improved day-to-day maintenance operations, they are going to continue over the next couple of releases. The feature set will be much more complete. And we are working on the scalability. So one thing that has not been mentioned explicitly yet is scalability is not yet optimal with Gloucester. It has a certain amount of memory requirements. And there is currently a limit of roughly 100, 150 volumes per Gloucester cluster at maximum. In the next release of 3.5, this will already be improved. Later, it will be improved very greatly so that it will be possible to host many hundreds to maybe even 1,000 or a few thousand volumes on a single cluster cluster. But currently, we are a little bit limited here like typically 100 to 150 volumes per cluster. And you need a good amount of memory, like 32 gigabytes for the storage container to host this number of volumes. So these are the limitations, but we are working on improving them. As I said, 3.5 will already be improved to some extent. 2 to 300 volumes likely will be possible. And the 3.6 timeline will have probably a factor of 5 to 10 that we can add to the number of volumes that we can host. So this is roughly the roadmap for the next year or so. And with that, I think we can conclude the presentation and move on to the last slide, which is basically some pointers. You can also see the links in the previous slides where I put out the communities of Gloucester Kubernetes. All those have GitHub presences and many of the things are present in Slack or an IRC on Freenote, etc. So you have both the upstream and the downstream contacts here basically. This has been really good. And it was a ton of information. And I know we didn't get to the demos and there's a few people commenting here. Can we bring you guys back to do more demos another time because they'd like to see it in action? Really, people just thanks very much for coming and doing this today. We only keep these things to an hour. Sure. When 3.5 comes out, let's schedule another session with you guys and maybe to do, we'll add the links to the videos into the blog post that should go up on Monday. And we'll reschedule another session to do some live demos so that people can ask questions during the demos too. Yeah, absolutely. Diane, I think, I'm sorry. I think what we should do is after 3.5 comes out, we'll actually be using container native storage, which is Gloucester running inside OpenShift to back the OpenShift registry. We can probably do a demo for that plus the demos that we missed today. So that would be a nice session to do in mid-April or May, whenever we can schedule this. I will definitely hit you up right after this talk and we'll put some pencil something in the calendar. I'll also capture all of the questions here and send them to you guys so that maybe you can write little responses and I can include them into the blog. Thanks again guys for doing this. I've been asking for it for a while and I know that it's been a moving target, but I'm really appreciative of you taking the time today and being here for us. So thanks again. Thanks for having us.