 Alright, we'll stay on the topic of containers, this time with Gluster. Give a hand for Kaushal. Thanks. Thank you everyone. Okay, so today I'll be talking about Gluster container storage. This is that is storage for containers running in your containers, similar to what Rook is doing with Seth. So this is what Gluster does for containers, right? So I'm Kaushal. I'm a senior engineer at Red Hat and I work on the Gluster project. I'm a maintainer and I work mainly on the Gluster management frameworks and right now I'm contributing to the container story around Gluster. So in today's talk I'll talk about what GCS is. We'll go into a little bit more of a detail about what GCS looks like, how it performs as in how it does what it does. And I'll let you know how you can try out GCS. Okay, so just a quick question first up. So has anyone already heard of Gluster in containers with something called Hackety, right? So yeah, this is GCS is built on our experiences that we gained with the Hackety project and running Gluster in containers with Hackety. So I'll touch a little bit about Hackety and try to compare what GCS does differently from the Hackety based stack in the stock. But yeah, so let's get started. So Gluster container storage. I'll be calling it GCS for short. Please don't get confused with GCS because in Kubernetes there is already GCS which stands for Google Cloud Storage. It's not Google Cloud Storage. It's Gluster container storage. We would welcome suggestions for a better name. So for now it's GCS. I'll be calling it as GCS. Okay, so what is GCS? GCS is a new effort from the Gluster community to provide persistent storage for containers, right? So we try to provide the two normal demands of container persistent storage that is the shared storage and the what do you say, exclusive storage. The RWX and read write many and read write ones volumes. We try to use the standard container interfaces as much as possible right now. So GCS implements one of the projects in GCS is to implement the container storage interface on CSI and using that we provide volumes when requested. For now we are Kubernetes only but maybe later on if we get help we'll move on to other container orchestrators. And GCS is being designed for a hyperconverged deployment right now. As in your storage runs along with your apps on the same servers that your apps ports are running, right? So yeah, so while the storage ports run along with your app ports on the same nodes, these are not your normal app ports. These need privileges because we do deal with underlying devices and underlying file systems and all. So we need to run in a privileged mode, right? Okay. So GCS, right? It's not just a single, it's not a single project. It brings together a lot of Glastro related projects and under a single umbrella. So I'll go into detail about what each of the project does for GCS specifically. So we bring in Glastro FS that everyone, I hope everyone already knows about, right? So the file system, there's Glastro E2 that's the new management tool that we have been building. There's the CSI driver specifically. There's Ant Hill. Ant Hill is a Kubernetes operator that will deploy and manage Glastro on Kubernetes. And there are other projects like Glastro Prometheus which helps monitor Glastro inside a Kubernetes deployment. So basically you can plug in Glastro into your Kubernetes dashboard and you can drill down into Glastro volumes. Glastro Mixon provides the dashboard configurations, provides the alerts and rules, etc., right? Okay, sir. So now why would you want to use GCS over what we have currently with Hecti, right? So the main theme of GCS is to simplify the whole experience as much as possible, right? So we simplify how the containers get deployed, how Glastro gets deployed on Kubernetes. We simplify the general overall management workflow intended for the admins or users, right? We also go ahead and we are trying to simplify the Glastro FS stack itself. So and try to simplify the stack, the file system stack and ensure that they work well for the container ecosystem. So what we do is that we try to make opinionated defaults, right? So right now the normal Glastro FS installation doesn't do any opinionated defaults. So it just leaves it open to the users to what you want. Everything is enabled. Everything is available. So GCS is in that. So there are, we choose features that need to be enabled. We choose options that need to be defaulted. And yeah, that's how GCS goes. Yeah, and all of this is going to be automated. So hopefully the most admin needs to do is to write a manifest for anthill, deploy anthill, write a cluster manifest on Kubernetes and done. So anthill takes care of deploying everything that is required. So all the projects that I mentioned earlier. So anthill takes care of deploying all of them, managing all of them in the future. They'll also do data operations like replacing a node, upgrades and other stuff, right? And all of this together, right? Allows us to improve our scale in the number of volumes at least. So Glastro FS originally wasn't designed for the container sort of environment. So Glastro FS was designed as a NAS replacement wherein you have one single large volume or a few single large volumes. So, but in containers, you generally deal with lots of very small volumes. So Glastro FS isn't particularly designed for that, but we are doing optimizations. We are trying to figure out ways so that we can scale better. Our goal is to, on it, I guess this is goal for a three node cluster that is like the minimum for, if I believe, open shift that is required. That we can provide 2000 read write, what do you say, read write many volumes and up to 5000 read write once volumes. That's the goal. These are like small, small volumes. These are not really huge. So something that can run in three node cluster, right? So this is what Jesus is going to provide. That is going to be different from the original stack. So I'm not going to talk too much about the current stack. As I mentioned, we have a hackety-based stack. So the hackety-based stack looks somewhat like this. So we have a component within Kubernetes itself. That's so we are pretty much like tied into the Kubernetes release cycle. So it's not so independent. There's a Glastro provisioner within Kubernetes. We have a separate hackety part, right? Hackety manages the Glastro parts. And the thing, one big problem with hackety-based deployment is that we have a hackety-db separate with a Glastro configuration separate. So there are two separate ways of the Glastro pool. Hackety has its own view. Glastro has its own view. And the problems that we see there are mainly about these two going out of sync. And recovery from that is hard. But a lot of work has been done on this. And right now we are a lot of it has been solved. But still there is a problem with this. And in general, this stack relies on the normal cluster deployment as the normal cluster installation as in without the opinionated defaults that the new GCS stack is going to provide. Right? So there are details available below in the slide. The link is available in the schedule. If you want to, you can look at this. So looking at it, it looks relatively simple. But deployment and managing this cluster is challenging. It's not easy. Right? So let's get into GCS itself. Right? So GCS looks like this right now. So there are few components. We have Kubernetes. We don't need to worry about that. So we have Anthel. Anthel is a Kubernetes operator, as I mentioned, that is going to deploy and manage the rest of these things. Right? In addition to Anthel, we have the CSI driver. That is going to be the point of interaction with Kubernetes for us. So Kubernetes talks to the Gluster CSI drivers. Gluster CSI driver will then talk to the Gluster parts. In the Gluster parts, we have the new management tool running Gluster D2. We have Gluster FS. We have the Prometheus exporter inside. And we also have an HCD cluster. That is a single source of truth right now. So the new management tool, Gluster D2, stores the cluster view in HCD. So we have, GCS runs its own HCD cluster and doesn't make use of the Kubernetes HCD cluster. Right? So yeah. Okay. I talked about all of them anyway. So let's go into Anthel first. Right? Anthel is a Kubernetes operator. Right? Which defines custom Gluster GCS resource definition. So Kubernetes provides ways for what do you say, other projects to define their own resource types. Right? Object types. So Anthel defines the GCS resource types in communities and it automates deployment, upgrades and day-to-day management of GCS. Right? At the current state, Anthel is very early. There isn't actually anything, other than the CRDs, there isn't actually anything present yet. But things are, we are working on stuff. Things will come out soon. Anthel is developed under the Gluster project in GitHub at github.com slash cluster slash Anthel. Next up, we have the CSI driver. So the CSI driver consists of, okay, so as I mentioned CSI driver translates the Kubernetes CSI request into the GD2Rest request, the Gluster D2Rest request. That's the main goal of these drivers. Right? So in addition to doing the translation of the request to create and delete volumes, right? It also will take care of doing the mounts. Unlike in Hackety, right? What would happen is the Gluster provisioner in the Kubernetes. Within Kubernetes would take care of reaching out to Gluster or reaching out to Hackety. That provisioner would take care of doing the mounts. But in this case, we do everything via the CSI driver. So it gives us more freedom to do things the way we want. Right? So the CSI driver consists of three different types of parts. There's a node plugin. There's a provisioner and an attacher. Right? I'll go through what each of them does. Attach your provisioner and node plugin. Right? The attacher is for us, at least in GCS world. It's a no-op. It's useless for us. But we just need to provide it because the Kubernetes CSI spec expects it. There is work going on to ensure, to allow that, allow CSI drivers without attachers. So not all CSI drivers actually require attachers' work. So the provisioner part, right? This is the main part for us right now. So this takes care of translate. This is the one part that listens for Kubernetes persistent volume requests or persistent volume claims, forwards them to Gluster. And yeah, that does it basically. It listens for the create and delete volume requests and forwards them to Gluster, translating them to the GD2 requests. So in this part, we have, sorry. Oh yeah, in this part we have four different things running. So there is a Kubernetes provisioner. Sidecar, so these, a few of them are provided by Kubernetes themselves. These are not developed by us. But yeah, so there's a Kubernetes provisioner. The Kubernetes provisioner's job is to listen for Kubernetes persistent volume object requests. And then the provisioner would talk via the CSI spec to the Gluster CSI driver, which would then relay that forward to the Gluster layer behind. So yeah, so there's a provisioner. We have a Snapshot or Sidecar which takes requests, handles Kubernetes requests for volume snapshots, persistent volume snapshots. And there is a cluster driver registrar which registers Gluster FS CSI as a cluster level driver. So it makes the Gluster storage available as a storage class. So that's what the provisioner does. This is deployed as a, right now if I am right, a state full set so that it's always available running. And yeah, then we have, then we have the node plugin. The node plugin basically runs on every Kubernetes node that you have, right? Its main function is to mount the volumes when requested or attach the volumes to the app parts when requested. So yeah, that's all it does. So there's a driver registrar. Again, it makes the Gluster driver available on that node for mounting. So that's, there are two containers. The driver registrar makes sure that the driver is registered for, so that Kubernetes knows to reach out to the driver to perform on. Right? So if the node plugin parts, right, also contain the Gluster FS client bits. So you don't need to have the Gluster FS client bits installed on the Kubernetes hosts. With the previous deployment with HackET, you needed to, you needed the client bits installed on your Kubernetes hosts. So that was one more issue with that. So yeah. So then now let's go into the Gluster part itself, right? So the Gluster part has Gluster D2, Gluster FS itself, and the Gluster FS exporter. So first I'll talk about Gluster D2, right? So Gluster D2 is the newer management framework for Gluster FS. It's intended to provide automated management operations for Gluster FS as an automatically create volumes based on requests from the user, right? So a user can request for a volume of a particular size of particular capabilities. And Gluster FS GD2 will automatically decide upon the layout of the volume and as the user requested and create the volume, right? In addition to that, it also tries to provide intelligent or automated volume management operations about, say, scaling a volume or shrinking a volume, replacing nodes, stuff like that, replacing bricks, stuff like that, right? So basically what it is doing is it is converging the functionality of different tools earlier that were there outside the cluster, right? So we had HackET running outside the Gluster cluster, Gluster pool, sorry. There was a project called Gluster Block which is used to provide the RWA volumes or block volumes, right? These were running outside the Gluster pool, right? They were not directly managed by the Gluster layer. All of this is converged into one tool right now, so Gluster D2. Gluster D2 does the work of HackET, right? Automated volume management. And it also provides block volumes. In a different sort of way compared to Gluster Block, we'll see how it does it in a little while, okay? So yeah, so GD2 provides a REST API like HackET, right? Against which you can program. So this allowed us to write our CSI driver. We have an API that can be reached out to over the network to do operations. So the CSI driver works on that. GD2 also has a very good orchestration engine built in, right? So it's more resilient to failures than the orchestration engine in the current cluster management layer, right? So we can do undoes, we can do rollbacks if some operation failed. We can do an operation and say a note comes back up later. It can rerun the operation if required stuff like that. So it brings in features to do more resilient orchestration of operations. So basically the management framework, what it does at the end is that it needs to go to each node and prepare the break, start processes and do stuff like that. So the GD2 brings in a much better framework to do that. In addition, as I mentioned earlier, there is just a single source of truth for everything. There is no split in cluster information between HackET and Gluster. So all the information is with Gluster D2. So Gluster D2 knows what all volumes we have, how the volumes are laid out, what all devices we have access to, stuff like that. So how many devices we have used, everything, right? So yeah, there is less possibility of failures because of HackET and Gluster going out of sync, right? In addition, GD2 also brings in support for Prometheus and OpenSensors so that you can trace or monitor GD2 with the standard tools that are used elsewhere in the container ecosystem, right? So Prometheus is widely used for monitoring within Kubernetes. OpenSensors implements the Open Tracing API. So again, we provide Open Tracing API endpoint so you can trace operations across the cluster if required, right? Yep. So as I mentioned, GD2 provides normal volumes as well as block, Gluster block. So the normal volumes are used to satisfy the Kubernetes RWX volumes or rewrite many volumes. The way how Gluster D2 helps there is that for normal volumes there is automatic provisioning of bricks. So we go ahead and we create the LVMs, we create the thin poles, we create the file system on top of that, we do the brick mounts. We do all of that, GD2 takes care of doing that. And in addition, one special thing that we bought in was support for volume templates, right? So the normal Gluster installation would contain a volume template with everything, right? All the features that we provide. But for the GCS use case, for the continued use case, you wouldn't need all of them. So if you had all of the features enabled in the graph, Gluster graph, it would be resource heavy, right? So you don't need that. So we can provide a custom simple template for RWX volumes for containers. Cool. In addition, we also provide RWO volumes, as in we also provide block devices, right? So block devices are meant for single users, right? Single client access. This is still a work in progress. Bits and pieces of it have been merged in. It's not complete yet, but yeah. So what this does differently from the Gluster block project is that this tries, does block exports in a very simple way using loop pack devices, right? So what it basically does is that you have a Gluster FS mount on your Kubernetes node. And from that mount, on that mount, we create an image, I mean device file or image file and then do a loop pack mount so that it's available as a block device, right? The Gluster block project did its exports via iSCSI and there was a lot of involved process in getting that set up. So that's more complex. Scaling was a problematic there. This one allows us to scale really easily, much well. So let's see the current status. Yep. So as of this moment, we can scale to again. All of these I'm talking about in a three node cluster, right? You can scale to about a thousand volumes, IWX volumes and hundreds of requests parallel as in hundreds of volume creation requests or volume decision requests happening parallel. So it doesn't bother you. We at this scale, we have begun hitting the limits of LVM scaling itself, right? So LVM, again, like Gluster previously, LVM itself wasn't designed for this sort of a thing. So Gluster uses LVM thin pools to provide snapshot capabilities. So we use LVM. So we need LVM, but right now we are hitting on LVM scale issues. We are trying to figure out optimizations in the way we perform LVM operations itself that can help scale a bit further. And we are also looking at other options if possible for IWO volumes. Yep. Yep. In addition to the LVM scaling, we also sort of hit HCD scaling when all of these requests come together. Again, we are tuning the HCD config itself, how we deploy HCD and we are tuning how we use HCD itself. So as in how we store data in HCD, how we access HCD, right? So we are doing that. And Gluster D2 is also, again, developed as a separate project under the Gluster community. So it's developed under GitHub, Gluster D2. Okay. So now let's look at what we have been doing in Gluster FS itself to help with GCS, right? So the main focus has been optimizing resource usage. So this has been in progress from the last few releases. So Gluster 5, one of the main topics was optimizing resource usage. Gluster 6, the upcoming release is also, there is a lot of focus, again, on optimizing resource usage. So basically, we are trying to reduce the memory that we use. So we are fixing a lot of the memory leaks that we have and everything there. We are trying to also optimize how we use threads, how we deploy it. And yeah. So we are optimizing resource usage. And again, there is this, if I don't know if you have heard of multiplexing before, this multiplexing is a feature in Gluster where we run multiple bricks in a single process, right? So again, before GCS, it wasn't such a priority. It was okay, we wouldn't have thousands of volumes running with GCS, with containers. We can't have thousands of processes running inside the pod, right? So we need to consolidate that. So yeah, we have multiplexing already for that. Multiplexing works for bricks, but there isn't a similar process for self-healing. So the volumes that we provide are redundant as in replica Gluster volumes, Gluster replica volumes, right? So they need to have self-healing demons running. Self-healing demons right now are individual per volumes. So again, there would be as many self-healing demons as volumes that we have. So we need to multiplex the self-healing demons as well. So that's work in progress, okay? And there are other features like fencing. What is fencing? Yeah, so fencing is like a feature where we can ensure that only one client can access a file at a given time. So this is required for block volumes so that we don't have clients or say, right? Say now a pod was rescheduled on some other node. Or we don't want them to like begin accessing the image file before some cleanup has happened or some release of blocks or whatever has happened, right? So things like that. So again, Gluster FS is developed under GitHub Gluster FS, right? Okay. Next on monitoring. Yep, so monitoring is developed under two projects. Again, under Gluster community, there's Gluster Prometheus and there's Gluster Mixense, right? The Gluster Prometheus project provides us with the Gluster FS exporter demon. This runs along with GD2 and exports the Gluster metrics available, right? So it's not just the cluster level metrics that GD2 could provide. It's like per volume metrics, per big metrics, process metrics as in how much CPU is being used, how much disk space is being used and all of that. So the Gluster Prometheus project provides this. And there's a Gluster Mixense project which provides rules and alerts, right? For Prometheus and for Grafana so that you can build dashboards out of it. So right now the Mixense projects provides a dashboard configuration for Gluster FS volume. There are others being built. Okay. So I'm going to skip this. I was just going to talk about how the flow of operations was going to happen for a volume creation, but in simple terms, it's just straightforward, right? So it's Kubernetes to CSI driver, CSI driver to Gluster. Again, the same flow. Create a volume, mount a volume. It's like one flow, single flow. It's the same for both sorts of volumes because everything is in one place. There's no split. There's no HKT so that the request needs to go to HKTR then to Gluster, right? Okay. So, okay. If you want to try out GCS, just keep in mind that it's still in very heavy development. We are planning on doing a one-dot release sometime soon where again it's not fully featured, but at least you can make use of it and just see how it works, right? So yeah, at the moment, we have an Ansible-based deployment tool written. It's not a deployment poll. It's just a playbook that deploys GCS, right? As anthill isn't complete yet, so we are depending on Ansible right now and this playbook deploy GCS.yaml. This performs the full deployment right now. But as anthill grows and gains more features, we'll be cutting down what the playbook does. So in the end, the hope is that the playbook would only deploy anthill and we are done with it, right? So, for more information on how to use the playbook to deploy GCS on your Kubernetes cluster, you can read the readme. So the readme gives you information on how to prepare the Ansible inventory files and how to run the playbook so that you can deploy GCS, right? But if you want to just do a quick test on your local laptop, so we have a vagrant file available which does this. It's vagrant-libert only for now because we do some adding additional tests and all that we haven't yet figured out on how to do for vagrant with virtualbox. Contributions are welcome to help do that. This vagrant-based tool will deploy Kubernetes and GCS for you on a three-note cluster, right? So Kubernetes deployment is handled using CubeSpray. We have a playbook that calls CubeSpray finally, but this playbook has some configurations that we need to enable for GCS to run, right? So we need to enable certain feature gates to allow the CSI driver to work. So CSI driver is still a work in progress in the Kubernetes community so not all the features are enabled out of the box. So we need to specially enable some features for the CSI drivers. Again, more information on how to use this is provided in the readme, right? So, yep, this deployment tool is available under the GCS project. So there is a github.com slash slash slash GCS. Under this we have a deployed directory where all of this is contained. So yep, and that's basically it. Again, as I mentioned, GCS is being developed under Github cluster GCS. So under this project we mainly have the overall issues tracking the full GCS, what do you say, worldview sort of a thing. And there is a waffle board which tracks the GCS related issues across the multiple different projects that I just described. So that's it. Please try out GCS and let us give us your feedback. And I'm open for questions now. Okay. So do we support extending volumes? GD2 itself supports it. The CSI driver doesn't yet, I don't know if the Kubernetes CSI spec supports it yet. So if that support comes in, we can build it. So our, the cluster management layer already has support for automatic expansion. So we just need to do one extra call when that is available. Is it possible that, so I suppose three nodes should have equal storage, equal hardware, yeah. It would be preferable? Preferable, yeah. It would be preferable, but, okay, sorry. The question was, in a three node cluster. It forms this big storage. And to answer the source sum, I provide only one revenue for that. And for some other volumes, I ask to have, for example, three per application. Is it possible? Okay. So I'll repeat the question. The first part was, is it possible to have non-homogeneous hosts? We prefer if you have homogeneous hosts, but GD2 should be able to work with non-homogeneous hosts. The second part was, if you are able to choose the, what do you say, redundancy count, the epic account. Right now, we aren't targeting that because we want to simplify the number of options that users have. For us to build confidence in what we are doing first. That might come in later on, right? So right now, we, the CSI driver is hard-coded to create three replica volumes. It's hard-coded. It's hard-coded. So those three replica volumes can be created on a heterogeneous cluster, as in, but the thing is, since it expects always three replicas to be present, if we can't individually provision bricks on three, on each of the three nodes, the request will fail. So that's going to happen. But yeah. Any further questions? Yes? Not really stable. Okay. The question is, how stable is GCS right now? GCS isn't really stable. We haven't reached something that we would ask users to deploy in production yet. Right? The upgrade stories and all should be pretty simple, but we haven't thought about that yet. We have been just working on getting what we want to deliver in the first release fixed. So the upgrades for GCS would be simple enough as, because it could be just deploying the new version of the blaster parts. Right? All our configuration right now stays in HCD. HCD itself has its own upgrade mechanisms already defined for Kubernetes, so that's taken care of. The GCS parts themselves always rely on the information in HCD. So once we reach that stage, yes, it should be simple enough to do upgrades. Yep. We need to do so. Okay. The question is, how do we intend to do the upgrades? Right? So if parts are mounting volumes and whatnot. So as I understand, there is some sort of, I don't know, gating sort of capabilities in the Kubernetes operator world. I don't really understand the operator part of it well. So the operator would take care of doing that. Right? So what the operator would do would be to like bring down one particular part. Right? Then get a new version of the part up and running, do whatever mounts is required inside that. Till that part is up and running, it would block the upgrades of the rest of the parts, do stuff like that. So it's not exactly clear at the moment, I believe there is some way to get operations to allow this to be handled in a smooth way. Yep. Any further questions? Nope. Thank you everyone. Please provide feedback on Boston.org.