 Good morning and thank you all for coming on a Sunday morning. My name is Gorka Ileor and I would like to start with a quick race of hands from those of you who have used persistent storage in your container workloads. It should be on. Okay, so seeing that many racing hats, probably I'm not the only one that thinks that it's a little bit of a mess. We have way too many interfaces to connect our storage to our container workloads. And sure, some of the interfaces are nicer than others, but it seems like the interfaces will not all that well thought of from the start. At the beginning we thought that we didn't need persistent storage in our containers and later on we started adding them as we needed and we ended up like this. Which is problematic for both users and vendors and storage vendors that have to have a lot of different drivers. So it's any time we simplify this and this is where the CSI or container storage interface comes in. It aims to be the sole interface for storage in any container platform. That's its aim. So, but haven't we heard this before? Like a standard suddenly comes a new standard that tries to be D1 and fails to do so and a year after you find out that it's just another one that comes into a list. So the question is, should we care about CSI? Is it actually going to make a difference and is it here to stay? And that's what I'm going to try to answer in this presentation. We'll go a little bit over the CSI spec, the first release that came out in November and we'll see how it works and what are the chances to make it. One of the things that CSI is actually trying really hard to do is to be a storage domestic and be open to all the different storage vendors, their deployment options, their features. And in this effort it supports many different infrastructure options. We'll go over the two most common ones. The first one is an architecture where all the storage can be accessed from all your nodes. So every node can access to create, delete and also connect the volumes. In this deployment we will have a CSI plug-in service running constantly on your nodes. Instead of having command line calls that are called from the orchestrator, the container orchestrator or CO, you have a service and using Google's RPC the orchestrator can make the request. This way we decouple the implementation and also makes it possible for the plug-in to be more efficient. It is the responsibility of the CSI plug-in to support the management of the resources as well as making them accessible to the nodes. This is a simple architecture but in many cases our deployments are a little bit more complex. So we have our storage that has different networks for the management than from the data path. So we will also have in our orchestrator deployment different types of nodes. We will have infrastructure nodes that can actually access the management to create and delete resources from the storage and we will have container workload nodes which only need to know about the storage transport protocol to be able to connect to the data paths. For example, if we have ISCASY, they only need to have the ISCASY initiator, they don't really need to know what backend they are using. Just that it's ISCASY, this is your connection information, go connect. These are the two most common deployment architectures in CSI and now we're going to have a look at the features. The features were developed or chosen between the orchestrator developers, the vendor, the storage vendor developers and third parties that were also interested in the CSI spec. So it is a joint effort between all the parts. It is not only the orchestrator that decided, hey, we're going to be doing this. So everybody's interests are represented in here. Sorry, in here. These are the features and we can think of them as two different groups. One group is the features that are meant to assist the orchestrator in doing its job in interacting with CSI plugins and the other group would be the features that are meant to manage the actual resources that we want. In the assistant or in the helping features, we have an info feature that helps identify the different plugins so that the orchestrator can know which controller goes with which node plugin because you can have multiple CSI plugins running and it needs to match them so that the request go to the right place. Then we have a capabilities feature because CSI actually, well, most of the, most if not all the features in CSI are optional. That way it can adapt to all the different storage solutions. So we need a way to check the plugins and see what features are actually available in a specific plugin that is deployed. Then we have a probe system that allows us to check the health of each of the different plugins. A way to find out how much available storage we have and a topology feature because we need to know which nodes can access which storage. You wouldn't want to, for example, try to create a container in a node and connect it to a fiber channel volume when the node doesn't have an HBA and it cannot connect. It doesn't make sense. So we have a topology feature that allows the orchestrator to do a smart scheduling of the containers according to the volumes it's going to be using. Finally, for the resource type of operations we have two different type of resources and we can create, delete, unlist volumes and snapshots and also we can get the stats on a specific volume to see how much available space is there and also attach and detach the volumes. These are basically all the operations that we have in the current CSI spec but the CSI spec is live. This is version one but we already have new features in there. For example, we can already increase the size of a volume which is not here but it will be released on the next version. We don't have time to go over all of them so I'm going to focus on three things that I think illustrate pretty well, the different mechanisms of CSI. The first one is the volume creation which like all storage related CSI functionality it must be added and potent. This means that you can receive multiple times the same request from the orchestrator and the CSI plugin must make sure that it only creates one volume in this case and it always returns the same volume. So the orchestrator will pass seven different arguments to the CSI plugin. The first one is called name and you can think of it as a request ID because it will be unique for each request and if you receive multiple times the same call you will have the same name. This is what is used by the plugins to actually attain item propensity. Then we have the parameters which is a key volume mapping that allows the CSI plugins and the storage vendors to expose their advanced features. This is how they support being storage agnostic while at the same time allow the storage to show their fancy features. For example, if you have a storage that supports compression and the CSI plugin says in the documentation, hey, you can use a key called compression, pass it through and your volumes will be compressed. Then the CSI can expose that and it will come as a parameter. These parameters are opaque to the orchestrator so the orchestrator doesn't care, just takes them and gives them to the CSI plugin so they will be different from CSI plugin to CSI plugin and you have to go to the documentation to check what you have available. Then we have the source argument which allows us to define what type of, what kind of volume we want. We can create three different types of volumes, empty volumes, cloned volumes which are volumes created from another volume and volumes created from a snapshot. Then we have capabilities which is the way of how the orchestrator tells the CSI plugin how these volumes are going to be used. Are they going to be mounted, used as a block device? Are they going to be mounted as a file system? What kind of file system do we want to be there? Is it going to be used by a single reader, single writer, reader-writer, multiple reader, single writer? You name it. So the orchestrator must ask the controller what it wants and if the controller is able to do it, it will return the ID of the volume, the actual size and also accessibility to tell where this volume is actually accessible from. This is a simple flow. It's a synchronous call. You call, it creates and when it's created, you get the return value. For a little more complex flow, we have the attach-detach operation which if you read the CSI spec, you will find out that they don't talk about attaching and detaching. They talk about publishing and unpublishing volumes and that's how they refer to it. First, we will see a flow in the first architecture we saw where every single node can access the backend so the container, the orchestrator will call the CSI plugin first to check the capabilities and see, okay, you are using architecture one, every single node can access the backend so it immediately goes to the node where it's going to create the container and it tells, please publish this specific volume and make it available on this target. And then, that's it. It is exposed. So it is the node publish obligation to make it accessible on a specific target. But what about architecture two where we had split the functionality between the manager and the nodes? Now, we have additional function. You call first the control capabilities and you find out that you first need to call the controller. So you get a little more info because the controller doesn't know where it's going to be publishing this volume. So you first query the node where you're going to place the container, get the information and with the idea of the node, you then call the controller and you tell it, hey, please publish this volume to this specific node. For example, if you were create a publishing an ISCSI volume, here is where you would be exporting and mapping the volume to the specific node with the initiator name and the IP address. And from the controller, then we call the node publish and make the final connection. But the node publish, if you have a multi-reader, multi-writer, can be called multiple times on the same node. So if you publish first, you publish for one container, you get one call. Then you create a second container accessing the same volume, you will get a second publish call. So this is something that the spec is very clear. You may receive multiple calls for node publish on the same node for different volumes. So to assist the CSI plugin, what you have is an extra, an optional call that can be made if the CSI plugin asked for it. It's called not a stage and it will go between the controller publish and the node publish calls right in the middle. It will always be called on the node where we are going to be running the container. And this allows drivers, for example, that use NFS to make the actual mount of the volume on the staging phase because they know that this is going to be called one and only once per node and per volume. And they can make it, they can mount it in a staging path that is passed by the orchestrator. And then when they receive a call to the node publish, as many calls that they want, all they have to do is beam mount the staging path and they are done. So they don't have to keep any kind of tracking how many containers are using this volume on the node. They just know. First, I will get a call to node stage. I mount it there, then beam mount. And the unpublished is the exact opposite. You get a node unpublished. They just need to unmount the beam mounts. And then when they get a call to the node on a stage, it is when they actually unmount the external storage. This is a little bit more complex. If you have all the in architecture 2 plus node staging, it is more complex. And finally, I have chosen the snapshot creation because for most drivers, for most plugins, this will be a simple synchronous call. The orchestrator calls and say, hey, create a snapshot. The driver cuts the snapshot. It is saved and it returns. It is a synchronous call and the snapshot is ready to use. But the CSI spec introduces one additional feature, which is post-cut processing. This means that your snapshot can be cut. It is ready. You return to the caller as a synchronous call and tell it, OK, I have made a cut, but the snapshot is not ready. I have to do additional work. So it returns and ready to use a false value. So the orchestrator knows that this is going to be an asynchronous call and it is going to be running in the background in the plugin. And it can take hours to complete. Let's say you are updating, uploading your snapshot to the cloud, for example, as a post-cut processing. And it is the orchestrator responsibility to pull and check, is it already ready to use? Can I use it? Can I use it? And since we, like all the resource operations, this is important, we have a very simple interface because you will be receiving the same parameters and you can easily check with the name if the operation has actually completed. This concludes the overview of the CSI spec that I had in mind. And now I would like to show you a little bit of the specifics in the orchestrator platforms. Right now the CSI is supported in Docker, Kubernetes, Cloud Foundry and Messos. And I'm going to give a brief overview of how Kubernetes implements CSI. Kubernetes has decided to implement it as a mix of sidecar containers, many sidecar containers, in a kubelet, the Kubernetes agent. So it has code in the kubelet and it has sidecars that you have to include in your pod. So for the architecture number two, where we have the controller and the node, we would have a pod in the nodes running your CSI plugin. Code as a service, then you would have the node driver, Racer, which is in charge of registering the CSI plugin into kubelet. So it knows that it's actually running on that node. And a Lightness Pro that can hook them into Kubernetes Pro system to report the health of the whole pod. The Lightness Pro is basically a gateway that receives HTTP request and makes calls via gRPC to the probe feature that we describe early. So HTTP request, check via gRPC the status of the plugin and return it via HTTP. On the controller or infrastructure nodes, we will have our controller side of the CSI plugin, optionally also the Lightness Pro, and then we start with the external provisioner, which is in charge of watching your persistent volume claims and trigger the workflow of creating or deleting them. If it is created, it will check the persistent volume claim or PVC and check the storage class, join the information from the two and pass it along to the CSI plugin. If you remember, we had the parameters in the create volume call that expose the extra features of the vendors. They are specified in the storage class. So you can specify your compression equals true in the storage class and the external provisioner will pass it to the controller CSI plugin. And then we'll create a PV, we'll bound it and it will do the whole operation. To attach, we have the external attacher sidecar that monitors volume attachments and calls the controller publish call. So this is only necessary if you have architecture number two where you have publish on the controller side and on the node side. And this is the first part of the attachment. The second part is done by Cubelet, which thanks to a node driver racer knows what is the socket to talk to the node CSI plugin and makes the GRPC calls from that node to complete the attachment calling publish or staging calls. Finally, for the snapshot, we have the external snapshot which monitors the volume snapshot and the volume snapshot class. Basically does the equivalent of the external provisioner. It joins all information parameters from the from the snapshot class, the request from the volume snapshot and makes the call to the controller CSI plugin. Right now we have this is version one. Now we have the resize. So we have yet another sidecar, which is the, I think it's resizer external resizer. And now I would like to do a quick demo of Ember CSI, which is a CSI plugin that implements supports like 80 different backends. It supports CSI dot two dot three and one zero in the same container. And the example, it's one of the that are included in the repository. It launches Kubernetes 113 with Ember CSI 1.0 using an LVM backend. I use LVM because I don't want to favor any storage vendor. And it deploys one infrastructure node and two workload nodes. It's recording because I didn't want to risk it and hopefully just a second because it is after a second. The quality I know why it's terrible. Let's see if it's. Oh, this is not going. Yeah, this is not going well. Anybody can see anything? No. All right. Awesome. I cannot make it bigger. Let's see if I get if it's just a second. No. Same thing here. All right. Just a second. I know if this is slightly better or if it's the same. Sorry about that. Basically, I'm going to describe it seen as nobody can see it. Clones the repository changes the directory into the examples one and runs a little script that uses big rent, Libbird KVM and Ansible to deploy to make the deployment. I have a sped up the Ansible part because it takes like 20 minutes. So now we have completed the deployment. We SSH into the master node. And now we're going to check that the pots are actually running. We have the controller and two node pots on the controller. We have five different containers. I'm not running the lightness pro container. So it's provisioner, a toucher, a snapshotter, the actual CSI plugin. And instead of the lightness, I'm running CSC, which is a command line tool that allows you to make requests to your CSI plugin. You don't need to go through Kubernetes. This is useful for debugging. Like, for example, you can run a list volumes without actually going through Kubernetes. On the nodes, we are running the driver racer. As I mentioned, the Ember CSI container running on node mode and again the CSI container, the tool. Now we have the driver racer registered with Kubernetes, with Kubelet. Right now it's CRD, which is CSI node info.csi.story.khs.io. And here we can see that it has registered the two nodes that we have and that the controller is not registered there. Now we are going to split the screen to see on the lower half the logs from the controller. And we can see that we have received multiple probes. We receive multiple probes because each the snapshotter and the attacher, each one probes the plugin to check that it's working. So they are coming from different side cards. And now we are going to create the PVC. Yeah, see that the new calls. All right, we get a request to get capabilities because to confirm that the controller can actually create the volumes, we get a little more capabilities, plugin info. And finally we receive the RPC create volume request with that is the name, what we call the request ID, the name argument, which was the request ID, which we use for item potency. And then we create a volume and we return that ID. So we have now a PVC that it's bound. And now we're going this is the get volumes is an internal representation for Embers CSI to to store the metadata of the volumes in Kubernetes. Now we create an app that it's going to be using that PVC. We receive the controller publish call and we return the value. We return that that the volume has been exported and mapped by the by the CSI plugin. So it can be used for from that specific note. Then on the upper half, we have the login of the note side that where the control and the container is going to be started. And we see that the driver is using a staging phase. So we could get a call to not a stage volume. And it returns now. It is request the request. Come gets the stage volume, then a published volume and the volume at this stage is already published in the target path. So Kubernetes can already use it. We check the status of the pod and we see that the application hasn't is not ready. And we wait a little bit and the application is running. We can check the volume attachments. That is the object that the attached sidecar is monitoring and we see that it is there. And this concludes the present, the demo. Just a second. Let me take it out from there. So to conclude, at the beginning we were asking, should we care about CSI? Is it going to make it? Is it going to make a difference? If you have a look and see the support that it's getting from the orchestrator and the storage vendors, the number of orchestrator platforms that already supported and the fact that all the new features are coming through CSI and the orchestrator are not bothering to add these features to the other interfaces. Only CSI is getting these features. It is clear that they are betting on this. And we already have support from the orchestrator. We have a good number of plugins. So in my opinion, it is going to make it. This is what it's going to look like, storage in the future for containers. And if you want to check it out easily, you can go to Embers CSI and try it. Thanks. And if there are any questions, I think we have like two minutes, four minutes. Okay. The question was, how can we track, for example, as a CSI plugin implementer, the evolution of the CSI spec? The development is being done in a GitHub repository. They have a mailing list and they have now, I think they have increased the cadence to buy weekly or monthly meetings. Everything is open to everybody. And you can check the status of the pull request, how they are advancing and check also the meeting agenda to see if they are discussing a topic. You can attend the meetings. So usually the best way to, if you are not as much interested in contributing to a CSI but implemented it, the best way is to just once a week to go check the GitHub repository, see the new patches that are going in and say, okay, these are the changes. Now we have a way to resize and we can resize offline or online. Let's see what my back end can do and just implement it. Does that answer the question? Okay. Anybody else? I don't think they publish, sorry. The question was whether they publish a roadmap or not. And I don't think they do. I haven't found it. I know they, when they are during the meetings at some point, they decide, okay, we want this feature. This is a blocker. They decide the priorities. But it is not published as such. You can probably, you can check the levels on the PRs and see what level they have been assigned as a blocker, as a nice to have, or as this is going on the next release. You don't have a page or something with this area. You have to query the PRs. Yep, I'm out of time. Thank you very much.