 Good afternoon, everybody. My name is Kapil Arora. I work at NetApp, and today we are going to talk about persistent volumes and stateful sets. So if you were there for the last session, there's a lot of overlap between the two sessions. I will also go through similar concepts. But I realized that when I was looking at that session, I was also learning some stuff from that. So I think it will be good for us as well to go through these concepts again. So what do I have on my agenda today? We are going to talk mostly about stateful applications. And I will try to explain what stateful applications really are. What do I mean by stateful applications? Then we will learn about the concepts in Kubernetes around storage. So we will go through the concepts of volumes, persistent volumes, persistent volume claims. And I will also talk about stateful sets, which was not part of the last presentation, which is very new to many Kubernetes users as well. I will talk about NetApp Trident, which is our integration into Kubernetes. And lastly, I also have a couple of demos to show you guys. So let's get started. So when we talk about application data, if we divide it on a very, very high level, we can classify them as structured and unstructured kind of data. So if I'm talking about media, or log files, pictures, they are mostly unstructured kind of data. And I can put these data into object storage, or I can use file shares or NAS storage for these kind of storage. And then you have application data, which is more relational kind of data, which is transactional, which you want to query maybe. And this kind of data is usually stored in your databases like MySQL or Postgres, which are SQL databases. Or you would store them in NoSQL databases, which are becoming very, very popular amongst people who are doing cloud-native kind of programming, because you can scale them horizontally, of course. So these are the data, or this is the kind of data that we are talking about today, and how do you run these applications within Kubernetes? So when you want to run such an application within Kubernetes, which needs persistent data, there are two ways to actually provide persistent data to your stateless application. So one is that you just use network storage or your host-based storage and provide storage to your containers where they can store their persistence. So that could be for an application which is written for object storage, that could be just object storage or file shares, or it could also be iSCSI learn or any kind of network storage that you are familiar with. But to actually store this data, you might want to use a database as well. And today, we want to talk about, does it really make sense to run these applications within a database? For example, I mean to say, does it make sense to run such applications which store your data? These data stores like MySQL databases or SQL databases or NoSQL database and key value stores within containers. So why stateful containers? Why do we want to do that? So there are lots of reasons that you would want to run your databases within containers. So there was a time when virtualization came into the picture and people thought that it's not a good idea to run your Oracle databases within your VMs. And people were scared to do that. And today, many people are skeptical about running databases within containers. But I'm sure that the technology will mature enough, will be mature soon enough for us to be able to actually run all kinds of databases within containers. So what advantages do I get out of doing that? So one of the major advantages that I see, especially for databases that scale, is that scaling these applications become really, really easy. And I have a demonstration with MongoDB in which we will actually show you how you can scale MongoDB with stateful sets. So that is one of the very, very good advantages that I see. And then when you are working in these environments, you know that you can write a YAML. You can write a YAML for your whole deployment. And then you can make it repeatable and also standardize how you deploy your applications. So that is another advantage that you get out of containerizing your databases, for example. And then handling node failures, handling upgrades, could be made much, much simpler by providing databases within containers. So these are some of the advantages that I believe are important and can actually help in all kinds of environments. And that's why we are talking about it today. So now that we have set a stage for what we mean by stateful applications and what we are going to talk about, let's get into the concepts within Kubernetes. So to provide persistent storage within Kubernetes, we have a concept called volumes, Kubernetes volumes. And Kubernetes understands a lot of different kinds of storage. So if you see on your left-hand side, you have RDB from Cep, iSCSI, NFS, EBS from Amazon, Cinder, Google Cloud Engine, persistent disks, and Azure as well. So all these storage is understood. Kubernetes understands all these storage types. And all these storage plugins are implemented within the Kubelet. So if you want to use within your pod or your application any of these kinds of storage, it is already available to you or Kubernetes already understands and can make available these different storage to your application. So if you see in this case, if I want NFS to be available to my pod, I can just provide information within my YAML. And then the pod can actually access this particular volume. Let's look at a couple of examples on how you do that. So I have two examples, one with iSCSI and one with Cinder. So if you see here, I'm using a Nginx pod. And I'm exposing my pod 80 on it. And I'm using a volume for my HTML data, which is iSCSI. And right here, I'm providing all the details about how to make this iSCSI connection, what IQ and to use, what type of file system it is, and is it read-only or not. So I'm providing the volume details within my pod definition. And I'm using this volume within my Nginx pod. What this means is that Kubernetes understands iSCSI and will be able to provide this storage or mount this storage within my pod. And similarly, we can do that with Cinder as well. And once you have provided Cinder or OpenStack as a cloud provider for your Kubernetes deployment, you can use Cinder volume IDs directly to actually provide to your pods. Now, what we just looked at was something which was directly connected to the pod. So if my pod dies, my volume is not available anymore. And I'm managing my volume within my pod. But I don't want to do that. I want to use persistent storage as a cluster resource or something that my developers can use. For example, I have Compute. And I can make available this Compute to my users of the cluster. Similarly, I want to provide storage to the users and make this storage available to them. There comes two new concepts that we will talk about. And first is persistent volumes. So what persistent volumes are, they are network storage. Mostly, they can be local storage as well. But today, we are talking about network storage. They are a cluster resource. And administrators need to actually provision this storage and make available within his cluster. It could be NFS, iSCSI, or any kind of cloud storage that you want to use. And it is independent of the pod lifecycle. So when it is actually provisioned, it has nothing to do with the pod at that moment. This is an example of a PV. So basically, I'm just providing NFS server details and the path to that particular volume. That's it. And it would be something similar for iSCSI that we saw in the volume. We need to provide all the details about the iSCSI connection. And I can provide a persistent volume or define a provision, a persistent volume like this within my environment. So this will be done by the cluster admin. Now, to consume this storage, a developer needs to create something called a persistent volume claim. So he needs to claim this storage. And he will do that within his pod. And he will define what kind of storage he needs, what kind of access mode he needs for that particular storage, what size he needs, and what is the reclaim policy. Basically, if the pod dies or if he deletes the claim, what happens to the volume? So this is an example of how a developer will use persistent volume claims. So he will actually define a persistent volume claim about the access mode. He will tell how large his volume should be. And then within his pod, you can see that he's providing details about the volume and just using persistent volume claim name to add volume to his pod. And then he's providing the volume mount location. And then he can use this persistent volume within his application. So what we just saw is basically static provisioning. The administrator will actually provision the storage beforehand. And then the developer or the user of the cluster is actually going to consume it. But it is also possible in Kubernetes now that the developer is creating a claim and the storage is getting provisioned dynamically for the user. So it's more self-service oriented. So that is also possible. And now we will actually discuss these two possibilities within Kubernetes. So here you can see I have a cluster admin. I have a Kubernetes cluster. And I have a user or a developer of this cluster. And the cluster admin has access to a storage system. In this case, it is not a solid fire. And of course, the user doesn't have direct access to the storage system. The cluster admin provision storage, so he creates persistent volumes, PVs. And these are now cluster resources available for a developer to consume. And now the developer writes two pods. And he says, I need two volumes. And he creates persistent volume claims. So if you can see, the left one is for five GBs. And the right one is eight GBs. So now what happens as soon as these persistent volume claims created, the PV controller from Kubernetes takes control. And he actually does the mapping of the claim and the persistent volume that exists. So in this first case, you can see the 5GV volume was mapped to the 5GV volume, which was easy for the controller. But the other case, the provision storage was 10 GBs. But the developer only asked for 8 GBs. So there is a slight mismatch. But still, he's able to fulfill his request. Just imagine if the other storage was 100 GB, then the 100 GB volume will be mapped. So there is some, because there is a manual effort required, there could be some mismatches in this case, which probably we don't want. So this is static provisioning in Kubernetes and how a cluster admin creates storage and a user consumes storage. Now we will talk about dynamic storage provisioning. So what is that? But before we go into dynamic storage provisioning, I want to introduce storage classes. It's not that storage classes can only be used in dynamic provisioning, but it is important to learn about storage classes before we actually learn about dynamic provisioning. So storage classes is a way for the cluster admin to define what kinds of storage he has available. Does he have SSDs or SATA disks? Does he have a storage system which can provide different kinds of quality of service levels like solid fire? Or does he have separate drives which are encrypted? So he has different kinds of storage. Storage classes gives him an opportunity to actually define a storage kind of a catalog for his users to consume. So it could be QoS levels. It could be gold, silver, bronze. It could be dev staging or production. So that's the idea of storage classes. Now let's also look at a couple of examples for that. So in this case, this is for Cinder. So I'm providing something called a provisioner, which we will discuss about in a bit. And I'm saying which availability zone should it be, which type of storage it should be. So this is an example of a Cinder fast NOVA availability zone storage class. Now if we talk about something that we can do with NetApp, I can define, OK, this storage system and gold type of storage. And this is something also that we will look at later. But basically, this is how you can define storage classes. And this is the whole point of storage classes within Kubernetes. Now let's look at how dynamic storage provisioning works. So I have the same environment that I had before. I have a developer. I have a Kubernetes cluster. And I have my cluster admin. But in this case, I have more storage class play. What the cluster admin needs to do now is basically create these three classes, or whatever classes he wants to do, and deploy a provisioner. So in this case, we are deploying the Trident provisioner. Kubernetes also comes with its own provisioner, which supports different storage types. But NetApp has written its own provisioner, which we are going to showcase today. So in this case, he deploys a provisioner, which is Trident, and he defines different storage classes. And we thought it makes sense for the SolidFire storage system to define it in terms of IOPS. SolidFire is a system which can give you guaranteed IOPS. So that's why we are using IOPS here as a mechanism of creating the storage catalog. So once he does that, he doesn't need to do anything more. And we can actually remove the cluster admin from the picture itself. His job is done. And now the developer says, I need, for my application, 8GB of gold storage, which should be redried once. What happens in this case is Trident is looking out or checking what kinds of PVs and PVCs or storage classes exist in this environment. And it finds out that there is a new request or new claim that has been created for storage. And it dynamically provisions the storage, talking to the SolidFire system. Directly, it provisions 8GB. Actually, this is wrong. It should be 8GB of storage for the user. And once he does that, this mapping is again done in a similar manner as previously by the PV controller, right? So what is the difference in this case? In this case, the developer only needed to know the storage class he wants and the size he wants for this particular storage. And the storage was dynamically provisioned for him. And he didn't need to wait for somebody to create it or have it pre-provisioned for him. So that's the difference, and that's the advantage that you get in dynamic provisioning. So there are lots of dynamic provisioners which are already available. So there's Trident from NetApp. Then you have OpenStackCenter, Amazon EBS, Google Cloud Engine, PersistentDiscs, GlusterFS, and SAPRDB. These are the ones I'm aware of, which you can already use with the Kubernetes provisioner, which is available by default. Now let me introduce the NetApp provisioner. So NetApp wrote this provisioner, which is open source, available on GitHub. And this will monitor your PVs and PVCs and provision the storage dynamically for you. It will help you abstract all the storage features behind NetApp storage systems, and the user doesn't need to be aware of encryption, compression, deduplication, or he doesn't need to be aware of any of the storage features. They can all be abstracted using that. And it supports all NetApp storage system that is on tap, solidify, and E-series. And depending on different types, like disk type, IOPS, compression, you can specify different kinds of storage classes. And it also has a REST API, so you can also directly talk to it if you like. So now let's look at a demo of how this actually works. I will try to do this live. And hopefully, everything will work out. So let's see what we have here. So you can see I have a four node cluster. Kubernetes cluster running in my lab. And if you see, if I do what pods I have, so I have a trident running as a pod. So this is the NetApp provisioner, which also runs as a pod. And I can see what storage classes I have defined. So I have two storage classes that I have defined here. So as a developer, I don't need to know much more than what storage classes I have available. That's all. So we are right now acting, or I'm pretending to be a developer who wants to deploy an application. So let's look at my application. So I have a WordPress and MySQL application that I'm going to show. So if you see here, I have created a persistent volume claim for my MySQL disk, which should be of the class gold, and which should be 50 gigabytes, and rewrite once. Rewrite once means that it should be available to one. It should only be mounted at one place at the time, right? So it shouldn't be like an NFS store. It's basically like your sand storage. So I've also created a persistent volume claim for my WordPress disk, which should be also of the gold type, which is a smaller share that I need or a smaller volume I need. So this is basically a WordPress and MySQL application. I have defined services for my MySQL to expose the port 3306 so that my WordPress can connect to it. I have a service for the web front end so that I can expose it to the outside world. And now this is my MySQL pod. And you can see that I have my volume here, which should be on this path. And I'm using my persistent volume claim that I created earlier to provide volume to my pod, right? And same is the case for my WordPress mount. So I'm providing my claim. So as a developer, from the storage point of view, I'm just creating a persistent volume claim and nothing more than that, right? And I need to know what storage class I want. So let's run this. kubectl create minus f. So my WordPress application is deployed. Let's see. So it's still creating it. So it will take a couple of minutes. And let's see if my service is also deployed. So you can see I have a front end service, which is running on this port. And my WordPress is running on the third node. So now my application is running. Let me see what's my IP of the third node. So let's see if it actually worked. So this is the port 31136. So my WordPress application is deployed. And let's see if persistent volumes were actually created for my application. So kubectl get pv. So this is the persistent volume. So there were no persistent volumes earlier in this environment. And as soon as I ran this YAML and asked for persistent volumes, they were on demand created by Trident for my developer. So the developer doesn't need to understand anything about storage. He just needs to know about the storage class. So that's the important part that you want to abstract everything complicated about the infrastructure from the developer and expose him only to a storage catalog. Let's get back to our presentation. So that was about dynamic provisioning in Kubernetes. We understood what volumes are in Kubernetes. We tried to understand what are persistent volumes. We tried to understand what are persistent volume claims. And we tried to understand what are storage classes. I hope it was clear from the graphic and also from the demo as to how a developer and a cluster admin will work together and how they will use storage classes, persistent volumes, or persistent volume claims to actually run their applications. So now there is a new concept that I want to introduce to you. That is called stateful sets. It's a very new concept within Kubernetes. And it is to run applications which have different kind of features. And let's see what do these applications need to have for me to consider stateful sets. So if you have an application, for example, like a MongoDB, and you want to scale it horizontally, what you need to do is whenever you create a new instance of MongoDB, you need new storage attached to it. And if you are running a regular replica set or deployment, whenever you scale this application, a regular replica set or a deployment within Kubernetes, you will see that the same storage gets attached or gets consumed by that pod. So that doesn't really work in the case of MongoDB. Other important point is that when we are creating a MongoDB replica set, the nodes need to talk to each other. And if they are getting scheduled again and again, their network identifiers will change. So if you need that your application has a unique network identifier and is able to talk to each other, for example, in case of a replica set, you need unique identifiers for that. So that is something that stateful sets can offer us. And then if you need order deployment of your pods. So in case if I'm deploying a three replica or three instance database and I want one node to come after the other and not all of them to come up at the same time, then probably I would want to use something like a stateful set. So these are some of the reasons or some of the use cases for applications like these you would use stateful sets. So how do you define a stateful set? So basically, there's a standard definition of how you do that. So you define a headless service. I will show you what that is. Basically, you don't provide any cluster IP to it. So for example, if it's the case of Mongo and I have three nodes, I can access my nodes by saying Mongo1.mongo if Mongo is my headless service, for example. So I'm able to access the nodes with their new identifier which is given to them as a number. Mongo1, Mongo2, Mongo3. Then the other, of course, the component when we are using stateful set is the stateful set where I can define what pod I need to run and the number of replicas, for example. And then lastly, I need to provide a volume claim template. So in this case, I'm not creating volume claims myself, persistent volume claims, but I'm providing a template. Every time I create a new node for this particular application, this template should be used and volume should be provisioned for me. And in that template, I provide the access mode, the size of the volume, and the storage class of this volume. So basically, this is even more dynamic than using persistent volume claims. So let's look at the MongoDB application that I'm going to show as a demo next. I will show you again this particular picture, but here you can see I have defined a stateful set. This is one YAML file, but I've divided it into three parts to show you the three sections. So I have my stateful set where I'm saying that replicas should be three. Then I'm defining which image to use, which Docker image, for example, in this case, what container port it exposes, and which path the volume should be mounted. And then I'm defining a volume claim template. So how should the volumes be created, which storage class should be used. And then finally, I'm defining a headless service. So this demo, I'm going to just use the video because it takes time for the nodes to come up. So I think it's better if we don't do it live for the sake of time. So let's look at the application again, which I already showed you. So I have this headless service. Its name is Mongo. It exposes this port. So the cluster is none, which shows that this is a headless service, basically. And then I have my stateful set, which is for Mongo, which has three replica sets. And I'm also defining the grace period for termination. If I'm starting or stopping the containers, how much time it should take. And I'm defining the command for my MongoDB. You will also notice that I have one more sidecar application, which will actually take care of creating the replica set. And I'm defining where I'm going to mount this particular volume. This is the volume claim template. I'm using the gold storage class. And each volume should be 100 gigs. So now let's run this application. Let's deploy it. I will deploy it in a very similar manner. With kubectl, create my MongoDB replica set. So you see I just created the service and the stateful set. Now let's watch how these MongoDB instances are coming up. So you can see that Mongo0 started to create. And I have fast-forwarded this. You can see the timer running on the side. So once Mongo0 is running, then the Mongo1 starts to come up. And then once Mongo1 comes up, after that Mongo2 will start to come up. So basically, it's ordered in this case. So now my MongoDB application is running with three nodes. So what happened in the background is basically three pods. I have three pods running. I have three new persistent volume claims that were created. And those were created dynamically for me. So these are claims that were created by the template or by the stateful set. And then you can see that actually these claims are bound with the new persistent volumes that were created. And these were dynamically created by the storage or the provisioner that we defined. So you can see I have three persistent volumes that were created, which were of the type code and 100 gigabytes. So now is a very, very important thing that stateful sets offer us. Now if I want to scale it to five nodes, what do I do? Which this can be done by a very simple application. Just imagine how would you scale a database in the earlier times. But here I can just say cube CTL scale, define the number of replicas, and give the stateful set name. And that's it. And now my MongoDB application will have five nodes as part of the replica set, and which will also come up in an ordered way. So that's the magic that stateful sets can offer you for applications or databases which can scale horizontally for databases which need to come up in an ordered way. Right? So one part that I still would like to show is how was the provisioner configured? So we saw that NetApp Trident is running as a pod here. But I've also configured this particular pod. So if we go to the Trident installer, and you can see that I've defined something called a back end. So we use SolidFire in this example. And let's look at how I can define a back end. So basically, I'm trying to show you how storage was actually defined or connected to this particular. So here I'm defining, or as a cluster admin, I am providing details about how to connect to my cluster. And you can see I'm also defining the different classes. So I'm saying bronze should have quality of service, which should be minimum this much, maximum that much. And burst IOPS, I'm defining right in my back end. And then I'm defining a storage class on top of that, which also we can look at in the definition. So basically, the administrator need to define a storage back end and storage classes. So let's also look at the storage class. So basically, the back end I defined, I'm choosing that back end and saying choose the gold definition that I've put in my back end. So that's it. So that was the administrator part. So whatever we looked at before was how the developer was using it in MongoDB example and also in the WordPress example. And now what I showed you was the part that the administrator had already configured, such as the back end and the storage class. So the MongoDB example that I showed you, I've written a blog on our netapp.io website if you are interested in how that is actually run. With Trident, if you're interested in netapp storage, you can actually look at it. And if you go to the blog, you will also find all the code and the GitHub link to where actually the code repository is for this particular example. These are some of the resources that you can use. So what are volumes? If you want to learn about volumes in Kubernetes, persistent volumes, persistent volume claims, stateful sets. If you want to learn about Trident, that is the GitHub link to Trident, which is an open source project. And this blog, and also everything that netapp does around containers, you can go to netapp.io and find about. Or also we have a Slack channel where you can discuss about how we implemented our provisioner, for example, or if you have any suggestions on how it is working today. Yeah, that's it I had. If you want to get in touch, these are my Twitter, LinkedIn, and GitHub IDs. You can get in touch. If you have any questions, I will be happy to answer them. Hello. The question is, so we're talking about persistent storage, persistent items. How do they persist the destruction of the VM or destruction of the whole set? Can I please elaborate on that? How do we reuse the data? How do we save what was in the database after we destroy the whole set and want to recreate, maybe with a newer version of container? So what we looked at today, like the MongoDB example, I'm using network storage in this case. So what stateful sets can provide us is those unique identifiers. So even if my pod dies and is rescheduled, the storage is network storage, and cluster has access to the storage. So the pod will be scheduled on a different node, or even the same node, and the storage will be made available again to that node. So that's the whole point of having stateful sets, where you are using network storage, and you can still handle these kinds of failures or node failures. So basically, all your data is within the network storage, and you are created for the scenario when actually the storage is away. That's why you are creating a replica set so that the storage can recover and provide redundancy there. Thank you. And I also saw there were recycling policies, something like that, recycle, delete. So actually, I made a slide on that as well, but I thought it would be too much detail. So basically, there are three types of reclaim policies. You can retain your data if you like, or you can choose to delete it. So if a persistent volume claim is deleted, so the data will be deleted as well, you can choose that, or you can say, I want to retain my volume if the claim is deleted, or you can say it should be recycled, so it will actually remove all the data, which is only supported in case of hostpath and NFS. So these are the three reclaim policies that are available. What about the resizing volumes on the fly? So that, as I do not know, if we can do that today already, for example, you can, of course, do that on the storage system. But through Kubernetes, or if there is an API to do that, I'm not sure if we have that already. Hi, right here. Cool. So nice presentation. So you show us how you use the NetApp Trident volume to provision the volumes. Is that volume already part of Kubernetes? So if you want to use Trident, you can just deploy it as a pod. So basically, it's a pod. You can, it's available, the container images are available on Docker Hub, and you can just run it as a pod in your environment, and it will be a provisioner. OK. So how does Kublin know about, understand Trident volumes? How does it know how to mount it to the host so that the part of the museum, how is that happening? So Trident provisions I SCSI, or NFS volumes, and Kublet understands those. So that's the point here. Thank you. How do you control the access to these volumes? Any access control policies? Which user is supposed to use? Which storage class or claims or whatever? So in Kubernetes, we have a concept of namespaces. So you can set quotas, number of volumes that a consumer can, for example, you can say this particular namespace, which is for one development team, this particular namespace has a quota of one terabyte gold storage. So you can do something like that. And once the storage is assigned to a particular pod, then it is available to that pod. But I'm not sure if, in that particular environment, in that particular namespace, if we can restrict access to one or the other volumes. But restricting, you can do it on a namespace. But beyond that, I'm not sure. Within a namespace, I'm not sure if you can do that. So my question is actually around like local attached volumes. So I run a lot of environments, and some of them are very small. And we're trying to keep costs down. So one thing that's thrown around is what if we could just use the storage that's directly attached to the compute nodes themselves? And maybe we have like a handful of compute nodes that maybe have very fast storage, and some that we're just kind of spending disk. Are there any options in Kubernetes to kind of manage that type of situation? Or is it really just kind of always provide network path and then remount that network path wherever the application spins up? So I don't know if you were there before my session, but that session was exactly about managing local storage. Just redo that session for me. So basically, exactly what you are pointing out, there are applications which might be able to leverage the local host storage, and which could actually be very, very fast compared to, or it could use it as a cache space or scratch space and still have a persistent, some level of persistence available. So she showed how you can use persistent volumes and persistent volume claims. Exactly what I showed for network storage. Now you can use that for PVs and PVCs. And somebody asked a question about, can you use actually block storage from the host? Which she said that people are working on that, and if you're interested, you can come to the SIG, Kubernetes Storage SIG, and discuss that. So there is work going on, and this use case is actually a use case that is coming up. And we had a talk about that. So maybe you can check it out on YouTube. All right, excellent. Thank you. Maybe we can take this question and then afterwards, sure. Yeah, just a simple question. I guess all this works with Trident. Would that work with the Cinder provisioning mechanism as well? So I would have loved to do a Cinder demo, but I really struggled with the documentation. There is resources available, and there is a Kubernetes Cinder plugin, for example. The provisioner from Kubernetes supports Cinder. It should work, but I have not tried it personally. So the question is, what do we do about other backends? And I showed NetApp as an example. Basically, I was showing you a provisioner that is NetApp, but Kubernetes comes with it on default provisioner, which I showed as Kubernetes.io. And it supports different storage types. So dynamic provision should already work for Google Cloud Engine, AWS, EBS, Cinder. All these should already be working, but I have not tried them myself. But Trident is one of the provisioners, and Kubernetes comes with a default provisioner and supports multiple different storage types. So if you want to use Cinder, then you would probably go for the Kubernetes default provisioner that is available. Any other questions? OK, thank you, everybody.