 Hi, everyone. Welcome to this tutorial. Well, I will introduce the container storage interface primitives and how to use those in Kubernetes. My name is Michael Madsen. I'm a tech marketing engineer and master technologist with the Hulu Packet Enterprise. I'm presenting this at KubeCon Virtual 2020. And if you're watching this live, thank you so much for hanging in there. This is on the last day. And if you're watching this as a rerun, thank you so much for watching this content. I hope you find it interesting. So this tutorial is basically all about CSI and what you can do with it in Kubernetes. And that is sort of the introduction to CSI that I'm going to talk about. Some of these CSI drivers are out there and what CSI is. The second part of the presentation will talk a little bit about dynamic provisioning of persistent volumes in Kubernetes. I think it's important that we kind of nail down the basics before we go into the more advanced topics. And I also want to talk a little bit about how parts and controllers attach to persistent storage as that is also a very important primitive to understand while working with persistent storage. The more fun stuff kind of begins when we start talking about CSI snapshots and using data sources and your persistent volume claims to be able to clone external storage into a new pod and leverage data sets that already exist on your storage system. Using raw block volumes is something that is introduced in CSI as well. It's been around for a while. I'm going to show you how that works and how those different use cases around using raw block volumes work. Another interesting concept is using ephemeral volumes with your pods and that basically makes your external persistent storage volume act like it is a container. There are various different ways on how to attach those ephemeral volumes to your pods and we're going to talk a little bit about how that works. At the end of the presentation I will kind of summarize what we talked about and there will also be a live Q&A. So this session is pre-recorded but at the end of the session there will be a live Q&A. I will be there in person, answer any questions you might have on this presentation or anything that kind of relates to this subject. I will also hang out in the Slack channel and I've been in the Slack channel throughout the event as well. In one important detail I kind of want to touch on as well here that I'm obviously going to deploy a lot of YAML files like that and run through a lot of hands-on labs. I've got 11 hands-on labs for you so this presentation really goes to 11 and I kind of put together a GitHub repo where all these config files and there's also Askinima Cast files so if there's anything you actually see in the demo in the video, you can use the Cast files and play them on your local computer and basically copy and paste the text from the demo because that is really difficult to capture from a video file into a terminal. So that is the repo that we are going to use throughout the entire tutorial. So let me start off here and kind of talk a little bit about what is CSI. You know that CSI stands for Container Storage Interface but what is actually behind it. So it's basically a specification. It's a set of specifications and the lack for a better comparison it's sort of like the cinder for Kubernetes with the benefit that the drivers and the entire frameworks live outside the main Kubernetes project. So the drivers and all the sidecars I'm going to talk a little bit about this later they're not part of the main Kubernetes upstream tree so the sidecars live in their own repos all the CSI drivers are delivered by vendors in various different ways. It's also governed by the Special Interest Group Kubernetes SIG storage they meet on our regular cadence and I also put some links here into the CSI documentation itself and I'm going to reference that in some of the demos as well on how you kind of find stuff and such. And the main goal for the CSI specification is to provide an interface standard on how to provision and attach storage to container orchestrators that is often referred to as a CEO. And we're obviously going to talk about Kubernetes today but you can also use CSI for Nomad or Cloud Foundry and Mezo and those are like the less popular ones I know that Nomad just got CSI support in that container orchestrator as well so we'll probably see a few more use cases for that coming up. So I just want to kind of touch a little bit on the history of CSI and what it kind of came to be. So if we kind of go back to the early days of Kubernetes we had these like entry persistent volume plugins and the two example plugins I have here like Fiber Channel and IceGaZ and they were kind of introduced in Kubernetes 101 IceGaZ in CO.15 but there's a slew of different plugins that is part of the main Kubernetes distribution and somehow this became really unwieldy really early on because what happens is that every vendor needed to learn how to contribute to Kubernetes and that wasn't really our easy feed back then and everything needs to be code reviewed and you also stuck to the release cadence of Kubernetes and that prohibits the vendor to kind of innovate at his pace and always have to wait for the next release for new features or bug fixes and things like that and that was pretty unmanageable. What came along in Kubernetes 1.2 is the entry flex volume plugin and that introduced the concept of allowing vendors to write flex volume drivers and flex volume drivers were kind of useful it was a pretty decent stopgap but somehow that became very unuser friendly as well because the flex volume driver itself it's a self-contained binary that needed to reside on every kubelet in the cluster that was supposed to attach persistent storage and it did not have any dynamic provisioning support either so if you were a vendor and want to provide a flex volume driver you also have to write your own provisioner to be able to satisfy persistent volume claims with that driver and that's what kind of when we can't move into the phase of the container storage interface it just completely lives outside of Kubernetes there's like nothing inside Kubernetes that depends on the CSI the delivery vehicles of CSI or the velocity of how things are introduced and such and the entire framework is actually deployed on top of Kubernetes so nothing lives in tree and that allows vendors then to once you have all the CSI side course and such in install on your Kubernetes cluster vendors can then provide their CSI controller driver and their CSI node driver and that is the vehicle that I'm going to talk about today on how to install CSI drivers and how to use the particular feature that a CSI driver provides so I also have this simplified architecture view of CSI so you kind of have the side course that I've been talking about first you have the node driver registrar and you will see like on every node that have CSI drivers on them you will see what drivers they have and some of the features that they provide like topology keys and such and then you have all the side course the provisioner, the attachee, the resize or the snapshotter they are all provided by the Kubernetes SIG storage community and then you have the external components which is the CSI controller driver and the CSI node driver that then talks to an external storage system the storage system can actually run on Kubernetes itself or it can be outside the cluster entirely so if you have like an external NFS server or a block storage server that can live entirely outside and then you have the container native storage or container attached storage solutions that are out there where you actually deploy everything on your cluster as well the communication that is between the sidecar images and the controller driver and the node driver is using a GRPC interface so some of these components they need to run on the same nodes but the external storage system or the storage system component they can talk over an entirely different interface like using REST and iSCSI and what not another detail you can see up there in the the light bulb there is that CSI drivers today they may provide either file or block storage and there are a few Kubernetes enhancement proposals around providing object storage with sort of similar semantics so we're surely looking forward to that and we all almost have over 100 drivers there is 90 some drivers available today so if you go to the URL that is on the slide there you will see the list of the different drivers what kind of features they support so I'm going to use the HPE CSI driver for Kubernetes in the hands-on labs and such but it doesn't really matter what CSI driver you use as long as it supports the different features and the different specification levels of the CSI specification and if you pull up this page you will then be able to see that there are certain aspects of each individual driver so I just took a good example here in the driver list CepFS and CepRDB which provides block storage and they provide sort of like different aspects of the spec that it supports and different features so we can see that the modes that the driver support is persistent it doesn't support ephemeral so the mode could be the persistent or ephemeral the access mode if you look at CepFS they can do read write multiple pods I'm going to talk about that later in this tutorial what that actually means but this is just a way for you to kind of assess the different drivers depending on your use case you can see that the block storage driver will only support read write a single pod and different features that are very similar the only difference here what you can see here is that the raw block or the block volume driver will support raw block and it will also support topology and those keys are not needed for using CepFS the file system component because it's a distributed file system will be available everywhere and it does not have any block capabilities so depending on your use case what kind of apps you're deploying you kind of want to assess the driver list the driver you're using support the specific capability that you're looking for alright so these are the different features that we're going to talk about today and kind of run through the different tutorials for provisioning storage and show you how to attach raw block volumes and so forth and these all have different maturity levels within Kubernetes today we have the features made GA and a few data features and we also have one alpha feature we're going to talk about today the generic ephemeral volumes that got introduced in 119 and it's a pretty neat addition to if you compare that to the ephemeral local volumes and I'm going to talk in depth about the difference around that when we get there and this is also something you need to consider when you want to use persistent storage with your workloads in Kubernetes and it's also the fact that different Kubernetes distributions they may mature these features on a different cadence or a faster cadence depending on the particular use cases that particular vendor want to cater for so that is also something to keep an eye out on and all these different features are described in depth on the Kubernetes CSI GitHub repo as well and we're going to cover most of these I'm not going to talk that much about topology I'm going to touch a little bit about that when we walk through the storage class volume limits I'm not going to talk about that either but that's a way for the CSI driver vendor to put a node limit on the node how many volumes you can provision from that particular driver to that particular node which is quite useful for the rest of the capabilities here I'm going to show volume expansion thermal local volumes volume snapshots and also use persistent volume clone using the data source stanza in the persistent volume claim so we have a full agenda for sure and so we're kind of approaching the first kind of hands-on lab here right so and that is basically installing and inspecting CSI driver right and drivers you can find most drivers on artifact hub.io most of them installed as helm charts some of them have fully blown operators and some of the drivers you just reference and a configuration file the points to a github repo or a web server and that will install the driver for you and once you have a driver or if you have access to a cluster now what you can do is just do a kubectl get CSI drivers that will list the drivers that you have installed on your cluster and the capabilities of the driver and if you do a kubectl get CSI nodes you will see what nodes in your cluster have CSI drivers on them like verbose, opportunity, what driver they actually have installed so now we are going to install a CSI driver that's our first hands-on lab in this tutorial so I'm going to switch over to my terminal and hang on for one second alright as I mentioned I'm going to use the HPE CSI driver for Kubernetes and you install that with helm and I'm just going to add the helm repo to my cluster do a helm repo update and then I'm going to create a separate namespace for that driver what I want to install it do a helm install namespace vendor and the CSI driver name I've obviously made a typo there I want to give the release a name because this is something switching from helm 2 to helm 3 obviously provides that headache and once the driver is installed you can do a cube color get CSI drivers you will see the capabilities of the driver and the driver name and you will also see on your nodes that you will have I have full working nodes in my cluster and they all have the driver installed the drivers also have a way to configure themselves so for the HPE CSI driver for Kubernetes I need to provide a secret that provide a means for me to find the back end that I want to use so for me to be able to start provisioning storage from storage classes and such I need to create that secret to make sure that the CSI provisioner or attacher and such will be able to find that secret when they need to talk to the HPE CSI drivers let me go ahead and create that and there we go driver installed so now when we have the driver installed we can do a lot of things right and the first thing I want to talk about is how we can do dynamic provisioning of persistent volumes and dynamic provisioning in Kubernetes is nothing that is exclusive to CSI drivers in any way right so if we just imagine for a second here that we might be using a Kubernetes entry storage plugin we might be delivering a cloud provider manage Kubernetes service to provide persistent storage and so forth or we might be using a CSI driver so on the left hand side here you will see that the cluster administrator he will create something called a storage class and that will reference the provisioner you want to use and you want to give it a name and you also might have a list of parameters that are specific to that particular provisioner for CSI drivers you need to have a list of keys that references the secret to the different sidecars and if you have your own sidecars you need to call them out there as well and you want to specify things like file system type and things like that and there are some other keys in this storage class that I'm going to talk about in the presentation the middle piece here the user aspect of it is that users create persistent volume claims and persistent volume claims are namespaced right so and that will in turn he will request the access mode I'm going to talk about that later as well what that is and then you request a storage size that is where you specify the capacity in you can do either terabyte, gigabytes, kilobytes megabytes whatever unit you want and you might want to call out the storage class there is an annotation you can do on the storage class I'm going to show you how that works as well that will allow you to specify default storage class and that means that any persistent volume claim without a storage class name called out explicitly will be provisioned from that storage class once a persistent volume claim has been submitted to the cluster the dynamic provisioner which listens to that specific provisioner name will provision something called a persistent volume and that is a stanza that basically describes the backend storage so that's where you will have your driver name and your implementation specific keys and such and how to find the volume on the backend so CSI you will also have a bunch of secrets and things to be able to attach and detach that storage from a particular note at any given time and you will also have the access mode and all that metadata that the Kubernetes needs to be able to attach and detach the volume and provision the volume as so forth so diving into the different object here if you look at the storage class here then having a default storage class is usually good hygiene right so if you provision a managed Kubernetes cluster on any of the public cloud providers you will see that you will have a storage class so do you have a kubectl get storage class you will see that there will be a storage class there it's more default and that will use the cloud providers storage solution to provide persistent storage to your workloads in the case of CSI you will have these keys that says CSI.storage.kates.io and a set of published secret names or secret namespace for the different sidecars and you need this for every sidecar right so the CSI expander sidecar we need a secret and the publishing and a few others that you need to reference in the storage class need to be pointed out there explicitly you also have the ability to set the reclaim policy on the persistent volume so it's basically when a user deletes a persistent volume claim what is going to happen when that persistent volume claim gets deleted will it be will the persistent volume that references the back end storage be retained on the cluster or will it be deleted another key here I'm showcasing is the volume binding mode and if that's going to be immediate or wait for first consumer and that is important if you're using topology within your cluster right so if you have a driver that supports topology you might want to be able to separate your different controllers or your pods in different zones in your cluster right and when you provision storage you want to make sure that you attach storage that is close to the node right so what happens is once Kubernetes have selected a node to provision your pod that's when the persistent volume gets provision and the persistent volume will then be attached to those set of nodes right because and it will then annotate the persistent volume with the with the affinity keys and that means that that persistent volume will only be able to be attached to those particular sets of nodes and if you want to allow or if the CSI driver you want to use allows expansion you will set that in the storage class as well right so if it allows expansion you will set that to true and you will then be able to resize your persistent volume claims as you desire or not resize that's the wrong term because you can only expand you cannot shrink and I'm going to show you how that works as well so looking at the persistent volume claim it's very straightforward very basic and you need to specify the access mode in my next slide I'm going to talk a little bit broader about what access modes are and then you need to specify the the resource request the capacity that I mentioned right so in this example I'm going to provision or I'm going to actually request a two terabyte volume the volume mode is set to file system per default but this is where you would specify volume mode block if you want to provision a block device and the underlying driver supports that and you will also be able to specify the storage class name or omit the storage class name if you have a default storage class but in clusters where you have multiple tiers of storage say you have a gold silver bronze style thing or you have like fast SSD slow media kind of segregation right it might be useful for users to be able to make that distinction between provisioning fast storage or slow storage because there's usually cost associated with that and the billing and accounting apartment is will be happier if the right workload is running at the right place so PVC access mode or persistent volume claim access mode this is a graphic I put together to kind of illustrate what types of applications require different types of storage right so one of the absolute most popular Kubernetes controller to use with persistent storage is a stateful send right and you will find like Mongo, Minio, Rook, Redis, Kafka all these different workloads that you run on top of Kubernetes they use something called a stateful set and stateful set in itself I'm not going to cover that in detail but that is a controller that has a has ordered starts persistent network naming and also persistent naming of the volumes that gets attached to each of the pods as well right and what happens is that each pod that starts up will basically have its own file system so each pod will benefit greatly by having a read write once persistent volume claim right because that storage will then be private to that pod for the duration of that stateful set right so if you delete the pod the pod will have the exact same storage at the same time and storage will also be provisioned dynamically and I'm going to talk about this in detail later in the tutorial as well another very popular pattern for deploying legacy applications on Kubernetes is to use something called a deployment right and in this case if you're using single replica deployments they usually leverage read write once persistent volume claims so as you can see here I have my sequel both in the legacy app single instance and the shared nothing distributed because you can run that database in two different modes right you can have the replicated one as a stateful set where you have one main instance and multiple replica instances and you can also run it as a single replica pod where you only have one single pod accessing one file system so that is a matter of preference however you would like to run that particular application and the same goes for postgres it also postgres also has the same pattern as my sequel in the stateful set as well you can run it in a shared nothing distributed architecture and when we get to scalable distributed applications that require shared storage right so say that you have NGINX distributed front end with a lot of content right it's really practical to kind of have them reference the exact same storage across the cluster so when you scale replicas up and down it will attach the exact same storage to that particular pod also a lot of the AI ML based workloads like running Jupyter Hub or Kubeflow they kind of see storage as a data lake right so every instance that you spin up of Jupyter or certain aspects in your AI ML pipeline requires all the different replicas of that particular workload to access the exact same storage at any given point in time and you accomplish that by using something called a read write many persistent volume claims so then multiple pods can access the same storage at any given time so if you look at the content serving aspect of it where you want to provide read write many access in read only mode essentially I see some of the use cases I see that you want to provide read only content to NGINX to serve static content you might have a Jenkins server that attach storage but you don't want your build jobs to screw up your storage sort of thing and running those jobs on a read only file system makes sense for that particular purpose right and in the way that you kind of attach read only many storage is essentially you request you have to specify it in the request that you do a read only many and then in the mount point you said I want to request this read only and that is basically how you request a read only many or RWO or RWX is called read write many so I hope that clarifies the different aspects of persistent volume claim access modes so with that said so the last slide kind of in the dynamic provisioning piece here is I just want to lay out the persistent volume overview here for our CSI driver and this is kind of slightly abbreviated this object contains a lot of information so once the CSI provisioner is provided instantiated this persistent volume it will have a lot of metadata around the driver what parameters to attach it all the secrets will be enumerated there will also be something called a claim reference which claim the pv is bound to and so forth also you see the volume mode there and the volume handle there actually references the back end that's basically the idea that you send to the back end storage to be able to see what to be able to look up the volume and attach the volume so forth and there's also a bunch of finalizers up there at the top to be able to see that essentially means that you cannot delete persistent volume if there's a pv holding a claim against it and such and you can have other finalizers on there that are all specific to your driver as well so that leads me to hands-on lab number two we're going to create a storage class we're going to create a persistent volume claim and we're going to attach a workload to that persistent volume claim and I'm also going to show you how to expand a volume for that running workload so hang on while I switch over to my demo alright so what I'm going to do here first is I'm going to show you the storage class that I'm going to use here I want to make this storage class the default storage class we're going to use for the entire tutorial and it supports all the different capabilities that I talked about in the introduction there and we need to specify all the different keys for the side cars and we also want to specify what file system we want to use in this case XFS we want to specify the reclaim policy and we want to allow volume expansion and we want to make sure that the volume binding mode is immediate so when the PV gets bound to the PVC we can use it immediately and we don't care where Kubernetes schedule is so we're going to create that the next step here is creating a persistent volume claim and you can also see here that in this particular storage class we marked it as default and this is the persistent volume claim I'm just going to give it a name specify access mode read write once because it's a block storage backend I'm using and I'm going to make that volume 32 gig initially and I'm going to resize this later and I'm going to show you how that works so we're just going to create that persistent volume claim created and we can see here that the status is bound and the volume is actually a reference there in the volume column is referencing the actual PV which shows here even more metadata about that particular persistent volume alright let's see what we can do next so I'm going to deploy MySQL with Hell I'm going to specify MySQL root password as admin I don't recommend that and I'm going to use my existing claim that I just created I'm just going to do a Hell install MySQL reference the values file specify stable slash MySQL and I managed to get the Hell syntax write in this example just going to wait for the MySQL deployment to come up we're waiting there we go successfully rolled out and then I'm going to do a container here and we're into the part I'd say and do some inspection here for you to show how things are wired up so I know that the mount points for MySQL is usually MySQL something I'm just going to grab that we'll see that we have in the we have a multi-path device mounted on slash warlib MySQL it's an XFS file system and we can see here with the disk free command that we have put a two gig volume mounted there and I'm just going to jump into the database here and create a new database and you will see that a database gets created on the on the volume there we go pop back out and we can now see that we have that database living in the file system which references that persistent volume alright pop out in the shell here again and I prepared a separate persistent volume claim YAML specification that will essentially expand the volume so I'm going to double the size of the volume and we also reference the same PVC there so I'm just going to apply that you can do different methods here so you can you can edit the PVC cube call edit and reference the PVC on the running cluster you can also use the cube cuddle patch command to patch it and it doesn't really matter which way to go right I mean you will get the exact same results so I'm just putting cube call get on my persistent volume claim here and I'm just going to wait for the capacity to expand here what happens here in the background is that the CSI resizer sidecar tells the underlying CSI driver to expand the volume the volume gets expanded on the back end storage system and once that operation completes then there's a node expansion operation that happens that talks to the CSI node driver and that will essentially expand the file system on the node itself so it will do XFS resize there we go 64 gig now and once we kind of clear this watch here we'll jump back into the container and see that we can actually leverage that extra capacity yeah so there we go 64 gig mounted on slash for live MySQL and you don't need to restore anything or doing any other operation so that is basically how simple it is to expand the volume in Kubernetes with a CSI driver that supports it and this is all user driven right so since the persistent volume claims are namespaced you will then be able to the end user will basically be able to expand the persistent volume claim all right that was the dynamic provisioning one I'd say I just want to talk a little bit about the workload controllers here you notice I deploy the application using a helm chart which is kind of practical on how how you would manage applications on on Kubernetes right but what you see here in these two windows here is sort of like most controllers they reference volumes in the port specification right so you specify a mount path and then you give the volume mount a name which is my mount and then you have a volume section which points to the persistent volume claim name and that's kind of how you key those two together right on the right hand side on this slide you will see the state full set and that is slightly different from the other controllers right there you have a construct called a volume claim template and that is basically it's more or less a inline specification persistent volume claim specification where you call out the storage class name if you want you can leave that to the default one to resolve it and you also provide much capacity each of the volumes are supposed to have but you still use that with the volume mount for a particular path in the volume claim template right so what happens here is when you deploy the state full set say you deploy a single replica state full set that will provision one volume and as you scale the state full set it will provision a new volume for each replica that comes online and that's why it's very practical to use read write one storage with a state full set because that individual part will have private access to that particular volume and that is that was a short slide because we're already on hands-on lab number three where we will deploy an application utilizing a state full set so hang on here while I switch over to my terminal so what I'm going to do here is deploy redis I'm going to deploy it with a Helm short as well and this is the values file I'm going to provide use password equals false and I'm also going to prepare a watch command here so this watch command will watch my state full sets my pods and my pvc as redis comes up so I'm just going to do a Helm install call it my redis reference my values file and I'm going to use the bitnami distribution of redis and here we go and here's my watch command that I had prepared and you will see here that the main instance is already coming up there's also a a replica instance being set up as well and down to the list of persistent volume claims you will see that there are two claims that has been fulfilled and now you can see that there's another pod coming up and it automatically created a new persistent volume claim and once all these pods have started that means that all the storage had been attached to persistent volume claims that were dynamically provisioned although you're using a Helm chart here it is a state full set that redis leverages and you can see by the persistent naming of the volumes here that they are derived from what you call the release name and so forth and that means that you have predictive naming of the volumes and the same thing with the network naming in state full sets as well that means that the pods and the instances that runs in those pods will be able to find each other I'm just going to insert a key here cube constatus what do we think? It's awesome so I'm just going to put that key in here make sure that we flush the store to disk and because I'm going to use this in subsequent labs throughout the presentation so we have cube constatus it's awesome I'm going to exit there and I will go back to my PowerPoint alright, in this next section we're going to talk about CSI snapshots and using PVC data sources and this is kind of where it starts to get interesting because we kind of passed all the basic stuff now so the way CSI snapshots work in Kubernetes is that it works very similar to how persistent storage works so you have something called a volume snapshot class which allows users to create something called a volume snapshot and you will have the CSI snapshotter will create something called a volume snapshot content that will basically point to the physical resource that references the external snapshot so we'll start the next hands on lab by creating the first of all I forgot to mention this so the CSI snapshot sidecar is not installed by default by the CSI driver the CSI snapshotter is provided by the Kubernetes distribution so you need to check with your Kubernetes distribution vendor if the CSI snapshotter sidecar is installed or not in this exercise since I'm leveraging vanilla upstream Kubernetes I'm not going to deploy the external snapshotter as part of the hands on lab but once we cannot pass this what you can do is once you have all these CRDs installed you can create a volume snapshot class and reference the again the CSI driver that you have installed on your cluster some some CSI drivers provide custom keys to set different values for the parameters that you provide to the volume snapshot class very similar to what you would do in a storage class if you have any specific parameters you want to supply to the driver when you provision a snapshot or provision a volume in the storage class case right and the volume snapshot is also very simple all you have to do is specify a source is just what actual persistent volume claim do you want to take a snapshot of and that then we'll create a point in time copy of that persistent volume claim with the content that's in that volume at that point in time and the volume snapshot content again is that is sort of like the physical representation on how your backend storage system will be able to find that piece of storage and reference that in that particular snapshot so with that I'm going to dive into hands on lab number four what I'm going to start deploying the CSI snapshot I'm going to create a volume snapshot class and I'm going to create some volume snapshots so let's go ahead I'm going to switch over to my terminal one second alright let's start by cloning the Kubernetes CSI external snapshot repository and then we need to create some resources that are provided in that repository so it's the the CRDs that provides the volume snapshot classes volume snapshot contents and volume snapshot that I just talked about and then you actually need to deploy the actual CSI snapshot controller itself and this you kind of need to do once per cluster once that's deployed you can go right ahead and create volume snapshot classes I'm going to create a default volume snapshot class I'm only going to leverage one in this particular exercise I want to point out which driver I'm going to use and I also need to reference the particular secret the side core needs to talk to the back end that we deployed alright we're going to create the volume snapshot class here we go and I prepared a bunch of volume snapshots and this will essentially create new snapshots of the Redis instance that I deployed right so here are the three parts that make up my running Redis instance and I'm going to create a snapshot on each of those persistent volume claims that got created when I deployed that Redis instance there we go I'm going to create those they're usually created quite fast snapshots are nothing usually I have a operation for the back end storage so I can do a get there we can see that they're immediately ready to use the source pvc that we're referencing and the restore size you can see in the column there are a lot of columns here that has actually been truncated and you can check the outputs in the in the asking ema cast files but there are a bunch of other columns there as well so and that was actually how easy it was to create a volume snapshot class and create a bunch of snapshots that references existing pvcs and now we have pointing time copies of our particular Redis instance so I'm going to switch back to my PowerPoint here in this first exercise where we created a volume snapshot class and a volume snapshot what I want to be able to do is to create a new persistent volume claim that references my volume snapshot so this is an example persistent volume claim how you would reference a snapshot when you create the pvc so you have the data source stanza here which calls out the snapshot name you want to use what kind it is it could be a volume snapshot or a persistent volume claim so far there are other things coming here that's not been released yet and then when you're using volume snapshots since volume snapshot is a beta feature you need to call out the api group as well that you want to look for this particular kind and here is a very important detail that I want to just call out here is that when you create persistent volume claims from existing snapshots and such is that the storage request needs to match what the actual snapshot is so you cannot say that you took out a snapshot of a volume and it was 2 terabytes you actually need to create a new pvc with the exact same size as the volume volume snapshot so without further ado I'm going to dive into lab number 5 here with the hands on lab number 5 I'm going to create a new pvc from the volume snapshots that were just created and attach a new redis instance to that so I'm just going to switch over to my terminal so I'm going to use the volume snapshots that were just created and how I'm going to attach those is basically I have a set of pvcs that references those particular snapshots so in my data source stanza here I reference the snapshots I want to use for each of the snapshots so I'm basically going to start up a new redis instance that I created initially in this tutorial and see if I have my key that I inserted in the first exercise so now I'm going to create those pvcs from the snapshots I have a bunch of yaml here that you might have already seen I now created the persistent volume claims and you can see here since the stateful sets have predictive naming I know what those persistent volumes are going to be called so when I bring up my redis instance I know what persistent volume claim is going to ask for so when I name my new instance my new redis that means that it will either dynamically provision a persistent volume claim of that name or use the existing one and since I pre-created these persistent volume claims what essentially happens is that those persistent volume claims will be attached to the redis instance there we go I just want to make sure that the instance come up here before we before we start poking around in it thankfully Helm makes it easy to find the resources that we want to look at so we can see here that my new redis instance is coming up here I do not list the persistent volume claims here because since I created a persistent volume claims and Helm did not create that resource I wouldn't be able to qualify the label with release equals my new redis but I know that the correct persistent volume claims are getting connected right we're up and running so I'm just going to exec into that redis instance that I just created typos or common I'm going to run the redis CLI directly and I should be able to get the kubecon status key there we go awesome so that is basically how you would clone an application leveraging multiple persistent volume claims from multiple snapshots using predictive naming thanks to the stateful set now you would be able to provision as many instances you want of that particular application and since the snapshot pvcs that you just created are not impacting production you would be able to do destructive changes on this particular application add keys, remove keys and connect an external application to it to do some testing and obviously this is a very tiny data set but it wouldn't really matter if this could have been a multi-terabyte database right and that is very popular for CICD use cases where you want to attach production like data into into your testing dev environments so I'm just going to switch back to my powerpoint here hang on the next demo I'm going to do is basically create a new persistent volume claim from an existing persistent volume claim and that means that you won't have an intermediate snapshot so you will create the new pvc directly from the source one and this does not require the external CSI snapshot so as long as the CSI driver supports data source persistent volume claim data source you will be able to use this capability and features so without further ado I'm going to switch over to hands-on lab number 6 and create a new pvc from an existing pvc and attach an application I'm going to switch back to my terminal so I'm just going to show you my pvcs from pvcs and this is very similar as I showed in the powerpoint slide here the data source what I'm going to use again I'm going to use predictive naming and I'm going to use my clone redis as the name the data source I'm going to specify is a persistent volume claim and these are the existing claims that got created initially I'm going to create those pvcs from pvcs and you will see that they will be instantly created as well I forgot to list them here but rest assured they will be provisioned so I'm going to install a third redis instance here and I'm going to call it my clone redis I'm going to watch it listed by label release equals my clone redis hopefully all my cluster nodes have the redis image by now so this should be fairly quickly it's creating there we go all the instances are up and running it's not ready yet there we go ready one one alright so we should now be able to exact into my new clone redis clone redis instance that's right that's the name run this redis cli and I should be able to observe the key that we inserted in the initial deployment and there we go it's awesome and just to illustrate here I'm just going to list the volume snapshots and volume snapshot contents that we had and this only references the original snapshots that we created what this basically means is that there is a snapshot created on the back end storage system but it's a snapshot that Kubernetes is unaware of so it's only the CSI driver that knows how to create a snapshot from that existing pvc and attach that to the pvc that is running in the Kubernetes cluster and resolve that and make sure it gets staged and attached properly alright that concludes lab number 6 I believe it was, number 6 I'm going to show something very similar here so in this particular case here what is fairly popular let's say that you accidentally delete a application but you still have the volume snapshot say that you delete the you do a helm on install and you wipe your pvcs and all of a sudden oh that was not my intention you just want to reinstall the app but you accidentally wipe the pvcs but if you have the volume snapshots you will then be able to create a new instance from those original names that you had when you initially deployed the application and sort of revert to your previous state if you so will so say that you would have that's the other use case for using the restoration procedure so you would essentially how would I describe this say that you're running your instance in production you would accidentally trash your data set but you know that you have a snapshot from 2 hours before what you can do is you can undeploy or uninstall the helm chart in this case and then remove your original pvcs and then create new pvcs from the snapshots that you just created so I'm just going to dive into hands-on lab number 7 and restore an application from a volume snapshot just going to switch over to my terminal we're back at the terminal and first I'm going to accept into my production instance and here is I'm going to delete my status key right so you will see that I will actually so this is kind of an accidental deletion use case right so you will see that I've deleted the key doesn't exist there anymore I know that I have a good snapshot that I want to be able to leverage to restore my redis instance I'm just going to inspect the pvcs that I want to recreate and I'm going to recreate these with the original names right and the original names of my volumes is just my redis no clone not my new or whatever and the data source I'm going to reference is the snapshots that I know I have and we know that those snapshots are good because we already restored to a new application with those particular snapshots we're going to do a helmet uninstall my redis and I'm also going to wipe the persistent volume claims that got created when we deployed that so I'm just going to filter by label and now the claims are gone and before we install the redis instance I just need to create my persistent volume claims make sure that those gets recreated I'm just going to list them here we can see that they're bound, they're good we want to then attach the redis instance I'm going to install that same instance name that we did for the initial production and this should now bring back ourselves to the state we were when we took the snapshots and basically be ready to serve production yet again we want to wait for the instance to come up here the main instance is running we have one of the replica instance instance is running right the second replica is up and we should be able to exec into the redis instance and list our key awesome we've now successfully restored the application to the status it had when we took the snapshot beautiful alright I'm going to switch back to my power point here and the next section we're going to talk about using raw block volumes and with that I just want to say that we kind of concluded the CSI snapshots and cloning and using volume data sources so using raw block volumes is quite nifty so you may have this in the persistent volume claim spec you will also have the same ability to specify a particular storage class but that is essentially everything you need to do to request a block volume from a CSI driver that supports block storage and the way you would reference a raw block volume in the pod specification is slightly different from how you would do if you were a file system because all of a sudden you're dealing with devices you don't have file systems on them so that's why the volume mounts stanza has been replaced by volume devices and instead of a mount path you have something called a device path and that's where you call out the virtual device and it's also ordered so the first device that comes to mind is the XVDA you would still have a volume stanza persistent volume claim and a claim name that you would reference and in this particular example I'm just going to switch over to the hands-on lab number 8 here and create a raw block device and attach a workload to that but this is a very simple example on how this works so bear with me here for a second so first I'm going to show you my block PVC here so I'm basically going to create a 2TB block device with a volume mode set to block and it's also always default set to file system it's going to create that and then I'm going to run the very simple part it's the same specification I had in the powerpoint actually and it's just leveraging a tool called IOping and it calls the claim name that I just created and here we have the stanza that calls out the volume device and the device path and the results of this is essentially that we will this tool IOping will perform an IO on that particular device once per second per default and you will see the results of that so I'm just going to wait for the pod to come up here and then we'll simply look at the logs and see what it looks like alright we're up and running we're going to tail the log here on that particular pod and yeah there we go it's up and running, it started and what you can see what happened here is by default so there's a 4K IO being read from that particular block device and you can see that the block device is a 2TB device that's exactly what we requested and it performs that IO once per second here and you can see the response time here in the request sequence and that is essentially how you would address raw block storage in Kubernetes alright and that concludes lab number 8 and I'm going to switch back to my power point here and you might wonder why would you need raw block storage in Kubernetes and to be quite frank with you there are not a lot of cloud native workloads out there that leverages this paradigm but there is one in particular that leverages raw block volumes and that is the CNCF project called Rook which leverages Ceph in the background and I'm not by any means a Rook or Ceph expert for that matter but I can talk a little bit about what it is so it's an open source cloud native storage solution for Kubernetes it provides both file block and object to Kubernetes and it uses Ceph as I mentioned the key differentiator here is that it allows you to manage distributed storage running on Kubernetes with native controllers on Kubernetes pretty effortless right so you install it as an operator and then you create something called a Ceph cluster CRD and it may and the key aspect here is that you can leverage existing PVCs right or you can create PVCs like volume mode with volume mode block the most common pattern is that you simply leverage a raw block device that are attached to the server so you don't have to use a external provisioner at all you would just use you would just statically map in like an NVMe device or whatever you have attached to a server but if you want to deploy multiple clusters and kind of get rid of the business of managing local hardware using a CSI driver that can provide volume mode block is quite practical actually so the hands-on lab number nine is I'm going to show you how you would deploy Rook with the with volume mode block with your CSI driver so I'm just going to switch over to my terminal here one second most of the labs that I've been running throughout this tutorial I have not sped up and this one I'm going to have to speed up because it pulls down a lot of images for the operator itself it doesn't pull down as much content but once you create your set cluster it took me around 10 minutes for the cluster to actually get created and come up so I am going to speed up this particular demo when we get to that point but first I'm just going to show you how the operator gets installed and then I'm going to show you the CF cluster the CRD that instantiate the cluster so let's see if this operator comes up here soon enough so I have prepared a CF cluster YAML file here and this is a vanilla CF cluster I just copy this from the Rook documentation and here you specify something called a volume claim template and you set the volume mode to block the access mode read write once I omit the storage class and once you kind of create this CF cluster it will dynamically provision storage resources that it needs from the default storage class so it uses actually there's three volumes being created once per replica for the CF application or the Rook application depending on how you see it and then you will have for the actual data pool the CF file system or the OSD as it's called in CF terminology will provision a block PVC and this is the point where I'm going to pause and speed up the presentation and once it comes back we're going to start chatting on what we're seeing so there, it's finally up now so what you can see here what got provisioned dynamically is that you have file system volumes that's been provisioned named Rook, Cephmon, ABC and then you have the data volumes that basically references a block device and that is because the application components require a file system and the actual Ceph file system requires block storage so that's why it's very practical to have a driver that can do both of those things to provide persistent storage to read write once stateful sets which Ceph presents so at this point you basically have a Ceph cluster running on your Kubernetes cluster so you would be able to install the Ceph toolbox and start creating new storage classes to provide data services on top of your Kubernetes but it's still backed by your back-end CSI driver being that storage appliance or it could be your cloud provider's storage block service so now you're more in the pattern of controlling your own destiny because you have your data services running on the Kubernetes cluster itself and that concludes the use case overview for Rook that was lab number 9 I'm not going to switch back to my PowerPoint alright the next section and the last section is actually about ephemeral volumes so something that is very comforting in the world of containers and kind of one of the biggest selling points in my opinion is that when you store a container you're more or less guaranteed that it will store the exact same way you store it on top in cloud in your data center it doesn't really matter and it kind of eliminates the whole problem of it runs on my computer but once we kind of start talking about huge amount of data it becomes very impractical I remember patterns back in the Docker days where we had customers shipping production databases inside the container itself imagine shipping this wasn't a large database it was less than a terabyte I remember that much but still that is a very impractical amount of content to store in a container because you need to pull that instance down from a registry and it's just all that bandwidth it needs to consume store that locally and it becomes a bit of a burden it just becomes worse and worse as the database grows basically and this was for a dev test use case obviously it wasn't for production or anything like that but that became very impractical so that is sort of where ephemeral local volumes and generic ephemeral volumes that I'm going to talk about in the next section is that that allows you to attach a piece of storage to your pod that will look exactly the same each time by default you will obviously just provision an empty volume but you can use a storage vendor's capability to perform a snapshot, a clone of volume or have other pre-populated content in that volume and that means that each time the pod gets restarted that data gets reattached the way it looks the obvious use case here is to use this as a means to store data like an empty file system is that this is the ideal use case to use for scratch disks and the other uptake on this is that imagine that you have a very high performant workload that consume a lot of capacity in the compute phase of that particular workload is that it needs to store a lot of capacity and by default if you would store the data inside the overlay file system of the container is that it will battle for IOPS and capacity of all your other applications on that particular mount point where you have all your containers running so provisioning an external volume as an ephemeral entity you will be able to temporarily use that scratch disk for that compute workload and you will be able to consume storage resources wherever you have the configured so it could be your cloud providers high performance tier for using flash disks so you might be using a cheaper tier for your temporary storage the thing is you cannot slice it however you want but at the end of the day is that you don't want to have your workloads that you know are ephemeral and you know that are IO intensive you don't want to have them in the gen pop you want to segregate them some vendors provides means to set very fine grained quality of service controls as well to throttle those workloads and also ensure the capacity limits are met so looking at this very simple stanza here is that you well familiar with this at this point where you have a volume mount the mount calls out a named volume and here is kind of where the secret source comes in you have the CSI stanza here you call out your driver and these sort of volume attributes that you will supply to your driver you will tell the driver I want an ephemeral volume if the driver supports ephemerality and then you specify the secret where you can find it and which namespace it is in and you specify our size and some vendors support multiple different options here like how you want a provision storage and so forth there is also a slightly different way to reference the secret so if you have your secret to your back in storage system sitting in a namespace with the application that you are actually deploying the pod or the stateful set the helm chart you can just reference it by name so you give the alternative syntax here by node publish secret reference and name it CSI and I already skipped over to my hands-on lab number 10 slide here about how to use ephemeral local volume so I am just going to switch over to my terminal here real quick I have a very simple example here in my inline dot yaml I will have a mount path of an nginx web server which is just a very simple stupid container and very stupid example that illustrates the point I want to make so we have this inline stanza where I am going to provision a volume I am going to wait for the pod to come up I am then going to exec into the container and simply put a file there to prove my point so I am going to go to slash user share nginx html echo, cubecon, rocks into index dot html there we go content is there and then I am going to do the brute thing of simply replacing the pod and this will essentially delete the pod and recreate the pod if we were referencing a persistent volume claim externally the content would persist on that particular mount points but the point I want to make here is we are going to have a new volume provision there so I am going to exec into the pod got my mount points there we go it's empty, there is no index dot html file there and that concludes the first part of the ephemeral volumes use case I am just going to switch back to my PowerPoint and I am going to talk about something that got introduced in Kubernetes 119 and that is called generic ephemeral volumes and this is a little bit simpler to comprehend in my opinion is because what this does essentially it kind of copies the behavior a little bit like you do with a stateful send so you would specify essentially a ephemeral volume claim template that looks very similar to an inline persistent volume claim and it supports labels annotations and all those things that you would expect to work in a persistent volume claim right and the key here is that it will be able to leverage any storage class you don't have to have a CSI driver that supports ephemerality you can just use the existing existing CSI driver to support persistence right so I am just going to switch over to my last lab here and show you real quick how this works I am going to switch over to my terminal so I have a YAML file here my ephemeral YAML and here you can see that I have a ephemeral volume claim template I input some metadata there so I will be able to find the application and you can see here the access mode how much storage I am requesting and the custom label I am going to create that and then I am going to do the exact same exercise I did in the previous example with the inline local volume example I am going to wait for the pod to come up it is running I am going to get the pvc by my app label you can see here that we have a determined name my pod my mount and that maps to the pod name and the volume mount name that you saw in the YAML stanza again I am going to exec into my pod there we see we have a multipath device mounted on user share engine x kubecon rocks into index.html there we go I am going to do the same replace operation that I did on the previous example I am going to delete the pod and it has been replaced let's exec back into it after it has come up I would say it has been a long day how are you hanging in there up and running we will see here that the directory is empty there is no file there the generic ephemeral volume functionality worked as advertised although it is an alpha feature be careful you will also see that the pvc has been replaced from my default storage class as you can see if you compare the IDs of the volume names they are different that concludes hands on lab number 11 I am going to switch back to my powerpoint congratulations you are done thank you so much for participating I am just going to iterate here what we have covered we introduced you to CSI drivers our dynamic provisioning works in Kubernetes how CSI snapshots CSI restore and using pvc cloning in depth and how you access raw block volumes and raw block storage the last few examples here how to use ephemeral volumes both local volumes using the inline stanza and using generic ephemeral volumes using your standard storage classes as I mentioned the source files, the yaml, the powerpoint the askinema cast files are all available in this particular github repository check out the CSI specification and you can also see all the past meetings and the members of the Kubernetes special interest group for storage on the github URL there and check out the CSI documentation for more information about the development of CSI and the different maturity levels of different drivers and features and God knows what so with that said thank you so much for watching if you're watching this live at kubecon please stick around we're going to shut down this video stream and we're going to switch over to the live q&a and until if you're watching this offline later feel free to reach out to me if you have any questions you can find me on twitter and all the social media networks out there the address is in the beginning of the video so thank you so much for watching take care