 They should at least trim it. I'm just going to trim it. That's it. There you go. Yeah, it's good. This is a satisfying thing to do. It's very careful. It'll lose piles. So that can manually, when you put it on the metal tapp, it's usually that way. I guess you like a multi-tool. Right now. Well, man, all the people that you have have to trim it. You know what I wrote down there? I don't know what to say. I don't know what to say. I don't know what to say. I don't know what to say. Hey guys, so please, calm, take a seat. Okay, so please quiet. So in the next 45 minutes, Jan Szapani is going to present you with a part of the persistent storage and open-shift. So in this talk, you will learn how open-shift brings persistent storage into the world of state-based containers. So Jan, it's your turn. By the way, one note, please don't regret the rate. Yeah. I will check it. So thanks for coming. Welcome to my talk. As Jerdra told, I will be talking about open-shift storage. I am a software engineer working in Red Hat. I work on Kubernetes, actually. I don't work on open-shift, but open-shift is just Kubernetes when it comes to storage. I will show you how to use existing storage inside open-shift. If you have some legacy application and you are now trying to containerize it and run it in open-shift, so it can access the old data. I will show you how you can create and delete storage dynamically in open-shift. And in the end, I will try to show you if time permits how to build a scalable application using persistent storage. That's the goal of this presentation. So before I start, I will very briefly talk about open-shift. Open-shift is a container orchestration framework. It runs containers on a cluster. And it doesn't run containers only. Like single entities. The smallest entity it runs is POT. POT is a collection of containers. Typically it's just one container, but you can have as many containers in the POT as you want. For the purpose of this talk, POT and container, you can take as synonyms. I don't... I won't talk about many containers in the POT. It's the same like one for me from the storage point of view. And as I've said, open-shift is Kubernetes plus additional features. And when it comes to storage, we don't have any additional features for open-shift. Everything I talk about will work in open-shift. It will work in Kubernetes. And I will talk about future open-shift. What you will see in the next release. 1.5 And that means I can show you Kubernetes 1.5 which has been already released. Talking about future open-shift, I will talk about current Kubernetes. 1. Difference between open-shift and Kubernetes is that in open-shift, we take security seriously and, well, more seriously. There is a distinction between administrator of open-shift cluster and user cluster. User of this cluster in this case is somebody who deploys applications on open-shift. Typically, it's a developer or some DevOps guys. This guy, this user in an open-shift sense is administrator of application inside open-shift. So, as I have said, open-shift and Kubernetes they run containers. The containers are fMRO, they are started, they delete it and every change that the container does everything what it writes is lost when the container dies. So, to bring some persistence into these containers we mount stuff into the containers and we can mount many things. We can mount third file system like NFS Blaster, CIF, we can mount full-block devices like iSCSI, FiberChannel, CIF, QuoBite, Flocker and Photon. If you don't know what it is, these are QuoBite and Flocker are new startups in Silicon Valley. Photon is something vSphere invented. And we can mount storage from cloud. If you run your open-shift on Amazon you can use AWS elastic block storage. I think it's called EBS. If you run on Google Cloud Engine, Google Compute Engine you can mount persistent disk you can mount native OpenStack storage, native Azure storage native vSphere storage and so on. We have also some special file systems which I won't cover in this talk because they are not persistent. In Kubernetes or in OpenShift you can define a secret which is just key value map where you are supposed to put credentials the application you are deploying needs to connect somewhere for example to the database so you create a special secret object inside Kubernetes and Kubernetes can mount or create a file system inside the containers which will hold the credentials for you so your applications can find it and can see the username and password and can connect to the database. And last but not least we have a flex volume which is only important thing all these plugins on the slide are hard coded inside OpenShift they are not plugins actually you cannot dynamically load and unload them they are hard coded in Kubernetes and OpenShift so if you have a new storage and you want to talk to it using OpenShift you have to write the plugin and the flex volume is for you if you don't want to write it you just distribute shell scripts which can mount the storage and unmount it and Kubernetes will call these scripts when it needs to mount the storage to your containers or unmount it and this will get extended soon I hope and most of the stuff here will move from OpenShift code and it will move into flex hopefully because this is hard to maintain there are so many plugins to maintain and everybody wants his own storage there and it's very pain to maintain so as I have said for you who don't know OpenShift a container is fmro and if you want to use some storage outside of OpenShift you can directly type the coordinates of the storage if this nfs share you type the server address and the share name into the pot directly into the pot you can if you want I have an example I think, yeah so this is the pot definition in OpenShift everything is defined by YAML files and this pot runs one container it runs a busybox inside the busybox it runs some busy loop that writes something to mnt.demo.txt and it mounts nfs server from this address this share and it mounts it into this directory so if I now cross your fingers because if I oops I have this pot running and it writes I can see that OpenShift mounted this magic share if somewhere inside OpenShift well as I've said I'm not running OpenShift I'm running current Kubernetes to show you future OpenShift so it mounted somewhere into Kubernetes internal tree and if I look on the nfs server I can see it's writing something so this way if the pot dies and it's restarted on a different node it gets the same share mounted and it can continue writing the data to the same file so that's how we ensure persistence in Kubernetes and OpenShift but to for application developers and OpenShift users they need to know where the storage is they need to know the share address the nfs server address the share name and this is not portable if I have this application in my test environment I have one share with data and if I move it to production then I have different share with the data and we don't know we don't want users to know details about the storage we want to hide the storage so in later I will show you how we get rid of this magic IP addresses and share names in definitions and move them somewhere else it can be managed by administrator and the user doesn't need to know anything about the storage at all so why am I using this direct connection to the storage it's quite useful if you know what you want if the user really knows hey, this is the storage I will never use I will never use anything else then this is useful for debugging purposes you don't need to create any other objects you can connect to the storage directly and this method where you put storage details inside the port definition this is useful for the non-persistent storage like the secrets, like the config map and I will show you empty directory which is kind of special storage it's not persistent when the port dies the empty directory dies and what does it do it provides just a storage for your data outside of the container so you don't pollute your container with files because you don't know how big it can be you can put typically some caches there and how does it work this is the same port up to this line it's the same like before you just won't mount NFS share into the port you will mount empty directory and memory that means you will get really fast tempfs based storage inside your port so even unprivileged and privileged port can use tempfs for some fast caches so that's basically the only reason why you should use the storage directly as I have said if you really know what you want for debugging purposes for non-persistent storage and for caches so now it becomes interesting so as I said we don't want the users to know letters about the storage before I start talking about the objects we have for persistent storage I have some analogy in OpenShift everybody use this every time in OpenShift if you start the port the port is basically a request to compute something request for CPU, for some memory and to run some container inside it and in OpenShift we have node object which represents some compute resources outside of Kubernetes and if you submit a port to OpenShift there is a scheduler and it schedules a port on a node and something very similar we have in storage we have request not for CPU and memory but for storage and we have a sort of scheduler that finds a storage for you so let's assume you have some storage outside of Kubernetes, be it NFS shares, iSCSI Amazon volumes, whatever the administrator, the OpenShift administrator creates a persistent volume objects representing the storage outside of Kubernetes so for each volume or NFS share or anything it creates one persistent volume object that represents the storage the user who wants to run the port he creates a claim as a request for storage the user says I want to store 5 gigabytes of data somewhere please OpenShift find me a place and what OpenShift does it has a sort of scheduler it finds a volume for you and finally the port OpenShift looks at the claim the claim points to persistent volume it points to the storage and the storage is mounted into the port and the port is happily running so is this understandable, any questions about this persistent volume represents the storage persistent volume claim represents the user's request for storage and it's bound together this binding between volume and claim is pretty strong nobody else can use the volume while it's bound so nobody else can steal your data basically the volume is bound only to the claim and the volume is available only through the claim and here is how it looks persistent volume it's a pointer to the remote data I mean there is the pointer the NFS IP address and share name it has also some metadata for example capacity which says how much storage it can hold and how it can be accessed for example is to rewrite many means that this volume can be mounted to several ports because this is an NFS this is a shared file system you can mount it several times it's typically mounted only once and it would say read-write once and on Google there is a special kind of storage that can be read-only many times so you can mount it read-only to several machines I will talk a lot later so the persistent volume represents the storage can be used only by the single claim and it's a global object so the OpenShift administrator creates the PVs ahead these PVs can be used by anyone I will talk later about how to restrict some volumes to some namespaces but by default they are global and they are managed by the cluster administrator there are some details each persistent volume has in the metadata it has its own reclaimed policy by default it is retained but it can be also deleted and recycled and the written policy tells what happens to the data when the user says that the data are not needed anymore so the user has a claim has pods running writing some data to the volume through the claim and when the user is done and says I don't need the data anymore then the reclaimed policy is executed and written means that the volume is there forever and it cannot be bound to a different user because there can be credit card details user names password on the volume and no, OpenShift doesn't know so we won't allow anybody to read the data but administrator can get to the data if the user changes his mind says oops, I deleted something accidentally please give me back the data then the retained policy you can actually get the data back with delete or recycle policy you won't get the data back if the user says I don't need the data anymore we either delete the volume completely or we delete all the data on the volume and make the volume available back to any user so I have an example how it works so user has asked for some storage using a claim, PVC and runs a pod a user can run as many pods as he wants with the claim the data is still available when the pod dies he can start the pod again or different pod and still it has access to the data the way how user says that he doesn't need the data is that he deletes the claim when you delete the claim as a user you say you don't need the data and if the reclaim policy is deleted then OpenShift will delete the volume and delete the persistent volume and OpenShift and you can't get the data anymore so please don't delete your claims unless you really know what you are doing and how does the claim look like as I have said it's a request for storage where user says I want to store in this case one megabyte of data somewhere please find this storage you can specify narrow the volumes the set of volumes you want to claim and important about the claim is that it is namespaced in OpenShift we have objects and pods and namespaces so different teams or different users don't clash with each other so if a user creates a claim and gets a volume eventually then the claim can be used only from the namespace you cannot share one claim among namespaces the claim can be used by multiple pods as I showed you and beware deleting claims is dangerous so some demo I have it depends on the persistent volume if it is shared file system you can use it from different pods at the same time if it is block device it depends on the file system typically you don't and if it is access mode read write only once is there something that checks that? the question is if the access mode is read write once there is nothing that checks that typically the second mount fails and the user gets some error message the retention policy is managed by the guy who creates the volume that means administrator but we ensured if you delete the claim you can't get your data anymore so it's either deleted recycled or it's retained that data exists but nobody can access it through OpenShift unless administrator do something so I have a volume again my favorite NFS share I created I have a claim which asks OpenShift for one megabyte of storage and come on I created a pvc and they get bounced together immediately so my claim says it is bound it is bound to this volume I have two megabytes I asked for one megabyte but I got two because the smallest volume we have I can access it from multiple pods and I can run a pod now I have two pods that use my NFS share so what happens and they are happy both have writing to the same file I should delete one I will delete both actually and if I look at the persistent volume details it has a status bound that means that it is bound to my claim and where is the retained policy here the claim policy is retained so what happens if I delete my claim that is bound to this volume it says that the volume is released now nobody can bind to this volume anymore so if I create the claim again the old claim the claim is pending that means there is no volume for you sorry and it will be pending forever until somebody creates some volume that can bind with so nobody can access this release volume so I will delete it and let's continue question the question is if the size of the volume is enforced and it depends on the volume if it is NFS share it is not enforced actually if it is Amazon basically block device restricted to the size of the block of the block device if it is shared for this time it's usually not so it's just the number that nobody enforces so now how can so now the administrator has created a lot of persistent volumes and if you want to what if the user wants a specific volume the persistent volume claim the user can say what kind of storage he wants and he can he can request something that can be shared among pods that means he wants read write many access mode this is for NFS cluster and this kind of shared file system or the user can require something can be read and written only in one pod that's the read to write one in Google there is one special kind of storage you can mount read only to many pods and it's block device actually it's not a shared file system so this way the user who submits a claim can say what he wants I want a shared file system I want the block device basically so each persistent volume lists in its metadata how it can be used for example in NFS share you can use it from single pod and you can use it from multiple pods and in the claimed user says what he wants I want something that can be shared and OpenShift then compares these requirements and the existing volumes and chooses something that fits their requirements and is small enough is as small as possible we are binding the smallest volumes first then you can require a storage class storage class is a quite new concept in OpenShift and storage class puts together volumes the same characteristics like speed or redundancy or price or anything this is defined by the class administrator what the class he wants so this way the administrator says that the persistent volume belongs to storage class fast the names are completely arbitrary administrator can define any classes he wants and to put a volume into a storage class he just says he just puts this magic annotation into the claim into the PV and when the user wants storage of some special class he puts magic annotation into the claim and Kubernetes matches this and finds the best PV with storage class fast in this case then we have something very similar to storage classes which is labels in Kubernetes and OpenShift we use labels for anything every object can have labels it can have as many labels as you want which is different to storage class storage class can be only one in a PV and the user then can request a volume with specific labels a volume that has labels shape square and color green and the user wants some persistent volume with shape square and this will match if the user will want shape, I don't know triangle it won't match so when to use classes and selectors these are kind of the same concept you label PV with labels or you take it with storage class this is the same basically but there is one difference if you don't say anything in the claim if you omit the selector then you say I want a persistent volume with any labels give me something, I don't care what it is on the other hand in the claim with no class requirements then the user says give me the default storage class that's the difference this way you can restrict some volumes to certain namespaces using quota because do I have slides about quota? yeah I have so I will say about quota soon so that's the difference between classes and selectors if you don't specify any selector you get anything if you don't put any class into the pvc you will get the default storage class and how can we use this quota? I have an example I hope where am I? it's 4 so I will create a quota that's just it says that a quota is part of the space so this namespace can now have 500 gigabytes of volumes with default storage class the class named default can have 0 because if I label my volumes I can say that this expensive ssd rate array is gold and I can make it available only to finance department because they are paying gold and default rotating hard drives I will give only to IT because they only dig bitcoins and they don't need storage so the quota and storage class is the way how to do it do you understand at least a bit? I see some notes it's per namespace quota is per namespace so I can create as many claims as I want I can create as many posts as I want but the claims I can't claim more than half a terabyte after that if I do I have a claim I have a claim that requires gold storage class and if I create the claim Kubernetes OpenShift immediately says that sorry that extends your quota so I cannot create just a claim that wants 10 gigabytes of gold and this namespace cannot claim anything in gold at all with labels and selectors you cannot do this if you label your storage like fast and slow there is no way how to restrict these labels oops only 10 minutes left so I'll speed up you cannot restrict it using this what I've talked about is I call it static provisioning somebody from OpenShift administrators creates the volumes ahead creates the persistent volumes inside Kubernetes and user can claim it this way the user doesn't need to know anything about IP addresses of your NFS servers, share names volume IDs anything it just creates a claim I want 5 gigabytes of something give me that but who is in trouble as your administrator he doesn't know how many TVs should I create in the lines how big should they be and for that we invented dynamic provisioning in dynamic provisioning we envision a situation when there is nothing created outside of Kubernetes no persistent volume exists and user creates a claim I want 5 gigabytes of something OpenShift creates the storage by itself creates the volume binds everything together and user can happily run the pod so for that we need a storage class object storage class as I've said it puts together persistent volumes with the same characteristics and it's a new object in OpenShift 1.4 and it contains parameters for dynamic provisioning this is the important enhancement now so the storage class it's again defined by the administrator it has some arbitrary name the admin can define as many storage classes it wants with different names what's important it has the name of the provisioner and parameters for the provisioner so that means in this case it will provision Amazon volume with the best you can get there so it's really fast and in OpenShift 1.4 1.5 we will allow a default storage class to be defined that means if user submits a claim that says just give me 5 gigabytes of something he will get the default one and how he does it the default storage class has this magic notation it will move somewhere here into the parameters but now it's notation in OpenShift 1.4 we have provisioners for Amazon Google OpenStack Cinder in 1.5 we will get more and what's important we have external dynamic provisioners that means that somebody can write a piece of software run it somewhere it can run as a pot inside OpenShift it can run outside of your OpenShift infrastructure if you want and it provisions the storage outside of OpenShift so you don't need to write a plug-in inside OpenShift which is, it's not hard but it's hard to get the code in and pass the reviews and such you can write something by your own and we have NFS Provisioner where is my NFS 05 we have created NFS Provisioner which runs as a pot and it creates NFS share inside the pot that means if somebody kills the pot the data is gone so it's not for production use definitely not but it's good for testing and development so on your local machine you can have something that's dynamically provisioned I see some question was this relation to OpenService broker? I don't know anything about that so probably none I have a storage class which is very simple it defines a storage class named default which is default and the Provisioner is it's a string it's nothing magic in that and this string is provisioned by the pot I'm running I hope it's not running because of stupid network so I need a pod with this now it's here it should work but from some reason oops no network is busted sorry I won't show that ah no time for wires I'm handling user permissions for persistent volumes but in case we run a container under a non-user I want to make sure that that user has permissions to run to the persistent volume the question is how user permission works if I have if the containers run as basically random users and the answer is you can in the definition you can specify supplementary groups and FS group parameters and open shift will then run the pod with these groups with a random UID but the groups will be there so if you make the storage available to the groups then it's kind of way how it should work so it's not bound to user IDs it's bound to groups so in this dynamic provisioning if it worked you would see that I have NFS share but I don't the user doesn't need to know anything about storage if he doesn't want to of course the user can use selected storage classes and whatever but if the user says I want 5GB of data that's fine administrator just configures the storage classes to provision something and everything magically happens so this is the way we envision for a new application when you are building a new application you don't have any legacy data you should use dynamic provisioning if it is possible