 Hi, my name is Tim Sarowitz. I'm a trainer and course developer for the Linux Foundation. Today, I'd like to talk to you about getting started with Rook inside of Kubernetes. Rook allows us to easily work with storage. Alright, so I opened up my browser, and we're looking at my home page, LinuxFundation.org, and I go under projects, all projects. When this updates, we'll see that there are quite a few projects that the Linux Foundation hosts. And there it is, CNCF, or the Cloud Native Computing Foundation. You'll also notice Ceph is another project that will be the storage backend when we configure Rook. So I'm going to select the CNCF page, and even inside of this project, there are quite a few other projects. Graduated projects like Kubernetes, which we'll be using to deploy Rook inside of. And if you scroll down, incubating. Incubating means that there might be some hiccups and some stuff that still needs work. Rook is one of those, but hopefully as you start using it and integrating it with Kubernetes, we'll all work together to get it to a graduated state and nice and stable and without hiccups. So what is Rook? As the page says, open source cloud native storage. What does that mean? It is a way for automating the tasks of a storage administrator. So the deployment, provisioning, scaling, migration, disaster recover, monitoring, and resource management at the end. So this is to make our management of storage, in this case we'll use Ceph, as easy as possible in our environment. As it mentions, it uses the power of Kubernetes. We're going to create some custom resources. So those are going to have our new objects and then we're going to add an operator to act with those new objects. So we're actually leveraging Kubernetes ideas and concepts here. Multiple storage providers. We're going to be deploying Ceph, but you have other ones to choose from. And I would not be surprised if more are added as this project grows in popularity. Everybody loves Rook, of course. Why? Well, these features. Simple and reliable, automated resource management. So from a storage admin perspective, there's a lot less hassle for me. I can scale or converge my clusters, distribute and replicate data. Okay, I like that. I minimize loss and provision file block and objects with multiple storage providers. So that gives me even more flexibility and I can change and adjust as necessary. So there's lots of reasons to want to work with Rook. Scroll back up and I'm going to choose the get started button. There's quick start guides for various storage providers. I'm going to select Ceph. There's a too long didn't read section that mentions a series of YAML files that we can work with. Common operator and then cluster test. That creates a nice simple cluster, Rook cluster, which is good for mini cube and other environments. We're going to actually take a look at the cluster.yaml file, which would be something you'd look at for a production environment. Let's get started by going to a terminal. Okay, I leveraged Amazon for this demo just because it's very easy to spin up nodes and add storage and so forth. Let's start off by seeing what kinds of nodes we have to work with. So you'll see there's four nodes in the cluster, a master and three workers, and they all show ready state. We're running version 1.16.1. To see the version and each one of the nodes is in Moon 2 1804. In this case it's a T2 large, two processors, and eight gig of memory. All four nodes are the same so that just for flexibility's sake. So let's begin by getting the software with the git command. Here we see git clone running, github.com. Rook, Rook.get, and it's been pulled down. As the webpage said, there was a Rook sub directory and inside of the git repository. And we see of course readmes and other stuff to take a look at. There's a sub directory called cluster, examples, then kubernetes, and the different backends that we can choose from. Again, we're going to choose sef. We see several yaml files here that we could work with. The webpage mentioned a common and operator, and they mentioned first a cluster test. You'll notice there's actually quite a few different cluster, external, minimal, and then a .yaml file. Let's take a closer look at some of them. The common .yaml file is where we see a namespace created, and then our new objects are being added to the API server. There's quite a few of them actually, and you can read through this at your leisure, but you can see there's quite a few. In fact, let's take a look, see what's being added. I did a grep for every line that had custom in it and showed the following line. You can see sef cluster, file systems, object resource users, and so forth. So lots of different custom resource definitions being added. The next one was the operator .yaml. This is what determines how the operator, which will be working with those objects, behaves. So we see the image it will run is rook, sef, master, various other directories, and then lots of possible settings, security settings, logging, and other resources. So should I allow multiple file systems? What's the logging level? How often to check the status? So notice it's a 60 second check. So when you're working with it, if you're expecting a very quick turnaround, wait at least a minute for that. The mon quorum, 45 seconds, duration of wait before trying to failover, 600 seconds, duration between discovering devices, 60 minutes. So if you add a new node and are expecting it to get a new OSD immediately, you'd have to wait at least 60 minutes. And other parameters that you guys can read through at your leisure, but that we're going to leave as it is. Then the cluster test, let's take a look at cluster test .yaml file. They mention that this is what they suggested as an easy getting to start because the monitor count is one. Now if we lose that one, we're out of luck. Normally we'd want three, so we have quorum. We can see other parameters here, such as the OSDs per device. There's one OSD, and we see that it's going to use Varlib Rook for that location. Of course with Minicube, that's not the persistent locations. You might want to change that to be something else. You can also declare which device to use if you don't want to use a directory. Let's get out of that, and let's take a look at the cluster .yaml file. This is the one that would be more production friendly, and we'll take a look at that. We see the image is being declared, whether or not to allow unsupported images to be used. We don't want to use that in production, so we're going to leave that as false. If you choose to use a DataDurr host path, Varlib Rook again is where it will use. Here we see monitor count of three, which would give us the ability to have quorum, and allow multiple per node. Typically we wouldn't want two on the same node, because if that node fails, we would lose our ability to have quorum. I'm going to change that just here in the demo, because I don't have that big of a cluster to work with. We'll keep going here. We'll see the dashboard. That's a SEF dashboard to view what's going on with the SEF cluster. We're going to leave that running. Monitoring, since we have not yet installed Prometheus, we're going to leave that as a false value. Then we're going to keep going down here, and eventually we'll see a commented section. If you want to determine where things go in a more automated fashion, you can use affinity and anti-affinity for where your various components should be put. OSD notations, and we're looking for an entry called nodes. Should be coming up here soon. Use all nodes. It would typically want to create OSDs as it goes. I'm going to show you how to declare a particular node. I'm going to set that to false. It won't just use all nodes. It will only use the ones that I tell it to use. Scroll down a little bit further, and we can see here's the nodes entry. It's commented out by default, so I'm going to uncomment that. Then, of course, I now have to tell it which nodes I want it to use. We see that it gives us good examples. We see a name and then a device name. What I'm going to do is deselect that part and then deselect that part. I haven't changed anything yet because I have to figure out what it is that I'm going to actually be using. I'm going to get out of this file and save what I did so far. Let's see what resources I have. fdisk sudo fdisk-l I've added, in this case, dev xdb. That's a 100-gig second drive on each one of the nodes that I can work with. Of course, we need to know what those nodes are. So, cube CTL, get nodes. Again, I have three worker nodes. Just to make my life a little easier, I'm going to put that at the bottom of the clustering YAML file. When I go back in the clustering YAML file, I hopefully have that nice and handy to work with. I don't have to go back and forth. I'm going to do this. Since this is all commented out, I'm going to delete quite a bit of it. Now that I have those, I'm going to do a Y3Y. Then paste and paste. Now, I can just swap that out with the different values. In fact, what I'm going to do is leave this commented out for now. We can see what it looks like then when I add it later. Of course, the spacing is wonderful for that. Let's get rid of what I added down here. Just to see what it is that's actually left inside of there, little prep-v for take away all the hash marks in that file. We can see the different possible values and permutations. Again, three monitors, but they can be on the same node. Then I've declared two nodes to be responsible for the OSDs. Let's see if my YAML is correct. I haven't created other ones, haven't I? Cube, CTL, Create. I looked at them, but I never... I'll have to actually create it. Cube, CTL, Create, Operator.YAML. Now, Cube, CTL, Create, Dash, F, Cluster.YAML. That looks a little better. If you recall, there's a new namespace called Rook-Seph. GetPod. We can see that the operator is still starting. The operator is going to query the API server and then be responsible for starting everything else. Now that the operator is running, it's starting all the other stuff, the discovery for the various nodes, and plugins for getting stuff to work continues. It can take a couple of minutes for this to finish, but we'll see how this goes. We see that our monitors are there, A, B, and C, so three monitors. A is terminating. I've noticed that. It will restart in a little bit. Let's take a look and give it another second, and hopefully we'll see some OSDs. Back to having an A and a B and hopefully a C. Now, a manager has been set up, so we're about two and a half minutes in and the manager is running. Now, we see that we have these Ceph OSD prepare. These are going to run on the particular nodes to configure the OSDs for us. They'll be creating the OSDs and making sure they're running. Give that a sec here. They'll actually show as completed, and then we should see the OSDs themselves. We're now, it's been about two and a half minutes, almost three minutes since we started and the operator had a chance to start. We notice our OSD zero is in the init phase. That's our first OSD is up. The monitors A, B, and C are there and our manager is there. The other one should be starting. There it is. Now, we have two OSDs, which is what we configured in the file, and then our monitors. At this point, we should be in pretty good shape. One of the nice things is, we have a YAML file called Toolbox, which you can take a look at. What that does for us is it gives us some of the tools that we need, like the Ceph tools. Before we get to that, we have one final step. We have to create the storage class. You can see that there's a couple of them. Let's take a look at this storage class. The first thing it has is a Ceph block pool. We have to have that with the replicated size being declared. We'll leave it at three. Then the storage class is declared next in this file. The cluster ID to use, a pool name, and then an image format. There might be a couple other parameters set in here, but the last one we're going to take a look at is the reclaim policy. This is what to do with volume when you're done with it. In this case, you'll notice it says delete. If you had it as a retain, it would stay there indefinitely until you managed to get around to deleting it. We're going to leave that as a delete, but I wanted to make you aware of where it was. Let's go ahead and create that. Now we have the storage class created as well. Let's take a look at our pods to see how those are doing. Everybody shows a running state. When we look at the containers, they're all running. Of course, the prepares have been completed. That got these two OSDs up and ready. Let's create that toolbox. It says it's been created. Let's take a look at that. We should see in here somewhere a toolbox coming up. There it is. RookSafTools is running three seconds now. We're going to execute the bash shell inside of it. I forgot to type bash. In this case, it looks like another prompt that's very familiar, but if you take a closer look, it is a little bit different. This is actually a CentOS image instead of the Ubuntu we were using before. This is an OS that is set up for us to look at stuff. So, CepStatus. Here we see it says health-lorn, but that's because we only have eight and they suggest 30 PGs, so placement groups. But we have three monitors and we have quorum. That's great. We have two OSDs up and, of course, we wanted to. We've configured a tool. We have one pool, eight PGs. It's actually in good shape. Let's take a closer look. Cep OSD status. And we see both of the nodes that I tried to configure are indeed actually there. So, that's good news. I'm going to exit out of the toolbox here and back to this cluster. And let's add the third node and see how that goes. So, again, Ctl, get node. And let's see here. So, six, eight, 123. So, this is the one that we need to add to it. Oh, I forgot I put it in here already. That's right. So, again, because of the way I did it, but let's just see did it work. I'm going to go back into the... Actually, let's go look at the pods. I hasn't picked up on it yet, but hopefully we'll see a new OSD show up soon. In the meantime, while we're waiting for that, let's go ahead and start working with it. See what we... I'm going to get out of this directory just to give it a little more space for the prompt. Ctl, get pv. No pvs in the default namespace. No pvcs in the default space. Let's create one. So, I'm going to create a file. I'm just going to call it pvc.yaml, something easy. And I have an example ready to go here. So, it's a persistent volume claim. I'm going to call it first pvc, very original name. Storage class name. So, this goes back to the YAML files we looked at. The storage class is named rooksefblock. Access mode of read write once. We'll take a look at that so you can tell what that really means. It's often misunderstood. And it will be a one gig claim. So, again, we didn't have any pvs or pvcs before. Now, let's see if we have any pvs. Not yet. What about a pvc? We have a pvc and it says pending. Status is pending. Let's see here. Go back to pv. And now it has created the volume. And we look at pvc. Instead of pending, it says bound. So, it took a little bit of time for the request to go to the operator for it to create the volume. And now our pvc has been bound. That's how easy it is. We're starting to use sef already. Now, let's create a simple pod to make use of that new pv. And again, I have some yaml ready to go. It's a pod. I named it web pod. Very original name again. And the volume will be known as first pvc vol. It's just a name I gave it. Then persistent volume claim, claim name. So, that's the pvc name. That was what I made in the previous yaml file I just edited. And it's going to be an nginx web server. And under mnt, inside of this pod, it will have the volume mounted. Okay. So, I created the pod, which should then make use of the new pvc. It says container creating. We'll give it a second. It's running. Let's go ahead and see what's inside of it. Okay. Here it is. And we see mnt is there, out of Rados block device. Give dash h. And it's listed. And we see that we have what appears to be a full gig. That's great. Let's put some IO into it. And then some, so we wrote to it. And then we'll do a find. A little bit more writing to that list. That's fine. And then mnt file list. So, there's IO. So, I wrote stuff. I read stuff. Get out of the pod. And let's go back to the tool. And again, let's take a look at stuff status. Okay. So, we have a cluster. We have three OSDs. The one I added is now in there. And we see that we had some IO actually happening. So, let's take a look another close. S-F-O-S-D status. And we can see there was written data across the various objects across the three different OSDs that I had. So, it's working. I'm working with it. Now, let's take a quick look at what it's like to work with those PVCs and how that read once means. So, I created an new deployment called Ingenix. And I'm going to scale it up. Just because there's a lot of them. And this we can see. I guess I could have scaled it here. But what I'm going to do is I'm going to add down here to the container itself. And I'm going to add some volumes. And I'm just going to copy this out of the same pod file so that I know that it's using the same one. Okay. And it helps if you put the right stuff in. How about that? I didn't copy it properly. So, let's try it this way. There we go. That has to go over. And in the container we need to put a declaration of the volume mount at the same depth as image. I believe we've gotten the space correct. And let's see if I did. Nope, something's off. Let's try that again. So, spec, two spaces in volumes. I guess two spaces in names, persistent claim, and two spaces. So that looks... Oops, nope. Not right. There we go. Now it looks better. See how that works? Okay. Now let's see what did I actually do. Cube CTL get pods. So some are creating. Some are not. Let's take a look at why. So here we see that some continue to run. Some will say container creating. Some will say running. And a 6.9. Cube CTL get RS, or replica sets. We have two of them. One of which has only two ready to the bonus three. Because in rolling updates, some are shutting down. Well, other ones are starting. But it will stop when it hits a certain percentage. Why? Well, if you look at what's running at what isn't, you'll notice, let's find our web pod. So web pod has been running before, and that's on 204. That's where the volume was mounted. If we find anybody else running on 204, in this case we don't. The old ones are running. The ones that weren't using the volume, but the new ones, the 29 second ones, these 5.9s, none of them that are coming up are being on some other pod. So actually it's, sorry, it's 6.8. So there we go, the 6.8 pod. And then this guy's running on 6.8. And thus that's why we see it running in this fashion. And let's see here. We'll do a get pods again. And this might help a little bit here. So again, the node, the ones that are running are and in 38.68, because that's where 38.68 already was. So when it says read write once, that means anybody on that node that has it configured can write to the volume. These other ones that still can container creating are on a different node. And let's take a look at what they report. That's when the container creating. And it says it can't. It's because the volume is already used by the web pod. So anybody not on the same host is not able to start. And that's why it works that way. So that was a quick start to finish understanding of how to get Sef and Rook running in your environment. Now, if you want to learn how to create the Kubernetes cluster, let's do some shameless promoting here. If you go back to Linux Foundation, there's a training link. And so this will bring you to a URL for training.linuxfoundation.org. And if you scroll down, there's a class, Kubernetes fundamentals, LFS 258. If you select that when it comes up, it is a self-paced course, 35 hours of material, hands-on labs and assignments. So you'll actually set up a multi-node cluster using the same tools that I used in this demo. You have to get 12 months of access, more videos, discussion forum, and there's a bundle. So if you want to get certified, you can buy this class and I think there's two attempts inside of that for a even less expensive price. So definitely take a look. Thanks for listening and if you have any other suggestions for what you'd like to hear about, please let us know. Thanks very much.