 All right, welcome everyone. Sounds like this is working. Glad to have you here at Extending Kubernetes 101. I'm Travis Nielsen. I'm a developer for Quantum Corporation working on the Rook project. I've been working with custom resources in Kubernetes for about the past year while working on Rook. So I'll tell you something about my journey along the way. And then we'll walk through some examples here about getting started, all right? So first of all, this is really the first talk after a great keynote this morning by Yen on extensibility, what you can do with it in Kubernetes. So I encourage you to look at these other talks today. If you didn't learn what you need to from this presentation, there's lots of other opportunity today. So a lot of great talks that I hope to go to as well. So agenda, since this is a 101 course, I wanted to start off by, well, why would you want to extend Kubernetes? And then how would it happen? What are the patterns surrounding Kubernetes? And then let's just get right into the code. That's what I do for my day job, I write code, and so that's why I wanna work through my slides pretty quickly so we can get to that demo. At the end, if we have a couple minutes for Q&A, I'll look forward to your questions. First of all, the most important pattern and I believe Kubernetes has gotten right and why it's successful is because resources are declarative. If you want to create something, you say, I'm going to just declare it, there's no code required, you define your manifest in YAML usually, and Kubernetes will go create it for you. For example, you wanna create a namespace, you say, I'm gonna create a namespace, it's kind is this, and its name is this. Simple, you can't get much simpler than that. Any kind of resource you want to create in Kubernetes you can do with a declaration. Another example, a pod, a pod is the most basic resource that allows you to run code in Kubernetes, any application you want. So I say, I've got a pod, here's my container, and here's the image you want to run. In this case, my hello world 1.0 container, and boom, you have a container. Kubernetes goes and starts it for you. So then you start to ask the question, well what happens if I want to run my own type of thing? What do I do? Well, I can create a pod that goes and runs my application, but what if I want to declare my own type of resource? What do I do? Well, Kubernetes team has done a great job of making this very natural and a very integrated thing. So they have this concept called custom resources, CRDs, which you can define and will follow exactly the same pattern as in Kubernetes. Last year as I was working with custom resources, known as third party resources at the time, I felt like I had this light come on like, oh, that's what they're for. I can define my own types and I can go write code that backs it up, but fundamentally I get resources in Kubernetes that are my type that anybody else can use and they're treated the same as any other type or resource in Kubernetes. Yeah, big light that came on for me. So there are lots of, it's a pretty new concept as far as how plugging into Kubernetes. And last year at KubeCon, a couple of examples were announced. The XED operator was one of those. And that's the first example that I found for custom resources, so I just thought I'd start there. What does that look like and why did they create it? So I went to their GitHub page and extracted an example of their CRD. So just some things to point out from this example, you see, okay, they've defined an API version that, whoops, it's just, it's their own. They define, okay, they're from coros.com. They define their version, v1, beta two. They've got their own kind, it's an XED cluster. But when you create this thing now, me as the user, I get to decide, well, what's my XED cluster called? I give it a name. And these other properties in here are very specific to the XED cluster. You want three of their XED servers. And here's what version, and now their XED operator is going to go orchestrate the XED cluster for you. You declare it and then they go make it happen. And as a consumer of an XED cluster, you say, wow, this is great. It's all integrated in Kubernetes in a way that I would expect. Another example, Prometheus follows the same pattern. They have a Prometheus operator that goes and applies these things. I kind of truncated it, ran out of some room here. But again, everything down here in the spec section, the bottom half of the Yaml there, that's all something that the Prometheus team decided, hey, we need these properties. We'll specify them. Our operator will go act on them, but you, the user, don't have to write any code. It's just a simple declaration. On the Rook team, again, same pattern. We define our own type. We have a cluster and our own properties, which I won't get into today so we can get into the sample. But really, that's the point. We have properties, you define them, you declare them, and then the code behind it worries all about it and treats them as Kubernetes resources for you. So as an example, where we were with Rook as we are working on this distributed data platform, it's a hard problem to solve. And we realized, well, we need deployment in a way that Kubernetes didn't quite satisfy. We need monitoring. We need to monitor our cluster to make sure your data is safe. We need failover that needs to happen a certain way. We need upgrade. When you have data that needs to be upgraded, the data path, Kubernetes just doesn't know anything about those details. So this, we were able to answer all these questions by using the custom resource patterns and then operator to back that up. So traditional approach, and actually where I started was, oh, well, how would I do this? Well, I'll just start a pod. It'll run my service. I'll implement it as a REST API. So create my REST API, great. Now I have a service endpoint where whoever needs to configure my service, they can connect to that and they can call my REST endpoints and okay, they can manage it. But then it starts to think, wait a minute, it's not integrated with the Kubernetes API or kubectl. How would they create my resource? Well, they have to go to the REST endpoint and that's just not a native Kubernetes way of doing it. Other things like security come up. Well, what if I want to secure my REST API? Well, now I have to go, I have to set up certificates to get an HTTPS endpoint. Why can't I just use RBAC security and do it the Kubernetes way? So the traditional approach for us, at least, it wasn't a good fit. We wanted to manage Kubernetes resources and this just didn't cut it for us. So instead of the traditional approach, the extension approach, where Kubernetes just makes it natural. So they're designed from the ground up to say, hey, this is part of Kubernetes. They're built, it looks like, feels like, smells like built-in resources. Again, CRDs use a declarative state and what that means is that if I have a YAML file, I can say kubectl create my cluster.yaml and boom, it'll be created. If I want to change the specification, the properties that are in that YAML file, I can, well, I can do it with kubectl edit or however I edit my YAML files. When I'm done, I can delete the thing and I can use the name, the proper name of my object. I can say delete my cluster and it'll be gone. So it's simple, right? But the power in this is that it's, your custom resources can be consistent across all of Kubernetes. If you want to be taken seriously, be consistent. So that means all the tools, whether it's kubectl or Helm, whether it's the ClientGo API, whether it's security and any other endless number of tools in Kubernetes. It's going to be consistent with how that works. So again, we've already talked about this, basically it's declarative and behind each of these resources, there's a controller in Kubernetes. The controller is what makes it happen and Yen this morning talked about this. She had a much, much prettier picture than me. But basically the controller sits there and it observes for what you want to happen or waits for you to declare your manifest files. The controller will see when you create it, it will analyze, oh, what do you wanna change in your cluster and then it will go ahead and act and apply changes in the cluster. Whether you want to add a new resource, whether you wanna update it, whether you wanna remove that resource, that is the controller's job to make that happen. All right, so that's the basic Kubernetes pattern that then we'll go through for custom resources. So I wanna walk through what does it take to write code and get a sample operator and custom resource running. The first thing we're going to do is so define, you need to design your thing. What properties do you want in your custom resource? Second, you think, well, how can clients consume my resource? With the, I live in Goleng land, I want clients to be able to use the Client Go API and use strongly typed API to say get or list or delete my objects all with that. So there in Kubernetes 1.8, actually, the Kubernetes team created this tool that will generate code for your CRDs so that you will have strongly typed resources. So I'll get to that in a minute to show you what that takes and what that looks like so that you, yeah, generated code, it's wonderful. It makes it feel again like Kubernetes and the Kubernetes types inside, the built-in types use these same code generators as I understand. So the third step after you've defined and generated some code, depending on how you want to act or what changes you want to make in your cluster with your resources, you will need to develop your custom controller or also known as an operator. So we, there's three basic steps to implement. You need to first register the CRD with Kubernetes so that it knows that this controller will be watching and applying these changes. You need to implement three simple methods for add, update, and delete, and then you start watching the CRD. Tell Kubernetes, hey, I'm ready to be notified when there's a change here, when you create, update, or delete anything. All right, when you're done writing code, of course, you need to build it. And that means, in my case, build a Go binary. As shown this morning in the keynote, you can also do it in other languages. My demo will all be about the Go lang implementation. But so you build your binary, you create a Docker container with it, and then we'll need to run that. So once you've built it, what comes next at runtime? First of all, since it is a pod that you will need to run a controller, though we will define its manifest, it will have some RBAC rules that say, hey, I need to access Kubernetes resources, depending on what your operator is going to do, if it's going to be creating other namespaces or pods or replicates, it just needs to declare that. So Kubernetes will allow the operator to do that. Roll bindings define the basic manifest. Second, go ahead and run the operator. It's a simple kubectl create command. No new pattern there. And then when you're ready to actually create one of your custom resources, you just say kubectl create your YAML file. All right, so let's jump right into what all this code looks like. We'll exit the slides and get to the fun part. All right, dismiss, all right. Is that big enough? All good. All right, we have our sample manifest for the simplest hello world CRD that I could come up with. So you get, you see all the properties here that you get to define. So this is myproject.io, that's my project name. I can declare what version, this is the very first thing. So we call it alpha. I'm going to call it sample. You can call it anything you want, we'll call it sample. And then the name of this instance I want to create in my sample. And then the interesting part though is when you get down to the spec. So in the spec, we get to define any properties here that we want to consume in our CRD. Hello world. All right, let's change this a little bit to make it a little more interesting. I don't want to say hello world, let's say hello kubicon, we're here. And we'll always still need a world property. So our world is Austin for the day. All right, so we've defined a simple YAML file. But in this spec, you can have any sort of structures, complex structures, lists of things. This is completely flexible as far as what you can do. If you can do it with a YAML file, you should be able to put it in the spec. So now that we have our spec, we need to get into the land of code. So I'm going to switch over to Golang and see how this translates into go code. Oops. All right, so I'm going to switch over. And by the way, all this code that I'm walking through, you can find on our GitHub and I'll have a link to it later for the sample resource. So we go up to types.go. So types.go, now you see the sample spec here. These are all the properties on the go side that I need to read from the YAML file. Okay, so we had hello. Now I changed it from the original sample. So if I want to deserialize that world property that I added, now I have to add world and it's going to be a type string. And if I want to deserialize it, it's called world in the YAML. And if I put the quotes all correctly, there we go. All right, so we have our two properties hello in world. Oops. And that's all I need to get them into the land of go. Now there's some other things surrounding this that will be used that you need to declare. So for the code generation as well. So let's look at that now. So the spec is contained by our sample type. So this sample is the same name as what we saw at the top of our CRD YAML file. So the sample has to define a couple of standard properties, the type meta and object meta. That's what brings us in the name and namespace properties and some other things. And then of course the spec is what contains our hello world properties. Above the sample, these gen client and deep copy attributes are what we need to add to our sample struct so that we can run the code generation tools to generate the strongly typed objects for people to consume from the client go client. All right, so that's pretty much it. Down here there is a sample list. CRDs, another pattern with them is that you can create any number of them so that you can list them. We have a simple list type with some items in them. And again, for code generation, you need to be able to deep copy them. And that's pretty much it. We have our types defined with the code generation attributes. So what's next? For code generation purposes, there's this doc.go which says what our group name is. That might be kind of hard to see in the gray there but it's called, so my project for generating the package name and things. But one line of code, simple. If we didn't have these copyrights at the top, it'd be a lot shorter. Now let's see, in this file we have about 60 lines of code. So a little bit more just to show what it takes to register. So we've defined our CRD. Now what does it take to register our type? Now in this library that we've got, a simple library that lets you define custom resources and register them, it takes all these properties that you saw in the YAML file will be defined here. And so its name is sample. When you have multiple of them, you're listing them, you need the plural name, samples. Our group name, myproject.io, our version, alpha, yep. So those are all the properties that we need for our custom resource. So the operator kit now will register those for us. I don't have time really to dig in and there's just kind of some framework around here, these other few lines of code that you can go refer to on the GitHub later. But basically, we register these two types, sample and sample list. And Kubernetes will, away we go. Now in our sample project here, we have two other Go files. So after those simple ones. So we've got main.go, we'll just start from the top. Main.go is going to create a context first of all. So the sample client set is what was generated by our code generator. Okay, if I look down here, this code for sample client, new for config, this was generated by the code generator. And I'll go generate that code actually in a minute after we see the basic code that we had to write to get this going. So after the context, we go ahead and ask our operator kit to create custom resources. So this takes our sample resource and registers it and tells Kubernetes, hey, I'm listening for this object. And now some basic things to watch for signals in the pod to manage the pod lifecycle. But then the important part is, okay, let's watch for, watch for the custom resource. So this, if I jump into this and you see, I'm gonna watch all namespaces for our new resource. So when I tell Kubernetes I wanna start watching, while we say, I've got some handler functions, I've got an add method, I've got an update, I've got a delete, okay. So here, they'll be below here implemented. I've got a strong REST client, I'm going to, this is the one that was generated and I'm going to watch this resource. So I tell go, okay, I'm ready to watch this, watch my sample. Now to make it as simple hello world application, when I add the resource, I'm gonna call this helper deep copy. And all I'm gonna do, it can't get much simpler than just printing to the console or printing to a log file, okay. So I add the sample with hello equals whatever. So it's just a simple go formatted string. Since I added the world property, I'll go ahead and add world equals percent s and that will print the value of what's in the world. With the go formatting, I also need to say, oh yeah, give me the spec and the world. And the code completion finds that, okay. So add will basically just print the values of what I just added. Now in a real operator, I would want to go make some change in the cluster. I would want to go start some pods, start, create whatever Kubernetes resources I need to create. Now update is just gonna be something similar. It's going to update from some old to some new value and delete, again, we'll just print the message. Very simple basic thing to get us going. All right, so let's go actually get this running. So I'm gonna go to the command line and sorry, before I go there, we need to generate the code. So to generate the code, there's a code generator in kubernetes.io repo where it's going to take the parameter, tell me the package name that I want to generate. So I'm gonna generate my strongly typed client resources in this client package. I'm gonna generate some deep copy methods into this APIs package. And that's where it finds the types that I've defined. And I want it to be a type V1 alpha one. Okay, so let's run this. So code gen will run pretty quickly. Generate the deep copy functions, the client sets, the listers, the informers. And now we have generated code. If I go look at my project again, we've got a few new files highlighted in green here. Let's see if I expand this a little bit. You see this deep copy.go, all generated. And the client set that's generated, if I just glance quickly into some of these files, you'll see we've got a sample.go and it has all these generated this interface and the implementation for them so that I can create, update, delete, get, list, watch, all of the standard kubernetes calls that I need to work with the resources. So we have generated code, a piece of cake, right? The next thing I need to do is take my go code and compile it. So let me go make sure I get the right command here. Now, if I didn't make any typos, this should complete in just a few seconds and I'll have my go binary. Okay, and after we're done building this, I'll just give that a minute. I'll come back here. We need to build the Docker container. So in the sample here, we go look at the Docker file. This is about as simple a Docker file as you can get. I'm just gonna say from scratch, add my new go binary to the local bin path. And then when this Docker container is launched, it's gonna simply launch my sample operator. Let's go see if this is done. Okay, the go build succeeded. Now I need to run the Docker build command, which I'll get over here. All right, Docker build. I have my container now, version 0.1. And I'm going to, now I need to get it into my cluster. I didn't publish it anywhere, so I need to copy it into my Minicube environment. So I take my container, put it in Minicube, which I have running locally. I didn't want to trust the network here at the conference. Now I have my container there. So just to make sure I'm running, okay, my Minicube is running. I have one node and now I'm ready to roll. In my, the first thing I need to do now is run the operator container that I just loaded. If I go back and look at the operator YAML, don't have time to look at this really, but the operator YAML defines my RBAC settings. The resources need to get list, watch, create. Since I'm watching those, I have to request that privilege. Got cluster role bindings, service account. This just makes it all work when I run the operator. But at the end of the day, I'm going to run a container, which says run my sample operator version O.1. So let's go ahead and do that. So if I create a kubectl create sample operator, it created my cluster role, my service account, my cluster role binding, my deployment. Let's run it, it's running. Now if I say get pods, just to make sure the operator started, good. So my pod is running, and what it's doing at this point, it's registered my CRD, and it's watching for me to create one. So if I look at my CRD, I've got this sample resource YAML. So again, my sample resource is going to create my sample with hello kubectl and world Austin. So let's go ahead and create that. kubectl create sample resource, create. Kubernetes says it's created. Now let's see if Kubernetes is what it did with it. So if I say kubectl get samples, so my type is called samples, and I'm going to say, what samples do I have? So it calls the list method. It says, oh, you've got my sample. Let's look inside the YAML to make sure it's got everything I wanted, and typo get samples, my sample, yes. There we go. So inside the YAML, here we've got hello kubectl, world Austin, great. Now we still don't really know if the controller responded to it, and if you remember the controller was simply going to print to the log. So let's look at that log now. So got all logs and I put a label on it so we could find it sample operator. So the log says, yes, added sample, my sample with hello equals kubectl and world equals Austin. All right, it worked. And then you can do other things like edit. Let's see, we'll try one more command here. If I edit samples, my sample, now we can go in, we can say hello kubectl 2017. And my world is only gonna be Austin for a couple more days and then I'm gonna go back home to Seattle. So let's say I wanna change to Seattle, all right? Oops, I missed something. VI is not my editor of choice. Oh no, it says it didn't edit it. Okay, let's try again. Well, you get the idea. I'll just change this one. All right, you get the idea. We'll keep the typo. All right, it saved it. And now if I look at the log again, it would say, oh, I changed from kubectl Austin to kubectl Seattle. And there we go, we have our custom resource. And if I did the same thing, I could go ahead and delete samples. By the way, you can fully qualify it if something else has a name like that, samples.myproject.io, my sample, and it would go ahead and delete and the operator would then print to the log file again. So there we go, we have a running sample operator that we created from scratch with the code generation and everything. And if I had more time, I would have spent more time on the client to see, well, what are those strongly types look like? Well, we would see that they look exactly like other Kubernetes resources. Okay, so we just got a couple minutes left to go back to the slides here. We went through the demo already. So takeaways, really CRDs make Kubernetes extensible. They follow all the same patterns that you'll find with other resources. CRDs really have a low overhead. Most of your time once you get set up is going to be inside your business logic, whatever you need to do when your resources are created. Since I work for the Rook team, if you wanna see a bigger example where CRDs are implemented, Rook defines five CRDs. Actually, we've got cluster, pool, object, file, volume, attachment. All of these are, anyway, fully implemented. You can go look for more examples there. Rook, by the way, it's, so it's file, object, and block storage for Kubernetes built on Ceph. We've submitted to CNCF and we're excited about getting that reviewed there. So thanks for your support there. Take a look at it. And I'll be in the Rook booth today, if you have any questions. Here's the links to the projects, the operator kit at CD, Prometheus, Rook. And there's lots of other examples out there. And it's starting to roll. If you search for operators on GitHub, you'll probably find a number of them. It's a community that's just growing more rapidly all the time. So if you have any questions, here's my contact information. If you want to talk to the experts, they're on the Kubernetes Slack at SIG API machinery, the creators of the custom resources. And again, here's the Rook site. So that's it. Let me know if you have any questions.