 So first of all, thanks for being here. This is actually my first talk at a great conference, so I hope I'm not too nervous for this. But today we're going to talk about operators, how to develop them, how they actually work, and then we're just going to touch on how to get one up and running inside of an OpenShift cluster. So the first question we have to ask ourselves is what do operators actually do? So we often see online, buzzword bingo style, yeah, we have an operator for that, or this is solved by an operator, there's an operator for anything, but what does an operator actually do? If you search for it online, you will find the following definitions, that it's a method of packaging, operational knowledge, you can control it via standard Kubernetes tooling, but this really didn't give me an intuition on how they actually work on a technical level. So for this, I've prepared kind of a short overview of how the operator architecture looks like. So in the end, we have a resource, which is your standard Kubernetes resource, like a pod, like a deployment configuration, or OpenShift resources, like an image stream, or a build configuration, and you change those inside of Kubernetes. So as of now, nothing has to do with the operator, you just change the resource inside of Kubernetes. So now the operator comes in, gets triggered by Kubernetes and performs some action. This could be, at this point, this can be any action at all. It could go ahead and delete the resource as soon as we created, it could attach a label based on some condition, or in this case, it could interact with an external system. So for now, the operator is just a black box, which takes a resource, performs some action on it, and then saves it back to the resource. And that's really all an operator is. We have a resource, we perform an action on it, and update it back into the cluster. So now, what do we do with those resources? We already have pods in Kubernetes, we already have deployments, we already have routes, we already have services. What kind of functionality could we actually implement using an operator? So we're not limited to only the Kubernetes resources, we can specify our own resources. This is called custom resources, which is a method introduced in Kubernetes to allow developers to package their own resources into Kubernetes. The whole thing works by a custom resource definitions, which is a schema for your resource, which you submit to your cluster, and after that point, you can do everything with your resource as it would be a default Kubernetes resource. Custom resources can be namespaced, they can be cluster-scoped, and they have a few advantages over just providing a plain old API, which I will get into later. So what does a custom resource actually look like? So a custom resource, in this case, I just defined a custom resource called key, and a single field called value, and now we are basically abusing Kubernetes as a key value store. Custom resources do absolutely nothing on their own, they get saved into Kubernetes like any other resource, but after that, that's it. There's no logic behind a custom resource, it's just a schema to store additional resources into Kubernetes. So after we have added this resource to Kubernetes, we can use our standard Qubectl tooling to get a key value store. So why would we actually want to do this? Why not just provide an API or use a default key value store? Well, you shouldn't actually use Kubernetes as a key value store, but the reason behind custom resources, why they are so awesome is they work with all Kubernetes clients out of the box. So if you have a Kubernetes client, you can use the custom resources as if they were native resources. You get role-based access control out of the box, you can specify role bindings, roles, user mappings, all the nice stuff of Kubernetes. You get API versioning, which is also something which is good to have. And if you already have a backup infrastructure in place to backup your whole cluster, custom resources will get backed up as well. Also, if you're using something like OpenShift or another UI tool, you will have UI integration for your custom resources. So in the latest OpenShift version, OpenShift 4, you also get a nice overview of all your custom resources. You can provision new custom resources from the catalog, but yeah, that's basically the use case for providing custom resources in contrast to just providing a separate server which is offering an API. So to get a feeling what we can achieve with operators, I've listed some example operators. So these are all community operators, most of them, and they just want to showcase what an operator can actually do. So we have the Prometheus operator which manages installation of Prometheus, but also configuration of Prometheus. So I can install a complete Prometheus setup and also control it via custom resources. So I could have a Prometheus server and then apply Prometheus rules. All inside of Kubernetes can manage this alongside with my application manifest, and it's all managed by an operator in the end. There's also the Yega operator, a Postgres operator, and those kind of operators all provide a resource. So we have a resource, we perform an action on it and in this case, the action is provide additional resources in our Kubernetes cluster. For example, if I want to go ahead and provide a new Prometheus instance, I create this resource and the operator will go ahead and hey, this guy requested a Prometheus server, I will go ahead and start a deployment config or I will go ahead and spawn new pods for this existing one. So those are all updating things inside of Kubernetes, but you could also use an operator to perform actions completely unrelated to Kubernetes. For this, there's the AWS S3 operator, which means you can now use Kubernetes to store information about your buckets, you can request new buckets, you can create buckets, you can edit quotas for those buckets. This is all inside of Kubernetes, so you never have to leave your Kubernetes cluster. And now to get a bit more concrete, this is what a pod monitor resource looks like with Prometheus. This is provided by the Prometheus operator and here again, we have a plain old YAML resource specifying a selector, match labels, and afterwards, the operator gets the Texas resource, takes this resource and reconfigures the existing Prometheus server to automatically scrape all pods with this label. So this is something you can do with operators. You could also go ahead and perform more complex in application configurations. So this is from an Elasticsearch operator where we can specify the version of the Elasticsearch cluster we want to provision, we can specify additional Elasticsearch specific configurations inside of Kubernetes. So you have this resource, I don't need to take care of any Elasticsearch specific configuration files, I don't have to modify any config maps, it's just all inside of this resource. Okay, so now we know roughly what an operator looks like and what kind of resources we can create. Now we need to actually develop our operator. So you could theoretically develop an operator in any language you want. You can even go as far and say, I want to write an operator in Bash, write a simple Y loop, checking with Kubernetes tooling, it has a new resource being created and if it has been created, perform some command line steps. But the reasonable way to go is there are existing frameworks to ease getting started with operator development. The biggest one right now is the operator SDK. There is also a Rust implementation of a Kubernetes controller. So if you're into that, you can write your operators in Rust. You could also go a bit more low level, which is where Qt Builder comes in. This is also a component which is included in the operator SDK, but you can take a low level approach and build it from the ground up. So for the rest of my talk, I will just go into examples of how to get started writing an operator using the operator SDK. The reason why I'm only talking about the operator SDK is because it's the easiest to get started. You don't have to learn any low level technical aspects of how to interact with the Kubernetes API and you can start with a simple operator in a simple framework and work your way from there. So this is a nice graph provided by the operator SDK, which lists the capabilities of the operators. So the operator SDK comes with three operator types out of the box. We have a Helm operator, an Ansible operator and a Go operator. And those support different levels of integration with the whole Kubernetes system. So I'll just go through them. You can always go back to this graph. It's a pretty popular graph which you'll find when you're searching for operators anywhere. So let's start with the Helm operator. I've provided some sample code. So if you want to check out our GitHub repository, you can see every code snippet I'm posting here in a full-fledged operator which is theoretically ready to be deployed but not production ready. So the Helm operator uses simple Helm charts to deploy resources. So you can also guess from this that you can't really integrate with something outside of Kubernetes. So if you already have a simple Helm chart, you can take the operator SDK and package it into an operator in no time. What this actually looks like is you have your operator SDK which will bootstrap a photo structure for you. This will look roughly like this. In this example, I'm creating a Redis operator using the operator SDK. You have your build folder which contains a Docker file to package all your operator resources. You have a deploy folder containing custom resource definitions, containing instructions on how to actually run your operator inside of Kubernetes and also what kinds of permission the operator needs. And then the important part is the Helm charts directory and inside of there are just plain old Helm charts. Nothing special about them. If you already have an existing Helm chart, you can just drop it in there and it should work. The way we interact, we now interact with these Helm charts. So we have Helm charts, but we also need to specify some values into this Helm chart. So if we just apply a single Helm chart over and over, we don't gain any benefits. So this is where the watchers file comes in. This looks roughly like this. In here, we actually create the mapping between our custom resources and the Helm charts to be applied. So inside of here, I specify my API version, I specify my group, I specify the kind of resource I want to watch and then which chart to actually execute. So this is the way the operator SDK then goes ahead and says, okay, I'm connecting to the Kubernetes API. Notify me every time Redis resource has changed and I will go ahead and apply this Helm chart. So for the Helm chart component, you just use your standard values.yaml file where you insert your default values for your Helm charts and then you can override this in your custom resource. So as seen here, I have a storage variable which is then directly referenced from my Redis custom resource where I can again specify a storage value. Inside of the Helm chart, this looks like any other Helm chart where you can just use the templating engine to insert this value into your concrete persistent volumes or pods or whatever. So this is the way we interact with our custom resource from Helm. So just to recap the Helm operator, it's best suited for existing Helm charts. If your organization is already working with Helm pretty much for everything, it should be simple to get started writing an operator out of it. It's best suited for very small applications so you can't really perform advanced lifecycle operations like let's say for example, I want to cluster my Redis instance. I would need to perform some custom logic to ensure that everything is set up. I have the right count of sentinels and masternodes. I can't do that with Helm. So that's why it's suited for small applications. And also again, you don't have the capability to perform complex integration with external systems. Since it's all using Helm, I can't go ahead and execute a remote API call. It's just limited to this. So the next operator is the Ansible operator. This is a more advanced operator in that it can perform any action that Ansible can do. So you get the full lifecycle integration. If you already know Ansible, you might already know some modules you can use to further enhance your operator. It runs Ansible playbooks using the Ansible runner, which is an external component. It supports full lifecycle management. This means I can include hooks to say, I want to just scale a resource up then I can use kubectl scale to scale this resource up. I can also detect, perform custom status updates. So in this example, you will find that the elastic search operator actually pre-populates the elastic search resource with the status of the elastic search server. So if my status of the server changes to red, I will get this as a status report from the custom resource. The integration with Kubernetes happens through the standard Ansible KHS module. So if you know this module, you will feel right at home. And the operator SDK additionally provides a module called KHS status, where you can update the status of your custom resources. So again, the folder structure looks roughly the same. We again have a build folder, which contains artifacts and the Docker file and whatever. The deploy folder again containing custom resources. But then we also get a generated molecules folder, which is a testing framework for Ansible. This is all pre-populated, so you can perform end-to-end tests from inside of the operator framework. And instead of a Helm charts directory, we now have a roles directory containing the roles of our custom resources. The watch.yaml file in this case doesn't map between Helm charts and custom resources, but between roles and custom resources. To integrate using the Ansible module, the Ansible operator, we again have to specify our default values. This is again just something specific to Ansible. So this is just your plain old default Ansible. And afterwards, you use it like any other top-level Ansible variable. And you can also access the metadata of your objects using the top-level meta object. So if you want to access labels from Ansible, you can also do this here. And yeah, again, all those values will be replaced by those you specify in the custom resource. To recap the Ansible operator, we have it's best suited for medium complexity. So if you already have an application, if you want to deploy an application which has an Ansible module available, you can reuse this and apply it to the created resource. It allows for integration with other components. So as I said before, it can do anything Ansible can do. And if you already have existing Ansible knowledge in your organization, you will feel right at home, you can use your existing Ansible playbooks inside of the operator SDK. But the limitation is it's hard to go beyond this. If you want to perform some other action, you have to basically write your own Ansible module for it. And at that point it might just be simpler to switch over to the Go operator. Now the Go operator is the last one I want to talk about. And the reason I don't have any sample code for this is that any operator you search for online will most probably be written in Go and follow the same kind of folder structure and basic layout. So the Prometheus operator is written in Go, most of the OpenShift operators are written in Go. If you want to take a look at sample code, just check out any existing operator. Yeah, as the name suggests, the Go operator is written in Go and it can do anything a Go program can do. This means you can include external libraries, you can perform file system operations, you can even go as far and create a separate web server providing a nice UI for your operator. It also has some advanced features like it can automatically create your custom resource definitions. This includes things like open API schema definitions, so at creation time of your resource, you already know, does this resource have this field available? Does it accept my value? You can perform regex matching in there and it allows for simpler external integrations because you can include any third party Go library. Just one thing I want to point out as many people get lost in here, when you bootstrap a Go operator, you will be left off with this function. This is the main entry point for your operator. So if you just ignore any other code and put your custom code in there, you should be ready to go. This is the function which we'll get called every time your resource changes, every time your resource updates, and in here you perform your custom logic. Of course you are free to change any other pre-existing boilerplate code, but this is really where your logic happens. To recap the Go operator, it's for medium to high complexity application. It allows for easy integration with external services and it's kind of the flagship operator as in most operators are written in Go and you can use any as a reference. So now just as I know I'm running short on time, I just want to have a brief overview on how to actually run your operator afterwards. The rough flow looks like this. We have our operator manifest. This is a YAML file containing all instructions on how to run your operator. This includes things like deployment configuration, image streams, information on which custom resources needs to be created, and those get packaged into a cluster catalog. There are many cluster catalogs available. For example, the operator hub.io, which offers community operators and you can also provide your own cluster catalog. So you can inside of your organization manage your own cluster catalog and this will provide all your users with the ability to subscribe to updates for the operator. I've also designed a more, more or less more conceptual overview, but I know I'm short on time so I will not go into detail. Just a quick overview. You have your operator catalog, which provides the manifest, which again results in the user's subscribing to those manifests, which creates an install plan, which in turn creates the resources. This looks like a hell of a lot complexity for nothing in return, but in reality you get the safety of creating separate update channels for your operator. You have versioned updates. You can say I want this namespace to subscribe to the Alpha channel and this namespace to subscribe to the stable channel and thus easily manage updates. So just to close this talk, I just want you to keep three things in mind. Operators perform actions on resources. They are not magic. You can do operators, don't do that much at all. You have resources, you perform an action on it. That's it. That's how OpenShift 4 works. So if you know this, you can now know OpenShift 4. You can define your own resources so feel free to experiment. If you have your custom application available, you can try to write an operator out of it to easily deploy new versions. And yeah, the most important thing. Operators are not magic. Do not be afraid to touch on operators. They could mess up your cluster, but probably won't. So yeah, that's it, yeah. So are there any questions? What do you mean by security? So the question was what about security? No, you can fine-grain the security permissions. You can say, I want to deploy this operator and affect only a specific namespace. So I want this operator to only handle this specific namespace. And you can also tune the permissions on that level. So if your operator only needs to perform actions on a single namespace, you do not need to provide any higher privileges. You only need to provide the privileges for actions your operator actually needs to perform. Yeah, then I will be available at the whole conference. If you see me, just approach me and ask me about operators.