 First of all, thank you for attending this talk. I expected a lot less attendance. So this talk is about writing OpenShift and Kubernetes controllers in Python. And my name is Subin. I am a senior software engineer in the AI COE team. In the last one, two years, I spent a lot of time writing controllers in different languages. And this talk is about sharing some of my experiences writing in Python. So what we're trying to talk about today, what I'm trying to talk about today is about controllers, which are the brains behind Kubernetes and OpenShift. Just to get an idea of what the audience does, anyone use OpenShift here or Kubernetes? How many of you are from, not from Red Hat and use OpenShift or Kubernetes? It's just one. That's good. So I didn't talk for today to understand more about OpenShift and Kubernetes operators. We need to understand what Kubernetes is and learn a little bit about controllers and CRDs and operators, and then talk about how to design a controller from scratch, like what are the considerations you need to take care of? And what are the options we have to write a controller? So we will pick two of those options. One is the Go Client and the Python Client. And then briefly talk about Operator SDK and do a couple of demos to showcase the Python controller. So what is Kubernetes? Many of you might have interacted with Kubernetes. We are the UI or the CLI. You might think that it is something like a master slave architecture. There's a master node and there's a couple of slave nodes. And it does something with containers. But that basic idea may not be enough to start out writing controllers. You need to learn a bit more about the components of Kubernetes to basically understand which component to interact with when you write controllers. So the Kubernetes components are shown in the diagram here. There's a master, which contains its CD, which is like a store, which stores different states of objects in a Kubernetes cluster. And you have API server, which is the main management entity. And then you have different nodes on which these application pods are deployed by the Kubernetes. So again, coming back to the question, what is Kubernetes? We started with the basic idea of this master slave. And then we talked about deploying containers. Kubernetes is a container orchestration engine which is designed to deploy containerized applications on a set of nodes. And so there's nothing new about anything like a cluster manager which does deployment of some objects on nodes. There has been in the last many decades countless such cluster managers. So what's the difference about Kubernetes and the previous cluster managers is that many of them previously implemented. They were monolithic in architecture. What that means is that there's a central brain behind these clusters, which knows the entire state of all the objects, which the cluster manages. And when it tries to achieve any action, it tries to base that action on the entire set of state of the entire objects in the cluster. Whereas Kubernetes is slightly different where it's a decentralized approach. It has multiple brains. And each of these brains are independent of each other. And these brains take care of specific parts of the cluster. And they interact with each other to basically manage this cluster in an autonomous way. So that autonomous is basically the big thing in Kubernetes. And we will learn more about how this autonomous behavior is achieved by these different brains of Kubernetes. Another thing about Kubernetes is that it's a declarative orchestration engine. What you mean by that is the objects which are there in the cluster, they start out the journey with a certain declared desired state. For example, what you see on the slide is a deployment specification, which defines that I need a deployment with three replicas of this particular container. And this is defined by a user. And the Kubernetes cluster basically looks at this declarative specification and starts about the journey of realizing this deployment. And what Kubernetes does is that when it tries to take actions on objects which it manages, the most important thing it always does is that it knows that it needs to take an object from point A to point B. That is a state A to state B. It always starts out by looking what's the current state. It's not looking at the previous states of an object. It takes the current state into consideration and then decides based on the current state what's the action it needs to take. And what it wants to do is that based on the current state, it wants to move one step further towards the final user defined state of this particular object. So this movement from one state to the final state can be a single step or it could be multiple actions. And each action changes the state of the object. And so we start out with the current state and the Kubernetes eventually reaches the final state. And when the final state is reached, there's no longer actions which needs to be taken on the state because the final state is the desired state of the object given in the specification. So once the final state is reached, the Kubernetes cluster has reached a steady state for managing that particular object. And all these state changes for different objects to be stored and they are stored in the HCD node. And these states are stored by the API server, which is the central management entity. The API server is a management entity which directly talks with HCD and manages the state of different objects. It also has endpoints which we can use to create a controller. So when we interact with the API server, we need to be careful about what kind of objects we deal with. In Kubernetes, there's a concept of each object is a type. And that type is called kind. For example, you have pods, images, application controllers, blah, blah, all the others. So all of them form a kind. When you create an instance of any of this kind, it's called a resource. It's called a resource because just like the HTTP resource like the rest resource, this instance of the object has HTTP endpoint where you can do the cloud operations of creating or updating the object. And another thing, the API, each of these resources may or may not have is the sub-resources. The thing which we need to be careful with sub-resources is that they may not be HTTP endpoints. There could be some other protocols. So the first demo I have is just trying the baby steps of how exactly we can explore or understand controllers and how we can start out this journey of understanding what if I am right in the middle of Kubernetes cluster as a person, what would I see? So I have here a demo where I'm creating a pod. And this pod has the OC or the kubectl command line. I hope people are familiar with the OC tool. And so let me just show you this particular grid. So I have here a pod. I basically named it as my controller dev environment. It's just a pod. And that pod has this OC tool. And it's running an infinite loop. I just have a bash infinite loop. But if I go to the terminal, I can actually execute the commands of OC. And I can do stuff like OC, get pods. So what I'm trying to show you is the first baby step where I am in the Kubernetes cluster. And as a program, I am in a pod. And I'm using something, a basic tool like OC, which we use outside the cluster to manage clusters. And we are using that tool inside a pod to do the same operations, like get pods and look at the events. And the API server exposes something called events. I'll just come to that later. We can also explore different options, command line options of OC in that pod by looking at different resources available in the API server. So this slide talks about the different kinds which are available. The kinds are the type of the entities which Kubernetes manages. And on the right hand side, you can see the different kinds. So what I'm trying to do with the baby step next is get more understanding about how Kubernetes work. So so far, we have an understanding that it's a decentralized cluster. There are multiple brains. And one important thing these brains do is that they look at the current state. They're not looking at the historical state and all that. And they also look at the desired state. So we understand it's a decentralized and there's a state change of objects. And the current state is important. So with this kind of basic information, we still haven't really understood controllers or how these brains work. So the brains or the controllers are these control loop. It's basically an application running in a pod, in an infinite loop, which does something. We'll come to what that is later on. So it's a process which constantly runs in a pod. And it's doing something to manage the resources and entities managed by Kubernetes. So this is just a diagram just to show how a controller basically looks. Just a sum in this infinite loop within a controller pod. But we still haven't come to how exactly does it do it. And so the controller, what it does in this infinite loop is to constantly read the state of the resources it manages. And as soon as change of state is received, what it does is it tries to find out what's the next action it needs to take on that state. And it does the change. So the controller does actions. It listens for events. And it does an action. And then the state is updated. And then it goes back into this infinite loop, waiting for the next event change. So if you want to write a program programmatically, it's like this. The algorithm is you get the first user defined state, which is the final desired state based on the specification. And you get the current state. And then you invoke, let's say, make changes. Basically, it figures out what's the action it needs to take based on the current state. So these state changes in Kubernetes object are called events. So every time, let's say, a pod starts or stops, an event is generated because the state of this pod has changed. And these events can be captured by this watch interface, which is exposed by every resource. This watch interface is like a change notification feed. Every time there's a change in the state, you can basically hook on to this feed and get to know what's the change. The most important kind of events we are looking at from this watch interface is listed here, added, modified, and deleted. I will show you a demo soon, what that means. So here's a demo. What this demo is doing is that I have written a simple Python code. So this is a pod in which this Python controller code is running for a long time in an infinite loop. And it lists, basically, the events seen by this pod. All these events are specific to pod. So this is a controller pod, which just looks at pod. And you can see that it lists all these events, so it is captured for the different pods created here. Just to explain, let's pick up the second last line. The second last line says, a modified event for this particular pod. And it shows the source version. The source version is a very important, it's like a number which basically identifies the state change. Like if there are n state changes, there would be n different source versions. And the last line shows a deleted event where this particular pod, my controller dev pod, has been deleted. And you can see that there's a corresponding source version change for that particular event. So this example what it shows is it's a Python. It's a very basic Python controller which is having an infinite loop. And it's listening to different events, specifically for pods. And it's trying to capture the output of all those state changes. So coming back to this question of how does Kubernetes work, we have understood that it's a decentralized. We have multiple brains in it. And all these brains are basically the controllers. And the controllers are infinite loops running in a pod. And these controllers consume events via the watch interface. And this interface is between the API server and the controller. And the API server is part of the control plane. So this slide basically diagrammatically explains how a Kubernetes cluster looks with respect to a controller view. So a Kubernetes cluster has many controllers. And each controller does specific things. For example, the demo I showed just now has a Python controller which just looks at pod. So you might have a pod controller. You have a replication controller. You have image controllers. For every entity type in Kubernetes, there's a separate controller. So we have some basic understanding about how Kubernetes works. And we now understand that controllers are important. And we also understand some basic idea about how controllers and this infinite control loop exist. So now let's dig deeper about how exactly we write a fully functioning controller. So we have also looked at how does if I am a controller code and I'm right in the middle of the Kubernetes cluster, what do I have? What are the tools I have? And to access the cluster and look into the cluster and do stuff in the cluster. But there's a lot more to writing a controller than just looking at the events and using the watch interfaces. Assume that a controller has crashed and it comes up again, how does it know what to do next? So there are thousands and thousands of resources in a Kubernetes cluster. And how does it identify where I am? Because it has completely lost track of what it was doing previously before the crash. So the first thing a controller needs to do after the crash or when it actually comes up is that it needs to get the entire state of the object it manages. So what it does is get to do a list function which you need to get all the object states. And once you get the object state via the events of each of these resources, you need to do some calculation like which is the most recent event of this object. And that is done by looking at the resource version. Another thing we may be tempted to do when we write controllers is that we may use the top level event resource which should not be used. It is mainly used for logging purpose. And it has a kind of cleanup activity where after every one are all the events in the top level event resource are deleted. So if you're writing a controller, don't use the top level event. Just use the individual events available for each resource. And we learned that watch interface is the interface which we use to capture all the events. But there's certain problems with watch calls. They are actually very expensive. And we shouldn't be using too many of the watch calls in a controller. Especially when you're writing Python application, we have to be careful because of the concurrency concerns. Our Python application might crash if we are using too many watch calls. So controller has come up. And the first thing it did was do a list call where it captured all the events. And then it looked at what's the most recent resource version of the pod is. And once the list function completes, the controller has all the state information until that moment, t is equal to 0 moment. But from that moment onwards, it only needs to do a watch call on the most recent resource version, meaning once it identifies the latest pod, all it needs to do from that point onwards is to find what's happening to that pod from that state onwards. So when you're writing a controller, you need to start with the list call but then immediately move to the watch call based on the resource version. And controllers are these infinite control loops running infinitely for long periods of time. So it needs to store a lot of event information. So it's very recommended that you use a data structure or some cache when you're writing in Python. So there are certain options available in Python. One is, of course, you can use a simple hash map. But I would recommend something like a bloom filter because a bloom filter data structure gives you certain benefits. One is you can use it to check if it is something like an indexing purpose. You can check if have I seen this pod before. Because when you capture, when you do the list and you capture all the state information of the pods, they may not come sequentially. They may come haphazardly. And you need to actually write some logic to figure out which is the most recent event. And if you're using a cache, there's one problem is that Python writing stuff in Python, there's always a performance delay. It's not like writing in CC++ or even in Go. So the whole flow of this API is to get the events and store it in a cache. There will be certain delays. And so there will be always certain lag between the events in the cache and what might be there in the HCD. So we need to do something like a re-sync. Re-sync is something like a real list every 30 minutes or 15 minutes, depending on how many events we have. And controllers, especially the ones written in Python, when you do the watch calls, they tend to break after a certain time. So what we need to do is we need to look for those network breakages and immediately restart the watch stream call. Else, your controller would constantly be re-initializing all the time, and it would never get a chance to do the business logic. And one other thing with the Python controllers is that when you receive these events from the watch interface, you need to do some kind of validation also. You need to validate the resources. For example, if you're creating a controller for a custom resource, you need to validate the input fields of this custom resource events. Because we don't have enough tools to do the schema check of the resources. So if there is a spelling mistake or there's an incorrect ASCII key in the event captured, it might lead to some net failures. So also have proper default values if the event state change of the resource does not have values for certain fields. So now we come to writing controllers. So we learned about some of the design considerations we need to take care to keep this controller code running for a long time and not failing and always doing thing what it is supposed to do. So there are some options to write it. One is, of course, the Go Client and the Python Client. The Go Client, you can access the Go Client here. I'm not going to talk much about the Go Client. But the most important thing I want to talk is that Go Clients have a few data structures which the Go Client have some data structures which are very useful because one is the Informer data structure. What it has is that it has a cache implementation and it has a wash interface and event handler and Lister so that when a controller is using the Informers, you can basically use this data structure for writing the controller. It can do the listing. And then it can just get the latest resource version and then watch that resource version. And the Go Clients also implement a re-sync functionality where it tries to update the cache frequently. And another data structure the Go Clients have is Work Queue. So there are situations where, for example, you create a pod and you delete a pod and then you create a pod. Let's say that's a flow. And in between this creation and deletion, assume that your controller fails. And when the controller restarts, it's going to capture a whole bunch of events related to creation and deletion. But since the deletion event and the control failure happened simultaneously, the controller never actually deleted the pods, although the delete event has come. So if you have a Work Queue, what you can do is you can prioritize deleting those pods before scheduling the next creation of those pods. Else what you'll have is in your cluster you'll have multiple pods which were supposed to be deleted but they were never deleted. Unfortunately, in the Python client, we don't have these two data structures. So if you're writing controllers in Python, you would eventually have to implement some of these. So the next thing I want to talk about briefly is an operator. So far we looked at a controller. Operator is pretty much a controller which manages a custom resource. So I previously talked about resources, like the pod entity resources and deployment configs and the application controllers, et cetera. They are the default object entities of Kubernetes. But you can go ahead and create your own custom resource. For example, here's a custom resource definition of a foo resource. It basically does nothing. On the left-hand side, I have mentioned how you define user-defined custom resource. And on the right-hand side is instance specification of the foo custom resource. It basically does nothing. The specification, if you look, all it has is just a version number. So you can basically write operator, which is in Python, which just looks at this foo resources and tries to do something with it. Maybe create a foo resource and then create 10 pods from the specification. I don't know. So when it comes to Python, the first option you have is the OpenShift REST client, which is based on the Kubernetes Python client. This is used, if you look at the topmost right corner. It is very heavily used in many examples. Used by almost 321 projects so far. And I have a few examples, like the example which I showed here is actually using this particular project to capture the events for pods. And what I wanted to show as part of the demo was to show the ad modified events for a pod. And this uses the OpenShift REST client. And the next option is the Python OpenShift Client Python. The difference between this and the previous one is that this is based upon using OC. So if you are happy with using OC in a pod, then this particular client can be helpful. The one drawback, of course, is that this does not implement a watch call. But I have seen a few examples where they have used both the clients. They have used OC using this particular project to do a few stuff. And it uses the other project to do a few other stuff in the controller logic. And I have recently, in the last two or three months, came to know about this particular project. It stands for Kubernetes operator framework from Zalando Company. So this particular project implements, it doesn't implement informers, but it implements certain work queue and event handlers. But it's not, I don't know. Personally, I don't know if it's very popular. Like if you look at the right hand side corner, it is used by about 14 projects compared to the Python client. I work in the AICOE team. And in the AICOE team, I work in the Thought project. And we have developed quite a few operators and controllers. So if you go on to this GitHub Thought Station repo, you can basically search for a couple of Python operators and controllers, which we have written. And if you plan to do something, you can kind of use this as reference. The last example I want to talk about is operator SDK. When you write controllers in Python, this is a lot of work. One is you write to your Python code, and you might have to implement informers or work queue or some event handlers. But you also need to package them in a Docker file. You need to create this container image. And you need to, let's say, take care of security policies all back. And what operator SDK does is that it basically does all this work for you pretty much very easily. So all you need to do is just focus on writing the controller code. But the problem is that this is go-client-based, and it's not based on Python. And what I've heard so far is that the operator SDK folks with the Python client, they're working on an informer in the Python client. So eventually, maybe in some time future, who knows? Maybe we will have a Python operator SDK. So I have one more demo where I have written a simple pod controller which is using the operator SDK. And this particular example, again, does the same thing. It just tries to capture the events of pods. All these examples are available on this GitHub repo. And you can look up and experiment with the examples. So the first folder contains the controller dev environment where you have the OC running in a pod. It has all the templates. If you want to just do an OC create template and do an OC new app, you have that for you to experiment. And so there's a MyGo controller which is a simple controller written using operator SDK. And there's a Python controller which contains the template. And it also has an example where if you want to write an operator with some CRDs, there's a sample CRD which you can experiment with. And the source has a couple of code for writing controllers for CRD or writing controllers for existing entities in Kubernetes. So that's it from my side. In conclusion, I just wanted to say that to write a very good controller, you need to have a good understanding about Kubernetes and how to interact with the API server. And also be a little conscious about Python layer itself that there are certain concurrency and scalability issues with that. And so do not, I would say, don't write very extensive hardcore controllers if you have a very reasonable amount of logic which can be done in Python. I think Python controllers are a good option. But if you need production grade requirements of a controller, you need to write in a Golang as of today. So thank you so much for coming here. I can take some questions. So if I'm writing Go code and I need to go off and do some other work, like the controller has to affect some change, somewhere else in the system, I can fire off a Go routine, which is relatively lightweight. In the Python world, you're battling the GIL. And my correct understanding, the controllers are single threaded, right? It's a single control loop that's going to be running continuously. Yeah, I mean, regardless of, because the controller is. Hi. The controller has a single control loop that's just cycling through continuously, right? So if you need to go off and do a significant amount of work in Python, is it just in case we say, OK, as soon as we get to this state, rewrite and Go? Or is there a way that you typically handle those kind of like long running IO tasks? So if you're using Python 2.7, they're limited options. But if you move to Python 3.6, this is a sync library. You can explore that to do that. And also, Python concurrency, there are problems with it. So there are certain patterns where you can, I think, use process, multiple process, where you have one process, which is the control loop. And you can have other processes, which are basically communicating with each other via, I don't know, locking mechanism or some shared cache or something. But it's tough. If you go to the thought station, one of the controllers, which I wrote, has three or four sets working together. And there's a lock mechanism where the locking mechanism is used to basically stop one on multiplex actions. Hello. We're using AKS. In that case, are we still need to write a controller for Kubernetes? I'm sorry, could you please repeat the question? Microsoft Azure has AKS. It's like an Azure Kubernetes service. It's already probably including maybe controller inside of it. Do you think under a work kind of a situation, we still need to write our own controller? It depends on the business logic. Let's say I'm just thinking top of my mind. If I want to write a controller which tries to connect to Azure service, and let's say spawn off something, I can, earlier I could have written some simple service. Maybe today I will write it as a controller with the CRD. And I can declare a specification. I can say, in the specification, connect to this Azure and spin off so and so thing. So it all depends upon how autonomous you want the system to be. If you're designing a system with autonomous thing in mind and you want to isolate the business logic to a certain, like a brain, you would try to do this. And basically it's helpful because if your other stuff or other business logic is also written in controllers, this kind of scales pretty well. So that's an advantage. I would say write everything in controllers and operators. Thank you. No more questions. Thank you so much for attending my talk. Thank you very much.