 So welcome. Good morning. So yeah, it's good that this started. So welcome to KubeCon. And today we are going to talk about when Kubernetes is your platform and some design patterns for extensible controllers. So my name is Rafael. I work for SUSE. I've been working with Kubernetes for five years and a half already. And yeah, I have contributed a little to Kube Admin, for example. And with me is Fabrizio. Hello, everyone. I am Fabrizio Pandini. I'm a staff engineer. I have one where I'm tech leading C-class lifecycle. And I also work in the C-class API project. So before starting today, let's take a look at the problem that we are trying to solve. So today, more and more people are developing Kubernetes extension. Developing a controller is becoming a mainstream practice to solve a problem in a cloud-native way. You can use a tool like KubeBuilder, operator SDK, to create a Kubernetes controller in few minutes. And the idea is that the controller can watch a customer resource and basically reconcile the state of your system to the desired state that you define in the spec. This is great. We all know this pattern. But what happened when your business problems became complex? Basically, you get to align that every developer must cross, where your controller alone is not enough to solve the problem. So today's presentation is about sharing some pattern, some lesson learned about how to cross this line. Before starting, I will just point out that what we are showing today is based on experience. It's based on experience of Rafael working on a project like KubeBuilder and is based on experience that we learned in cluster API. And there are also some idea on how we can improve this in the future. Yeah, so not that the problem statement is clear. It's fine to start introducing the answer to the question or to the problem. And so when we are designing extensible controllers, there are mainly two ways to do that. And one way is to extend the capabilities of a single controller. But that quickly gets out of hand because it's hard to maintain. It's hard to keep track of everything. And the other, it's in case you need more than, you can have more than one controller in order to reconcile all these resources. But then you need to keep track of everything. And you need to orchestrate them somehow. So in the next slides, what we are going to do is to check this last part for now. So before starting, let's clarify some terms because there are some loaded terms in this thing. And so, for example, the controller, what we refer as a controller in this talk in general, is just a single reconciled loop of a resource. So what we are doing is just going through a resource and reconciling the expected state with the real estate of the resource. Then a manager or an operator is a set of controllers. So you can think of it as the QController manager, for example. A custom resource is an instance of a custom resource definition. And then we have the owners in concept, which is a controller owns a resource. And is this controller responsible for driving the observed state of a resource to the desired state of this resource? And then the act of watching is being interested in something happening in the cluster. So we need to act because of that in order to drive our resource to the desired state of what we observed in the world. So finally, it's time to start talking about design pattern. And for warming up, let's start to focus on a first family of design pattern. And what we are focusing on is how to orchestrate many controllers. It could seem counter-intuitive, starting from many controllers. But in reality, we have this kind of pattern in Kubernetes since the beginning. So we start from this set of pattern because we hope that they are familiar to the audience. And this will basically help to build up for the rest of the presentation. More specifically, we are going to look at three pattern today. A controller watch, two controllers watching the same object, or many controllers watching a set of related object. And the last one is API contracts. So let's start talking about the same object. So the idea is pretty simple. We have one controller that is responsible, that owns a custom resource, and is responsible for driving this resource. In this case, for example, if you think about cluster API, we can say that a machine controller owns the machine custom resource. So just as a reminder, I'm going to repeat that again. The idea is that you have the external world that is real machine somewhere at some cloud, for example. And what I'm going to do here is to drive the way I observe from the world to the desired state of this resource. So this would mean zoning. And so say that we want to extend this controller. And we don't want to introduce more logic into it. So one straightforward way would be that we have yet another controller. In this case, for example, we want to create backups. So we have the volumes backup controller. And this is going to do backups from time to time when we want it to do. But for example, we are going to delete a machine right now. And we want to make a backup right before this machine is really deleted. So what we can do, for example, instead of putting that logic into the machine controller, which would make it harder to maintain, harder to understand, we have just the volumes backup controller that is just doing the backup. And what it's going to do is something is going to add a finalizer to our resource. And this volumes backup controller, for example, is going to watch for this object. And when it finds a delete on time stamp on it, it's going to do all the triggering of the backup. And then when it's done, it will remove the finalizer. And so Kubernetes will garbage collect the resource. So in the end, the machine custom resource will get garbage collected when we have the final backup of the machine before even it's completely gone. So some lessons learned here. Use when you want to make progress after a precondition is met. In this case, for example, the delete on time stamp of the controller is non-null. And I want to do something. I want to do a backup right now. The pros is that the responsibility is very well defined. I'm not adding complexity to the controller that I have. I am adding this on another controller that is doing only this. It's a battle-tested pattern. We know it very well in Kubernetes. And we see this on lots of controllers that are very widely used. And some cons, for example, is that you need to orchestrate somehow. So in this case, you could add annotations or you could have something to sequence your operations if you are working with a single resource. And then there is behavioral dependencies between controllers that are not very well documented. We'll go through that later on. But yeah, we need to document these kind of behavioral dependencies. And this is a kind of this pattern. OK, let's move to a slightly more complex pattern. And also this is existing Kubernetes since the early days. So we are using an example from Kubernetes. You all know that there is a deployment controller that owns the deployment. And when a deployment is reconciled, what happens is that it creates another resource, which is the replica set. So let's stop for a second on what is happening. Is that there is a relation between two customer resources. In the simplest way to document this relation is saying that this is a parent-child relation. And second, there is also like an intimate knowledge between those pieces. So the deployment controller knows exactly what the replica set is, how the spec is done, how the status updates. So those are the principles of this pattern. And the last bit is that we have a second controller that basically owns and is responsible for the replica set. The first controller closed the loop by watching how the replica set is progressing and basically rolling up the state at deployment alone. So this is a pattern that I hope we all know. But generalizing a little bit, when we can reuse this pattern. So the main point is that we can reuse this pattern when there is a parent-child semantic relation between the object that you control. What are the pros is that this is battle tested. It exists in Kubernetes. And it scales well. You can have different layering of complexity, machine deployment, replica set, and pod which are controlled by the kubelet. Also, we all know very well that it could become pretty sophisticated. You can add history of change, rollbacks, deployment strategy. So it is a pretty powerful pattern. What are the cons is that we have two customer resources. We have two controllers. And whenever the progressing of your process gets stuck, it's harder to debug. You don't know exactly to which controller locks to look, et cetera, et cetera. And the same is that given this intimate knowledge of the object, whenever you want to change something, basically you have impact on two controllers. OK. After introducing these two first patterns, which are our history, let's start talking a little bit about something more complex or more fun. So we start talking about a pattern that we call API contract. We are using an example for cluster API. So we have a machine controller that basically owns the machine. And we all know that a machine could be creating on different type of infrastructure. In this example, we are using the sphere infrastructure. But what if we want to create our machine on a different infrastructure, AWS, GKE, whatever? We don't want to rewrite a different machine controller. We want to use the same. And what is the key to make this possible? The key to make this possible is basically to identify a small set of fields that are common to all the machine infrastructure implementation. For making this simple, but it is pretty realistic, we assume that all the machine at the end has to have a field that is called machine infrastructure ready, that basically feedback to the machine controller, the only information that it cares about, my machine is ready. This field basically is our API contract. And by using this API contract, what we can do? We can create a controller that owns the machine implementation. But given that the dependencies between this layer down and the layer up is a well-known field, basically the lower level is swappable. And this is pretty powerful. In cluster API, we are using these extensives. So let's learn using this path. When we can use it, when there is this semantic relation between an object and a set of related objects, the key word in this case is set. And the example was machine and all the machine infrastructure implementation. Second, and for most important part, is that we can use this pattern when it is possible to identify a set of common API fields between all the machine implementation. This makes the pattern generic. In the example, we discussed about status infrastructure ready. The pro, it allows swappable implementation. It is powerful. In cluster API, we have, I don't know, more than 30 implementation of machine infrastructure. So it works well. We use this extensively. What are the cons? Is that, in order to agree to this contract, is something that requires discussion among all the stakeholders of the stakeholder. So it requires a certain degree of communication. And I thank the cluster API community to make this possible. Another downside, also the contract is defined. Most probably, over time, you have to evolve it. And these, again, require coordination, et cetera, et cetera. And the last bit of information is that, given that you cannot basically, when you implement your machine controller, you don't know the concrete implementation of the machine infrastructure. When you implement your controller, you are required to use unstructured client or the generic client from client go for assessing the fields. And this makes writing your code a little bit more complex. Yeah, one question for Richard. So in that case, what is cool is that, since this is the contract, it's so small, it's also easier to maintain, I guess, right? Yeah, if I can give a recommendation, is that one tip to make this pattern sustainable is that when you define your contract, so the common surface of API between all the implementation, you have to keep as small as possible. This makes it easier to manage and evolve over time. OK, so great. So far, so good. So we started from looking at how we can observe one resource with different controllers. Then we saw how we can have strictly custom resources or even hierarchical custom resources manage it. And then we saw how we can have a very small set of fields that we can put as a contract. Now it's time to shift perspective. So instead of working on this space, we are going to see how we can extend a single controller. And so the first one is the simpler one. So when it comes to extending the capabilities of a single controller, the first thing to notice is that this approach is not used extensively in Kubernetes. So we can see that, for example, you can plug your own scheduler, which is slightly different, but you can do that. You also have dynamic admission control, but this is not really a controller. So you could say that you already have some kind of extensibility. But for example, the scheduler example that I put is just replacing the scheduler. It's not like extending it. So despite the idea of extending a controller is not yet used in core Kubernetes extensively, we see that on the ecosystem, this is getting traction more and more. So we are going to talk about two main designs that will allow us to do this. So one of it is in process extensions, and the other is out of process extensions. So the first one is in process extensions. So the simplest way to extend a controller is to just introduce support for plugins. So you could think of it as selling out to a binary that you just call from the controller or using some more sophisticated things, like, for example, a Golan plugin or a Hasicore plugin, for example. For instance, one thing that you should have into account is how you are going to deploy this binary, how you are going to make this binary visible to your controller. So for example, the Valero CLI, what it does, when you perform the installation, you say which backend you want to use. So it will download the binary for you on a sidecar. And so you have other examples, like, for example, core DNS. If you want to extend core DNS with your specific logic, you have to rebuild it. So it's like different approaches to the same problem. But for example, the core DNS one is a simpler approach to the problem. So let's look into the lessons learned about this one. So it's cool when you can execute external binaries, because this is the other problem. This is running in the same name space as our controller. Can we trust this binary? What is it going to do for the pros? This is similar to the CNI and the Valero model. We already know it works. We have already people using that. And also, the error space is narrow in the sense that we don't have the network in the middle, for example. So we just have a binary. We call it, and we wait. And for cons, we need to do some kind of binary, either download or registration ahead of time. So that's something that we have to do. So it's also hard to prevent side effects, for example, produced by the binary. We need to understand what the binary is doing. Is it, can you run it again? It's the kind of problems that we find with maybe binaries that we don't control completely. And of course, you need to have binary compatibility. This usually is not a problem, maybe. But you need the OS and the architecture of your machine to match the controller where the controller is running. And so because of that, there is a variant of that one that we see also this coming more and more over time. I'm not going to put a lot of time into that, but this is just the variant of WebAssembly. Instead of having a binary that needs to match our OS and our architecture, it's a WebAssembly binary. And our controller just creates a runtime for WebAssembly and runs it. And a good thing about this, it's basically the same. It's only, you can see that the pros is going to run everywhere. And it's also more isolated than the previous one, because this is going to run on our WebAssembly runtime. And so this is more controlled. OK. So now the alternative to in-process plugin, of course, is out-of-process plugin. Let's see how this can work. So in the solution that we are presenting, what we are proposing is that your controller needs an additional component. We are calling it extension management. This extension management is something that is responsible to watch for which plugin exists in your system. How we can define which plugin exists in your system? An idea is that to use another, an additional CR. In the example, we are calling webbook configuration customer source that basically instruct or define where your plugin is. So it gives the URL, it gives the CI bundle to connect it, and everything is necessary for your system to talk with the plugin. It acts as a discovery mechanism, basically. When the plugin is discovered, the next step is you also, inside your controller, you also have your controller core logic. So the plugin gets registered in your core logic. So whenever customer source gets reconciled and whenever something has to be done, your core logic, at this point, is capable to call the plugin, which is responsible for some additional behavior that basically it has turned your controller. So taking a little bit step back on this pattern, few things are worth to notice. So the core idea of this pattern is to run plugin in a separate process. That's the main goal. Everything else, the discovery mechanism that we discussed, is plumbing. It's something that helps us to reach the final goal that is to run the plugin. It could be implemented in different flavor. We showed this implementation because it reminds how the dynamic extension of a webbook works. So maybe that some of you are familiar to the web mutation webbook configuration and stuff like that. So it is the same, let me say, approach. Second thing worth to notice is that with respect to the previous pattern, there is a lot of moving parts. This is something that we have to keep in mind. So let's talk about lesson learned. So when you have to use it, when it makes sense basically to take charge of all these components, point number one is that when your plugin basically requires a different security context, I make you an example in cluster API, our controller have a pretty powerful airbag permission. Most specifically, they can access, for instance, the cloud identity of stuff. And we don't want that the plugin to be run with the same identity because it could be dangerous. So by running the plugin into a separate process, into a separate pod, we can assign the plugin a dedicated security context with very limited airbag permission. The second reason for using this pattern is this pattern basically allows you to add or remove plugins at a time, which is a pretty powerful feature to have. So another pro is that your plugin runs a separate process. So if the plugin basically cannot screw up the core controller, the worst thing that it can do is that it does not give an answer. But your controller continues to run. Another nice thing is that you have the option to deploy a plugin like you deploy any application on Kubernetes. At the end, it is a pod with deployment, with a security account, and all this stuff. Let's talk about the cons a little bit. The controller has a dependencies. Now it's a dependencies with an external service. And this is sometimes annoying because you have to take care of authorization, authentication, network policies, network failure policies, back pressure, basically everything that gets into the picture when you call an external service, or when you make a network call. The last point is that there are also some organizational implications in this pattern because you can have basically two different persons which are responsible of what gets into the system. So if you have a person that is responsible to manage the web book configuration, this person is basically able to gate which plugin gets used. And this is a sort of security mechanism that can help in organizations. Yeah, I won't talk too much about this one, but it's the same as the previous one, only that the service, instead of being outside of the network, is inside of our same pod. So it serves the network namespace of the kernel. So you can just reach your service through the local host. So that's the variant of this one. So yeah, this one has also implications because it's, again, going to be in the same namespace, but also you don't have to go to the network, which is you have to measure. OK, we are nearly at the end of this presentation about extensible design pattern. And there is one and maybe the most important question that we still have to answer. What is next in this journey? What is next in an extensible controller? So before trying to answer to the question, I think there are two points that we should recognize. First is that everyone in this room have a common goal to use Kubernetes to solve complex which are more and more complex, more challenging with these cases. The second thing that we have to recognize is that this is a hard problem. It's something that we cannot solve alone. It's something that we have to solve as a community. Yeah, so with this in mind, there is a set of problems to get solved. There is a lot. This list is very long. But for example, the first question is how to document behavioral dependencies. So we know how to document APIs. We have API documentation, but how we document behavioral dependencies. Like this controller is going to be that, and then this one is going to pause until this happens, this kind of thing. Textual, it's not so good, right? So how we do that? How to make it easier to debug when a resource is not progressing? Why is this resource is still on pending? What is going on? We have ways to do that. And sync instrumentation is working on that. It's providing us really good tools to do that. But still, this is an open question. And also how to define contracts for extensibility points, how we are going to do that, how we are going to also document that. So we have a lot of questions. So the next point is, how as a community we can react to this? So do we need some framework? In cluster API, we are developing. We have a proposal. And we are managing some code with something related that we call runtime SDK, which is, at the end, it is a small framework that is designed to help you to manage out-of-process extension. Can we generalize this as a community? Can we make it reusable across projects so everyone don't have to reinvent the wheel? Same applies to decision-like, binary versus web assembly. If as a community we can agree on which one of the two we prefer, basically we can start working together to a set of library that will make the life of the next person implementing a controller much easier. And finally, we can have also the same approach to problem-like, how do we package our plugin? How do we distribute them? Do we have to reinvent something, or we can rely on standard like OCI, or something else that exists today? So in conclusion, extensible controller is a very interesting topic. And despite the work that we did in some project, we agree that there is still much to explore in this space. And the only way that we know is by try and even sometime failing. So it is a challenge, but also an opportunity. So let's work together to improve the art of developing extensible controllers. Thank you, everyone. Thank you so much. A great session. We have now five minutes for Q&A. If anyone has any questions, we're gonna repeat them for the virtual audience. So any hands for questions? No, it was such a good, great session. So no one has any questions, but you can obviously probably speak with the speakers after the session and within KubeCon as well. So thank you, everyone. Thank you, enjoy. Thank you.