 Hi everybody, I'm Flavio Castelli from Sousa Rancher. I'm here today to talk about this crazy idea about extending the Kubernetes control plane using WebAssembly. So for the ones who are not familiar with WebAssembly, WebAssembly is a boundary instruction format. That means you can pick your favorite programming language, write some code, and then compile this code into what is being called a WebAssembly module. So instead of building a Linux binary for the Intel architecture, you are instead targeting the WebAssembly architecture. So WebAssembly is already being supported by many programming languages, and more of them are currently working towards, you know, supporting WebAssembly. It's actually a great time for WebAssembly. There is a lot of momentum. The reason is because WebAssembly is a pretty interesting technology with many advantages. So let's take a closer look at these WebAssembly modules. So the first thing you will notice about them is that they are pretty small. So these are small independent computational units that take really little space. And this is, I think, particularly refreshing, you know, considering, you know, how much space it takes a Linux container, for example. These WebAssembly modules, in addition to being small, they're also portable. That means I can write some code, compile that, let's say, on my Apple Silicon machine, have a WebAssembly module, and then ship this small WebAssembly module into, I don't know, a Linux machine running on the Intel architecture, and then just run it. That's because there is a thin abstraction layer in between the WebAssembly module and the operating system, which is the WebAssembly runtime. WebAssembly runtimes, historically speaking, have been placed inside of the browser, because WebAssembly has been a technology created for web application development. But nowadays, this is no longer true. And as a matter of fact, there are ways to run WebAssembly outside of a browser. And we will come back to that later. So there is one interesting point here. Given that WebAssembly was conceived as a technology for the web, there are some nice side effects. The first one is security. So if you think about that, when you visit a web page with your browser, and this web page as a WebAssembly module inside of it, your web browser is going to download this binary blob from the internet, and then is going to run it on your local machine. Does it sound scary to you? A bit. Luckily, when drafting up the WebAssembly specification, a lot of efforts have been put into the security implication of that. And thanks to all this thinking, there are certain types of attacks that are not doable with WebAssembly, others that are harder to be pulled off. And of course, I mean, we're talking about security. So I'm not saying that WebAssembly is the silver bullet. It has its own issues, vulnerabilities that are being found. But this is, you know, definitely a step forward given that when it was designed, security was a centerpiece of the proposal, and it still is. On top of that, when you're executing WebAssembly on your machine, each WebAssembly module is placed inside of its own sandbox. That means that it cannot interact with other WebAssembly modules running on the same machine. It cannot see them. It cannot exchange data with them. And in the same way, it cannot access the whole system. So it cannot see processes running on the whole system. It cannot access the file system, the network, the devices. You can think about the sandbox as a bit of, you know, when you put the sandbox that is being used by Linux containers from a certain point of view. You can remove some of these isolation features, for example. But there are other things that you can't even yet do with WebAssembly because it's not even supported right now. So this is an interesting thing to keep in consideration. So I think because of all of that, there is this interest of running WebAssembly not just inside of a web browser, but also as a regular standalone application. And as a matter of fact, there are already quite some people that are experimenting and building systems that allow you to run WebAssembly functions and services. So this is a great use case for WebAssembly. Another great use case is to leverage it to build plugin systems. So you start by, you know, it's a regular plugin system. You have a program. You want to add extra feature capabilities to it. And you do it through plugins, external plugins, but instead of, you know, writing some native plugin, one for Linux, one for Windows, one for Mac for all the different architectures, you just write it once, you compile it into WebAssembly, and then you just use this WebAssembly module. So this is exactly what we are going to be talking about today. We're going to expand an existing program. And the program for today is the Kubernetes Control Plane. If you think about that, the Kubernetes Control Plane has already some extension points. You can bring your own scheduler, or what we're going to be more interested about today, you can bring your own admission controller. So Kubernetes already ships with some admission controllers that are built into its own binaries. Think about pod security, limits, quotas, all of that. But they cannot satisfy all the different use cases that Kubernetes operators face on a daily basis. Because of that, there are dynamic admission controllers that work in a really simple way. So the API server is receiving requests coming from users or other application, and then is going to send this request to be evaluated to external webbook servers. These webbook servers are usually running inside a very same Kubernetes cluster. They just receive this information about something that is about to happen. And then they answer back with a response, which could be, yes, accept the creation of this service, or no, reject the creation of this pod because we do not allow privileged pods, or yes, let's create this pod, but let's also add these labels because we have to be compliant with these internal rules of our company. So dynamic admission controller are a really well-established mechanism of Kubernetes. And it is the foundation of many open source projects such as Open Policy Agent, Gatekeeper, Kyvernal, and last but not least, Kubewarden, which is an open source project. I contribute to a project that we submitted for inclusion to CNCF. What differentiates Kubewarden from the other projects I mentioned is the way policies are written and distributed. Kubewarden policies are written with regular programming languages, or if you want, you can write them using Rigo, which is the query language created by Open Policy Agent. The difference is these policies are then compiled into WebAssembly, and they are distributed using container registries. And then our dynamic admission controller provides a webbook server, which is capable of pulling down these policies and using these policies to evaluate incoming requests. So when I created the Kubewarden project, I also thought about, this is just the first step. The next step would be to eliminate this discrepancy, this differences between dynamic admission controllers and the ones bundled into the Kubernetes API server. So the idea I had was wouldn't be great to be able to extend the API server so that I can write my own custom policies, my own custom rules, but instead of leveraging an external webbook server, why don't I run them straight into the Kubernetes API server? So that was the idea. And you might be wondering, why should I be doing that? I mean, after all, dynamic admission controllers have been out since a long time, they work great. True, true. But there are a couple of things we could gain by going this direction, by further thinking about this idea. So the first thing is being more predictable. So right now you have API server, which receives a request, and then forwards this information to a webbook server, and then waits for an answer, which could be accept, reject, mutate, or it could be nothing because there was a timeout because of a network leech, or because I don't know, the webbook server is overloaded, or it could be an HTTP status code because there is something wrong going on with the webbook server. So what to do next? This is where the failure policy setting that you can see on this slide, this is what determines the next action for the API server. So you can pretend nothing bad happened. So you just ignore the failures, and you just let this request go through, accept it. And maybe you got something, you know, okay, or maybe you got a workload inside of your cluster that is dangerous. And by the time you notice that it might be too late. On the other end, you can be more strict, and you can say, I'm going to fail every time. And this is what the fail setting over there would be doing. But in setting failure policy in this way, you risk off blocking everything, because if a webbook server is down or unresponsive, whatever, then a lot of requests will fail, even the ones that are supposed to be accepted. But there is no clear answer, basically, you know, there is no clear path towards, you know, ignore or fail. It is something that you have to think about carefully. But what if you don't, you wouldn't have to worry about that. And this is actually the case when you run all these rules or this policy straight into the API server, because you suddenly get rid of all kind of a bunch of failure scenarios. Another interesting aspect is, and this is kind of personal, I don't know how you feel about that. But I have this impression that a lot of CPU cycles and memories inside of Kubernetes clusters are not being used to actually run user workloads, but are instead taken by infrastructure code. So let's think, for example, about the keyboard and stack. We have a webbook server, which is there waiting for requests to be evaluated. There could be a lot of requests if there is a lot of activity going on in the cluster, or there could be nothing. And this is CPU and memory being kind of wasted, just waiting for something to happen. And the same happens with the controller, which is there reconciling custom resources. And if there is a lot of changes in terms of policies being added, removed, updated, then a lot of activity is going on. But otherwise, it's again something which sits idle. And I feel like we came to accept this kind of situation. But there are scenarios where it feels better, it feels good to not have all this overload. So what if we could get rid of the webbook server and the controller, because we have a way to run policies inside of the API server, and basically we have kind of a function as a service platform inside of the API server just to evaluate all these policies. I think if we go down that road, we would have more resources for our end users. And that would be great inside of constrained environments, like edge clusters, where the hardware cannot just scale up and down in an easy way, because you have to actually send someone to add new boxes into these faraway locations. So last but not least, there is one final reason why we should maybe keep toying with this idea, which is it sounds fun, doesn't it? At least it sounded fun to me. And that's why I went ahead and I did a POC of it. So big disclaimer, I know this topic is going to be a bit controversial. There are going to be people who are going to like this idea. Other people are going to be against that, and it's totally fine. So there are good chances that this idea will never be carried forward, because everybody is against that. And so all this code is going to be dropped. I don't want to maintain a fork of Kubernetes just to have that. And so what does it mean? It means that this POC I've been working on, it had to be done in a short amount of time. And to do that, I try to reuse as much as possible what I know about Kubernetes, what I know about this problem space, and what I have already at my hand. That means I reused Kubewarden technology, pieces of a stack, like all the policies we already have with our WebAssembly. So I just want to run them. And if we want to move forward with that, that doesn't mean we have to stick with the Kubewarden stack, not at all. All right. Having said that, these proof-of-compset I created is basically introducing a new feature gate of Kubernetes, which has an understanding of WebAssembly policies and the kind of contract that Kubewarden policies offer. So without further ado, let's see this proof-of-compset in action. So here I have a K3S node, which is running the vanilla stack of K3S. There is no Kubewarden involved over there. This is, however, a K3S binary, which has been patched with the code of this POC. So the feature gate I was talking about is turned on and this feature gate has a configuration file. So let's take a look at that. So here you can see that we have one policy called privileged. The policy is provided by this WebAssembly module, which is downloaded from a container registry. This is a vanilla Kubewarden policy, the very same one you would use for the Kubewarden running in as a dynamic admission controller. This policy is interested about pods, is interested about creation, update operation against pods, and this policy is enforced only inside of a default namespace. Next to it, we have another policy called Trusted Dripos, again, taken from a container registry. Also, this policy is interested about operation related with pods, create, update, and this policy is enforced inside of the default namespace as well. This policy can be tuned by operators, so it has some settings, and this policy is going to prevent the creation of pods that have at least one container image coming from the Docker app. I think this is a pretty common use case as a policy that I see many organizations. So let's try to create a privileged pod. Let's try to create it inside of a default namespace, and as you can see, it fails. It is rejected, and we get a message coming back from the API server with the reason why this operation has been rejected. This is the same user experience you get when you're dealing with a dynamic admission controller. Now, let's try to create a pod which is using a container alpine coming from the Docker app. Again, we try to create that inside of the default namespace, and again, as expected, we get rejection because the policy is preventing us from doing this operation. Now, to show you that policies are enforced only inside of the default namespace, let's try to create a privileged pod again, but this time inside of the test namespace. As you can see, this time, everything worked as expected because the policies are enforced only inside of the default namespace. Now, this POC code, it does one extra thing. So every time a request is rejected because there is a policy which is against the operation, the code is going to emit a Kubernetes event of a certain type. Now, instead of writing Qubectl get events whatsoever, I just created a Qubectl plugin that does that. Now, as you can see, I have a fancy table which tells me all the recent events about policies that have been blocking some operation from happening. Now, there is one interesting thing over there. This Qubwarden plugin of Qubectl that I wrote is not written using Go or any regular programming language like you would normally do, but is instead delivered as a WebAssembly module. So this is another POC I don't want to talk about today, but we can talk about that in the Q&A if you want to. Basically, I think WebAssembly can be used also instead of other places of the Kubernetes core ecosystem, like, for example, writing plugins for Qubectl. I'm going to give you a link to this POC at the end of the talk. So now, let's talk a bit about the challenges that I faced while working on this POC. So the biggest challenge has been actually to be able to evaluate WebAssembly inside of the API server. That's because all the major WebAssembly runtimes are not written in Go. They are written in Rust. They are written in C. They do have bindings for Go, and this is what I've been using as a first iteration. But as soon as I started to use them, I then realized that the final binary, so the API server, would not be a statically linked Go binary, but instead it would be a dynamically linked one. And that was kind of an issue because this is not how Kubernetes binaries are being built. So in the first iteration, I made some patches to the Kubernetes build system in addition to the actual code of the feature gate. And that got me, you know, I got to the point where everything was working. I had a container image, which contained the API server binary plus this shared library that provided the WebAssembly runtime. And that was working. I got it up and running inside of Minikube. It was it was all great, but I was not, you know, I wasn't super happy about having this dynamically linked binary and have all these patches to the build system. Next thing I tried, given that, as I said before, I think this is going to be great for, you know, edge environment. Edge environment triggers to me K3S. I immediately think about that. And so I went ahead and I backported the patches I did for Kubernetes. I backported them into K3S. And then I realized that, you know, this shared library thing was a big problem for K3S because K3S is not delivered as a container image. It's a single statically linked Go binary. And so I had to find a way to get rid of this shared library. And so after some experiments, I was able to take the Go bindings of Wozum time, the WebAssembly runtime from the byte code alliance and rebuild them in a custom way with muzzle C and then have one Go binary able to do that. I wasn't super happy to be honest about, you know, having this custom Wozum time Go build, but it worked. But then I stumbled over Woz0. Woz0 is a project from the Istio folks. They basically are in the same, you know, kind of problem that I had because they are using WebAssembly to extend Istio to write rules for Istio, but Istio is written in Go. And so I guess they had the same problems I described before. They went pretty ahead into this problem and they came up with a WebAssembly runtime, which is written in pure Go. This is a fantastic project because you just include it. You don't need to do any patching to the Kubernetes build system or, you know, maintain custom builds of Wozum time Go. You just have it working. You include it like any other kind of Go library. So thanks a lot folks. This is an amazing project. There is also another positive side effect of using Woz0, which is the PC now is no longer using Sego. And for the ones who are not familiar with that, every time you use Sego, you're incurring some performance penalties whenever, you know, you invoke this external shared library code. So with Woz0, you don't even have that. So I'm super happy about that. There are some things that are missing from the PC because as I said before, I wanted to prove that this is an idea that makes sense, that could be used, that could be accomplished, but they didn't want to, you know, invest too much time on it because maybe we're just going to be dropping that because nobody agrees that this is a good idea. So because of that, the PC is not able to do mutating policies. It would be trivial to add them. Also, there are some features that are offered to the policies by CubeBorden, like access information from the Kubernetes cluster to do context aware policies, or, you know, add information from six store. So you can, you know, write an admission controller, which prevents image not signed by trusted party to be used inside of the cluster and many other things. But all of that is done on the host side. And CubeBorden, the policy evaluator of CubeBorden is written in Rust. So I couldn't just copy and paste the code of CubeBorden into this POC. This code has to be rewritten in Go. It's not rocket science. It's just a matter of time. And I didn't have time to do that, but it's doable if you want to. Also, there is another important aspect. So right now the API server is going to download the policies, which are WebAssembly modules from container registries, and it's just going to execute them. But this is a bit dangerous, of course. So the proper solution would be to do something like CubeBorden does, which is verify what is being pulled down from an OCR registry by using six store so that you end up running only the trusted policies that you trust. But I didn't have time to do that. It's good. It would be easy as well. Last point, performances. So I didn't spend time doing performance tests against this solution. There is also the big question mark about Y0, which is not optimized for performances. It's written on their project page as well. So this is something that needs to be investigated more. What should we do next? Well, for me, the purpose of this talk and this PUC is, first of all, to kickstart a conversation around the usage of WebAssembly inside of core pieces of Kubernetes, like the API server over there. What do you think about that? Is it a good idea? Is it a bad idea? Let's have a conversation about that. Also, I think there are other places inside of the world Kubernetes ecosystem where WebAssembly could be pretty useful. Think about the kubectl plugin I've shown to you before. This is something that could be done inside of other projects as well. I hope I sparked your interest in doing something like that. And we can also, of course, talk about that. There are other things that I think could be extended with WebAssembly. So having said that, this is all for today. These are a bunch of links that I'm going to share with you. So this is the source code of the PUC I've shown to you. There is also a ready-to-consume K3S build that you can download and run on your machine. There is also a link to CrewWasm, which is this project that I started that allows you to create kubectl plugins with WebAssembly. And so this is it. Thanks a lot for joining, for attending, and have a great day.