 Hi, everybody. Welcome to our talk. Thanks for joining us today. I'm Flavio Castelli from SUSE, and I have with me a colleague of mine. Hello, I'm Rafael. I'm really glad to be here. So we are here today to talk about how to keep your Kubernetes cluster secure by writing policies, tailor-made policies for them. So security is a really big topic for Kubernetes as well. There was a recent survey that provided a really staggering information. So 97 percent of organizations are concerned about the security of our Kubernetes cluster. Digging deeper into this number, there are many areas of concern for them. But if you look at the first one, the one at the top, this is about having policies that are applied in a consistent way across different teams, Kubernetes clusters, and it's not there. But looking into this state report, another important piece is not just applying policies for the sake of keeping things secure, but also to keep the infrastructure compliant. So there are policies that are not just securing the infrastructure but are making sure that compliance is happy as well and is enforced. So this is impossible to be done by hand to have something that scales that is consistent. So the answer to that is to handle policies as code as well, in the same way as we handle infrastructure as code, we can write code that describe policies and Kubernetes has already some powerful mechanism built into it. You have row-based access control that basically defines who can do what inside of the cluster, like let's say the user alpha can create parts inside of a staging namespace, but he can't create them inside of the production namespace and he can't list them only inside of certain namespaces. So our back policies are really important and the key aspect of keeping your cluster secure, but they are just focused on one problem and that's fine. To have a completely secure cluster, you have to pull in different solution, each one of them tackling an aspect like pod security policies which are there to basically prevent user from creating containers that have more privileges, that can offer a wider attack surface. By the way, pod security policies are also deprecated right now. They're going to be dropped soon by Kubernetes ever going to be replaced by new mechanism or by admission controllers, which we're going to talk more about pretty soon. There are networking policies, I just mentioned over them. They are there to prevent network communication between unwanted containers. So you might want to say, I have these database running inside of my cluster, and I want only the pods running the front-end service gas book to be able to talk with it. So this is what the networking policies are about. Then you have admission controllers. Admission controllers are there to basically decide whether something with a request that has already been accepted by our back can be completed or not. So what happens is that the user Rafa, before he wants to create a pod. So our back says, Rafa wants to create a pod inside of the namespace staging, that's fine. I am going to approve that. But then after this is approved, there are one or more admission controllers that might be interested in evaluating this request. So what these controllers do, they look at the JSON object which describe what is about to happen. So this user is starting to create this pod with this specification instead of this namespace, and then using some business logic, they are going to decide whether this is something that can be completed, so it's accepted or something that should not be allowed. So it's rejected. There are also cases where you have admission controllers who have the ability to mutate an incoming request. So they can approve the request, but change that on the fly, so that the end result is something different. To make a concrete example, let's say that you are an operator, you want to enforce quotas on your cluster. To do that, you also want all your users to specify resources whenever they declare a workload. So when a user is not declaring the resources for his workload, a mutation admission controller can fill in some default value so that the user is happy because his request is accepted and the operator is happy as well because the default values for resource consumption are enforced and are put into place. So this is what mutation admission controllers are about. So Kubernetes provides a lot of built-in admission controllers that are actually part of the binaries of a control plane, but of course they can't cover any possible scenarios. So let's say that, for example, you want to prevent users from deploying containers that are coming from the Docker hub. So there's no way that one of the built-in admission controllers of Kubernetes can do that for you. But luckily, Kubernetes has a way to extend its admission controllers. And this is done through the dynamic admission controllers. In this case, you have an external admission controller, an external as in it's not part of the control plane, it's not part of the Kubernetes official binaries. This is a workload that usually is running on top of the same Kubernetes cluster. They are implemented as webbook endpoints. So they are simple web application that receive the JSON blob to examine from the Kubernetes API server and then they provide back an answer to the API server. And then the API server based on what the admission controller responded back is going to accept or reject or accept the mutated version of the original request made by the user. So they're really powerful, but if you want to write a dynamic admission controller from scratch, this becomes pretty boring and repetitive because there are many operations that you have to keep doing over and over again every time you want to create a new policy. And this is basically a big waste of time. So unless you have a real specific use case for which we start from scratch, what everybody is doing is to leverage what are called Kubernetes policy frameworks. So here you can see open policy agent, there is also gatekeeper, which is basically based on that. You have Kaiverno and there is Q Warden, which is what we're going to talk about. So these are basically, you can see them as platform as a service. So you provide some your policy, which is just the business logic of it. And then you deploy on top of this pass that is then going to do the heavy lifting of integrating with Kubernetes and doing many other things for you. So you just focus on writing the policy. Talking about writing policies, who is supposed to do that? So again, from a really recent survey, you can see that there are clear expectation here. So there are still security teams instead of organization that are just doing security work and they are expected to provide some policies for their clusters, but the expectation is that developers and operators, they are going to provide ways to secure their cluster, which includes also the act of writing policies. So the reason why we started this project called Q Warden is because we think there is a better way for writing and maintaining policies. We really want to lower the barrier here to make it easier for these people to cope with the amount of policies they have to write and maintain. Also, we want to provide a different and simpler way to consolidate all these policies and distribute them. So how did we do that? Well, as you might have guessed from the title of this presentation, we leverage WebAssembly. WebAssembly is a key point of this solution. So for the ones who don't know what WebAssembly is, I'm going to do a quick overview. So WebAssembly is a binary instruction format. What it means, it means that you start from a program written using a programming language like C++, Rust, assembly script, which is kind of JavaScript, a Go, and then you run it through a compiler and the compiler, instead of producing a Linux x86 binary, it's going to produce a WebAssembly module. These binary artifacts, you can take it and then you run it on top of a WebAssembly runtime. The WebAssembly was born for the web. So the first WebAssembly runtimes were browsers that were using WebAssembly that are using WebAssembly to extend their capabilities. But WebAssembly is so interesting and powerful that recently there have been a lot of efforts to run WebAssembly to standardize the usage of WebAssembly also outside of the browser. So you have WebAssembly runtimes that are not browsers that you can run on top of any operating system. So the interesting aspect here is that regardless of the runtime, be it the browser or a standalone runtime, WebAssembly modules do not care about what is the underlying operating system or architecture, and this is pretty powerful. We will see that pretty soon. So why is WebAssembly a game changer for policy offer? Well, that allows the usage of WebAssembly allows policy offer to be productive since the beginning. They don't have to learn domain specific languages just for the sake of writing policies. And so by just using a language they are comfortable with they can just be productive since day one. And this is not just about writing policies. Policies as code means that you must have someone who is also going to review these policies, hopefully someone who is not the author of the policy. So by using WebAssembly, by being able to use regular programming languages inside of your team, inside of your company, you suddenly have many people who are able to review policies, write them, maintain them. So this is pretty powerful. Also by using regular programming languages you can have your teams keep using their coding conventions, their tooling, like let's imagine linters for example, they are just applying them against regular Rust Go program. Also, let's say that you want to write a policy that performs some semantic versioning check. Well, this is easy. You just go ahead and look for a semantic versioning library for Rust and just consume that as you would do with any kind of regular program. Also, you can go ahead and reuse the existing CI CD pipelines that you have in place to ship your code. You just reuse them to ship policies. So what can you do with the policies of Kubewarden? Well, we provide to policy offer several SDKs. We have SDKs for Rust, for Go, for Swift and many more are coming. With these SDK, you can easily write policies that validate or mutate objects. So you have a full spectrum of possibilities covered. Also, we offer you the ability to pull some extra information from the cluster while evaluating the incoming objects. So for example, let's say that you want to write a policy that is validating ingress objects. So looking just at the ingress object that is about to be created is not enough because you need to know what are the other ingress objects that are already part of the cluster. So this is what we call context-aware policies and this is something that we cover as well. Now you are a policy offer, you created your policy, you're happy with it. The next step is to distribute the policy. So here, WebAssembly can be pretty handy for us because WebAssembly modules, first of all, are really small. It's refreshing if you compare them to container images. So to give you some ideas, a WebAssembly policy written in Go is about 300K. A Rust one is 101 megabyte and half. A Swift one is nine megabytes and half. The size depends on how mature the compiler is when it comes to producing WebAssembly modules. As you can see, Swift is not there yet but things are improving. So these are really small compared to regular container images and most important of all, they are portable. As I said before, WebAssembly doesn't care at runtime whether it's running on Linux or Mac, on ARM, on Intel. That means that as a policy offer you can build your policy on your Apple machine running on Apple Silicon and then run that very same binary unchanged on top of the Linux cluster using Intel as the underlying platform. So this is really interesting. The way the policies are distributed then, actually, practically speaking, we are distributing them using container registries, which is pretty handy, we think, for container operators, sorry, for cluster operators because inside of the regular container registry you end up having your container images and also the policies that are going to regulate this application or your cluster. And that means that all the processes that you use to distribute container images, you can apply that also to the distribution of a keyboard and policies. All the mechanism you have in place to secure the container registries for images, you have them in place also for your policy. So this is really a smooth transition for operators. Also, when it comes to the policy themselves, the actual end binary, WebAssembly is really flexible because it allows us to enrich the final file with some additional metadata. And we took advantage of that to bundle into the single WebAssembly file information such as the name of the policy, the version, the license, all of that. But I think it's really interesting that we also bundled inside of it the documentation of the policy. So if you find a policy, you already have bundled the usage of the policy inside of the same file. Also, we provide some metadata that is then used to scaffold the Kubernetes custom resource definition that you use to enforce a policy. And Rafa will show that into the demo later on. So you have all this information in a single place in a single file moving across different registries always in sync. And last but not least, these policies, you can run them also outside of Kubernetes. Rafa is going to show that to you. And this is pretty handy because an operator can discover a policy, download it, run it outside of Kubernetes to see how it behaves and then automate all of that inside of maybe a CI system to make sure that the policy is still doing what is expected to be doing. So this is really flexible and handy. Another key aspect of WebAssembly that we're leveraging is security. So as I said before, WebAssembly was originally conceived for the web to be used by browsers, which are exposed to many kinds of attacks. So when the WebAssembly format was designed there were a lot of efforts spent to make sure that certain type of attacks cannot be done when executing WebAssembly binaries. I don't want to dig into the details. There is a link on this slide. Go ahead and look at it. It's pretty interesting stuff. So, given that we are a project in the security space is also pretty useful for us to know that WebAssembly is offering some safety when it comes to run these binaries that you find on the policy up, which Rafa is going to show to you later. Also another key aspect that WebAssembly provides is isolation. So all the policies, all the WebAssembly modules, generally speaking, all WebAssembly modules are run inside of their own sandbox, be it inside of the browser, be it outside of the browser. In our case, we have a component which is called the Polish server, which is the place where all the policies are run and are used to evaluate incoming requests. So these software is written using Rust and using a WebAssembly runtime call wasn't time and wasn't time is provided to us for free sandbox and inside of each sandbox, we have a policy. So all the policies are unaware of other policies that cannot see each other and they cannot even interact with the host. On top of that, when we deploy a policy server inside of the Kubernetes cluster, we are running that inside of a regular Linux container. So we have even more isolation added on top of that. One last important aspect of WebAssembly, another feature that we're leveraging is that WebAssembly is kind of, it's a universal format. So as soon as you have something built into WebAssembly, you can just reuse that. And so I mentioned in the beginning that there are other ways to write policies for Kubernetes. The first project in this space was Open Policy Agent and there is also Gatekeeper, which is based on that. So Open Policy Agent has policies that are written using a query language, which is called Rego. So the interesting fact is you can take a Rego policy and then build that into a WebAssembly module. So you can do the very same thing with the Gatekeeper policy, they're all Rego files. The interesting thing is we instructed our WebAssembly runtime so it can understand policies of OPA and Gatekeeper and that leads to a really interesting scenario where you have a policy server of Q Warden, which is not just running a Q Warden native policies, like I don't know a policy written in Rust or one written in Go. It's also writing side-by-side policies that were written for OPA or for Gatekeeper. And we also updated all our tooling that Rafa is going to show to you so that you can distribute these WebAssembly binaries created by OPA or Gatekeeper. You can distribute them with OCI registries. You can enrich them with the metadata so that you have the documentation, the usage, the scaffolding all together. So this is really leading towards Q Warden being a universal platforms for your policies, which means that if you already have invested into OPA or Gatekeeper, you can reuse all these technologies. Now, another, one last topic I want to touch and this is not about strictly speaking WebAssembly, but more about policies code. We want to have policy treated in the same way as all the other parts of your infrastructure. So you're already, you're interested. You want to know how your application are performing, how your infrastructure is performing. You want to have observability on your policies as well. So because of that, we extended the SDKs of Q Warden so that when you write a Q Warden policy, you can also generate trace events from it. And then we instructed the policy server so that it can collect all these events and then push them to an open telemetry collector. So by doing that, we enable deep introspection into how policies are behaving. So here you can see that we collected an event which describes the execution of the policy, which is basically the evaluation of an incoming request from Kubernetes. And we pushed that through open telemetry to a Yeager endpoint. And here with a Yeager view, we can drill down into the policies and see how they are behaving, how they are performing and deep down into a single request. We can see all the different steps that the policy took to evaluate. So this is all for me, this is all. Now I want to show to you how everything looks like in action. So Rafa is going to give a really nice demo to you. All right, thank you, Flavio. Okay, so let's go ahead and define the problem first that we want to show because we are going to work with policy for this demo and I would like to explain first what we want to achieve. So let's say that we have a cluster on our organization and that this cluster is only exposed in services to be consumed by our users. And so we want these services to have valid certificates. Let's see what we are using as manager as well. And so what we are doing is on our ingresses, we are annotating our ingresses with the cluster issuer that we want for let's encrypt using sort of manager. And so we have two cluster issuers. One is the, let's say, the staging one and the other is the production one. And so in this case, despite the ingresses valid in itself, what we want to achieve is to only allow ingresses that have the production annotation one because if someone by mistake deploys the ingress with the staging annotation, then it will be exposed to the world. And then everyone could see that the certificate is not valid early. It comes from a staging let's encrypt certificate. So for that, I know that I need to change the annotation. So let me go ahead and share with you the policy hub because this is what we use to find new policies and this is how it looks. So I know that I need to work with annotations. So what I would do in this case is look and search for annotations and we see a policy that actually builds perfectly. This policy, you can see that it works on any kind of resource, it's a super generic policy. It has a homepage, it has more information. But the important thing that we want to see here is that we can copy this URL and this is where this is coming from. So this is where it's published. We can just go ahead and pull that one. So, okay, I copy that and then I will go back to my terminal and you can see now, let's go ahead and share the terminal. And now you remember that we copied that URL. So I'm going to use KWCTL. KWCTL is the CLI go-to tool for Q Warden and KWCTL allows you to do a lot of things. For example, it allows you to run policies locally without the need of having any Kubernetes cluster. You just have a policy, a Wasm policy and then use KWCTL to run that policy with some settings. For example, with KWCTL, you are also able to annotate the policy. As Flavio said, our policies have also metadata that goes with the binary itself and with KWCTL, you are able to annotate the binary. So let's say that it's expanded with this metadata. With KWCTL, you're also able to pull policies from a registry or an HTTP server or you're able to push policies to an existing OCI registry. And so KWCTL works in a sense like if you think about Pullman or Docker where you have like Docker images or Pullman images. So in this case, you can run KWCTL policies. We have a store, a local store in your home directory here. So KWCTL is only for your local development or local testing. So if I run that, I have an empty output because I didn't pull anything. And since I copied and pasted before on the policy hub, the URL, now I can just go ahead and pull this policy. And this is going to pull this policy inside the store that I was talking about. That was empty on the previous step. So now it finished, I just pulled it. And now if we try to list policies again, we see now that we have one policy only in our store and that it reads some information from the metadata because these policies are already annotated inside the OCI registry. And so it can read some information from it. And as you can see, it says that it's not a mutating policy, it's just a validating one. It's not context aware because this is just going to evaluate the request in isolation. It also provides some SHA, SHA-SAM for the policy itself, the size of it. So you can see some basic information about it. But you can also see more information. And this is going to go ahead and read the annotations from the binary itself. So we can run KDVC in inspect for the policy that we pulled. And now if we go to the top of this command, we see that it has some details. Like who is the author or the authors, the URL, source, license, whether it's mutating or not, context aware, what is the execution mode. So you can see some things. Also the rules, for example, this policy is super generic because since it's only checking annotations, this works on the metadata and basically it works on any kind of resource. So what this policy says is that I can work on any kind of API group, any kind of API version, any kind of resource and for operations create an update. And I'm validating this kind of resource. Of course, then when you deploy this policy on a real cluster, you can narrow down what requests it is evaluating. An important bit here is that this policy says that it can understand any kind of thing. And if you had another policy, for example, that is very tight to validating some kind of pod, for example, so you are sure that it has, I don't know, validating. It has a liveness probe, for example, and to reject any kind of pod without liveness probes, you would say that it would target pods, for example. So that would be a very tight policy to the result of this evaluating. And so you also have a free form text usage where the policy offer can write anything they want here. So you know how to use it. And this is, as I said, free from text and you can see here what kind of settings it allows, what kind of settings it understands. And we're interested in constraining annotations because we want to ensure that any kind of fingers that this created inside this cluster will only be tied to a production, let's include cluster issuer from CERN Manager. And so let's go ahead and look for a request, this one that has a production one. So if we look at the request, this is our request that got recorded from the API server. You can see that the cluster issuer in this case is let's include production. So this would be a request that we actually want to accept. And so we can go ahead and evaluate the request with KWCTL run. And we are able to provide some settings to it in a JSON format, you can also provide that in a file that contains this JSON. But here we are saying, okay, we constrain the annotations and this annotation in particular, so it's the cluster issuer from CERN Manager has to be let's include production. And so the request path is a request that I previously stored from the API server. I recorded it somehow or I generated it. It doesn't really matter but it needs to be a valid request from the API server because this is what the API server will ask for if you deploy that to production. And then you say what policies you want to run. So in this case is the one that we saw on the policy hub is basically the same URL. And now we see that this is a load. So this is the answer from KWCTL. This is the answer from our policy and this is the expectation because this is actually a production one. Now let's go ahead and see a staging one. So if I look for issuer again here you can see that this one has a staging that's in feed cluster issuer. This should not be accepted in this cluster because we don't want that. And so if I try to basically run these policies with the same settings because we still want only production ones but I target this request that is the staging one. What I'm getting is that the value of sermanet as issuer doesn't pass the user defined constraint which is the expected response from it and it's not a load. And now let's go ahead and now that we are able to run this locally to go to the policy hub, search for it, pull that, inspect it, see how it works, see the documentation of it in an offline mode, let's say like locally in my machine, in my laptop or desktop. Now I want to go ahead and deploy this to a real cluster, to a real Kubernetes cluster. And this is where QBotan is super useful because it will do a lot of the paperwork or Kubernetes of Kubernetes for us. It will do many, many things for us and we just have to deploy the custom resource from QBotan. In this case, this is the cluster and mission policy. And the cluster and mission policy is a resource that is able to say, you can deploy as many of cluster and mission policies as you want in a cluster. They can target cluster wide resources, they can also target namespaces resources in any namespace. And so you are able to basically deploy all your policies targeting wherever you want. We will introduce also namespace one, so people who don't have these privileges to create this kind of cluster wide resources can also perform some kind of compliance into their own namespaces. But for now, this is the way we work with this custom resource. And we have a sub command for KWCTL which is able to scaffold a manifest for us. And so we can run the manifest sub command with a type cluster and mission policy. This is the only one that we have right now. So I was saying, and we can request KWCTL manifest to fill some settings for us. So this is the settings we are interested in because the constraint and this is the one that we want, the production one. And this is the policy we want to generate the manifest for. And so if I go ahead and generate that, it goes into the standard output and I can see that this cluster and mission policy, it gets a name that we would like to configure later on. But yeah, did we get the module over there? This is the URL again that we will see before and the settings that I requested the manifest will also provide. And the rules are automatically filled from the manifest from the metadata of the policy. Of course, I could narrow down this for right now and just target ingress because this is what I'm interested in. But for now, it's fine. You can just keep it as it is. For example, it also says it's mutating in false because it also reads this from the metadata of the policy. So this is the manifest itself. So now let's go ahead and actually deploy that on Kubernetes. So I'm running the same command on the type that KWCTL apply. If I run that, it will take a second and I will get the generated policy created. And now this is being deployed by the keyboard and controller that is doing all this paperwork for us. So we just need to wait for the webhook to be configured and then we will be set. So now it's our policy right now is completely working. Like it's completely activated. Now let's go ahead and look again for a production ingress resource. In this case, again, as you can see, I am going to create this ingress resource that has the cluster user let's include production, which is the expected one. So if I go ahead and deploy this one, it should be accepted. Okay, that is fine. And now let's try to do the same thing. But this time, if you look at that, it's the let's include the staging one. And if I try to apply this one, we get an error back from the policy. The server is coming from the policy server and it's coming from the policy on the same and from the policy in the end. And so this is what we have in terms of how you can go ahead, find some policy on the policy hub, pull them into your local store, inspect them, generate some manifest for Kubernetes and deploy it on top of Kubernetes. And you can see that everything is pulled from the registry. And in this case, the GitHub registry, but any kind of OCI registry works and everything works out of the box and in a nice way. And now let's go for the second part of the demo in which I have a gatekeeper policy and I went ahead directly to their GitHub project. And I went ahead and just downloaded, copied and paste one policy called required labels. So what I want to do right now with this policy is to enforce that we have on every resource that we create. For example, this could be a rule inside the forward company. Every kind of resource needs to have an owner. And so the required labels is super useful for that. And so if we look at the required labels policy from gatekeeper, we see that this is coming from Regal, we have the packets, we have the entry point that is called the violation and basically I don't want to go too much into the details but we have like this, this is the query. This is the entry point that we want to call. And so we have two ways of violating this policy. One is by not providing the label that we are requesting from the settings. And the other is by providing a valid value for the key provided on the setting. So there are two ways to violate this policy. And so let's go ahead and build a policy. And for that, I need to run OPPA build because this is a Regal policy. It's starting gatekeeper but it's written in Regal. And so I can use OPPA build and target wasn't. I can provide the entry point, which is the packets, Kubernetes required labels slash the entry point. In this case, the violation are constrained. And then I just need to provide the files where my policy is. It can be in different files across different files. So I just build that and OPPA build will generate a table for me. And I extract the policy that wasn't far from there. And then I'm going to show a request that is valid now. So in this case, this ingress resource has the label. So if you look into that, we have the owner team label here and it's pointing to an email of my company. And so this is actually fine. And if I try to run that with KW-CTL, in this case, we need to provide what is the execution mode of this policy because with Rego, you can either build policies targeting the OPPA framework or the gatekeeper OPPA is open policy again or the gatekeeper framework. And so we cannot tell the difference between them just by looking at the wasn't file. And so you need to provide some information to KW-CTL on how to execute this, how to provide the inputs, how to read the outputs from it. And so we provide the execution mode of gatekeeper. We also provide some settings over there and we also provide the request path that we are evaluating. And we also provide, of course, the wasn't file. Of course, you can annotate policies written for gatekeeper for OPPA as well. There wasn't files. So you can also provide annotations over there. You can run KW-CTL annotate on them. So you can also set this gatekeeper or open policy again the execution mode on the metadata of the policy itself and it will fly with the metadata, with the policy itself anywhere it goes. And so you wouldn't need to provide this parameter, the gatekeeper one, but right now I need to do that this ID and annotate this policy, I just built it. And so as expected, this is a load and this is fine. And now let's look at a request that is invalid. In this case, as you can see on the metadata it is lacking the labels part. So we don't have an owner over here. So this request should get rejected by the policy. And now I run that with the same settings. You can see labels, key, owner team, we are forcing that. And now if I just run that, I see that I get an error, this is not a load and this is the error that I get back from the policy. And this is everything that we have regarding the written policies. And so let me go back to the presentation. Answer that with all of you. Okay. So how do we get involved after all this? So we are, everything we do on QBorden is in the open. So everything that you want to say to us, everything that if you want to get involved in anything, just go ahead and tell us. So we have this main website called QBorden.io. You can see some information over there. We also have a blog over there where we write from time to time. We also have some documentation over there. So have a look at that. We have the policy hub. This is where we expect people to share policies, publish new policies for policy authors, discover new policies for class administrators. We have the GitHub QBorden organization where we have all our projects over there regarding QBorden. We have a number of them because it's a kind of a split project across different repos. We also have a QBorden channel on the QBorden at this workspace on Slack. So feel free to join there. And we also have the Twitter account, this is a one for QBorden. And I don't know if I'm missing anything, Flavio. No, no, everything was great. So I would say thanks to everybody for joining us today. I'm really looking forward to talk with you, to share ideas and collect your feedback. Thanks. Thank you for your time and yeah, have a great day. Thank you for staying here. Bye-bye.