 Thank you. Hi, hello, everyone. Welcome to our talk. My name is Raul, and here we have my colleague, Victor. We are both software engineer at Sousa Rancher. And today, we're going to talk about how to enforce a secure supply chain on Kubernetes. And let's start by defining what's a supply chain. A supply chain is everything that's needed to deliver your product. If we think of software and Kubernetes systems that might involve your source code that you probably are using source control, so you push this source code into source control, you probably have a built-in pipeline that will take this code alongside the dependencies. We create an artifact, which is a container image. And you will put this image into an OCI registry. And then this image will eventually be on production inside a bot on your Kubernetes cluster. So that's everything that's involved in your supply chain. And as you can imagine, there are many threats. We took this image from the Sousa security framework. And if we start from the left side of the image, first thing we need to make sure is our source code is never compromised. Imagine someone gets access to your source control and pushes on malicious code. This could eventually go into production. And then we have the built-in integrity, which is the part we are going to focus on today's presentation. We want to make sure that what we build from our source code is what we actually see in our production environment. And in terms of Kubernetes, we want to make sure that the image we build from our source code is what we actually see in our production cluster. And the way we want to do this is we're going to sign our container images when we build it. And then we're going to verify the signature inside a mission controller inside the Kubernetes. So before deploying a pod, this mission controller will verify the signature of the containers. If the signature is satisfied, it will allow the pod. If not, it will reject it. As you can see, there are many threats there. If you go to the salsa security framework, you can see a real-world example for all attacks that happened last year. I'd like to talk about the F, which is a compromised, modified package. There's a company called CodeCop. And somehow, the Google Cloud Storage credential got leaked. And someone pushes a malicious image. And then people started using this malicious code because they didn't have a way of verifying this image. They could have been prevented if they were using some kind of provenance check, like signing images and then verifying these images. OK, so let's talk about signing. And for that, let me introduce Sisto, which is a combination of open-source technologies that includes Cosign. That's the tool that we use for the actual signing and verifying container images. Then we have Fulgio, which acts as a certificate of authority that will issue certificates that we can later use for signing. And then we have RICO, which is a transparency log. It's a transparency log where all the signature will be stored, so we can later verify them. In addition to container images, Sisto also supports signing binary blobs, any OCI artifacts, such as Hemchart, Watson modules, many more formats. And we're seeing a lot of adoption in Sisto, especially in the open-source community. Even Kubernetes itself, they started signing all their active start innovation 124, which means that you can verify not just your images, but also third-party images if they are using Sisto for designing. OK, so let's start about signing. And for that, let's talk about Cosign. There are two ways of signing. One is the traditional way, which is using a keeper. You can use Cosign to generate your own keeper. Or you can bring your picker if you already are using and exiting other PKI. That's totally fine. That's supported by Sisto. It also has a KMS support for Google Cloud, Amazon, and Microsoft Azure, and also Harsicov Vault. So that could help you to store your keys. But there is another way of signing that's the keyless way. That doesn't really involve it. I mean, it's not really keyless. You still have ephemeral keys that are generated by FUGIO. And then the signature are stored into the RICO transparency log. Yeah, we will get back to this keyless workflow later. Let's start seeing the traditional way. How you would sign using a keeper. As you can see here, we are using Cosign to generate a keeper. You need to introduce a private key. And then it will generate a public key and a private key. And you need to keep this password and the private key secure. That's your impossibility. You can use a KMS provider that might help you. But yeah, you need to keep it secure. And then for verification, you need to distribute the public key, which is not ideal. We can see here, for verification, you need to provide the public key. Then you need to distribute the public key to everyone for verification. So now let's talk about the keyless mode. First, still experimental. Hopefully it will be ready for production soon. And it's using OpenID Connect for identity. OpenID is an identity framework built on top of OAuth 2. And it allows third party application to identify the end user using our authorization server, or our IDC provider. And the way it integrates with CISTO is that you request a certificate to Fulgio, which is axed as the certificate authority, passing an IDC token. Fulgio verifies your identity and creates a certificate with your email address. Then you use this ephemeral certificate to sign your container image. And this signature is starting the record transparency log. So anyone can query later this transparency log to verify that the signature is there, and no one has modified anything. It has support for automated environment, so it can be integrated in your CI CD environment, as we will see later. So let's see an example. This is how you would sign using your terminal. You type a cosine sign, and then the image. And as you can see here, it starts generating the ephemeral keys. And then your browser will be open. So it requires some human interaction. You will see something like this. You need to log in with one of these OIDC providers. In this case, we chose GitHub. Then once you log in, if you go back to your terminal, you can see that the signature was successfully pushed to the OCI registry and also transparency log. You can see the T log entry created with index. You can use this index to query a record. It provides a CLI, so you can query this transparency log to see the entry. And also, if we go to our OIC registry, we can see that the signature is also stored there as an artifact. How can we verify? As you can see here, we don't need any public key. We just need the email address with this subject. And then the issue in this case was GitHub. So there is no need to distribute anything. Just an email address, which is a public key. It also has a great support for CI. It has automatically discovery functionality for GitHub actions at Google Cloud at the moment. That doesn't mean that you cannot use another CI. You can use any CI. You can pass the identity token to Cosign providing a flag. But for GitHub, as you can see here, we are not even requesting the OIDC token. Cosign is clever enough to take a token. I'm running inside a GitHub action, so I will fetch the token, pass it to Furshu, and then all the certificates will be generated for us. And then the way we verify this is the issue. In this case, it was a GitHub action. And then we have the information about the workflow. So we need to check which workflow we trust. In this case, it's the one we use for generating this. OK, so we now know how to sign and verifying images using Systo. Let's see how this fits together in Kubernetes. How we can verify this inside of a mission controller. For that, I'll hand it back to Victor. Yeah. So yeah, Raul here has given us a primer on how to sign and verify with Systo, which is a vendor neutral effort from the Linux Foundation. So everybody is welcome to use it. And yeah, how do we make it work inside of a cluster? Well, let's see a cluster. Here we have a cluster. We can see the blue part, the blue box, which is how a Kubernetes cluster would be. We have the happy users there to the left and then ATCD to the right, which once the request combined and so on, the info gets saved on ATCD and gets taken by the reconciled loops of Kubernetes and things happen. And Kubernetes provides us with one thing, which is called the dynamic admission control. We can just use a dynamic admission controller. For those that don't know, maybe here in this conference, you are not so used to Kubernetes. A dynamic admission controller allows us is to connect some webhooks to the path of a request. So we have the webhooks there for mutating and for validating. And with those webhooks, we can just change the request or validate it. For example, let's say that you try to put a pod in the cluster. So then it's a JSON request. The JSON request that specifies some keys and values on the pod, on the JSON, and so on. So then the user sends that JSON request to the QAPI, goes through authentication and authorization for that user, and then it arrives to the mutating admission part. There, we want our pod to have an annotation with prod, because that's what we want. No, we want our cluster to have everything annotated with prod for a specific namespace, for example. And that's what we do here. We are there on the webhook. We wait for the request to arrive, and then we change it and add the prod, annotation prod. Then we go to schema validation, which there is where the JSON gets checked again and see that the JSON is conforming with the schema. And then we go to the validating admission part, where we have other webhook, where we check that the annotation actually is prod and not maybe test. And with that, yeah, it passes on and goes to a DCD, and we have our pod, or our happy user has our pod. OK, this is the concept of dynamic admission controller. We need a specific one. We need one to enforce the signatures and verify the signatures. In our case, we're going to use Q-warden, because it's the one that we are used to, and it's the one that we are building. So yeah, it's the same. You would have a project. In this case, it's Q-warden, which is a policy engine. There's other policy engines, maybe you know of others. And in our case, the policy engine has policies, and those policies are written in wasm. We would talk about wasm later, and why it's important for designing and verifying and simplifying things. And in the case of Q-warden, it's written in Rust, mainly, and also the controller is written in Go. And the policy server, being Rust, needs some libraries. So they work with SixTor. In our case, for Rust, we didn't have any SixTor crates, so we built a SixTor crate. And we contributed it upstream to the SixTor organization, and now it's taking a life of its own. So if you are into Rust, please check the Rust crate for it. And we are talking about policies being wasm. What does that mean? Maybe you know about other policy engines that work with OPA or Rigo and so on, which are domain-specific languages. But being wasm has other benefits besides that. For those that don't know about wasm, wasm is small. It's just a binary instruction format. So imagine it as an architecture. You compile against wasm, and then you get a binary, and then the binary is super small compared to other architectures, and then you just run it wherever you want. Why can't you run it wherever you want? Because it's a VM, a VM that comes from the browser world. So it's quite secure. It has very little things open to the outside, and you can just connect to the outside using a POSIX way of working, POSIX-like syntax, and so on. It's also polyglot because being just an architecture, you can compile a lot of languages. You can compile REST, Go, Swift, even Rigo policies, and so on, OPA policies. This is great because it allows you, I mean, maybe you don't know about OPA or maybe you don't know about Rigo, but for sure you know some language, no? And maybe you have your colleagues knowing a language already. Then just keep using the libraries of that language. Keep using your pipelines in whatever CI you have, and so on. And writing policies, either in Rigo or whatever, it's pretty simple because you just take a JSON object, check for a value or two, and then return true or false, or maybe change a bit the JSON. So it's not like rocket science, in a sense. So it's doable. And by using all of that, you can just integrate it in your CI CD, and so on. One thing that is important here for signing and verifying is that wasm modules, the wasm binary, those are first-class citizens in OCI registries, the same as container images, or the same as Helm charts. So now you start seeing that everything kind of starts becoming together. And apart from that, being just a binary, a wasm binary, you can just run it outside of Kubernetes, which is great because just getting a Kubernetes cluster is just to test a policy, just to see if something in one JSON, nah, just run it outside, no? It's very simple, and that's it. And personally, maybe this is a bit off topic. But personally, I start seeing, there's some talks there that talk about the complexity of computer science. It's like, everything gets more complex, and more layers, and so on. With wasm, being that it's already secure and contained, you maybe can reduce your layers, the amount of layers that your whole stack has, which it's nice. And it's picking a bit of steam everywhere, and so on. So coming back to signing and verifying, and making your whole cluster secure. Raul has talked about how to sign and verify images with six stores, Cosign, CLI. Well, if you know how to verify and sign contained images, you know how to verify and check the policies, because they are just an artifact in the registry. So in this case, it would be Cosign, Sign, and then Cosign, Verify. You would go through the whole thing. Of course, it's difficult to explain. We don't pretend to explain Cosign and Six store in five minutes in a presentation. Not used to get the whole thing, but it's doable. And in the case of our policies, we can just check them outside of the cluster with KWCTL, Verify, which is a CLI that we provide. For the policies of Q Warden, we have a hub, there in hub.qwarden.io, where all the policies are marked as signed or not, and then you can just check those things. So far, only the Q Warden devs policies are signed, but I suppose in the future it will expand. OK, we have signed our policies. How do we ensure that the policies, how do we check the signatures inside of the cluster? We configure Q Warden for that. Q Warden being help charts and so on. You just have some configuration that you put in a config map. And in this case, it's pretty simple. You can get it with our little utility to get the thing. And here is where we start seeing patterns, because this seems somehow simple, but there are some best practices that you need to follow. Here, Raul has talked about issuer and subject. But here, we are going a step forward and saying, OK, if it's something from GitHub Action, then we can just look at the owner and the repo. In this case, Q Warden. Everything that comes from the Q Warden organization in GitHub would be checked and verified and that's it, because there's little details on how things work that may come to bite you. So that's why having best practices, it's a good idea. OK, we have our cluster deployed. We have signed our policies. Now, we are verifying all the policies. All the policies that end up in the cluster are verified. So if we use them now to check other things in the cluster, we can start trusting them, because we are building our own just chain of trust. And now, we need all the images inside of the cluster to be verified and trustworthy. We can do that with a policy. In this case, we are using this policy that we have written with the RAS SDK, but you could just write your own and use whatever language that you want with the SDK that we provide, and that's it. This policy seems simple. It's a policy that looks on the pods for containers listed there, so it's going to be containers, init containers, and fmerial containers. That's all the things that could go inside of a pod. And you get the JSON request, and the policy decides, looks at the signature of the container, and maybe it says, no, I don't know this person, this organization, rejected. But then it can be fine, and then it approves. But ah, there's a trick there. Imagine that you are just verifying an image up example with a tag 1.0. The tags are mutable. Are they? No? You can just push today a tag 1.0, but tomorrow you can push again the 1.0 and rewrite it. So then you should not trust the tags themselves, never. What you need to do is check for the signature, and check for the signature of the digest, which is the checksum of the image. And then what you do is mutate the JSON request that contains the pod, and append the digest, the checksum, to your app example version 1.0, and then the checksum. So you know that you are instantiating only the things that you want to instantiate, and nothing gets changed below. OK, we have talked about the policy. Now how does a policy get instantiated in the cluster, a keyboard and policy? Well, it's a custom resource, cluster and mission policy. They're in line three. You have the spec, spec.module that points to your policy wasm module, which is kind of like a container image. You remember because wasm modules are at the same level as container images, the rules then we are putting it for pods, for operations create an update when you change a pod. And mutating through, because we want to make it change the digest, and then some settings. In this case, the image has an asterisk, so it would try to, it would check for all the images that end up in your cluster. And in this case, it could be the GitHub action and so on, but we can also use the issuer and subject. Here we are looking for the subject that comes from GitHub actions. You can see the whole URL. Imagine if we get an asterisk after github.com big quad, then only that user would be trusted, and everything that comes from that user would be trusted. Here we are only trusting a specific tag from a specific pipeline that comes from GitHub actions and so on. So we do a kube control apply of that policy. We wait. It's a CRD, it's a custom resource. We can just do a kube control get and see the list of them. And once here at the end of the ride, we see status active, then that policy is protecting and enforcing. What happens if we try to instantiate a pod with untrusted images? This is a pod that comes from somebody, and it's not signed. Then you can see there, we are going to do a kube control apply, and it's going to be saying, pod nginx not accepted. Verification of image nginx failed. No signatures found for image. It's not signed. It's not going to get into the cluster. It's just going to get rejected. The user is going to get here a one as a return code, and that's it. And for the case of a pod that has container images that are signed, well, we do the thing. It's transparent to the user, and that's it. And you can see, if you look at the specific pod, you can see now that it has the digest appended. The last line, maybe it's a bit smaller, but yeah, it's just the URL of the container, 0.1.0, and then the digest appended there. OK, what's next? We have talked about SIGSTOR. I mean, there's a lot of problems with the secure supply chain. We need to check it and verify the supply chain. We have talked about SIGSTOR and our own images, but we have not talked about dependencies. How do you handle dependencies? Well, what you need to do is append a server bill of materials to your container images. That server bill of materials will contain just a list of your other image layers that you are depending on, what are the projects that are there, libraries, and so on. And that's partly already in cosine and SIGSTOR, and we intend to put that also in Q-worden. Also, we need to sign and verify everything. If you get something from this talk, please be the SIGSTOR vendor neutral coming from the Linux Foundation and how everybody depends on something. We don't have our own stack on our own, and we all depend on some dependencies. And those dependencies need to be signed and verified if we want everything to be secure and we want just to trust the whole stack and forget about it. So if you're interested in this, think about signing and verifying. With cosine, it's quite easy and simple, so it's doable. And for us, in Q-worden, it's expanding the CI integration also for Q-worden and SIGSTOR upstream because it's really nice when you can just take the OIDC tokens that come from inside of the CI, for example, in GitHub, or from your cluster if you are passing the OIDC token by exposing the token inside of the node and so on. But it would be nice to have the same ability for, I don't know, GitLab because it's open source. It's open. It would be nice to have it for other CI schedulers, drone, tecton, and so on. And that's one thing that should improve. And yeah, with this, I think that's it. If you're interested in SIGSTOR and secure supply chains, please come by and stop us later. If you're interested in Q-worden, have a look there. You can see the information there, the Slack channel just to ask some questions and so on. And now we are open for questions. So if you have any questions, please feel free. Yep. Yes. So the question is, OK, I have used Notary before for signing images. How is this different compared to SIGSTOR's workflow? It's a complex question, no? What did you need for Notary? At the end, you were signing your images, no? And you were giving your users some public key to check. What would happen if your private key got compromised? You would need to revoke your public key somehow, no? It's kind of like this private key cryptography thing. How do you ensure that the users end up with the revocation certificate at some point because they could just not care about the revocation certificate and then you are in problems and so on? SIGSTOR, it's a difficult topic. We can talk about it later. Let me just put here the whole diagram. SIGSTOR works by delegating the trust to the transparency lock and by using a key that is fmdrl. So you only have the key that you had, the private key and so on. In SIGSTOR, you also have it. But you only are going to use it for the 20 minutes that you're going to sign things and then it gets discarded. And it cannot be used more because the signature goes into the transparency lock, which means that people in the future could just see that with that key you only sign a specific moment that allows you to sign and not later and it would not be validated. So that's one kind of attacks that are not there. By doing all of these dance with the full certificate authority and so on, you can delegate some trust into your OIDC, into your OpenID Connect service. Maybe your organization is big enough that has an OpenIDC service. Maybe it's an open source octa or things like that. Or maybe it's GitHub or maybe it's whatever your organization uses. Or maybe you are super small and you don't have that capability on having that infrastructure. So you can delegate to GitHub OIDC and delegate all the things. And then at the end you trust in GitHub. But if you are small that you are building also on the GitHub builders, you are already trust in GitHub. So what's the difference? And you are using GitHub as a source for. So you are already trust in GitHub. So you are not shifting so much trust. But it allows for an open source. Everything is distributed. It's not so centralized. And you have a lot of dependencies coming from different parts and big players and small players. And this way of working allows for small players to be signed because you need all the dependencies to be signed for the whole security of the supply chain to be there and so on. We can talk about it. I'm so sorry if I'm just expanding too much. We can talk about it and we can, if you want we can sit. And we can go through the whole diagram on what it means for SIGSTOR to do this. But it would take some time. Yep. No, no, no. It's making us as Raul has said, I need to go back. The signature is stored into the OCI. He registered as another artifact. So there is no additional overhead of existing image. It's just this new artifact, which is quite small to be honest. Yeah, it's gonna be a JSON with some strings and that's it. So what, four kilobytes? Yeah. I just wanted to show this one. No, one less. But we don't have the, yeah. You have the image here. This is in the OCI registry of GitHub. You have here the image 0.1.0 and then you have another tag which ends in .sig. And the first part is the checksum of the image of 0.1.0. And that's it. Another tag on the image and it's super small. Yeah, you have a point there. So the question is, okay, you are trusting on the whole thing, but you are also trusting on Fulsio and Record, which comes from SIGSTOR and they are services running on the web and they are registry authority. Fulsio is a registry authority plus a certificate authority. What happens if those get compromised? You now have a new vector for a compromise. Okay, yeah, that's a good point. First, Fulsio and Record, which are the transparency log and the certificate authority. The code is open and you can run your own for starters. Second, all the utilities can be redirected to your own Fulsio and Record services. Third, we go back to, okay, if I'm small enough, I cannot run my own certificate authority. Then I'm gonna delegate, but if I'm small enough, maybe I'm already delegating on a lot of things. I take things from GitHub anyways or I build things on GitHub or I build things on AWS or whatever. And if you are big enough, then maybe you just deploy your own Fulsio and Record and so on. And maybe you say, yeah, not even that. Then I can just go back and go back to the public key and private key authority, the way of working. And then you don't need Fulsio or Record and so on and you have the escape hatch if that's what you would prefer. So that's possible. For sure there's something. If the six store guys would hear us, they were gonna say, yeah, you're missing this and that, that actually helps. But yeah, that's it. Any more questions? Okay. Then, I think that's it then. Many thanks. Thank you.