 Hi, folks. Good afternoon. Thank you so much for being here. I hope you're having a great time. Awesome. So in the recent history, we've seen a lot of security attacks. We've seen solar winds, log4j, even sunbursts. This has very rightly switched or, you know, transitioned our focus to securing supply chain. In this session, Vishal and I, we will, you know, we'll talk about, or we'll walk you through starting from what container images are, what it's made up of to going into the importance of signing and verifying images, and finally talking about where and how Kivarno fits in. Right. By the end of this session, we hope that you'll be equipped with the right tools, insights, some understanding that will help you achieve a secure supply chain. Yeah. So let's start with the software part of the software supply chain, right? So a container image is basically a standard way of packaging the executable code along with all of its dependency in a way that can easily be shared across users, right? And in terms of Kubernetes, container image is a basic unit that can be used to run your own code inside of a cluster. By that, what I mean is that, let's say you want to run your binary inside of a cluster, what you can do is you can create an image from it, and then you can run a pod and then expose it for it to be accessed, right? And let's say you want to create an API endpoint and you will create a Kubernetes service and expose it to all of its users using HTTPS. One more thing to note is that a container image is stored in a content addressable storage, which is basically a way of saying that the URL of a container image is the proof of the content that lives there. So the URL of a container image has a digest, and when you go to the URL and you pass it through a SHA256 checksum generator, you will get the digest back. This is a guarantee of immutability, which means that if the content changes, the digest will change too, and this will change the URL as well. And this immutability comes in really handy when we want to ensure supply chain security. So the basic is out of the way. Now let's dive into what are the contents of a container image. So a container image has configs and layers as well, right? So let's start with the config. So config, as the name suggests, is the configuration of the container image, which can be used to create the container properly and make sure that everything is in place. So it is a JSON object that contains the history, which contains a list of the authors. It contains the OS information as well as the informant variables that should be used while we start the container. Another thing is layers. So a layer is basically a chunk of the executable code of the container image, and when we go to a container image layer and fetch its content and pass it through the right parser based on its media type, we get the binary of the container image that was placed in that layer. So here we have a layer of type tar, and when we pass it through the tar CLI, we get the binary of the stored in that layer. So I actually like to think of this as like layers of a cake. So for example, a cake needs multiple layers and icing on top to make it, you know, perfect. Similarly, we need a container image to have all of its layer in the right order, along with the config to wrap it all up to make the container image work. Okay. So moving on. So the question is now we have the config and the layers. How do we distribute this to the users, right? So to do that, we have another data type called an OCI manifest. So what a manifest has, it is basically a JSON object that has all the layers, the information of all the layers as well as the configuration that needs to be used. And when you fetch the image, you basically fetch this JSON object. And the client that you're using will use the config and the layers to start the container with all of everything put in place, right? But what if you have a container image that supports multiple platform, right? So for that, there is another type called a manifest list. This will be stored at the root of the container image. So when you pull a multi-platform container image, it will give you a list of all the manifest with the information of the OS that they support. And you can use the right manifest in your environment to do what you want. Okay. After that, these were the basic until last year. And recently, there has been another addition to the OCI spec, which is referrer. So basically, referrer is another OCI artifact that can be tied to an image without sacrificing the immutability. So let's say you have a use case where the image has been published, but you want to add additional data, additional information besides of the executable code, and you want to attach it to the image. So to do that, what you will do is you will create another OCI artifact in a registry and make that artifact point to the image. So what will happen is that when you want to access everything that is attached to the image, the referrer's API will look for all of those pointers and will give you a list of all of them. So here we have an image, and when we use the ORA CLI to fetch all of the reference, it finds that it has a vulnerability scan report, an S-bomb, and everything attached to it. So before we summarize everything, I would like to give the shout out to Brendan Mitchell for this amazing image and an amazing explainer video on what OCI images are. It is highly recommended to watch that if you want to learn more. So let's explain what we just discussed. So here we, on the left side, we have the tag, which is pointing to the manifest, and the manifest has a config and multiple layers, and on the right, we have all the referrer's that are pointing to that image. So this explains how container images work and the contents of a container image. So moving on. Great. So now there are scary things that can creep into your images. What I'm talking about is vulnerabilities. What is your vulnerability? Now that's a security risk that can be embedded into your container image. Why do we care about it? We care about it because if you have a vulnerability in your workload, this could leave space for a security attack. Now that's definitely a big no-no, right? So we came up with, this is a norm now where we scan our images, we scan our repositories for vulnerabilities, we produce reports, we deal with these vulnerabilities, we resolve them according to the threat they pose. But vulnerability reports are sort of like a snapshot of what vulnerabilities existed at a certain point in time. They're not stable. It's actually a moving target because there are new vulnerabilities that come up every day. And when this happens, this is not a stable way to communicate about vulnerability information. Enter SBOM. So SBOM, SBOM refers to a software bill of materials. This contains a list of everything in your software, right? A list of everything that makes up your software. Once you have this, this provides a static way of keeping a track of all the updates, of all the known vulnerabilities in your software, making vulnerability, you know, dealing with vulnerabilities much easier. We've even got a certain image showing how a developer builds the image, pushes it along with SBOM and vulnerability reports. This is together pushed to container registries and when pulled from it, we verify the SBOMs, we verify all vulnerabilities, make sure we are okay with what exists and then push it to a workload. Awesome. But now, there are some problems that need to be addressed when, you know, you've introduced metadata like SBOM and, you know, reports. So the problems that we're talking about, the first one is how do you store this metadata? Your image is immutable. But now, with every artifact, you have a SBOM or report and whatnot. So this was actually covered by Vishal a few slides ago where he talked about the referrer's API where you can essentially have a SBOM and point it to the image that it's, you know, supposed to be a SBOM for. And the next problem that we want to look at is how do I trust the metadata? Now, any Tom, Dick and Harry could have, you know, some artifact pointing to my image. How do I know that this is a trustable metadata or how do I trust this information? Yeah. So as Anushka just mentioned that since anyone can attach anything to your container image, how can you be so sure that the manifest or the SBOM that you're looking at is the one that is created by a developer, right? It can be created by someone who wants to take advantage of you, right? A more general question would be how can you be so sure that the container image that you're using is a trustable image out of all the bad images that can be used to do container breakout attacks on a cluster, right? So we need a way to communicate this information of trust from the consumer to the developer to the consumer of the image, right? We can achieve this by using signing. So an image signing method can be used to sign the digest of the image, which is immutable in nature. So when you sign the digest of an image, what will happen is that it will provide your identity and it will attach it to the current digest. So anything that is added on top of the current image will lead to a new digest, right? And since the image signature was attached to the previous digest, the consumer of the image will know that up to this point was the code that you came up with and anything that has been added on top of it has been created by someone who is not trustworthy. So they will know who to trust and whom to not, right? This image signing and verification flow can then be used by any Kubernetes dynamic admission controller to enforce rules inside of a cluster to build integrity and create a trusted environment, right? This will ensure that no bad actors can get access to your cluster. So now that we have discussed how why we sign and verify image, let's discuss the options that we have for signing and verification. Right, so a signature can be created using multiple ways. You have multiple tools available. There are even many mechanisms involved, you know, there could be a C authority verifying your signature. You could, you know, have an OIDC provider verifying that you are who you say you are. There are mechanisms like storing the timestamp of when a signature was made, having transparency log of signatures and so on. The two tools that we'll be discussing today are six tools, cosine and notary. These tools are used for image signing and verification. Right, so talking about notary, notary is like this simple, easy to understand, straightforward way to achieve image signing and verification. It achieves so by using X509 key cert pairs, right? It supports OCI 1.1 attestations, it is a CNCF open source project and yeah, it's a pretty cool tool to check out about cosine. It adds certain features and complexities to image signing and verification. It supports both keyed and keyless based signing and verification. It even supports in-toto attestations. Now cosine helps you store, you can store the signature timestamps and verify that the signature was created at some point in time that you expected it to be created, not before or after. It also helps storing a transparency log of signatures. Now this can be then used to monitor and detect malicious activity. So if you don't want this transparency log to be public, you can even have a private infrastructure which cosine supports using TUF. Now both these tools, cosine as well as notary have new releases with some really cool features to check out. Right, so now that we have talked about how we can sign and verify images, now let's talk about how we can incorporate all of that inside of your Kubernetes cluster to ensure security and enforce policies to make sure that no one bad can get access to it. Right, so for that we can use Kiverno, which is a CNCF incubating project, and Kiverno at a very high level is a policy engine that is built to work with cloud-native ecosystem and Kubernetes. So Kiverno is native to Kubernetes, which means that it only requires you to know YAML and you don't need to know anything else to get started. So basically to get started, what you can do is just install Kiverno in your cluster and start writing policies using YAML and you can start on a journey on improving security in your cluster. Right, a Kiverno policy can be used to do validation, mutation, mutate existing generation of new resources as well as scaling up old resources, but for this demo, we will be focusing on image verification rule. So what a Kiverno image verification rule allows you to do is that it allows you to verify signatures on an image, as well as signature on image attestations, as well as the payload of the attestation to make sure that that also complies with your requirements. Right, so in Kiverno image verification rule, you can do image signature verification using both cosine and notary. You can verify in total attestation as well as reference API attestation and you can use any OSIR industry. So we have support for all of the possible OSIR industry, be it public or private. We also have support for the newly added Jeffers API and we support keyed and keyless verification using cosine as well as cert pair verification using notary. Right, and to speed this up, we also have caching, so once you have verified an image once, when you try to verify the image for the next time, it will hit the cache and it will try to do it more efficiently so you save some time on subsequent verification. Right, and you can be very flexible with your checks in an image verification policy. You can verify all the conditions as well as any as well as make a required count so that if that count is hit once, like you don't have to do anything else after that. So this actually allows you to basically implement any type of requirement that you can have with a rule. Right, so let's say you want to verify that your image has a vulnerability scan report attached to it, which is signed and it has no critical vulnerability. You can easily do that using image verified rules. So to see examples of the image verified rule, we will be showing some demo, but we also have a very vast library of policies on our website, which is open source and anyone can get access to it. Great, so now let's move to the demo. Awesome, so now we have a good understanding of, decent understanding of what Kivono is, what cosine is, what notary is. Let's see how this all fits in in a couple of demos we have prepared. Yeah, let's start with cosine. So the first demo I have for you here is, you know, using Kivono and cosine together. Great, of course I had to record this because I was in short of the internet situation here. So let's see how this goes. So we have a policy YAML that, you know, looks something like this. We are creating a namespace to test this situation. We have a cluster policy. Now let's examine the cluster policy in detail. It's called keyed basic policy. In the specs we have set it to enforce. That means we want to block a resource if it doesn't comply. Great, so you'll see that there is a single rule. It's called keyed basic rule. It is trying to match a resource of kind pod. In the verify images block we have an image reference given. So here essentially you have a pattern to match with the image in your resource. Right, next block that we have right after image reference is attesters. Here we are giving the public key that will be used while we run cosine verify to, you know, ensure that the signature on the resource is correct. Awesome. So now that we've looked at the policy, let's also look at the resource that we'll be trying to apply. So it's a simple pod. It's running in a namespace test verify images. It has an image which will match the image reference in the policy. So in the policy essentially we want to match the resource being a pod as well as the image reference being the pattern that we saw earlier. Now that we know this let's go and try applying the policy and then the resource. So you do a cube CTL apply for the policy, create the policy and now I'll try to apply the resource. So what will happen in the background is that Kivarno will look that oh a resource is being created it's of kind pod and it matches the image reference that is stored in the cluster policy. Now I'll use the public key stored and check if the image running is signed correctly using cosine or not. It takes a second or two and you see the pod was created. That means the signature was verified and the pod is good to go. That was you know using keyed signature and cosine along with Kivarno. Now let's look at keyless signatures using cosine. Great. So again let's examine the policy. It's a cluster policy called check SLSA attestations. In the spec we have again set it to enforce. We have a single rule. In this rule again we are checking for resource of kind pod. In the verify images block here we have again given a pattern for image reference. In attestations we have provided the type of attestation. In this case in my policy it is of type provenance. It's provenance information. In attestors we've given we've specified keyless and we've given some details. So what it'll essentially do is check that the image is signed in a github workflow and the signature will be verified using the github OIDC provider in this case github. Also you'll see there's a conditions block at the end. So this place is yeah so we are trying to add conditions to the payload the attestation payload. So after seeing that the attestation is signed correctly we also want to ensure that the attestation itself is compliant for our requirements. Right we want to be able to check the attestations themselves. So we are looking for some condition and making sure it's equal to true. That's your policy. Let's go back to the resource and look at what we're trying to apply now. Now again it's a pod. It will match the image reference will match the image given here and Kivono will try to apply the policy against the resource. I've created the policy and now I'm trying to apply the resource. So again a similar thing will happen here. Kivono will see oh again there's a pod trying to come up which matches my requirements of being a pod and the image reference that I had earlier given and it'll see if the image is correctly signed the attestation sorry is correctly signed and the attestation is in compliance with the condition that was mentioned in the policy report I mean policy not report. I think I took it back. Sorry about that. We were looking at the policy being applied and once that's applied we want to see the resource being applied and we see the pod is created. Awesome. Now that we've looked at some core sign examples let's look at notary signature and verification examples. The first one I have for you is right here. So let's examine the policy. Here again we are trying to create a namespace in which we'll test and we are storing the certificate like we had earlier mentioned. Notary use key cert pair for image signing and verification right. In the policy that you see it's a cluster policy which is again set to enforce and you have a single rule in it. The rule will look for a resource of type pod in a namespace notary verify images and the verify images part of the policy that you see have type mentioned notary because we're checking for notary signatures and we'll pass the certificate that was stored earlier in the policy that will be used to verify the signature. So the certificate is corresponding to the private key which was used to sign the which we want to make sure the was used to sign the image. Sorry. Yeah. So now that we've looked at the policy we'll quickly look at the resource as well. The resource again would be a pod. Yeah. You'll see a pod that matches the image reference that was given in the cluster policy. Let's apply it. We're creating the policy. We are creating trying to create the report just like before Kivirna will try to ensure that the resources compliant the signatures valid verify that signature using the serve that was provided. And once all the verification is done great. I have my pod test ready. With that I come to the last demo bit that I have here for you today. It's a little lengthy but should be fun. Great. So in this case what I'm going to do is I will take an image which has some attached attestations. I'll first pull these attestations. Great. So I see I have some four attestations. I want you to specifically pay attention to the second one because I'll try to play around with the vulnerability scan attestation here. So first I'll pull it. Wait for it to be successfully downloaded. Now that it is let's let's you know walk through that station and a very particular part that is a timestamp that you see. Now in this timestamp let's focus on it. And you'll see that it was created on 25th May 2023. That's a couple of months back. Right. Cool. Duly noted. Let's go back and look at what we can do with the policy and the timestamp. So that's my policy. Again I'm creating a namespace to test and storing the certificate. Let's go down to the cluster policy in place. Great. The cluster policy is of name check image attestation. It's set to enforce. We have a rule. It's looking for a pod, a resource of kind pod. And in the verify images part we are specifying that it's of type notary. In image references you've given a pattern to match. And in attestations first you're specifying what type of attestation is being looked for. In our case it's a vulnerability scan. In attesters part we are giving or we're providing the certificate corresponding to the private key which was used, which should have been used to sign the image. And now here we have the condition. Now this is what I want to like spend maybe a minute on. So the condition says the key is a timestamp that we looked at before. And we want that value to be less than 730 hours. So essentially 730 hours is one month and we want whatever scan reports we have in our attestations to be not older than one month. If it's older than one month we don't want it. Great. And we just saw the attestation and we know it will, it's not compliant. So what I'll try to do is I'll apply the policy and I'll apply a resource which is using that image I pulled. Takes a minute, does a check, checks the attestation, pulls the attestation, verifies the image signature, and finally fails giving you the reason why it failed. It says scan reports generated more than one month ago are prohibited. Great. Now let's play around with the policy a little more and change that 730 hours to 7300 hours. That's 10 months and that should be allowed. Let's do that, that's 10 months. Go back, configure our policy and now try to apply the resource hoping that it won't be blocked. And as expected the resource port test was created because the attestation vulnerability scan was like five months, it was created five months ago. Awesome. So I think with that we are done with the demo bit that went fast. So here in the slides I have linked a couple of links to understand Kivono better, join some channels to ask questions, to discuss more, to understand where Kivono can fit in your use cases and how it can help you. Of course there's also links to the monthly community meeting and the weekly contributors meeting. We are always looking forward to having more contributors and more discussions. And before we wrap the session up, don't be a stranger, reach out to us. Let's talk about kibernities, kivono and whatnot. We've added some of our handles looking forward to more conversations. And that would be all. Thank you very much. And I think we do have like a minute or two to address some questions, some quick questions. And if we're not able to deal with them right now, we'll be walking around. Let's meet and chat. OK. Hi. Is there a way to? I think there's a mic. There's one. This is Manas. Actually when this policy is applied and typically in a loaded environment when the pods are coming up, if it takes that much time, what we saw in the demo, it may slow that thing because especially if you are running in something which is more on the serverless side of it, where not of what's are coming in. So is there any caching enabled where already verified images are cached and not checked again? Yeah, I think we shall. Yes. So when you verify an image, it has to fetch the data of the image as well, right? So that is making a network call that is taking so much time. So if you are in a private environment where the both of them are in the same network, it won't take this long. And to fix this, we also have a caching. So once you have verified the image using a policy, it will be cached. And for the next time when you try to do it, yeah, here, when the next time you try to do it, it will hit the cache and it won't make network call. It will be instantaneous. OK. Yeah. So we, sorry, go ahead. Yes, this is fine. Yeah. So we have decision caching to sort of fasten the process and make sure we can, you know, handle performance better wherever we can. So this cache is maintained centrally at the API server level or it's maintained at the node level or at the repository level. Yeah. So it is maintained at the Kibirno admission controller level, right? So it is maintained in the pod of the admission controller that you have that we have. We store at ETL-based cache and yeah, that's fine. OK. And is it persisted or it's an in-memory cache? I'm sorry? It's persisted cache or in-memory cache? Oh, it's persisted. Persistent cache. Yeah. And later this referrals, 1.1 referrals API support, is it a new support that has come up or? So this reference API was added in OCI 1.1 spec. So it was added earlier this year and we have added this, added support for it in Kibirno 1.11, which was released a few weeks ago. Yeah. So referrals API is recent and support for it in Kibirno has recently come up like about a month back. Yeah. And is there a limitation in size of it? Sorry? The reference artifacts that we are putting in, is there any limitation on size because at runtime when it is downloading, it may cause delays due to that? I don't think, are you? Yeah, OCI spec does not define any limitations on size of any artifact, right? So there's no limitation of size. OK. OK. Thank you. Awesome. I think we've, you know, that was our 30 minutes folks. Thank you so much for attending and we'll be around to have a conversation later. Yeah, we'll be answering your question. Oh. Hi. He had a question. Can you hear me? Yeah. So what level of Salsa does this comply to? What level of? Salsa, the SLSA that you were talking about. You know, so what level does it? I mean, Salsa has like level 0, level 1, level 2. So this process of signing attestation and verification. So what level does that comply to? I think you can like verify any JSON object, right? So it does not have anything to do with the SLSA level. So you have attached any JSON object. You can run conditions like using James Path on that JSON object. So I don't think like that will be a limitation here. OK. Awesome. Thank you guys. Thank you so much.