 אוקיי, אז let's start with the basics, vulnerabilities, anyone sounds familiar, I'm sure it is, kind of on news by now, right, vulnerability scanning, I'm assuming that everyone here knows what it means and you are scanning your images or your code repositories for known vulnerabilities, and if you do so, you probably also know that it's kind of a moving target, right, asking the question, is my image vulnerability free or is my code clean, because new vulnerabilities have been discovered every day, many of them, so we need to keep asking this question all the time, literally every day, it's a common and a good practice to re-scan your images like on a nightly basis, let's say, and this is why, because it's a moving target, the currency for communicating vulnerability information has to be something more stable than the vulnerability report itself, and this is why it has to be something like the SBOM, which is the description of the contents of the code repository of the image, and that's around the time when we started to have a lot of discussion in the industry about SBOM, you probably heard about it as well, way too much, about why SBOM is important, tooling that generates SBOM, standards and formats around the SBOM ecosystem, I think this is where we were like maybe last year, and then the next logical challenge was answering the question of where do we actually store this SBOM, how do we store them, okay, now I have plenty of them for every single artifact, what do I do with them, how do I make the SBOM available, easily available, when it's needed in relevance to the artifact that it's describing, so that's the first part, you see the numbers 1, 2, 3 here, this is also the agenda slide, so this is the first part of our talk where we start to describe the new OCI specifications that helps you attach the SBOM to the container images in the registry, and then we'll move on to the logical next challenge of how do I actually trust what is there, because this is not just any artifact, these are security artifacts, we use them to make security decisions, right, critical security decisions, we have to be absolutely sure that we can trust the integrity and authenticity of these artifacts, so we'll talk a little bit about signing of this and how the artifact spec relates to that, and finally in the third part of the talk we will show the use case of why do we even talk about all of this, what are these, how are these building blocks fit together into a use case and end to end use case that solves a real problem, so we'll try to simulate actually this picture in our demos, it's a very typical example of a work folder, let's say that you can see in the enterprises, you have on the left side the open source site where open source developers are producing images, they are publishing them to Docker Hub or GHCR, and on the right side you have the enterprise, so the enterprise consumes those base images and expects that actually there is some information in the base images to give them a clue, is that base image really secure, should I use it as my base image for my development purposes or not, so if you look from the floor from left to the right, so you will see that you can already, based on the new development in the OCI specification, also publish S-bombs, also publish even vulnerability reports in the open source repositories, when somebody in the enterprise decides to use this base image they can verify for example the authenticity of the S-bombs before they decide actually to use it for their builds and internally they can continue this process by continuing to generate vulnerability reports, sign them and enforce them at deployment to their Kubernetes cluster, so this is once again remember that picture because this is what the demos will be and how this is enabled very quickly, let's talk about how registries used to work and how they are supposed to work in the future, everybody knows that container registries started a while back when Docker actually become popular and so far we were used to actually push container images in the registries only, however a few years back people started actually pushing other stuff, Helm charts was another thing that got pushed to the registries and about two years ago Cosine and Seekstore, they started pushing signatures and attestations, but everything was pushed in a flat structure and it was really hard to enforce the relationship between the artifacts in the registry itself, here is where OCI comes into place, so thanks to OCI now we can also relate artifacts in the registries, for example when I have a container image and if I have signatures for that container image I can relate those and they can actually work together and this is enforced not by just the tagging convention, but referring the digest of the artifacts in the actual manifests of the artifacts, which means it's really harder to let's say override the tag and let's say change the signature for a particular image and not only that actually we can create a much deeper hierarchies in the registry, so I can actually push an S-bomb that is related to the image, I can also sign this S-bomb, so you can create these arbitrary structures and you can traverse these three very easily thanks to the new developments in the OCI APIs. If you need more details actually the link below, Brandon has a very good talk that you can watch on YouTube with much more details how OCI works and how it is implemented. So the referrer API is the thing that actually enables this functionality, so everybody knows the Docker V2 APIs, the new additions are the last two APIs, what they allow you to do is to get all the referrers for a specific artifact and one additional capability is to get not only all the referrers, but you can filter them based on the artifact type. Looking at examples for that, so this is the first example of calling the first API where I want to get all the referrers for a specific image and as you can see here I have vulnerability reports, I have SARIF, I have signatures for the SARIF, I have also S-bombs and so on, everything is linked to the image. If I want for example to filter something I can actually create a filter and filter only specific artifact types and just work with those artifact types. Okay, you try. Okay, so I want to show this, the new OCI stuff in practice, before we do that let's just examine something. Trivi, open source vulnerability scanner, this is the typical flow of what's happening when you are scanning an image. First of all, Trivi has to pull the image locally and then it can analyze the image, basically create an inventory of what's there and then it can match this inventory with an up-to-date vulnerability database. So essentially these three steps, pooling, analyzing, matching usually happens when you scan an image and then you have the report itself. Now, the one way we can leverage the new reference types is by splitting the process in two because if we are doing a nightly scan like I described earlier, the first couple of steps, analyzing, pooling and analyzing the image, they're going to be redundant in the second, in the third scan because the S-bomb is not supposed to change. So I can do the first couple of steps pooling and analyzing once, have an S-bomb and then when I do a re-scan, I can re-scan the S-bomb, basically just step three, just the matching. So to do that, we can pool the image once, analyze it once, create an S-bomb and then where do I store the S-bomb? I can store it right next to the image in the registry so it is very easily available there. And then when I want to do a scan or a re-scan, I just pool the S-bomb and re-scan it, match it with the vulnerability database and I have the result. In this case, I can also push the result, push the vulnerability report itself to the registry to sit there next to the image if anyone wants to make decisions based of it. We'll take the decisions part to the last part but this is the thing you should have, the mental model you should have as we go into the demo. Okay, so let's start. So what I'm going to do first is to run a regular image scan, so trivii image, name of the image, it's going to run the full scan, it's pooling the image, it's analyzing the image, it's matching it and we have, we got some vulnerabilities in this image. So the first step that we wanted to do is to split the process so let's stop after step one and two. So I'm typing the same command here, this time I'm saying, I just want the cyclone-dx format which essentially says generate an S-bomb and stop there. So I'm running this and then trivii is pooling the image, analyzing the image and just printing the result of the analysis, this is a cyclone-dx S-bomb. I can take this S-bomb, maybe store it in a file locally, so here I'm just redirecting the output to a local file. Same thing happens, this time the contents are stored to the file and now instead of scanning the image like I did earlier, I typed trivii image, instead of scanning the image I'm going to scan the S-bomb, so the command's a little bit different, trivii S-bomb and I'm pointing it to the S-bomb that needs to scan. Now it's doing only step three, it happened a lot quicker and we get pretty much the same result as in the first step except this time I can repeat this process much faster. All right, we split the process but we still haven't leveraged the new OCR references, so let's push the S-bomb that we generated to the registry to sit there right next to the image. How can we do that? So first of all you can use any generic registry manipulation tool, Toddy just talked about ORAS which is one of the options, Crane is another option, I'm going to show here RageCTL which is another option, so you can do something like trivii generate an S-bomb and then pipe the result to RageCTL artifact put, you need to tell it which image it is referring to, the source image, you need to tell it which type of artifact you are pushing, so this is a Cyclone DX mine type and you can add any annotations that you want with this artifact and this will be very helpful later when we want to find an artifact that refers to the image. So this is how a command that pushes the S-bomb to the artifact store looks like but there is a much easier way with trivii, so first of all you need to install a plugin, so I'm doing a trivii plugin install and the plugin is called referrer, it's a new one that we built basically for this new spec and now we can simplify it by taking the S-bomb scan and just piping it to trivii referrer put, that's pretty much it, trivii will understand that this is an S-bomb, that this is Cyclone DX S-bomb, it will understand which image it's supposed to represent, it will also add the necessary annotations, so let's run this and we have our first artifact in the registry, you see here that it was pushed and we have an image, an artifact digest for it, that's good, let's push some more artifacts just for fun, so here I'm generating another S-bomb this time in a different format, it's an spdx S-bomb and again I'm sending the result to the registry with trivii referrer put, run this and we have a second artifact and let's also generate a sarif report, sarif is a format that represents the vulnerability report itself, so steps one two and three right including the vulnerability information and yes I just said it will probably be out of date or irrelevant in a week or in a month, but we're still going to use it later on in the demo so let's push it as well to the registry, all right, so we have our three artifacts there, to see them I can run Red City Artifact Lift with the name of the image and it dumps everything it knows about the referrer of this image, so we have a lot of information here, trivii can simplify this as well so I can do a trivii referrer's list and to see everything in a nicer way as a table, so we can see our three artifacts here, we can see a cyclone dx, spdx and sarif reports, something to note here is the trivii also added annotations that to mention that this was created by trivii and most importantly when it was created, this is not part of the spec and it's super important because think about a scenario where you scan your images nightly and you push every night a sarif report or something like that and then you want to make a decision based on the vulnerability posture of your image, you would probably have tens of or hundreds I don't know how many hundreds of artifacts of the same type, so the type itself is not enough, you probably want the latest one, how do you know which is the latest one, you need to keep the timestamp there so with the trivii plugin we also do that, so we can list the artifacts as well, if we want to get the contents of one of them because for example I want to scan the sbomb right so I need the contents of the sbomb, I can run the list command maybe find the one that I need to and then I can take the you see every one of those artifacts has a digest so I can take the digest with a generic tool I can do an artifact get and the SHA itself which points it to the precise location, it would have gotten me the sbomb itself but again an easier way to do that if you're dealing with security content in the registry is with the trivii plugin so you can do a trivii refer or get name of the image just tell it I want to cycle on the x and it will do all of those three or four steps automatically for you it will use the refers API, see which artifacts are there, filter by the type and then sort them by date and then get the contents by digest and the result of this one command is just the sbomb itself so now that I have the sbomb for the image I can scan it right we've seen earlier that I can do a trivii sbomb and scan this thing that's one way to do that actually an easier way to do that is to tell trivii to scan the image like we did in the first time when it did all the three steps but this time I'm giving it a hint I'm adding sbomb sources as OCI I'm telling it scan this image but before you pulling it check to see if there is an sbomb attached to this image in the registry if there is an sbomb just skip the first couple of steps and use the sbomb so I'm going to run this command slowly to show you that the first thing that trivii prints here is that it found an sbomb in the OCI refers which is cyclone DX meaning that it worked like we have the sbomb there so it was able to leverage that and the results are printed immediately and now you see how actually you can use sbombs to really do the scanning of the image and improve the performance when you do to when you need to actually scan often the problem that we are facing now is like do you trust these sbombs right because everybody can go and eventually push an sbomb to the registry right if somebody takes over the registry they can replace the sbombs and and so on so just having the sbomb attached to the image is not reliable what we need to do is to really ensure that there is some integrity in the every artifact that we are pushing to the registry here is where two open source projects come in place one is the cosine from six store and the other one is the notary project how that works so I will show you how you can use cosine to sign all the images that ita just pushed remember the picture that we had at the beginning so you have the open source site where actually open source developers can push images to ghcr or docker hub they can sign them with cosine they can also push vulnerability reports sbombs they can also sign them with cosine and the next step will be we'll take these images we'll copy them in a private registry and we will re-sign them with a notary so let's start on that first we'll look at again what was the structure that ita introduced so as you can see we have an image with a cyclone dx spdx and a sarif j zone we will sign the image with cosine first so this is the signing of the image and now we will go and sign each individual artifact starting with the cyclone dx the actually the referrer apis they are relatively simplistic right now so we need to do some magic with jq in order to select the right artifacts there and to find the digest of the artifacts in order to to sign them so we signed already the cyclone dx now we are signing the spdx with cosine and the last thing that we'll do is we'll sign the sarif with cosine let it finish that will be pretty fast during this process actually this can be completely automated in cicd pipelines for open source development typical example is github actions now what we are doing is actually we are copying all these artifacts that are stored in ghcr into a private enterprise registry and this is the the new structure it keeps all the digests in the private registry so all the signatures are still valid from cosine and as you can see we are verifying the image here the signature of the image and we are resigning now with notary the reason we are doing that is very often enterprises they actually want to verify the artifacts before they allow them to be used internally and also they want to sign it with some internal key that is trusted within the enterprise because else they need to actually add too many trust sources if they actually take images base images from various open source developers so we signed the image we are resigned verifying the cyclone dx and the spdx and the sarif and resigning them internally so it's it's a lot of text but those are all the steps that actually are going through last we are actually linking the tree and as you can see in the tree now we have everything that we copied from ghcr and added the signatures for let's say notary so each artifact has two signatures the cosine signature and the notary signature now inside the enterprise i can go and actually verify the notary signature and move forward with my deployments there so this is the process that we went through once again to summarize so we pulled the image we analyzed it we generated the s-bomb signed the s-bomb pushed the s-bomb and then uh when we actually want to uh re-scan again we can pull the s-bomb verify the signature of the s-bomb and uh do the scanning all right so to recap everything in one end-to-end example um let's uh in the in the left part of this slide you see everything we've gone through so far we have an image it has some artifacts in the registry everything is signed everything is ready to be used now let's use it what do we do with this image let's deploy it to kubernetes this is on the right side so when we are about to deploy the image we will also want to enforce some policies uh at the deploy time at admission time on kubernetes for that we're going to use kyverno which is a cncf open source project that were quick enough to implement the new spec for us for this talk to demonstrate so kudos to kyverno for the hard work and what we're going to do is to create a policy now usually when you think about an admission policy it gets the the pot spec for example so you have everything in the pot spec to make decision on for example where is my image coming from what attributes are in the spec or think something like that in this case we want to tell kyverno that the attestation is not on the contents of the spec it's on something else so kyverno needs to go to the registry pull something else like the s-bomb for the image and then we can only make the attestation so let's see how this can look like so first of all um just to recap three refer to three we have all the we have our image everything is uh is signed and everything is there the s-bomb and the vulnerability report itself this is a starting point now let's look at the first policy the first kyverno policy uh that is just verifying the image signature a kyverno policy is just a yamel uh a kubernetes resource so it's a cluster policy kind and we are enforcing that um any pod this is the resource selector what this policy applies to any pod and what we want to do is to verify the images using notary there are different kinds of verifiers also cosine in this case we're using notary for the signature it will apply to any image coming from this registry our private registry which we copied everything to and uh the attestation is for the public certificate so we want to make sure that the image was signed with the same public certificate that we expect uh you can see that uh i i don't actually print the certificate contents here i could have just pasted the certificate contents here but just for readability i stored the the certificate as a config map you see here i'm referring to a config map in the kyverno namespace i created a config map called keys and it has the certificate itself so here i can just refer to it like so so this is a very simple kind of almost like a copy paste take it to your cluster and it will also work a policy so to create it we just it's a kubernetes resource right so we just kubernetes create it and once once it's deployed we can try to deploy our pod the same one that we signed it succeeded that's a good sign it it didn't just succeed it actually passed through the policy because it is signed with the right certificate so it worked we can also double check that it worked because the pod is running so let's clean it up and look at the next policy the next policy is going to ask kyverno to pull the s-bomb so it starts the same exactly the same until we get to this part when we verify the image using notary and here in the attestation we say that it needs a predicate of type cyclone dx this is the line that tells kyverno uh to find the s-bomb for the image that i deployed and first of all before we make any attestation we just verify the signature of the s-bomb this is different from before before we verified the signature of the image now we're verifying that the s-bomb is signed that we can trust the s-bomb itself after we know we can trust it we can check inside of the s-bomb so here i'm saying let's look at all of the components all of the licenses of all of the components and i want to see that none of them are gpl2 or 3 typical enterprise policy i don't want any gpl license dependencies running so this is how you can enforce something like that and let's create it kubectl create this policy and now we can deploy our pod this time it fails it should fail because we do have a gpl dependency which is good and um we can actually double check that there is no pod running right now because kyverno blocked it so let's clean up and look at the third policy so the third policy is looking at the vulnerability report itself not the s-bomb the one including the vulnerability information starts exactly the same until we get to this part when we attest the predicate this time i'm looking for a sorry kind of artifact in the registry and if we find something like this we first of all verify that we trust it using the signature and then i can look into the vulnerability report itself now because this is a sarif report is a very generic kind of standard um there is no easy way to look at the summary of vulnerabilities so if trivy generated it we know that trivy would record the severity of the vulnerabilities in a certain way so first of all before we do anything i just want to make sure that the tool that created this sarif and this is one of the fields that's recorded in the report equals to trivy this is just to make sure that we can work with this and then i can do another test that basically checks that there are no critical or high vulnerabilities this is what i want to see and if we kubectl create this policy and try to deploy our image then it failed it's a good sign because we do have a critical vulnerability if we are going to want to see the the other side of this so let's say we have a fixed version of this so this is a real image that doesn't have this vulnerability to make it work we had to rescan it and push the sarif report to the registry and sign it right in order to see this work and this is what this test image fixed means and if we run this image it works so it passed through the policy and it's the policy was happy with everything it saw there so claps all right yeah we can take claps all right so that's the grand finale and what did we learn here yeah so what we learned is with the new oci changes it's making it easier for us to start managing not only the images but also vulnerabilities s bombs and everything is stored in one place however of course we we hit certain challenges right now so oci is not very specific about the let's say the artifact types for example when you push s bombs to to the registries we cannot use let's say generic s bomb type we need to be very specific right now we use the aiana media types for the artifact types the question is are they actually specific enough or generic enough or go through all the scenarios so one of the things that that we hid is like as i mentioned the sarif stuff so sarif can't describe also other problems not vulnerabilities only so how do we differentiate between a sarif report that is for vulnerabilities and sarif report that is for other type of bugs in the software annotation standardization was another thing we had to create custom annotations like created by trivi because currently oci doesn't have any guidance on how for example we should store the tool that generated either the s bomb or the report in the artifact itself we looked at some suggestions that oci had like for example or open containers image created as for example date for that or vendor however vendor is too generic in our case so that's why we went with created by trivi another thing we learned is that getting the artifact is much the right artifact is much harder than just pushing it there because the new specs is basically just a way to put stuff arbitrary stuff in the registry and link them to the image now if i want i know that i pushed like every day sarif report there and now i want the right one to make decisions on there is a certain step that i need to make some of them are manual some that can be automated first of all i can ask the reference api give me all the refers of this image the only filterable thing i can do there is by type but as tody said if i'm asking for a sarif i don't really know what i'm going to get it's like saying give me a jason what is it is it a vulnerability report or what is it another thing is like after i got the refers i can maybe look at the annotations that it has but this is up to the client that pushed the image to populate the right annotations and this is a place where conventions will come into play probably but this is another step that you need to take on the client side to further locate the right artifact that you're looking for and that might not even be enough because sometimes if it doesn't have the annotation you need to actually look into the contents of the file itself to to decide if it's right for you or not for example if it didn't have the time step as a notation maybe you need to look at the time stamp from inside of the file which is another step this is pretty much all of those three steps are encapsulated in the 3vrefer get command this is what we actually do there and finally the specifications are not exactly accurate i mean a 3v s-bomb is not the same as a regular s-bomb we had to push some more information there to make the vulnerability report more accurate some of it is the related to the s-bomb spec some of this is like adjacent to it like the purl spec or something like that but it's another thing to keep in mind that even though everything is standardized it's not always the same exactly the same implementation all right so thank you very much thank you very much