 So, we're here to talk about securing the container supply chain with Notary. I'm Justin Cormac, I'm the CEO at Docker. I've been a Notary maintainer for many years and this is Toddy. Yeah, I'm Toddy Moaldanov. I'm a principal product manager at Microsoft. I am a maintainer for the last couple of months on Notary and I work on secure supply chain for containers for Microsoft and Microsoft customers. So, let's start with, you know, the end-to-end software supply chain, what we're, you know, what we're really trying to achieve. We've got, you know, developers, you know, building applications, pushing them to registries and what we're really seeing is that people are building up and trying to, you know, trying to add these components to build the secure supply chain on top of, you know, what started out as an ecosystem which sadly was built without any security properties at all. Yeah, so when you think about, like, when you have piece of software right, how do you ensure that it comes from, let's say, trusted source and how do you ensure that when it moves from one place to another place to third place to fourth place, actually the integrity of the software doesn't get compromised, right? That's what we are kind of thinking about the software secure supply chain and you need to ensure that actually it is trustable and not tampered with throughout the whole pipeline. And this example is specifically around, like, developer whether it's open source or inside the enterprise producing a piece of software, signing it, for example and making sure that actually it's signed during the build time and then it can go through the whole pipeline into the registry, get deployed to the Kubernetes cluster and you can add that mission on the Kubernetes cluster and ensure that, for example, the software is really produced by who you think it is and it's not changed. Meeting your policy requirements. So the Notary Project's really been driving innovation around a set of standards around the secure supply chain and we haven't been doing all these things like in the Notary Project and so we're doing them all over the ecosystem where it's most appropriate and where the standardization is best done. So we did, for example, there was the first question years ago it was like, where do we store signatures? How can we build this into the standards around OCI? What are the requirements? How are we going to do this? OCI had a bunch of limitations that made it kind of difficult so one of the big initiatives we did was work with the AuraS project around the artifact manifest and the referrers API. So the referrers API, one of the customer requirements that we got strongly from customers is that they want to be able to add things to containers and container images after they've been built to add supply chain information that wasn't known at build time and there's a lot of kind of thing to frame but it's like you've got immutable artifacts and if you change them they're a different artifact it's the kind of model that we built for containers and so the referrers API was built as a way to say well I've got this immutable artifact but I want to point to it with a referrer and add additional data to it that adds for example a later SBAR a detached signature and then if I want I can refer directly to the signature which points at the artifact or I can go to the artifact and ask the registry what are the things that people have added what's the additional data and so we prototype that we proposed it to OCI and that's part of the OCI 1.1 processor that's kind of shipping in OCI and you can now, this is gradually being supported in more in the registry ecosystem so there's work in distribution in order to have a reference implementation for that so that people can, we're in process of implementing a Docker hub and you've implemented it in ACR ACR so there's places you can test it now there's places where you can test out the work and make sure that it's providing the customer value and then the next stage is really helping people build stuff on top of that and working out the next stages and what the additional work is The most important part is actually using the referrers APIs and ensures that actually those artifacts are linked using kind of the digest which are kind of the shas and they are not linked by let's say the tag as for example some other products are trying to do because tag is mutable so you can very easily actually override the tag and say okay now actually I'll override the tag with different signatures which is my personal signature whether I'm trustable or not separate question Yep, so And I think a lot of what's come out of the process is there's a lot of different kinds of customer requirements around what the kind of workflows people want around signing and secure supply chain are there's still, there's a bill of materials which is becoming very important but then there's like Entei to ASA stations which people are really starting to become much more interested in adding to their images and so on so there's a whole ecosystem of pieces around images that we're building up standards for how to support them and still more work to do on standardization and helping people use OCI most effectively to do these things So really kind of emphasizing that notary is based on standards we're not trying to necessarily make the notary project a home of standards but there's a place where we collaborate on building those on building those standards so we've taken Seabourn for signing from IETF for example and JWS and things like that so we're taking existing existing standards from outside the ecosystem as well Right now we support both like JWS and COSI which is kind of the signing and encryption format for Seabourn the one reason actually we went with Seabourn is it's actually a very compact and binary format which means that signatures that are using COSI they can be also verified on a low power device if you have Raspberry Pi or even kind of smaller devices it doesn't need the computing power that for example to parse the JSON so you can save on energy when you use the COSI format Since last KubeCon this is kind of the progress that we made I don't know whether some of you attended last KubeCon they were actually just in one of my colleagues did demos but we are in RC stage right now so we had our first RC was in December we finalized the signature specification there and also the way that we can actually implement plugins to Notary so you can implement your own plugins in Notary if you want to for example support key votes that we do not support right now it's very easy to go and plug it in and use it in Notary One change that we had to do I don't know how much you are aware but there was significant kind of step back on the OCI side so in February we needed to do a change in the specification just to revert from using the artifact manifest to previous image manifest so that's on the spec side on the tool that we use for actually signing we call it Notation so we have almost four releases actually by end of this week or maybe next week we'll have RC4 and we are waiting on the security order to be completed in order to release actually the tool in the first version which will be kind of officially supported version for that but there are a lot of things that happen so we have remote signing trust store and trust policy so I'll demonstrate this in the demos later on you can see a bunch of features that actually you can use there but it's a full-fledged actually signing experience that you can use in Notary demo it's the demo time so I will need to switch to my actually duplicate the screens because it will be hard to for me to look with my back so let me just do that change we can jump in the demo so the first demo is really how the signing works so oops actually no that's the wrong demo the demo goals are not good today so sorry about that I need to re-run all these things that I thought I run right before we started but and it is failing last time if not I will run it from a video so okay sorry about that let me go to the video recording of that so we can save the time okay once again apologize for the demo goals what we are doing here is actually I'm setting up my environment so I have two images that I would like to sign one is the application image which is stored in the github container registry and the other one is the test image which is stored also in github container registry the flask sample is the one that actually I consider my application the net monitor is one which is the test image and now sign it with test key I have notation version 100RC4 which is a pre-release for RC4 RC4 will come as I said either later today Pacific time or on Monday the next thing is I have a plugin that I have configured AKV plugin that's Azure Key Vault that's where my actually remote key is stored so it's stored in very secure location I don't have the remote key on my machine so I will demonstrate how we can use this remote key to do the signing there are two tags for the images they both are called KubeCon demo v1 for both the flask sample and the net monitor image and I think that's the complete setup for my environment now what I will do is first I will create a test key on my machine and I will sign the test image with this test key so I will call the test key Wabbit Networks IO and there is a convenience command for that so if you want to actually test how the signing works you can use this convenience command it will create not only the test key but also will create the certificate that you can use to validate the signed image so if you lose the keys you will see that actually this is the network's key it can give you the keypad the certificate pad and the plugin name because the test key is associated with the core notation CLI it doesn't use any plugins it doesn't have a plugin name and of course if you lose the certificates you will see that there are also the certificates so this is the command that I can use to do the signing so as you can see I will use the test key to sign my test image and I will use the causing format for this signature so it's as simple as that you issue the command as always it gives you warning because I am trying to sign the tag and as we already mentioned the tag is mutable tag and what we do actually we sign the image with its digest not with the tag itself but it will give you this warning to say you should be signing tags you should be signing digest because that's the mutable part of the image and of course we have the notation list command that you can use to list the signatures there so if you do that you will see that the image which is the net monitor image with this particular digest has a notary signature and it gives you the digest of the signature the digest are linked so if you try to modify the signature then that will be actually tampering with the signature itself and it will invalidate the signature for that particular image so that is the first demo and having in mind that the demo gods are not working I will run the next one sorry again with the video so the next one is troubleshooting actually notation so when one of the things that actually we noticed when all the changes with OCI happens is really hard for kind of users and also our developers to go and figure out what's happening because different registries they provide different support for OCI so we need to troubleshoot a lot and either you debug it directly as a developer but sometimes people like me PMS we don't have all the gold environment setup and we needed to get some more information so I implemented a very detailed debugging capability so if you run it with for example the signing command with debug tag or switch you will get a very detailed actually output what is happening so as you can see we trace all the calls registry what we are sending what we are receiving back and of course at the end of the day I get an error from DHCR oops a little bit so I get an error and I can go and actually figure out eventually that error what that means so I can tell you at the moment because we investigated that error so many times some of the registries actually do not support deletions and as you can see the error here is really that delete failed the way that Github container registry is actually 1.0 compliant registry so they do not support yet the 1.1 reference types but there is a backwards compatibility with that the only problem is that as I mentioned many of the registries do not support the delete the way OCI works is that actually for backward compatibility we use manifest indexes and every time when you add a new signature you actually create a new manifest index and the tool tries to delete the old one and that's where actually it's failing right now that doesn't mean though that the new signature is not added and we will just see that actually this call succeeded so we have another command for helping you kind of investigate issues with notation which is inspect of the signatures so I can run notation aspect and this is my application image and if I run it I'll get a very detailed information about the signature itself so you can see it gives you not only the digest of the signature but also gives you additional information like the algorithm that is when it is signed, when it expires so if you had hit some for example errors like verification errors you can actually go and inspect it with that and again just to show that the second signing with Teschi succeeded so if I don't expect I can see that actually there are two signatures here there is the first one it's a little bit cut on the video and the second one the last demo that I will show is also the verification so let's go through the verification demo so the first thing that I will do is I will set up actually look at the trust or vocation so we have the concept of trust of course and trust policies in notation as you can see I have only one trustor which was for my test key and if you what I'm gonna do now is I will add additional certificate that I will trust for my application image so this additional certificate comes out of Key Vault as I mentioned we use Key Vault in this particular case where we store the keys and the certificate so I will just pull the certificate down store it locally and add it to my trustor this happening ok that is done so if you list the vocation you will see that's my certificate file and I am just adding this file to the trustor so this is a public certificate so there is nothing secret to it so you can go and pull it down and now if you list the trustor vocation you will see that I have two trust stores certificates that I trust one is Wabbit Networks which is my test certificate and the other one is the GHCR which came out of Key Vault now if I try to verify the image that will fail now the reason it fails is because I have not defined any trust policy I just pulled the keys but that doesn't mean that I trust this key for everything that I have in my registries so what we need to do is really to define a trust policy that will tell us what keys do we trust for which registries what that means is that you can actually provide different keys for different registries or different images that you want to deploy or run on your work route in this case I am just creating a trust policy for my application image I will not create a trust policy for the test image and the assumption is that I trust only the application image and the key on my production work route so if I do notation verify application image it succeeds if I do notation verify test image that will not succeed once again I should come on too fast I guess you are able to see the error anyway so those are the two demos apologize that we had to run them from the from the videos and let's get back to the slides so the next thing that we'll talk about is the notary security so notary we are actually we started first testing with notary back in December and we completed actually the first pass right now we have first tests that are continuously running on all the sub projects in notary there were two issues that were discovered one of those issues actually was filed as a security advisory and we have cv this issue was fixed in one of the RC versions the second issue was just a dependency issue and we updated the dependency and everything is good so there is a first report that you can go and read there is also a CNCF blog post describing that we would like to thank other logics for the first testing and all the great collaboration that we had with them we are also in the process as I mentioned to a security audit so we are not required to do that as an incubating project but it was something that actually we are really happy to do and we would like to do this on a regular basis not just once the security audit started in March we have the preliminary results from the security audit there was nothing severe discovered in notary as a security issue there will be report that will be published so the audit will be scheduled to complete by end of this month and the report will most probably show up in May we are just waiting for this security audit to complete and to have the report published in order to release the first version of Notation CLI on the community update so over the last year we had 52 contributors to the notary project 15 of those contributors were very active so they had more than 10 PRs in the project as we mentioned so since the last KubeCon we had 9 releases and we are just waiting for the security audit to get the first official release and supported release and that's for Notation we have also libraries which you can use in order to incorporate notary notary signing into tools like let's say Kiverno, Kiverno is doing that right now so they can verify notary signatures and we had 3 revisions on the signing specification and of course a website update what is next for Notation after 1.0 we will continue to add new capabilities like timestamp authorities, we will add certificate revocation we will add capabilities to verify more than one signature we are working on many usability improvements that is one thing that actually we will have significant improvements there and of course more debugging and verbose options there is one more demo that I will go through so one thing that we implemented is also signing so we have 6 minutes and we have few more things so what is the preference should I go through the demo or do you guys want to hear more what's coming from Notary and I will be happy to do a demo offline also shall we go through the slides and if we have time we can do it let's go through the slides so the one important thing for us in Notary is really the portability one of the things that we would like to make sure is that signatures that are created with Notary can be moved from registry to registry to registry they can be verified doesn't matter how many hops they had and the most important part is you can use it in an air gap environment of course you cannot rely on any communication outside this air gap environment like public endpoints for verification so everything that we create in Notary is with that in mind that customers are not actually freely open to go and communicate from their let's say Kubernetes clusters randomly to some internet endpoints we're also looking at with the standards around identity and verifiable identities so one of the important things around supply chain actions is like who did the action can you verify who it is can you trust who the person is supposed to be rather than dealing with typo squatting and things like that so we're also looking at working with the SCIT project which is an IETF project which is basically builds a transparency log and helps record identities and signatures so that's an ongoing IETF project that there's a lot of interest in as an external standard for supply chain security that we're working several people are working along with to work out how we should incorporate and build that with Notary there's a lot of interest which has been driving uptake of this around the additional metadata around supply chain so S-bombs, SBDX and other documents also in Toto attestations and more fine grained information about where things were built and so on like the testifiers that we're doing so we're really looking at how we can add all this kind of information to software in the container ecosystem how we can help people build all those things together and tools so you can have more fine grained controls about there's a specific issues with specific releases because you can trust the S-bomb work out what's in it work out whether you trust that particular release because you trust the dependencies or not and so on so this whole flow of information and the decisions based on real-time updated information that you have about what's going on is really important so that's another area again because I think we need more standards we're looking at we've done some prototyping around in Toto attestations from Docker build but we're looking at again putting that through the standard process now we've started prototyping it we've also just started we're doing some internal prototyping work but we've decided to go forward with a project for storing tough directly in the registry at Docker so that we're going to propose this as a new sub-project of NATURY so this is to directly store tough repository metadata directly in registry one of the initial issues we had with NATURY v1 was it stored in a separate database and part of the original plan was like how do we migrate everything out of these sort of sidecars that are not part of ACI and are separate and put them internally tough provides lots of advantages that we kind of don't have without that ability to distribute and rotate keys and delegate different people for different targets and sign the individual tags so this is all part of the fact that we see NATURY projects as a home for a set of standards around supply chain security that you can put together and build all the different things that people are asking for in this space we are right on time actually those are a couple of other sessions that unfortunately they are already passed but you can go and watch the recording so we had a vulnerability management session where we demonstrated together with 3V how you can manage vulnerabilities sign them, verify them and also use Kiverno to when actually deploy some work-outs on Kubernetes you can verify that let's say the SBOM or the image is signed that the vulnerability doesn't have high severity vulnerabilities of course Kiverno has a session that they also talk about NATURY there and there was another session with Ratify and Gatekeeper kind of using the same approach we don't have time for the last demo the last demo is actually how you can sign local artifacts so you don't need to have the image or the SBOM in the registry before signing it which is important for certain scenarios so if somebody is interested I'll be happy to run it like after the session or you will find it linked from the slides and you can watch it yourself. Thank you very much