 Welcome everyone, nice to see you in person and online people, nice to not see you too. I'm Justin Cormack, I'm a CTO at Docker and I've been involved in Naturey Project for many years through its different iterations as the original version and then the now V2. Naturey originally was a project that started at Docker but is now a big community effort across all sorts of companies. And this is Steve. So I'm Steve Lasker, I work in Azure and a couple years ago we thought we was important to revive the signing efforts in containers because it was a pretty important effort and this was about two and a half, three, well it was pre-COVID, so it was about three years ago. And we basically got together a bunch of the people across the various clouds and providers and we were like, we need to actually solve the signing problems with the core issues of promotion workflows and things we'll talk about today. So with that, we can jump in. Yeah, so we're gonna talk about a little bit about the goals in Azure V2. We're gonna talk about some of the workflows that people wanted us to support and want us to support. Gonna talk about the kind of supply chain artifacts that people are interested in storing in registries. We're gonna deal with the kind of question of who do you trust and why for signatures and we're gonna go through some of the open issues that are still being resolved and then what the state of the project really is. The goals in Azure V2 have been really to build upon existing fundamentals. We're not trying to radically make things different. We're trying to build things that people understand and know how to use and use existing specs as much as possible but put them together in ways that are really useful for the container ecosystem. So we've been doing lots of work around X509 because a lot of people ask for support for that. We've been a very, I'd say, very customer-driven project. As businesses, we've all been talking to our customers about what they would like in signing and that's really driven the kind of way we've looked at trying to solve the problem. We're looking to really invest in and extend existing services in particular registries. Registries are really important and everyone has them integrated in their workflow and they understand the security properties and things like that so it's very important that we just build on that infrastructure rather than try and replace it so we're just trying to add things in an additive, migratable way and we've been really working on kind of working out what best practice actually looks like. Obviously, all these things are kind of new and you have to make best practice up as you go to some extent and iterate on it but it's very important we feel that and it was one of the problems with kind of Azure V1 that the signatures and the other supply chain artifacts were not really stored with the images. They were stored kind of in a separate sidecar service and that's been a kind of driving point the whole through the V2 is like this integrate supply chain metadata into the image. It's really all about getting integrity from build all the way through to production making it the same image and you understand what's going on happening and people have a lot of workflows where they take say public images like official images moving to private registries or have different dev and staging registries and all those kind of things where people are moving images around that's been really important support again that was one of the problems with Azure V1 was that signatures didn't move with the images and these kind of workflows where you want your images to be in different places for different functions for different production systems so if it become really important we'll talk a bit more about that in a bit as well. So if we think about like signing what exactly does it promise? Sometimes we hear this term well it has to be signed and as long as content is signed I'm fine. Well is that really true? So who is it signed by? Who do you trust of all those things that are signed and who don't you trust and who do you trust and not trust for a particular environment? What you might want in staging is different than production or development or different production environments. Financial services has very different requirements than say gaming so there's very different profiles that you have to think about. So in the concept of who do you trust? If I navigate to evil.com it's a real site it's got random stuff on there but the evil.com is not signed. There is no HTTPS certificate on it and the idea here is that we wanted to be able to make a comparison to how we think about the browser model because there are a lot of analogies here. And the browser model's been very successful. We've got encryption everywhere in a short period of time so it's kind of interesting to look at how this compares to containers and what the differences and similarities are. So in this case there's no cert the browser tells you there's no cert you can kind of navigate through and you can find more information so you kind of make a very easy decision. There's no certificate there's no identity to who this evil.com people are so no cert, no trust, kind of logical. But let's look at some other ones. So my kids are now out of college but my daughter was doing high school research or I don't know if it was high school at the time pre-browser pop-ups. So the only thing on the internet at the time was .com so she went off and did whitehouse.com and was kind of a little surprised about what she found. Next thing I heard in the house was dad, dad, please make it stop, make it stop because pop-ups kept on popping up and every time she closed one another one popped up and it wasn't something that my kids should be seeing. So it turns out that that site is signed it's got a certificate. If you drilled through you'd be able to see it's got a full certificate chain internet security research group like SoundsLegitimate totally is, go through and there's lots of properties on the certificate chain that says like this is a legitimate site. If I go to whitehouse.gov, the real one for what at least my daughter was trying to find and where I was more comfortable with her looking it's also got a certificate and the certificate actually is the same root chain. There isn't any real discernible difference between the two. They're both completely signed. So it's like which one am I supposed to trust there? If I go to microsoft.com there's a search chain there as a company there's some extra information that's popped up that I can do more research on if I so choose. But if I look at the search chain there's a certificate there. It roots up it happens to root up to a different root chain but that doesn't really matter much because there are a lot of software companies and other companies that have a lot of business that they count on right? What is the internet based on? There's various sites out there that it's very important for them to encrypt and have good security models on there. So it's very difficult to discern any of these from any level of difference. So we have different levels of trust and different kind of rules about who you trust. It's not quite the same as the browser model because you're not just trying to authenticate that you just get encryption you're actually trying to make decisions about policies about things like what you put in production. So it's what you put in production on your developer laptop might be very different from what you put in production on a banking site that's serving your end customers. So we need to actually think about how to have rules that let you configure different kind of policies for these different kinds of use case. And they're similar in the same way also, right? Like what I might want to allow is sites that I can browse and do research on are probably different than the filtered sites that I allow my kid to look at. Dev production, very similar things, right? Then browsers have ways to set up safe limits and you can opt in sites that you allow or not and your company can provide group policy on things. It's very much a policy based configuration. We also have different kind of routes and kinds of trust and we know that we really want to make this very configurable for the end user to make a choice. So you might trust Microsoft to sign Excel and Word and nothing else because that's the software you get from Microsoft and you trust the upstream. You might want everything else to come from your own organization certificate. So perhaps you want to have a spiffy setup with ephemeral signatures for your build going back to your route and that's fine. You might want to use Tough which has a trust group per application and it's built through on the application. You might want to use some different mixture of these for different kinds of application. And then another thing we're seeing a lot now is you might want trust, not just based on signatures but on more detailed metadata. You might want to trust things that you've inspected the bit of materials on it and you've made certain choices about what's in the image and whether you're prepared to run an image based on the contents or other kinds of attestation about how it was built, where it was built, who it was built by and we're seeing a lot more of that. So we really want to provide a very flexible, configurable model where you can make these decisions and perhaps change them over time. I mean, think about it, in your dev environment you're probably doing some pretty scary stuff with compilers and file parsers and other powerful tools that you need in your build environment. That same software company might produce images or project, project or company might produce images that you do want to use in production as well. So the same signing authority in that case won't let you differentiate dev to production. There might be some other claims or metadata that you want to make sure is available as well. That claims a metadata, you want to make sure it's signed because this guy Justin over here might make some claim about something. Doesn't mean that you actually trust his decision. So you want to have identity associated with all the information that you have. So this one's kind of interesting. So this is where we think about, and this is how I think in some ways in Notary V1 the environment wasn't mature enough. This was several years ago at this point. And in Notary V2 we've learned. If we look at what workflows customers are trying to do is they're trying to bring the content into an environment they trust. So in this case, we basically have these two companies, there's this small software vendor called Wabbit Networks and they produce this net monitor image. Intentionally a small company you haven't necessarily heard of unless you watch Bugs Bunny. And then Acri Rockets, a typical customer just trying to run software. So in this case, they're trying to pull some public image and they can deploy it on their node. Maybe they should want to check the signature first to see if it's really from a company they trust before they deploy it. So in that case it was a little reversed. But what we also see is the customers that are trying to do this have security expectations. And this was the interesting thing is, been at Microsoft a while, I've been working on enterprise customers for a long time is the containers are just the latest tech. They have a set of security bars that they want where things are signed, things are in vNets, especially in clouds, right? When you're on prem, you're on vNets. When you're in the cloud, they want to simulate those environments. If I've got a vNet around my production environment or even my dev environment, how do I go and get all this new information like S-bombs or claims or scan results if those are outside of that environment, right? They're blocked. That's the vNet barrier there. Yeah, we're seeing much more of a trend that people really want to control what comes into their production environments. They want to gate everything first. They don't want to pull in uncontrolled ways. They want to basically pull and gate, which is partly why we need the signatures attached and all the metadata attached to the artifact so that you can bring it all into the environment, vet it sort of at entry points and then basically use it safely from then on. Yeah, I mean, it's great to pull, like having the internet as the source of content, awesome. Do you really want to pull something from the internet directly into your production environment, right? Like, who's got a refrigerator at home or in their hotel? We keep things close. When you want something to eat or drink, you want to make sure it's there. You go and replenish it, but there's nothing much difference between the technology and what we do in many other real-world scenarios. So everybody's got a fridge in their environment where they're storing their images and all their artifacts. And for each one of those environments, they probably have another registry. So they have a registry, whether it's one shared or different. And then that content that's coming from various sources, they're not, they could pull it into each environment, but what we've also seen is many of these customers are setting up what they call golden images or other registries where they're keeping that content they depend on. And then all of those different teams and those different environments are promoting that within their workflow. So, and there's multiple companies, right? This is not, people aren't using just one project or one company, there's multiple out there. And they want to be able to look to some third parties necessarily to, hey, is there some clearinghouse for some of this? So this is where Docker Hub can come in really well and they've had this historic model for there's a certain set of content you can get from Docker Hub that they've put effort into to make sure it can be trusted content. And then there is the open space for all kinds of other things that you can discover as well, but there's a differentiation there. So in this case, the small company Web at Networks can publish their stuff to Docker Hub. Notice there's a single signature there because it comes from them. And notice it's not just the container image, but it's the signatures, it's the claims, it's the scan results, it's other metadata that they want to make available. There's a subset of that content on Docker Hub that they will say is trusted content. They've done some extra effort, whether they built it or scanned it or done some policy on it and said, look, we've taken some special look at this and this content is different and therefore it gets a Docker Hub signature. There's other content that isn't certified by them or trusted, right? It could just be out there for discoverability purposes. The whole idea here is that Acre Rockets can consume content from multiple registries. We see lots of public registries popping up all the time. How can a company configure what that policy is? So in this case, what we've set up is there's a trust policy that says they actually don't trust Web at Networks because they don't know who they are, but they trust Docker Hub. And they trust a couple other vendors as well. So this is a way that they can choose what content they will allow in to their environment. And then as they scan that, provide the policies they want for their company, they could sign it with their own signature. Now they can say everything in my environment must be signed with Acre Rockets, let's say. You can basically configure policies like per environment in different ways. So you can say which keys you trust and we're integrating with admission control and all types of things that are gatekeeper and other policy managers. So you can basically configure those policies which are in the demo. Yeah, again, the ideas we're leaning into what's available today. So the signatures are associated with, we've basically got a model here where we have kind of separated signatures so that you can assign, you can reference the artifact that they sign. We make sure that you can have multiple signatures on objects in case you want that kind of workflow. Signatures move around with the artifact, you can, we sign, basically, we don't require you to pull the entire object to check the signature, we could just check the hash or the manifest hash. And you can have a whole different sets of artifacts attached to S-bombs and scan results and any claims, annotations, anything else you want. I mean, the model we did with signatures to be able to support to attach the workflows is we wanna be able to put things in a registry that a registry doesn't even know it's a signature, it doesn't care. So we can associate anything because if you look at your deployment scripts, they probably say like net monitor colon V1 or net monitor even with a digest. How do you find out the S-bomb for that thing? How do you find out the scan or the signature? You actually, your deployment files have a named reference so we wanted to build on that capability. So why don't we just jump into some demos now? And let's see here. All right, so every good demo starts with a Docker build. So we'll just come over here and I'm just building an image from in this case a Git repo, right? So nothing special, just kind of the standard baseline there. And then we'll do a standard Docker push, right? Nothing magic. The point here is that I'm building and pushing to a private registry. Doesn't matter what cloud, wherever you're running your workload, use the registries that are closest because your fridges nearby, your registries should be nearby for, we won't minimize the things that can fail between your source and destination and you wanna maximize the performance. So I wanna start with just a very simple sign and verify. Now, I'll use a test cert here that we can generate. It's a self-signed cert. Just, we'll just start here, we'll show where we can have a remote sign cert here in a minute. But I just got an X519 cert, wherever you might have gotten it from. Now I can sign because I've, if I look closely, notice I've said default. So when I generated, I've actually set it up so that I can sign with that key by default. I happen to, so I mentioned the key and I'm just saying notation sign, the image or any artifact that I'm referencing. This happens to be a container image, the net monitor V1 image. And it comes back, it's like, okay, it's signed. It's literally that simple. It's based on a notation client and a registry. I can see what has been, actually, let me clear this. Get this at the top of the monitor. If I look at what's associated with the net monitor image, because that's what's in my deployment chart. That's the only information I know. So let me ask the registry, is there anything else related to that? So we have this ORAS client, OCI Registry of Storage. And it can discover what is associated with that reference. And I can use a tag or a digest. And it comes back and says, hey, that image has an ordinary V2 signature hanging off of it. Now, if I want to verify that container image, I just say notation verify, pass it the path. And of course it fails, because we haven't, we've got to fail by default policy because this is a self-signed cert. We don't know anything about it. We haven't set it up in our environment. So we actually need to configure that we actually want to use the certificate. So we say cert add. This is what the interaction is now. We will have more of trust or policies coming this month. And I can now, now I've configured that that's what I want. Now, if I say notation verify, of course it'll pass because I've said that's what I want to be allowed in this particular environment. So the idea is your private keys should be kept private. This is kind of the model. Yeah, most people don't really want to keep their private keys on their laptops and things like that. So one of the really strong asks we had was that for most, almost everyone really wants to keep signing keys in hardware, which means in some sense remote it might be as remote as a UB key or the TPM on your machine, or it might be actually in a cloud provider key storage or somewhere else that's not basically on the file system accessible to the client. So it's really important that we support that use case because pretty much everyone wants keys in hardware for everything nowadays. And this is where we just wanted to lean into the infrastructure that our customers already have. We've already got a lot of infrastructure here. We want to leverage it. That's their security bar. So the model is you keep your private keys private. You might have access to sign them and those key providers have lots of logging to know who signed them. But you can't, you don't have the rights to take the private key out and put it on the internet. You can do signing and it's locked down to nodes in a machine or certain environments. And these can be ephemeral keys and things that are generated through Spiffy or something like that as well. So they might just be a key that's generated just for that signature. Exactly. So the next part is, well, there's lots of key vault providers out there. So there's lots of signing services out there. We don't want notation to have to track every one of those. We don't want anybody who wants to write a key vault provider, sorry, a plugin to have to come to the notary maintainers and like, can I please check this into your source code? There should be a specification that says, here's how you write one. So that's what we've set up and we've basically, it doesn't even matter what language you write it in because it just does binary calls and does standard and standard out to get the information to it. So really nice, it doesn't require you to even use, in this case we use go for the notation client. And as you want to rev those, you can completely build it all on your own. You can completely keep it all on your own. You can service it and nobody has to know about it outside of whoever's doing it. So let's take a look at how we can do that here. So what we've got is, we've happened to write one for Azure Key Vault. AWS is writing theirs. We want to get one for Hashi Key Vault as an example. We haven't had a chance to do that yet. So if somebody wants to write one, that would be great. I've already configured that with a Azure Key Vault provider. Again, anybody can write one. And I'm just gonna, I need to get the key ID. Now, this is the path that when it's doing a remote signing, what should it use? Now, in this case I happen to be using Azure Key Vault. You can use whatever you want. And then it comes back and it gives you the key ID. So if I say key ID, if I spell it right, it's just a path. And most key vault providers have some kind of way to reference it. And then I'm just gonna say, let's go add that name to the notary config. I've said, this is the name of it, the key. And by the way, I'm using a particular plugin name. So there are no to route. When I say sign of that name, route it to that plugin or it to do all the heavy lifting. If I look at the key list, we point out that we've been gotten used to very wide monitors at home. So we probably need to rethink about how we format that output. But then I can just simply say, no different than I did with the self sign key, I'm now doing a sign with a key that we got locked up in a key vault. I don't have any access to other than to do sign. I just reference it by name. And when I configured it, it notices that name, uses that plugin, so go route it over there. And now I've got my signature set up. And if I look at the list of keys associated with that image, right? That's the only reference I have is net monitor V1. It will go and say, well, I actually have two signatures now. Now, then it's a matter of, well, which one do I want to allow? It's up to you for what policy you would configure. So we can see there's our net monitor and then there will be a hash of the different type, artifact types, in this case, it's signatures, will show when we have S-bombs and so forth in a minute. Okay, so that's remote signing. So if you notice the simplicity of signing with a local cert and signing with a remote cert, it's still just notation sign, notation verify, but you're configuring where you want those to come from. Now, that's great. I can do that on my laptop, but I don't think we're running too many workloads probably from our laptop. Maybe some ML scenarios. I mean, that's not completely true, but there's definitely some. How do I get this into some kind of production or other environments? And this is where we wanted to integrate with the existing systems. So we've got a Kubernetes cluster here and I've got two namespaces. I've made it very... Try security, wanna... Yeah, there's a security and a not-secured namespace. We're gonna leverage the policy managers, in this case, Gatekeeper, and we've got a project we call Radify, which knows how to read those references and do validations, could do validations on signatures, could do validations on S-bombs. It's a completely pluggable model. So let's go and jump in there. All right, so we're gonna create the two namespaces. So that's simple. And then we're gonna take that public image we're gonna run an nginx image from Docker Hub, right? So, and we're running it in the not-secured namespace, so surprise, surprise, it works. And then we can see our pod running. But now I wanna lock down that secured namespace, right? We wanna be able to say that only things running in that, should be allowed to be running that namespace that are actually signed by a key that I'm configuring. And I happen to have that public key stored in a key vault in this case. So I'm gonna go grab that key because these are, you know, affirmable nodes. I don't have them deployed everywhere. If it's public software, like Microsoft, we're signing our content, then that public key will be available and you can curl that into your environment. We need to clean up some of the key format conversion stuff. There's a little too many pipings going on here, but suffice to say, I just needed to get a public key into an environment variable. So then I'm just gonna take, I'm gonna do a helm chart for ratify and I'm gonna give it the public key that I want it to secure. So I'm just doing a helm install and then I'm saying, you know, there's our registry credits set up, but basically here is the public key that I want you to use. Then I'm gonna create a constraint. The constraint says I wanna take the namespace secured and apply this ratify constraint to it. So that's starting to glue the connection there. So I just got this config file and then I'll apply that to the cluster. Just apply this constraint which locks down that namespace. So now if I try to run that nginx image in that secured namespace, forbidden, right? The failure that you want, right? This is the good failures here. Okay, so great, we've locked that down. Can I run anything in there? Well, remember, we signed our net monitor image with two keys. We signed it with a self-signed private key, which it goes, whatever, because I've configured that cluster to use the active rockets or the web networks key in this case. So there is a key that I've set up only allow stuff that's signed with that key and so it was scheduled and life is good. Now, we talked about the promotion workflow. What if I do want to run the nginx image in that environment? Should I just allow anything that's unsigned if it equals nginx? Well, how does that, is that really what I want to? I want to constantly pull updates from Docker Hub and deploy it. Well, there was a hundred different connections that could be down, having nothing to do with Docker Hub between that point. There could be a security update to nginx that might actually not work for my environment. Remember, SolarWinds was an update that got applied. It wasn't software that was bad to begin with. So I'll pull, we'll simulate that import workflow. So I'm going to pull the nginx image and we'll then scan it. So we're basically just looking at it to see if we think this is okay to run in a production environment based on criteria around scanning. And it'll take a look and it's going to run against the dependencies and whatever, this is a point in time. Vulnerabilities are discovered on things that are already shipped. So I only know what I know today. So there's a set of issues, but effectively it is the best thing that we have today. It's like, okay, I'm making that decision to be able to promote it. Now, I'm actually going to do something a little interesting here is I'm actually going to take the output of that scan and I'm actually going to save it to nginx scan. Okay, I forgot the .json file. So I'll have to, I'll remember how to fix that. So it'll save it as nginx scan, not .json, it should have been, but I forgot to do that. So now I'm going to tag that image so that it is going to be set up for the registry that I want to deploy from. I don't want to deploy from Docker Hub. I want to import from Docker Hub into my environment. I've scanned it, it's good. And then I'm going to push to my private registry, right? We're doing that promotion workflow. And then I'm also, and I'll remember to fix this. I'm also going, so it's doing its push, it's all good. So now I've got the container image in my private registry, almost. We've been good, lucky with the wifi so far. We're going to show how duplicated images are, or we're going to show de-duping. So it'll finish it up again, but it'll show that those layers already exist now. So give it a second. We're not being lucky with the wifi. There we go, see, right? So it just has to do the item potency thing and finish up the last piece, come on. There we go. Okay. So the image is there, but remember, I also want to associate that nginx scan file that I forgot to put .json on, that's why I was just fixing there. I'm going to push that up also. So this I'm using the RS client, and basically I want to push to the nginx repo. I'm going to name it SnickScan v1, and the subject is that net, sorry, the nginx121.6 image. So when I push that up, I'm now saying, hey, please store that scan result with the image as well. Going to sign, come on internet. There we go. So now I'm going to sign that image. So now I've actually got two things that I've associated with it, right? We've got the scan result and the signature. So now if I do an RS discover, again, take the nginx image, the tag, and that's the only name reference I know, what else is hanging off of that? And you see that now I've got the SnickScan and the signature, both, right? So that's kind of nice. I can now have my objects together. The only instance that I have here is a registry and the notation in Norris clients. Okay, so now I can get back to, I do want to run that nginx image. So now I can say, run nginx, I'm naming it nginx, but notice I'm pulling it from my private registry that has been signed with the key that I trust because I've done the verification on that workflow and it's often scheduled, right? That's kind of the beauty of it. That's basically it. So cool, let's kind of think about that for a second. So we have several different tools that we were using. In this case, we were signing and we were generating scan results. We want all of that information to be promoted. Remember that animation we did where we said we want to take the public stuff, put it into our private registries because that starts to get inside of our environment. We've secured it and we can promote it. Well, I want to show one more quick one here where if I want to copy that content from one registry to another, this Aura's client doesn't know anything about scan results, doesn't know anything about signatures. It just says copy things from this named reference in Web at Networks to Acme Rockets. And because it can read those references, it doesn't need to know what they are. They just know that here's a bunch of things that are associated and I'm going to copy them across. So we'll just finish that up. Again, fun with this. Bring these go on. Okay, I'll come back and show that in a second. So, go on. So yeah, so we've got policy and management for our environment. You can configure which keys you trust for our environment and it's just integrated with normal tools such as Gatekeeper or other, many other tools available, whichever tools you're using today to do that. So where we are really, we're finishing up release candidate one this month. Microsoft is shipping a load of example sign things that you can test right now. We'll be starting to ship Docker official images signed with us shortly as well. So there'll be more tests, but if you want some test images now and I'll see one will be compatible with future format. So it should be good to test going on. There's more links there to give you some information. We have to, we're a software company as Microsoft. We need to ship our software with signatures and S-bombs, the executive order in the US and all kinds of other things. We wanna make sure we can start testing this workflow. So that's kind of some stuff that's available there. Yeah, go ahead. Yeah, we're finalizing the specs at the moment just trying to put things together. And we've got a alpha release working towards the RC and we've shown the things that some of the key vault providers and some of these other example stuffs. There's more going on with AWS is doing work on there. Key signing provider, as I said, we're working on getting the official images signed and things so you can kind of test this all out and provide actual workable useful workflows to start with. So it's just, it's okay. So the copy was done. Let me just kind of do one more. So we were able to just let that sink in there for a second. We're gonna be able to copy that graph from one registry to another. And so notice I'm now said, hey, Aura's discover what's on the Acme Rockets Registry for what I copied. So it will give us that list. It didn't finish yet. Sorry, slightly. It's traveling across the pond. There we go. Okay, so and now you can see that I actually didn't clear out my previous demo because there's actually two of them. So I ran the demo once before too. So you can kind of see it. That's even more important. Like that was emptied before. Now it's copying everything over that I want. And of course there's filtering commands and so forth. So you can copy the subset if you want. We still got a bunch of things that going on. We're looking at additional work on identities. Not everyone has an X509 infrastructure. Lots of larger companies do and lots of software publishing organizations do but individual developers generally don't. So we're looking at support for SSH keys and policy around which identities you trust for what. There's kind of a lot of options there. Still lots of work to do on that. We're looking at things like inline signatures for when we are originating content with signatures straight off. But this is all designed to be extensible framework that supports lots of different use cases. So come along and tell us what your use cases are, what you need. And just the idea there is the same way that we want policies on the son because we want policies on different identity types. So that's what you kind of see in there. And there's all the links. The wall of links. Yes. So I think we... I think we're kind of at time. Thank you. Appreciate it. I don't know we have... I think we've run out of time so we haven't got questions now but we'll be around. We'll be around in the hall and you'll find us on Slack on our info that's here and we're happy. We'd love to hear more of your info. So thank you.