 Alright, so when Justin and I titled this talk, it was early in the year that was just like, hey, there's some people working on some stuff and it's kind of important. We should really keep on going. My health things have changed this year. So it's really exciting actually to see all of the focus. I'm Steve Lasker. I'm a PM architect in Azure. I focus on our container registries, ACR, MCR, and a lot of the content that flows through there for various other services that we provide. Justin couldn't make it today because of the travel restrictions, so we've been giving this talk a couple times now, so it's unfortunate he couldn't make it, but we'll get started. So yeah, when I got this email a couple of months before this came out of, hey, there's this executive order coming out. I'm like, which executive of Microsoft's putting something out? Like, okay, this will be interesting. And I was reading the email more and it started talking about things like in a government way and yeah, you scan email to see if you're really going to pay attention to this thing or not. I was like, government, didn't we do all those Congress stuff where they were interviewing tech companies? Like, this is not going to go well. And sure enough, it was a real thing. And there was a lot of stuff that's been happening over the last several years, but the past couple years has been ramping up. So it was great to see it. So there is over the last several months, there are a lot. I mean, there's been many efforts for a long time. This is not a new thing. It's just brought a lot more focus. There are on the various efforts of focused on various stages of the supply chain. There is some overlap, and that's good. It's much better have overlap than gaps, right? Because we really need an end to end solution. I think of this a little bit of the gold rush. There's a lot of focus and a lot of piling on and there's a lot of opportunity to do great work. And that's awesome. But we also think about today, like, how many people know people are actually doing gold mining? Gold mining, not Bitcoin mining. I mean, we know that there's a TV show around gold rush, and that's some interesting characters, and that's about it. But every one of our devices has gold in it because it's a good conductor. There's gold is prevalent everywhere. We just don't think about it because it's just part of the ecosystem that we work with. And that's what I'm hoping we'll see with the supply chain because the historical problems with trying to do secure supply chain is just too hard. So I can't do things that's too hard when I'm being asked to do too much. So if we can get it to the point where it's just the normal way of things are done and we don't even realize it, but it's getting done, then we've actually hit the right point. So there's definitely lots of work happening. Innovation is happening at the speed of whatever you think is fast, because whatever I list there, you'll tell me something could be faster. So that's just awesome. So when I think of the stages of the supply chain, I just kind of think of it as the CDC, which obviously has a search problem. But it's just the way I think about it. There's the creation of those binaries, the creation of images, packages, all of those pieces that go into it, the source code, whatever it is that goes into the creation. And then I want to distribute it to the various places that's going to get consumed. Now, I like making analogies back to real world, other world problems. And if I think of how products are created, distributed and consumed, I can go get a Kohler faucet from Home Depot, Lowe's, and a hardware store. It's distributed in multiple hops. Food is distributed in multiple places. And that's how it gets spread out. And that's how we have reliable, scalable solutions. So that's a distribution. And then when I'm consuming it, I want to know that this is the Kohler faucet. This is the Rolex watch. I want to know that this image is from somebody that I trust. So Notary V2 is focused on the distribution and consumption. That's the prime area. We're not worried about how the source code was built and so forth. That's not a focus that we're doing. But when you create content, your from statement is the consumption of distribution from somewhere else. So there's a little bit there. And we want to make sure that when you're consuming something that you know what you're consuming is what you intend to be. So there is a bit of that, but that's where Notary V2 focuses on. And with V1, we don't really talk much about V1 here. It's more about, I'll just stay focused on V2 of the main thing is the signatures can flow with the content. So how do you deploy today is an interesting model. Do you deploy by unique tags? Tags are nice because they're human readable. I can see it. I can remember it. I can write it down. I can track it. Part of the problem with tags, they're not immutable. They're not, you can't lock a tag as a standard across all registries. That's a registry specific implementation. Do you deploy on digests? They're unique. That's great. They're really long when they're 256. Imagine when they go to 512, 513, whatever. There's a balance there of usability that needs to be incorporated. Do you convert a tag to a digest? What a digest promise? They promise that they're unique. No question. But do you actually know who created the digest? Do you track all the digest that were created and are you consuming it? There's a lot of complexity there. Somebody's tracking those. A digest doesn't tell you who created it. It just tells you that is the unique identifier for the thing across the way. So when we think of, we get these analyst questions all the time, do you support all content is deployed? Sorry, do you support all content deployed is signed? Okay. By who? That's the interesting part. I can sign, I can deploy content signed by badco. Does that pass the test? Who do you trust? Just because somebody else might trust somebody, it doesn't mean that's trustworthy in your environment. You might have a high security environment where it's much more important than maybe a typical environment where it's just where most of the software sits. Who don't you trust? Who do you and don't you trust for a given environment? Right? That's also important. This isn't just the big switch that here is the official sign content that the world should trust. What does that mean? So part of the question, and this is not mutually exclusive is if you could sign content from the end and then configure the entities you trust, how much do you need to be tracking digests? Not for, I'm not talking about for forensic purposes. I'm just talking about how you track your deployments. So just interesting. It's just an interesting side effect. Now I mentioned I like real world analogies. So I wanted to use this as an example to try to bounce back a little bit. Most of you came here. Did you travel by a plane? Right? Okay. This is probably some people from your family and the people at home, I'm sure at some point you've been on a plane and you'd like to get one and get out and get more. So you get to the airport and agent 99 says, whoa, whoa, whoa, you're not going anywhere just yet. Give me your pod spec. Where are you going? Okay, here it is. And agent 99 says, well, like, that's great. Okay, you want to get on that plane and go to that destination, but who are you? Well, my name is Steve Laskaris here. That's great. But who's telling me that there? How do I know that's really you? Is it a note from my parents? Is it something I just copied? Or is it some kind of document that can be verified? So driver's licenses from various states. There's always this interesting extra check that they're trying to do now for some states or all states not all states have finished them yet, or passports from certain countries are allowed certain countries are not allowed. Right? So there's a set of identities that are allowed to prove who when I say who I am. And then agent 99 there can scan my driver's license or my passport and see if I'm on some do not fly list or it's been revoked. Right? That's a normal process that people do today. Now, I don't have to have a printed paper. I could give my digital version of that as well on my phone or whatever. And once agent 99 has said, yep, you're good. I've checked. You are who you are because maybe you're still outside. They can stamp my boarding pass physically or digitally that says, Hey, I'm now allowing you into the staging area. So and as I go through the staging area, what happens there? You get scammed. It's kind of interesting, right? Even though you've proved in your your identity, you've given it a pod spec of where you want to be, you're still going to get scanned and verified that no, that's still good. You're good to go. But you get to the gate. No, no, no. Agent 44 is now saying no, no, no, you can't come in yet. You have to prove who you are. Now, when I give him the boarding pass digitally or physically, I don't give him my driver's license again. I don't give him my passport. That was a precheck that was done just to get into the staging area. Now, Agent 44, he's checking. Do I have a signature for being in that environment? And that's the check that's being done. So there's a bit of a promotion scenario happening. The content, me got promoted into a different environment, and I'm being checked again before I can go into the next stage. And then Agent 44 will stamp another signature on that and says, you can now be promoted onto the plane. All right, so there's a promotion sequence that's happening there. So let's take a look at maybe something we're doing with our code, as opposed to our travels. So we have our proverbial customer, we think about a notary with this Acre Rockets company, and they're trying to deploy some software from Wabbit Networks. And so they pull an image, but we've got some policy management. This is good, right? We've got OPA gatekeeper saying, no, no, no, you're not going yet. You have to prove who you are. So let's go get the signature across the web. And if the signature passes, life's good, and the image can be deployed. Okay. How many of you have environments where there's a virtual network in configured and you're not allowed to have egress to the public internet? And for the ones that are not raising your hands is some security person yelling at you, you should be doing it. And you keep on giving excuses, and they're too busy, and they leave you alone for a while, but it's not going to last. Right? It's this is the everyone's trying to shut these down. There's been technical issues. We're all as cloud providers and products and, you know, projects, we're all trying to enable those vnet scenarios. So now that sbom that's sitting over there in Wabbit Networks, that can't get in. The image couldn't get in at this point either. But, you know, the scan results, the source, whatever it is, all that stuff is blocked, you can't get to it. So the answer is have a registry that's within that vnet. And all the cloud providers have registries that now can project into vnets as well, and limit all egress and ingress from external. So that's great. And you're replicating that across multiple environments, multiple development teams, each development team has a different set of deb test staging, you know, prod as well. So lots of those, all of them looking to have vnets of various sources. Now, are you going to pull the content from that external source into each one of those registries? We just said we have this vnet setup, so you can't. So what we're seeing is customers will set up some kind of shared registry inside their boundary. I want to say corporate boundary, whatever the boundary is that they define. And there is some content that's approved for it. So now how do I get that content in and do I verify it? Now, by the way, the content isn't coming from just one place, right? There's Docker hub is a distributor, a redistributor of content, just like you go to Home Depot and Lowe's, there's lots of vendors that are there. But there's other registries out there, Microsoft, we ship our software directly, Oracle, NVIDIA, IBM, well, IBM different story, but anyway, sorry, I was remembering something else. But there's software vendors that do direct shipment, just like some products you direct ship and some you get it from redistributors. So in this example, Acme Rockets doesn't know enough about web and networks to trust them. So the signed content that web and networks produces with the signatures on all of that, will go to Docker hub and Docker hub will stamp another signature on it if they certify that content. There's some checks that they're doing, whatever that is, just like if you go to Home Depot and you see some products there, you probably trust it's good product, right? It is a brand there that you trust, they're doing some vetting. Now, the other content in the Docker hub example, it could be more like an eBay example if you will. I don't know if I trust it. I have to put trust in that individual company. So the way you can do this is you can pull this content in. So now you're bringing that content in and now you can stamp it with a signature that Acme Rockets has done the scanning, the policy management, whatever they decide that they're the Agent 99 there that says, I trust these sources and once it's in, I will put it in the shared registry and I can put another signature on it. And as part of that import process, you can set up a trust policy. Just like Agent 99 knew to trust most states in the US that have their little tree on it or certain passports are allowed, all of that complexity of who can be trusted can be set up as a trust policy. And you can set that up, let's say on your import model. Sorry. But then inside of each environment, you might have a different trust policy. The trust policy in that top right one might say as long as it's got the Acme Rockets shared, you know, signature on it's good. The other ones might be a more critical environment and they have a more critical set of bars that might be some Acme Rockets critical content that has a higher bar and what's allowed in there. Now that's single content promoted across a sequence, right, across the public to the internal to dev staging, test, prod, so forth. What we're also seeing just like you can get that Kohler faucet from multiple distributors is we're seeing this with the shared content as well, like the public content. So there's a group making the Debian image, they're building that Debian image, they're putting an S-bomb on it, they're signing it. It's all awesome. But where are you getting it from? You can get that same image for multiple places. So when the Debian image is built by whoever, Docker, Bitnami, whoever, what's the location that's in those documents? Could be any one of those. And then when it goes into your registry and which is an event, can you get to those other locations? So there gets to be some interesting challenges and this has been the requirements that we've been working on with Notary V2 is how do we make sure we support these scenarios? Notary V1, Docker content trust, had a bunch of usability problems. There was trust on first use. All of that is difficult. You can get around it. You can't get around the fact that the signature wouldn't move with the content. That was the critical blocking thing. At the end of the day, that was like the head decapitator. So we want to make sure that we can support artifacts promoted across environments. They're in vNets. How do we deal with that? How do we do validation? And because they're getting in that vNet, it can't go back out to get the S-bomb, to get the signature, to get these other things. We want to make sure that that content can be promoted so that you can validate it inside that environment. So the way we've been doing this, and like I said, we started trying to do signing and that was our main focus. But as we were going through this, we always have the eye. What is the reusable patterns that we're seeing? And is there some general applicability of this as we're going through it? It took us a couple of months to realize we can't change the digest and we can't change the tag. That would mess up all your kube deploy scripts, all your helm charts, all your pod specs. It needs to be transparent to all the workflows you have now. Use the digest, use the tag, and now let's go find the signature. Now let's go find the S-bomb. So these things become detached objects that get added to it. When I went to the airport and I wanted to give my pod spec to get in, I didn't cross the line. I was still outside. I had to hand my information through the little plastic hole. They scanned it and only if I was approved was I allowed in. If I walked in with the ID, it's the Trojan horse attack. I'm already in. That's exactly what we're trying to avoid with these detachable objects. We talked about multiple signatures and so forth. So let's take a look at how all this works. So I've got a doc that I walked through. I've got it on my presentations and I'll just copy and paste coming out of this to make this a little simpler. So what we have is two registries. They're running locally. There's low close 50-50 and 5,000 is the public one. So the first thing I'm going to do is just generate a certificate for Wabbit networks. This is a key pair for my public and private keys. They're X509 keys. I can use the key management systems that you're using today that'll have a full extensibility so you can use whatever cloud provider key vault or your on-prem key vault or a hashy corp or whatever project you want. So now I've got a key pair. And now I'll do just, whoops, I'll paste it here. I'll do a Docker build Docker push of that public image, right? I'm building the Wabbit networks net monitor image. And now the Notary V2 client is notation. And I'm just going to say notation sign that public image. Now the config that I've set up has that key already in it. If we look closely up here, when I said generate cert, there is added dash dash default. So I can configure that information. We'll talk, we talked a little bit more about policy. Now I can look at the signatures that are associated with it. And I'm just getting the unique identifier of it. Here's where it gets a little bit more interesting. Oras is an OCI registry storage client that we've been working on for a couple of years. That's part of what we use for a lot of the projects used for artifacts for like Helm and Wasm and others. The idea is that you can push and pull things to a registry and so forth. And there's a library and a client. We've been putting this extra sauce into oras to make to enable things like these references. So I've got the net monitor image at the top here. And what I've done is said I'm going to add a signature that's of a Notary V2. That's the artifact type. And then that's the digest of that signature. But you can see it's linked. Now let's go and make an S-bomb. And we'll push the S-bomb. So all I'm doing here is oras push to a public repo. There's no tag on it. I'm associating the, I want to push it to that repo. And I want to link it to the public image. So public image is net monitor v1. And I'm just saying, hey, just take the S-bomb and make the media type at JSON file. So now if I look, well, before I do that, let me sign it because an S-bomb is only good if you trust it. Every Star Trek episode that starts with somebody doing something bad was what's the first thing they did? They deleted the entry from the log and then data was able to go figure out there's a hole in the log and I'll go and find the magic bytes and it can reconstruct it and whatever. So now that I've got that on there, if I say oras discover with the tree output of that public image, now I'm seeing there's the net monitor image, the signature, the S-bomb. There's the S-bomb. And notice there's another indentation because now there's a signature of the S-bomb. Now I have some piece of information that I can trust that that S-bomb is what it says it is. But wait, there's more. I might do a scan result. So we'll use Docker scan has got some SNCC stickers in there. And if I scan it, sorry, if I cat it out and we can see there's some scan results from a notary perspective, I don't care. Somebody says that's the S-bomb and that's the scan results. We'll just sign it. So we'll push it. No different than we did before. I need to get the digest of the thing we pushed because I'm going to sign that digest. Now think about what we've done here. This could be the SNCC client, right? I'm showing a little bit more details there. That could be SNCC scan. And here's the reference. And then it would internally to that CLI would say, okay, I'm going to scan it. I'm going to push it and I'm going to sign it. So the fact that I use three commands is just showing you some of the details there. And now when I look at it again, we've now got a richer tree, right? Now in addition to everything else, I go back to the root again because the S-bomb is on there. Sorry, the SNCC results are hanging off the image and it itself is signed. So I got an interesting graph of content here. Okay. So, you know, we talked about this promotion scenario. We want to make sure we can get this content across. When you're copying files in one directory to another directory, do you go and find Word, Illustrator, you know, your media player? And do you use all those individual tools to copy the content from one file to it from folder to another? Of course not. There's a file system API. And the file system API knows how to think about files that are in a directory. And it can copy it. The file system API doesn't care what the content is. There is a standard. There is a spec that says when you want to put something in a file system, here's how you write it. It doesn't care what the content is and how it's structured. It has attributes. When you look at Explorer, you do an LS by default, you don't see all of the attributes and all of the noise. You just see the list of files because that's all you care about. Even by default, the extension isn't there, but you get an icon, right? The point is those are details that you're not thinking about. So what we wanted to do is we really needed to incorporate that as a first class object. We didn't want the signatures and the scan results and all those things to be populating that list when I do the repo listing because I want to see all the builds. That's what I'm trying to see. I'm trying to see v1, v2, v3, v4, all of those tags of all the content that I'm actually thinking about. So when we're building those images, we're building those other artifacts, we can link those to each other. And this is a reverse index so that we can find it because when you're trying to find a signature, you don't know what signature, the deployment file says net monitor v1. So you want to be able to go to the registry and says, can I have the signatures, the S-bombs, the things for net monitor v1? It's a reverse query. And of course, you can have lots of signatures that are associated with it as well. So let's take a look at how we can do this. So now I talked about promoting. So it's not just as simple as copying. What I want to do, well, actually, first of all, nothing on my sleeve, right? There is the public registry now, sorry, the private registry, and there's no repositories. It's completely empty. Before, because I do an import of my Wabbit network's image into my private environment, I want to verify that it shouldn't come in. What is the policy by which I'm allowing content to come in? What is AG99 allowing to come in? Well, I didn't actually give AG99 information yet. They don't know the list to say to the passports. So it fails. There's no certificate available. So all I'm going to do is configure what certs are trusted, what public keys are trusted. Now, I'm using my key management system. So I got the key on my machine, right? That's part of what we're trying to do is leverage the existing infrastructure. But I'm configuring the policy to say I'm allowing that Wabbit network signature to come in. So now when I do verify, now it does pass. Now I want to create another cert for inside of Acme Rockets. So I create the Acme Rockets library key. And then I can do the standard Docker pull, Docker tag, Docker push to take that image, put it in. And now I'm going to sign it with the Acme Rockets key because I changed the default. Now, if I look at the content, we're only going to see the image and the signature. That's the Acme Rockets key. That's all I've done at this point because I did the Trojan horse verification like, oh, not letting this image in until I verify it. Okay, it's good. Now my steps will proceed. But I want to get the rest of the content. So I'm going to do an RS copy recursive of the public image to the private image location. So it just copied this graph of content across. So now I've got, I don't need all of the individual tools. I've done the equivalent of the file system API, because the registry has the knowledge of those references. So I can pull that I can discover it, pull it across. If I look at the registry, it's still only got one repo, the net monitor repo, and it's only got one tag. I'm only seeing one file in my directory. I don't need to see all the attributes yet. I'm not looking at the attributes yet. Those are attributes of that image. If I want to see the attributes, I can ask for them. So they're there, but they're known to be attributes. They're known to be references of that image. I could even filter the attributes. So here I said, or as discover, artifact type or CNCF notary v2. So filter to that artifact type. So now I've got the web and networks and the acme rockets key out. I don't see the S bomb. So that is how we've been trying to pull all this stuff together. Now I talked about a little bit about the policy management. We want to make sure that you can configure what keys you're trusting for particular environments. That's up to you. You can figure how you want that. We're not trying to create yet another policy manager, just like we're not trying to create yet another storage service. We want to integrate with all the other policy managers. This is just information that can go into those. And I talked about the key provider extensibility. We'll have key revocation, absolutely key and signature revocation, right? When you scan your driver's license, your passport, we need to know which of the bad entities and don't move forward. It turns out the feedback we've gotten, you know, and tell us differently is that's really important for the public content. Internally, there are the systems that are already managing what gets deployed. And it actually gets more complicated internally to try to do revocation, not just because of the implementation. It's the management of it. We're working with the tough team. We're definitely trying to get tough integrated into this. The interesting thing about tough in a lot of the update framework is how it's been used. It's for single streaming kind of registries, the single Docker Hub registry, a single car stream that goes to car devices, a single update to nest devices or something, that there is one registry. There's no content promotion in that. And they're looking for the most recent updates. As we're trying to promote content across registries, just pieces of that content, we need to make sure that that scales. We also have an OCI registry one fallback mechanism as well, because we know that this is using work that we've done in ORAS to enhance registries to know about these reference types. We'll have that in ACR. I don't want to speak for other cloud providers and when they're doing the work, we're working with the other ones and Docker. But we know it would take a while for everybody to get there. So we will have some fallback support, but it's hard to know what you're falling back to if you don't know what you're falling back from. So we released the draft spec a couple of weeks ago. We're feeling pretty good. We're looking at it. You can't really do multiple signatures with some of the tag schemes that have been thinking about. So we're trying to figure out what's the best model there. So just wrapping up, the signature allows you to decouple location from identity. Doesn't matter where it came from. I want to be able to verify this independent of it. I want to know this Kohler product is still the faucet, whether I bought it from one of the manufacturers, one of the distributors, I bought it on Craig's list or whatever. I can verify that. You choose what you trust. As you promote, stamp it for the next environments. You don't have to reevaluate everything. And we've been generalizing the references and what else is on here with that. So I'm looking at the clock. We've got a couple of projects here. This has generated a couple of projects so that we can use these things not just for notary, but for other signatures or other supply chain artifacts, or if you want to associate the pictures of this conference for the image or whatever. All that's there. And I will stop talking and stop presenting and take any questions in the 30 seconds I have left. Thank you. If y'all have questions for Steve, we actually are at time, but Steve is welcome to hover in the hallway and answer questions from y'all. So, Steve, thanks. Thank you. Good job. Thanks. Staying to talk, bro.