 Hello and welcome to KubeCon. Today we're going to talk about NaturallyV2 supply chain security for containers. I'm Justin Cormack. I'm the CTR at Docker. I'm a maintainer on Notary and a member of the CNCF Technical Oversight Committee. I work a lot with security and I've been working on supply chain stuff for quite some time. And this is Steve. Hi, I'm Steve Lasker. I'm a PM architect in Azure work on container registries, including ACR and MCR. And as part of that, you know, with registries, we're sitting at the middle of all of the services in Azure that use containers from production services, the DevTools. So we see quite a lot of different scenarios. We talked to a lot of customers around the various scenarios. And of course, one of them has been around how do I sign my content so that as I consume upstream content from the community and bring it into my environment, how do I know it's that and how do I promote it within my environment, you know, it still has the same integrity. So notary has been a key focus for us. And it kind of builds on the stuff we did with OCI artifacts. I'm a TOB member for OCI and a maintainer for OCI artifacts, which is how we leverage registries to store all kinds of content. And we'd like to be able to sign all kinds of content so that there's integrity around it. So with that. So why are we doing this? So until last year, really, when you... I used to have a much longer piece in here about why supply chain security, but SolarWinds last year really kind of helped people understand what was going on, that someone could attack your supplier and insert malicious code into their software, you would bring it on-prem and it would start attacking you. This is kind of not what you want to happen. And it's really brought this into the sort of public view. And it's like something that people are much more aware of now. And the SolarWinds one was interesting because it was this attack. Well, where the attack happened was interesting. They had to figure that out. It wasn't based on containers, it was an old different technology, but the binaries were signed. So they were able to realize that this wasn't some kind of production deployment environment hack. It actually came from the build system. And having that signed content that was deployed gave them the forensics to be able to go back and figure out where the breach happened. And that's something we need to be able to bring to containers as well so that we can know that when things go into containers that we can attest to their flow. So it was just interesting how that breach affected so many different areas. One of the things about cloud native software is that it's you know, modern software in general, everything is controlled by software. So if someone controls your software, they control everything that's running in production and they really own everything. And often if people want to attack, then sometimes the supply chain build servers and things are the weakest link. And you know, people have hardened their production environments, but they haven't hardened build. So let's attack build. So it really is a, you know, something that's been growing in importance. We're just recording this a bit ahead of time, but I think the CNCF security software supply chain best practices white paper should be out for KubeCon. It's a really great bit of work if you want to have more background on supply chain. So take a look at that. And container security, I think it's, it's kind of interesting because for a long time with containers, people just talk about things like, you know, can I escape from a container like treating it like, like those boxes, can I get out of a box? That's not really what's important about containers and shipping. The box is really just about standardization. The important thing is the whole process by which you know what's on your container, you get the bill of lading, you can trust that when it arrives, you can put it straight into your supply chain, your factory, you've got the metadata with it about what it is. And you can, you know, as long as you're, as long as you don't get stuck in the sewage canal for a few weeks, you're pretty much sure that you can order things remotely and get them and incorporate them into your products. And then later when, you know, when you have a product, if you have a problem with it, you can trace back which components went into it from which containers they were in when they were being shipped and which where they were at the factory. And we're trying to bring that kind of approach to containers. It was always something that was promised, but it's actually something that hasn't really been delivered for everyone in their day to day work yet. The interesting thing about this also is, you know, you've got the containers on the left and then you have this, you know, bill lading or bill of materials, if you will, that says what's in it. And like, okay, Justin could have made this himself on some, you know, document. But there is a stamp on it. And the stamp has, you know, an authenticity to it that says this bill lading was approved. The bill lading represents what's in the container. And there's a level of trust associated with that. And that's what we're trying to bring with containers. There's a there's a lock on the container that says it wasn't tampered with. There's a document that says what's in the container, the bill of materials. And there's a lock on the document, the stamp that says this, this is accurate. That's a really good accurate representation of what we're trying to accomplish here with the container workflow. Best thing to move forward. So what are we working on? So Nature Review One was created back in the 2015 and seems a long time ago in container land. All containers were pretty new based on the update framework, which is a CNCF project. And it's been a CNCF project since 2017. So it's, you know, one of the, one of the earlier projects is supported by some registries such as Docker and Azure and IBM and Harbor, but not everywhere. So users can't rely on it always being in the registry, which is kind of frustrating. It's got some users, but it never really was a widespread thing that, you know, the majority of people would actually use on a day-to-day basis. We learned a huge amount from the rollout, how people used it, what they tried to do and all those things, which is really feeding into a lot of the design for v2. And some areas we really want to fix. You know, these are some of them, but they're probably the most important ones. First of all, the signature is not being native to registries and compatible with all registries is a real problem. It was a separate database and there were portability issues about moving the signatures around. It wasn't just, it didn't have the APIs, but things were actually tied to the names of the locations. Nowadays people use a lot of registries. You know, you might, you might use, you might get some images from Docker Hub. You might have a private registry on Azure. You might have JFrog on-prem, you know, all sorts of combinations of registries and for different purposes and different stages of the life cycle. So that's really important now. It wasn't so important back in 2015 to be, I mean, where, you know... It was one registry? It was one, yeah, Docker Hub. That was it. We don't need any more registries. But part of this is that we continue to hear from customers that, you know, it's not just dependent on a single source of failure that's out on the unit that has nothing to do with Docker Hub. It's the network connections. But it's also the update cycle. Right? We want to be able to consume public content, whether it be containers or package managers or other things, but we want to bring them into your, you know, into our environment, into the environment that that customer represents and says, this is the environment that I trust. There's a security boundary on it and nothing's randomly getting changed on me, even for good or bad intentions. But how do you promote that workflow? How do you take the content from upstream, bring it into your environment, promote it across your own environment, and know that that content was never tampered with? Yeah, absolutely. Other issues were a lot of keys to manage. A separate key... Well, a separate set of keys for each repository, which when you were managing repositories at scale was really difficult to manage. And debugging issues was complicated. The introspection tooling wasn't that great. Security engineers assume that when something fails, verifying a signature, that that's probably a hostile attack that they've just thwarted. Yay! But actually... That's what they can do. They're ready to engage. Yeah, but unfortunately, sometimes it's like users using the wrong key, users, cert has expired, like all sorts of other reasons that I actually need better debugging tooling. But it's more than debugging to it, right? Also, it's just that balance of how complex does something have to be to solve a problem? Because the more complex it is, the more places for things to fail, the more difficult it is to use. And if it's difficult to use and complex, which's got more failures, it's not going to get as much adoption. Also, I think it's important that people understand what's going on. And I think that's debuggability to me. It means that the mental model of what's going on fits with the reality of what's really going on behind the scenes. And I think there was a few issues around that as well. And then there was issues around registries. The tough model was designed around packaged repos, where you only have the current content in it, whereas registries have everything in it, and that didn't fit so well with the update model and the update framework. And there was just kind of a bit of a mismatch where the security models weren't quite right. And we needed to kind of tidy that up a bit. We also got a whole bunch of new requirements. One of the first things people start asking about is really being able to add more signatures later on. So people often have this workflow. They want to do where they want to mark the state of approval of something by signing it. And so when this is approved for production, we want to put a signature, and then we just want to check that signature for prod. But that wasn't terribly easy to do. It wasn't the kind of security model it was designed for with V1. And then there's this really important thing about adding additional mess data. So you want to add that bill of materials. You want to add other things, scan reports on the image, and things like that, that you want to add after build. And you want to sign the addition of that so that people can validate that. You can attest that the SBOM goes with the image. And so that's really been something that's become particularly important in the last year or so as the SBOM standards get approved. And people understand what the value of being able to add extra metadata afterwards is. It's not just SBOMs. And then it's not just about containers. As Steve mentioned before, people are putting lots of things. It's time to put lots of things into registries. And these have different requirements as well in different formats. We're really trying to do incremental improvements. Right now, hardly anyone uses container image signing at all. And so we really want to make it easy to use it at scale. It's a community project with very broad involvement from registries and end users and cloud providers, all sorts of people, which is great. So we're getting lots of different requirements, but we're picking the things that we can ship earlier that help people right now. And laying a foundation for people to be able to experiment and build more stuff later on. We want to get the base infrastructure out there, standard centralization and OCI as well in terms of how do we put this stuff in the registry, formats, extensibility. And then we want to have tooling, which we're going to demo to shortly as our prototype tooling and tools for policy management and wide availability and interop. But we want to let, we'll start with flexible base tools and build up. People want to do, people have specialized requirements and want to build specialized tools. That's great as well. It's the libraries. It's not just the tools. The tools will be backed by libraries and specs so that you can use these libraries in other scenarios that we hadn't thought of. We're not trying to, containers are used in so many different places. We don't, we'll talk about open policy agent as a Kubernetes solution, but we want to make sure that you can do wherever you're running containers, like with container D libraries and so forth. Absolutely. Yes. So the first stage work in progress design is being finalized by the time you hear this talk, it might be finalized, hopefully. So we'll be able to give you more notes in the live chat during while you're watching this. Kind of scope it as kind of equivalent to get commit signing for containers. So it's like individual one-off signatures, not the kind of tough update, sign a sequence of containers type thing for now. Just point-in-time signatures. There's going to be X509 support because lots of people have, some people have X509 infrastructure and people want to use that. Not everyone has infrastructure now, so we'll have some easy tooling. Anyway, there's lots of requirements we've got around having key management. For example, if you want to use a cloud, a cloud service to manage your keys, or a vault or something like that, or you want to use hardware keys, like the UBK I've got right here, those kinds of things. So we've got a lot of people who want to use remote key management and things like that. If necessary, if you want to add some support, but we're going to have a lot more. I just want to make it more flexible, like the whole idea is don't lock you in. If you've got a way for managing your keys, you should be able to use that. That's the main point. Yeah, and then as we said before, like IPER and container D libraries and all the kind of tooling, we need to build this into tools like Docker and so on. So the plan is to ship a usable turning solution in the second half of 2021, and support widespread usage, then go ahead and iterate on that, work out what's working, what else people need, improvements we can make, all the things that people need to make it easy to use and widespread. While we're also establishing a working group on specifically on pipeline security for software factories, which is an area where the guarantees from Toughen and Toto who make a lot of sense are actually guaranteeing that something went through a pipeline stages and this software is one piece of a flow of updates of software that you can feed into your production systems. So that's intended to be signed in NCI probably with ephemeral keys using spiffy things like that. So that's a working group that we're going to have going also for future work. I'm going to do a demo of where things are now. So this is really a demonstration of signing artifacts and s-bombs and how things will work. It's not like picture perfect, but it's like directionally what we're looking at. Basically, we have container images that were stored in registries and the interesting thing is about how that is stored because that turns out to be very generally useful for a number of different things that we've been able to put in registries. Up until today or up until artifacts were created, the OC artifacts project, people were stuffing all kinds of things in, but making them look like container images like the old way we used to change the extension on the file when we try to attach it to an email. So what we're going to be able to do is support adding no reV2 signatures into a registry that would be stored as a blob in that artifact, which looks like an image, but it knows it's a signature. The new thing that we're adding is the ability to add a reference from that signature to the image, and the directionality there is pretty important. If you notice the image points down to the layers and the signature points to its blob that contains a signature, but it also has this reverse lookup reference that says, hey, by the way, this signature is associated with that image. So that reference concept is really important because it allows us to add multiple signatures, or in this case, if we wanted to add a Bill of Materials document, then it also is just another document. We're a call about an S-bomb here, but if you want to add yet another artifact type, yeah, you can do that as well. Store it as an artifact in a registry, give it the reference to what it associates with, and now you're starting to build this graph. And just like we saw with the Bill of Lating of the ship, the Bill of Materials here, we have a signature on it to make sure it's attested to. So if you look, there's this interesting graph of artifacts of which one of those things, artifacts types, is a signature. And that's the fundamental concept that we've been working on. So with that, let me see, I'll do a little screen share switch. And so what we've done is, for the purpose of this experience, we've added a plugin for Docker. And if you notice there's BuildX, for instance, as a plugin, there's NV2 that we've also put as a plugin as well. So just to make things a little easier to not have to type Docker NV2 every time, I'm just going to create an alias. And I'm also going to just create an environment variable for my artifact name, the net monitor view on image, I'll just create as an alias. Now, the first thing we'll do is just kind of do a Docker build. So I'm going to build, and I'll tag it that fully qualified name. And, you know, while I now have an image, the net monitor image here. Now, we want to be able to support ephemeral clients that are booted up in each time they're being run, they're always around a refresh state. But we also want to be able to support signing before it ever leaves that environment. So what I'm going to do is enable Notary. And again, these are just temporary names, naming is hard. This is just what we've done for now. And now I'm going to say Docker, Docker Notary sign. And what we're doing is signing it with a key and a cert. And I'm just saying, here's the image that I want to sign. So we generated a manifest, we signed it, and the signature itself was saved locally so that we can push it later. But the idea is now I've got an artifact that signed locally. So now I can say Docker push. And because we have this plugin that we've extended for Docker, which represents the experience we're shooting for, it'll change. But an example that what we've done is we've pushed the image and the signature. And there's a link between the signature and the image. So now I've actually already pushed that graph into the registry. So we talked about a set of libraries that we're adding. Let me just paste that book clearly. One of the libraries we've been using around artifacts is something called OREZ, OCI Registry S Storage. It's something a couple of some of the folks put together to enable this experience. And we've added a Discover API that says, let me discover the things that are associated with that image. And we'll just output through jQuery to get a better view of it. So what you're seeing is here's the image. And we've got a collection of references. And if we look, here is the manifest of that, that there is a, there's one reference in here. In this case, it is an artifact type of cncfnuderyv2. There is the signature in a Blob's collection. And the manifest collection points back to the digest of that image. So that kind of gives you a pretty powerful experience that I can now push these two things into the registry. Now, that's great. We've got the image, we've got the signature in the registry. If I want to get rid of actually, I want to spell images, right? So we've got that here. If I want to we'll just kind of cheat. And of course, it'll, so now if I do it, minus a minus q, minus a minus a, you talk to, what's that? You talk minus a minus a, not minus a minus q. So I don't have anything, I got nothing on my sleeves, right? We only have the registry running is the point here. So now I want to simulate another client. So I'll say Docker pull, pull that image. And what's happening here is it's saying that, oh wait, I've got my signature already set up. Sorry, hold on a second. See, this is the demo failure. Let me, it's one thing jumped ahead. So let me reset that because, so if I say Docker image poll, if I do not have my client configured, which is what I want, is it found a signature, but there is no keys on my machine. So it basically says, I found this image, but you have no keys that match it. So I'm not going to allow that image to be pulled. What I want to be able to do is configure which keys I want to be seen. So I want to be used. So give me a second here. If I cat the output of our configuration for the NV2 client configuration, there's no certificates referenced. That's why it didn't work. If I go and change that and put a certificate in the path for it to find, now when I say Docker pull, because I've got that signature, it'll say I'm going to pull and the pull has got an extension to it. So the pull will first pull the manifest, find any references, and it'll go iterate through the references, look for signatures, specific signature types, and then match those with the key. All of that's happened before the image was ever even pulled. So there's no Trojan horse style attack. It was able to verify that one of those signatures matches one of the keys I have locally, and then it proceeds with the image pull. So if I look at my image cache locally, now I've got the net monitor image here because I have the certificate, but wait, there's more. So we said we wanted to be able to talk about S-bombs. So we're going to go create the world's most simplest S-bomb. And if we look at it here, we'll see. It's got some version, which is none. It references an image and it says it's good. So it must be good. It's not great. It's not awful. It's just good. Clearly, this is not the kind of S-bomb we want to shift in production. The idea is there could be anything, any files, could be pictures of cats and dogs, whatever. There's some document that gives some information around that image. And I want to be able to push that to a registry as an artifact that is linked to the image. So I'm going to push this S-bomb JSON document, the thing we just created. I'm going to push it to this repo. Notice there's no tag. We don't see linked artifacts as needing tags because they really have no value unto themselves. They're in addition to the original image, original artifact. So if the net monitor image is deleted, we want the S-bomb, the signatures and everything to delete with it, just like when we copy it, we want them to go with it. So we're going to push this file to this repo. We'll say it's an artifact type of example S-bomb v0. So clearly nothing. It's referencing an artifact reference to the net monitor image. We referenced it as a tag, but it will convert it to a digest. We don't link to tags. We link to digests. And we're going to save that manifest because we're going to assign that in the next step. So that's simple. We've created that, taken that S-bomb and pushed it to the registry. And we can see that by simply using our or as discovery API again. So we're going to discover the anything that's referenced to the net monitor v1 image. And if we look, we see here's the net monitor v1 image. And we notice here's our artifact type of an S-bomb v1, v0, sorry. And there's our S-bomb content is stored in this tar file. And then the manifested points to it's pointing at our container image. So that's that reverse lookup reference back to the container image. And notice we also have an artifact type of another v2 signature. So those things are both linked to the artifact, to the image at that point, because notice the signature also points back to that same image. All right. So we got the S-bomb, but I don't have that stamp on it. So let's put the stamp on that S-bomb. And that's unfortunate. Hold on. So I need that digest. So let me grab that digest. We haven't quite finished all the linking experiences. So the first thing I need to do here is save that digest for the signature. And now when I say NV2 sign or NV2 is our binary, but there's libraries that back it up, you can use. I'm going to use that X509 cert of the Rabbit Networks key. And I'm going to reference the thing I'm signing, which is this S-bomb document. I want to sign it and push it associated with the S-bomb in this case, not the image, but the S-bomb. So now if I do that discover one more time, I'll now see there's yet another. So I have our notary v2 signature for the actual image. I have another notary v2 signature for the S-bomb. And then here's my S-bomb. Then I have the S-bomb itself. So I now have that graph of objects that I put in the registry. They're linked to each other through references so that when I do deletes, I want all that content to go with it. And when I want to do a copy, we can also copy that content across as a whole as well. So that whole graph can travel as one. And that's really what we're trying to achieve, being able to sign things that are in a registry and being able to establish that chain of references. So you can put lots of things in the registry that reference, whether it be home charts or get up, read jobs, kind of documents or something we haven't even thought of, because that's the main thing. We're trying to enable an ecosystem to grow without us having to know about each and every type. So that's it. So how to get involved? This is our places to find us. The easiest sort of central thing is the CNCF Slack notary v2. Meetings are on the CNCF calendar. Projects on GitHub. Meeting notes. It's all very, everything's open and available. They have recordings on YouTube. So please come along and get involved. Come and talk to us if you're interested. And we'll be in the chat with more to tell you about what's going on right now, rather than a few weeks ago, because everything's moving pretty fast right now. So thanks very much. Thank you.