 Well, hello everybody, it's getting towards the end of the week, hopefully everybody's starting to think about how they're going to enjoy a wonderful Valencia. So I'm here to talk about distributing supply chain artifacts. Anybody thinking about security these days? Kind of a little bit of a focus. There's lots of new supply chain artifacts. Where are we going to store them? How are you going to handle them? What do we do with these things? So I'm Steve Lasker. I'm a PM architect in Azure. And this is an area that I've been spending a lot of time thinking about for how we store and distribute them and something we've been working on for a while actually. So I'll kind of talk about what we've been thinking about and how it's been evolving. So lots of supply chain artifacts. Where are you going to store them? How are you going to distribute them? It turns out the what matters because of some of the best practices for how we think about to distribute and deploy these. Then we'll talk a little bit about should we be building new services or should we invest and extend some of the existing services that you're all using already. So I'll spend a little time talking about this elegance and evolution of registries which will put context to the whole thing of OCI and Aura's artifacts. So there's lots of them. Oh my. It's just every day there's a new artifact type that's coming out whether it's a new S-bomb format or claims or out of stations. There's just lots of different types that are applying to the software already thinking about. So we think about containers, but there's packages. There's loose binaries. Turns out there's IoT deployments that we should be thinking about from the killer cars that could be taken over to the medical devices that are in the people that got hit by the killer cars. These are all components that we should be thinking about of how do we have supply chain artifacts around these new types or new types around the existing things we're already deploying. But it also applies to our VMs that are under the covers for all of our containers that we're running. This is not a new problem. It's just a matter of how do we think about it across the spectrum of software and hardware quite frankly. So as we started this effort several years ago, a lot of these formats didn't exist. They continue to pop up. So how can we think about associating all of those with the type of software we're building? So where we store them? Does it matter? Do you need to distribute them or if I just make them available on a website, everything's good. And by the way, what's the life cycle of these? So we'll spend a little time talking about that. Now, we spend a time thinking about the best practices for how we're going to consume public content or consume software in general. On your way here, you've made some software available. You might have built a new image or whatever you were working on. As you're leaving, the cluster needs to scale. Should that cluster be pulling across the internet and a thousand different types of connections from Docker Hub or some other public registries like Elastic? How many of you have a refrigerator in your house? Why? You can go to the store and get fresher milk every day, right? Is it going to be in stock? We go to the refrigerator in our house, in our hotel room, because it's right there. It's faster. It's reliable because it's there right there. Now, I keep on updating it. I keep on replenishing it. It's the same model with our production nodes. We don't want to be reaching out across the public internet or outside of our control for content that the connection might be bad or it might have gotten updated. Let's not forget SolarWinds, the exploit there, was a software update. Lots of updates are well intended and break us because everything that changes is a change of some sort. We want to make sure that we've got a tested copy of the software we depend on in a location we can depend upon and then have a constant stream of updates that could come in after we've tested them to apply. The concept here is basically we want to keep a copy of the stuff we depend on as close as possible, even in the same region of your deployments. If you're deploying in US and UK and Australia and Asia, you want to keep copies everywhere that you're deploying from. Shouldn't we have that other supply chain artifacts with it as well? We want to ship all of that software together. As I'm trying to do verification, whether I should be deploying this thing because I'm making a decision based on the S-bomb or the Git-bomb or any of the other claims and details, we want to make sure that's with the software itself. We have automated builds. We probably build more than once a week these days or once a month. It's probably multiple times a day, maybe even every get commit. For every one of those builds, how many do you test? How many tests succeed? And how much do you actually deploy? For each one of those, are you generating all the new supply chain artifacts? So you start to think about, wow, this is starting to get to be a lot. What about all that non-deployed content? There's some liabilities. There's assets and liabilities here. But I'm not just talking about the storage costs of this. That's the least of the problems. All that software, there's vulnerabilities everywhere. But do you care? If you have a vulnerability in software you built you never deployed, does it matter? The scanner doesn't know any different. The alerts don't know any different. So we want to get rid of the stuff we don't care about. But there's still a lot of stuff that we do care about. You've all the software that we've deployed that's a subset but we still need to keep track of that as well. How many of you have compliance requirements that for any software you deployed you need to keep for some amount of time? 18 months tends to be the magic number. For all of that software you've deployed but it's not deployed now, are you scanning it? Well, you probably should because you want to know if there was any exploits. Do you patch the archive software? Because you've got an alert. You're supposed to go patch it. If I patching the archive software which I'm keeping for compliance then it's not the software I deployed. So there's just lots of complexity here. As you want to move your deployed software that was now archived you want to keep that in a separate place. You're going to need all of those supply chain artifacts to travel with it as well. What was the S-bomb at the time? Hey, I've got a vulnerability now. Why did you deploy that thing? Well in January when I deployed it look the scan result didn't know about it. That's why we deployed it. The scans didn't show that there was an exploit in it until a month after it was taken out of production. So it actually wasn't a problem. But you need to be able to keep track of all these things. Now is everything publicly accessible by everybody? It's one thing to have OSS software that's available but we consume OSS in software that we either ship with intellectual property that we don't make publicly available the ingredients or it's internal within our firewalls within our companies that we need to keep. What's the scoping of access to all of that? Does everybody have equal access to all things? Does everybody get create update and delete rights? Or we just trust everybody? Kind of defeats the purpose of... Well actually if we trusted everybody that we wouldn't have to worry about security but of course that's not the case. And then what about just listing the information out? What about the metadata on it? All of these are the things that we think about the more... the days next around what these storage services have to provide. And what about all the itties for production capabilities? These needs to be geo-replicated into multiple regions. Within a region there are multiple availability zones. That's the expectations that are put on cloud providers for these services. How do you promote that content? How many of your customers are multi-cloud? Are you a multi-cloud? Will this work across multiple clouds? Or on-prem? And when stuff goes wrong how do I think about the diagnostics and all the troubleshooting? So you kind of get the pattern here in firewalls. We talked about how this software needs to be able to be available within a VNet. Because all of our customers want their security that they had on-prem in the cloud. No public access, lock it down. So these are all the requirements that keep on coming up on these various storage services. Oh, and by the way, who's going to support these things? So running a storage service is complex, right? We start with some blob storage. We add some REST APIs for discoverability. We put authentication on it. There's some caching for the multiple requests for the same thing. There's some kind of support. There's integrity and signing that's available on all of these things. And then all these other capabilities keep on coming up. So it turns out that it is complex for all of these. And all of the supply chain artifacts are going to have the same check boxes for requirements. So as we're looking all of this, should we be building new services for these things? Do I need a new SPDX service? Do I need a new Cyclone DX service or an SBOM service? Or a scan result service? The idea is that we'd want to be able to leverage what we already have. So that's kind of the setup here is, well, which one should we invest in? So Git turns out pretty popular. Very tailored towards your build environments, right? It's optimized for developers making changes and tracking those changes. It is not optimized for production deployments. We have lots of things trying to pull from it at the same time. And we've seen that with some of the various technologies that are trying to do more Git-based deployments. There's lots of existing package managers, and most of them are underfunded. They just don't have the capabilities. We're always trying to get them to improve, but it's not the critical business. Sure, we could use raw storage accounts, which is underneath all of that, but it's not robust enough. And then we can get to various ISV products, but then we're starting to compete for things. And this is not the place that we're trying to compete. The package managers and the storage services and registries, they're lost leaders. They're there to enable the rest of the workloads, which brings us to what we've been doing under the OCI distribution spec as leveraging that capabilities to which every one of your cloud providers have it. You have them on-prem. If you're at KubeCon, I'm sure you have a registry somewhere, at least one. How about we extend into that infrastructure? So let's take a little bit of the elegance of registries, because it turns out it's actually pretty well designed. So let's walk through a couple of pieces to it. So first of all, if you say Docker run, and you're doing anything other than Docker Hub, there's going to be some registry URL that is the endpoint. It could be a login endpoint, the rest endpoint, the registry name, all kind of it's synonymous. The idea is there is some address for where you find this thing. It's kind of a little weird that registries are the only package managers that you have the addressability in it. Like every other package manager, it's a config option. So that's a different effort we're trying to make that a little easier so you can promote these things and it's a configuration for the registry. There's some namespace where your stuff is stuffed into. And depending on the particular registry, it could be one, two, or n levels deep. It's really just a path with magic slashes in it. Then we finally get to the actual image, and there's a version. Turns out it doesn't actually need to be an image. It could just be what we call an artifact. So what's under the covers for all of this? That net monitor V1 image is backed by a manifest. A manifest is just a JSON document that has some descriptive information about it. It's intentionally lightweight, and we'll talk about that a little bit more. Then there's one or more collections of content, the actual content that makes it up. Those are blobs. It's all stored in some kind of blob storage, usually. Some of the lightweight registries that are on-prem or on IoT snares, they might be using regular files for storage, but typically there's some kind of blob storage, and there's this REST API we talked about. If we map that into a registry, the manifest, there's a lot of them. They're stored, and the blobs are in blob storage. And now we're starting to see how this is framed together. If I walk through what a Docker pull looks like, this gets a little interesting, too. So I have the client here on the left, and it says, hey, Docker pull this net monitor image from this registry. The client calls that registry and says, hey, I need this thing. Can you tell me about this thing? Sure. Let me give you back a manifest. The manifest is meant to be extremely lightweight because I want to be able to pull it often. It's absolutely minimal information as possible into the manifest because it's going to give me some metadata, including the list of blobs. I might even have one of those blobs on disk already because I already pulled the base layer for some other image. So I've already got some of that information. I don't really want more information to come into the manifest than possible. It now says, hey, from that list of layers or blobs that's in that manifest, I already have this one, but I don't have the other one. Hey, registry, can you give me a URL for this blob? And this is another interesting part about the way the registries work. Even though the registry endpoint, in this case, is registrywebitnetworks.io, the response to the request for the ID for layer two can be a completely different URL. That's a protocol that's set up. That's how we enable CDNs that can be serving the blob content from different endpoints. It's how we do regional endpoints. It allows us to kind of, there's a negotiation that it says, I know you asked for registrywebitnetworks.io, when I'm going to give you the content, it actually can be a different endpoint. So those are things just to be aware of when you're configuring firewall rules too, is to look at what your cloud provider is doing or your registry is doing. So then it finally says, hey, here's the URL. And the client goes, okay, great, let me go pull them. And I might pull them concurrently. And of course, this is the model we see. We see that in the Docker model, we see the layers being expanded. And once all those layers are there, it goes, hey, I'm done. Here's the container image and off it goes. So if we look at what's in that manifest, there's this collection of blobs. They're called layers in the container image format. There's a collection of layers and there's this config object. The config object is just another layer. It's just another blob, just a specialized one. Everything in a registry is stored as a descriptor, which is nothing more than the type, the hash, and a checksum. And the hash is the unique identifier. And then there's some additional metadata, which we'd love to do more with. This is like, we think what people are using labels for. This is what annotations were intended for. We're hopeful that as we can start indexing these more, that there'll actually be a nice circle that'll come around and go like, well, I can actually do something with them. So maybe I'll produce more annotations. Now, the files, the blobs are really just a pre-structured way to kind of segment stuff. It's really optimized around that shared content. But it's really no different than what your disk subsystems do when they were slower. We had these defragmenter things and they were fun to watch the little bits going around. It's just a way to chunk up the files. They're just pre-allocated. So a manifest is an artifact. That's the way to kind of think about it. When a manifest is submitted, it is that net monitor v1 image. The blobs are the content that make it up. And it turns out there's actually several manifest types that are in registries. There's the one that's probably the most popular because if you're doing Docker build, that's what the Docker manifest is. It's the original one. When it was submitted to OCI, there was a little bit of reformatting of it to make it a little more vendor-neutral and standard. So that's the OCI manifest. Same functionality. If you have multi-arc images, whether it be Windows and Linux or ARM or whatever, there's another manifest list that basically says, here's the separate individual manifest for ARM, Linux, Windows, and so forth. But you can associate it with a single tag. It's a nice little way to kind of get an abstraction there. And then the OCI index is the multi-arc version of that. But there could be more. So we'll talk about that. Let's first talk about tags because this is another aspect of it. When you push something to a registry, you're pushing it as a tag usually. The NetMonitor v1 image, that's fine. If I want to reuse that tag, there's another manifest and potentially more blobs that get uploaded. And this synthetic thing over here that you've called a tag is pointed at the new manifest. That old manifest is still there. It's just called an untag manifest, as usually the terminology. And you might even set up that automatically delete untag manifest as a thing. But there's a virtual concept of a tag and it has a pointer that can float. So recapping manifest or artifacts, blogs of the content. We can secure them in namespaces. All the content has unique identifiers, which is really cool. If you're pushing from two regions to the same registry, we can easily handle. There's no conflicts. We talked about the tags or pointers. And the artifacts in a registry can be from a K to multiple gigabyte. Machine learning folks, man, you guys got some really big images. It is pretty amazing how you stretch the extent of what a registry is capable of. So as I've gone through this, is there anything really unique around a container image in this flow that we just walked through? It's pretty generically useful. So I'll take you through a little journey on how we wound up doing this generic model. So when we, this was back in 2018, we were trying to figure out how to make Helm available within registries because it's great way to deploy. Why not use the registry for it? So we started going down this AZACR Helm repo ad. And before we could even, and it worked and mapped out pretty well. But it was unique to Azure. But as we started, we didn't even get finished and CNAB was becoming a thing. And Singularity was saying, like, we really don't want to host our own registry. Can we just get this working in the existing registries? We don't want to run yet another storage service. So we just kept on going down this list. And we're like, AZACR, AZACR this, AZACR this. Like, this is not sustainable. So we basically said, all right, let's invert this model. How do we get the Helm team, who actually worked for Azure at the time, to want to build this into the Helm client? It only makes sense if it's going to work across all cloud providers. So what we did is we went back and said, what if we leverage that same infrastructure and generalize it so you can store Helm charts in OCI registries and it'll work in Azure, AWS, Google, on-prem, wherever. Any place has a registry. It should just work. Now, all of a sudden, it makes sense for the Helm team and the Terraform team and all these other teams to say, well, why should I create my own registry? Why don't I leverage those ones there? It'll work across all cloud providers. So that was the evolution from container registries to becoming these artifact registries. And all we did was basically relax some strings that were in the registries for how these things were mapped. So we're able to say, not yet another storage service. So that was kind of the evolution there. Now all these things can be stored in them and it turns out that it's so generically useful that we've actually done a bunch of prototypes where a lot of the package managers can be stored in them as well. But that's a whole other thing. So OCI artifacts was about storing other individual things like Helm, like OPA, like Singularity, like all these different things we don't even know about. That's the beauty of this. There's so many different types now being stored within companies and shared across projects. What if I want to be able to associate other things with my existing one? Like how do I associate an S-bomb with my NetMonitor image? How do I create a relationship between those? Because when you look at your deployment chart, all you know is NetMonitor v1, right? Engine X 1.12.6. What we want to be able to do is say, hey, based on that thing, well, first of all, is it signed? Great. Can I attest to whether it's signed by entity I trust? And now I can go back and say, hey, by the way, can you give me the S-bomb for this thing? Can you give me the list of scan results? What is this current revocation status? So when we started down this work, we originally started this reference type stuff with Notary because we wanted to be able to associate detached signatures. Turns out that infrastructure is available for all kinds of different types. So how can we generalize from individuals to enabling how do I find other things based on that same name? I don't want to change the tag or digest because that's what's locked in my deployment file. Don't mess with that. That's what I know. But I want to discover related things. I want to filter on those related things. I might want to sort them because my scan results I do a lot of, so I don't have just one. I want the latest. Or maybe I want the first to know what it was originally scanned as. Oh, and then because we talked about the lifecycle, if I want to delete the stuff that was never deployed, I want to get rid of all of the S-bombs and scan results and all the other stuff because all of that's gone. I never deployed it. Get it out of my system. If I want to copy it to an archive location because I need to keep it for 18 months, I want all those things to go with it as well. I don't want to have to correlate 12 different storage systems to figure out where is all this content that I need to save or delete. So that leads us to the evolution. Does everything have to be a brand new service or can we invest in the existing service that have all those iddies that I don't want to have to think about and users don't want to have to configure? So we have a registry that can store all sorts of stuff. What if you could ask the registry what else it has related to that thing? So we have this project called ORA's OCR Registry of Storage. And I can say from a client perspective, hey, discover what's related to the net monitor image. And now the results can come back and say, hey, here's some signatures. Here's a bunch of scan results. Here's two different S-bomb formats. And a gitbomb. I can hang all of those off my net monitor image. I could even filter. Maybe I only want the scan results. Well, there's three of them. Well, I can order them by date to sending or ascending. And I can also say, just give me the top one. That would be kind of cool. Let's show it. Let's dig into how we can do this today. So I've got just a copy of distribution, actually a version of distribution that's got this functionality in it. Usually I start with a container image. But what I want to show is it doesn't have to be a container image. So at Microsoft, we have to abide by the U.S. Executive Order for providing S-bombs and claims and details, not just on the software we ship, like Office and other things, but on the services that we run. If we run Office 365, we're not going to host the Office 365 service as a container, but we do want to provide S-bombs for it. So here I'm using the Web at Networks company and they have this network monitor service that they run. And they're going to have, let's just call it the 2022 Q1M1 version, that we're just going to create an arbitrarily named thing in the registry to make a reference to. So I just created some JSON document. No big deal. I'm going to push this to the registry using ORAS. Now, this is the original CI artifact stuff we did a couple of years ago. All I'm doing is saying, localhost 5000, that's the registry, and then the namespace is services, and then the artifact is net monitor, and I've got a version associated with it. The manifest config thing, that's again, that's how we did ORAS, the OCR artifact is the first version. There's a way to have a type in there, so I said it's an application JSON, just made up a type. And then the service.json is the stub. It's just that little file that just represents something. There is, if I want to find some information about the service, whatever you want to make. Really, the main thing that I'm trying to do is just create a reference, that tag. Now, let's say I have an S-bomb. I've got a very fancy S-bomb here. I think it says it's good. So it's just version whatever. It might be 200 megabytes long. Some of our S-bombs for some of our services are huge. So we need something that can scale from very small to very large. So I have an S-bomb. I want to push it to the registry and associate it with that NetMonitor 2022 Q1M1 service. Because that's the thing I know. I know of Office 365, some version. I know of the NetMonitor service, this version. And now what's interesting is now I can start saying, show me what's related to that service. So I now have an S-bomb. I've got the S-bomb example. I know of the NetMonitor service. I don't know what the identifier is for the S-bomb, but I know my service name. I know the software I'm looking for. Let's say I want to add some claims. We'll make this a little fancy here. Wait, is that the one I did? Yeah. So I've got some claims. We'll do it there. Let's say that it conforms to some SDF compliance. Again, I've got some document. I want to make it available for people that are looking for the information on the service. I'm going to push this one also as a reference. So push to the repo. Basically, it's push the claims example. That's the type. Think of it as like the file extension. The .json file, the .steve file, whatever. There's a type. I'm taking the claims.json file and I'm just pushing it up. And the subject, the thing it's associating with is that NetMonitor image. Wait, did I push it? Yes, I did. Sorry. I might have some additional annotations. So this is what I talk about. Annotations would be really cool. Do you know the end of life for software when you build it? You think you do. Usually don't. It changes. Turns out it might be more useful. You might have to extend it. You don't know what the next version. I originally had M1, I think it was the version I had up there. Turns out the replacement M2 turned out not to be so good. So we're going to say the new version of the software if somebody's looking for an update is actually M3.1. So if I can add new information to the existing information that's in the registry that I can improve it over time. I can start to add more information. So again, I'm just going to push this. Again, it's just this one says the artifact type is, I just call this artifact annotations. And same subject. I keep on going back and saying I'm adding more information to that specific version of the NetMonitor image. But notice there's no file here. That was the interesting part. And again, if I just look at the tree of information, let's just make this a little easier. I'm starting to grow a pretty rich graph of information that if I'm building all of this in my build system, I can associate them all together. If I want to get something out, this one we can do some better usability on. This is where we are right now. We're using the Aura's Discover command. I'm filtering by the artifact type of claims, because that's the one, and I'm just doing some parsing out of it. It says, hey, give me the first entry out and give me the digest of it because everything's got a unique digest. And then I can say Aura's pull. And now it will pull into a download directory because I want to get it out of the same directory I just created it. And I've given it the reference, right? Because everything in the registry has a unique reference. And if I were to now take a look at the output, surprise, surprise, I was able to round-trip that information back out of the registry. I was able to pull just the claims file. I didn't have to pull all the other stuff. I said, hey, the net monitor reference? By the way, I want the claims for it. Now, here's where it gets really cool. I've used several different tools. I had an SBOM tool. Maybe it's SPDX, maybe it's Cyclone DX. I had a claims tool. I had some annotations that I added to it. If you're trying to copy the files from one folder to another on your computer, do you open up PowerPoint and Visual Studio Code or some other tool like Adobe Illustrator or whatever, you use the file system API. You just say copy from here to here. It doesn't care what the type is. Because we've set up this relationship with the objects in the registry, I can literally go, hey, copy this from point A to point B and take the entire graph. Copy from services net monitor to Wabbit Networks net monitor. It could be a completely different registry. And now if I look at the destination, I got the same graph, but it's at the other end. That's kind of what's powerful by saying, hey, the registries now know about references. I can copy all of that with me, and this is the content I need to archive. This was the thing that was deployed. It's no longer deployed. Let me archive this and keep it for 18 months. And maybe one of the annotations is you can delete this 18 months from today. So that's the model that we've been kind of focused on. Let's check on time here. So it turns out registries already support manifests. We're just adding another manifest type. It's totally within the spec to add another manifest type. This is how index got added. The new manifest just says, look, it's a new type. That's the media type says this is a new schema. We've taken the hack we did in OCI artifacts via the original one, because that's the way we just took the config media type and we surfaced it as a first class object of property so that we can do the filtering by. We've taken the layers and we've named that to blobs because layers are unique to containers. Turns out blobs are pretty generic. If you really, really, really, really need a config object, there's a media type. Just stuff it in the blobs collection and call it the same config object so you can still get it out. Turns out most of them don't need configs. The subject is really the magic piece. The subject is the thing that says, hey, this S-bomb is related to the net monitor image or the goofy service reference that I created. That's the back pointer. It's very similar to how tags work. Tags, there's no document. It's something that we cache or index, but index is a loaded word. In the registry, we store it in a database that says these things are related, but there's no document that gets out of sync. So we can do things like annotations because there's no blobs required. I can just set up, hey, here's just a new annotation. Now, to get things back out, there's a gazintas and the gazoutas. If I'm putting new manifests in with the subject property, we index that information. So when I call this new refers API, it's the opposite. I call an API to get the content, and I have another API that says, I want the related objects to that net monitor image. So it just returns a list of descriptors with the artifact type so I can decide whether I actually want the signature or the S-bomb. That's how the filtering works. So it's kind of going back. This is actually originally what we did for the notary work, is that we can sign stuff without changing the original digest. Turns out we want to be able to add S-bombs without changing the original digest. We want to keep them alongside so that we can copy them and we can pull them independently. Just like when you go to the airport, you don't walk through with your passport. You give your passport through the hole, and if they like you and they approve you because you weren't revoked, they download the rest of you through the security checkpoint or onto the node. Let's see what else. That's basically it. So there's a couple of links here for how we've been doing the work. We originally had OCI artifacts. That's there. That's the single individual objects. The ORAS artifacts, we think it was reference types. We've done the incubation under ORAS. We're working with OCI to get that submitted as a standard. ORAS and OCI obviously both are different projects, so they're available for everybody to use. Then the ORAS CLI is how I was using it there. Think of it as the file system API. But if I'm in an application, I can do file save and file open within an application. Those are libraries for how to interact with the file system. We have ORAS Go, which is how you can build your S-bomb tool to interact with the registry. We really want to make sure that these things are available for other people to use in their tools. Your users shouldn't have to think about ORAS. They want to think about your thing, your S-bomb tool, your scan results, whatever it is you guys are going to creatively come up with. As you can use the ORAS Go libraries to say, look, I don't want to be dealing with storage things. I want to leverage what's already there. So let me just use this library and I'll do the push-discover pull. And then we've got a fork of distribution that supports all of this. So that's it for the talk. We have this running in Azure. AWS is coming soon. Docker is working on their version of it. We're looking for anybody else that's looking for support and new innovations, new ideas. Our goal here was to not worry about a specific type. It was to enable the ecosystem to store all these things and not have to create yet another storage solution. So I hope that was helpful. I'll stick around for any questions and thank you for coming.