 I know there are a few more people that are going to show up, but I'll go ahead and get this thing kicked off here. So we're going to talk today about modifying the mutable. This is how we're going to attach additional data to our OCI images as we ship them out into the field. And so quick background on me. My name is Brandon Mitchell, as the title said there. I do work for BoxBoot. We are now an IBM company. Big Blue bought us a while back, and we're always looking for people that want to either be our clients or engineers that want to join us as well. We do a lot of consulting around this stuff. Relevant to what we're talking about today, I am an OCI maintainer on the image and distribution specs. So when you see things we're doing wrong, feel free to complain. You know, you can throw them directly at me. And you'll see me picking on Reg Client, a little command line called Reg Cuttle and a bunch of my demos. That's me picking the tool that I know the best because I'm a maintainer. I happen to be very familiar with them because I'll be able to get things fixed on these slides because it's me. So I'm not trying to pick favorites there. It's just the one that I know the best. All right. For container images that are out there today, I'm hoping that a lot of people here know what a container image is already. You've probably seen it. There are layers that build up these container images. You've likely run the build commands. You kind of have a rough idea what's in there. You've probably seen there's metadata about it. You say, this is the command I want to kick off. This is the environment that's going to be shipped with. Labels, things like that. And lately we've probably been seeing a lot of these multi-platform images out there. People get the Mac M1s and they say, we want to be able to run this stuff natively on our machine. We want to start shipping an image that can run both on the Intel and the Mac all over the place or maybe the Windows machines before they were doing that. Being able to have these cross-platform images. So all of that falls into how this works as we're shipping these images on the specs. And I want to hit a little bit of that and give you an idea what's happening there. But first to talk about the lowest level is we have a content addressable store. The content that we ship across the container registry is named by the hash of that content itself. And so whenever you're looking for something to give it the hash, it gives back content that extracts to that hash. And so as you see, I'm running this thing through there. I run it through a SHA-256 checksum command and it outputs the same hash as what I put in there. And so that gives you some guarantee that nobody changed in the process there. You know that content was never modified in transit. It also tells us, in this case, the content that I'm looking at is a layer. And so if you know container images, you know layers. You've probably seen more than enough of these in your time. This is just a tar file with Gzip compression. And we can extract this. And this is one chunk of our images that we see out there today. So when you have these images shipped around, this is probably what most people think of as the image. The other part you might not think about as much is all the other metadata, the JSON data in there. In this case, we're shipping also the same blob API. So we were shipping before just a stream of tar. This is now a stream of JSON. And in this case, I've got the path, the environment that I'm using, labels, the history in there. If you're curious on that, you can do like a Docker history command and investigate all the steps that were used to build up the image before, and you can do a little bit of history checking on that. All of this is just metadata on image. So we've got the metadata that's there. We've got all the layers. You need to assemble those two things together. And to do that, we had a different API and OCI, which is the manifest API. And it outputs some different JSON. We're good with JSON and OCI spec. So if you ever need somewhere with JSON, we're a good place to be. In this case, we have got a config. And so that was the config JSON. And then we've got an array of layers. And so these are all your layers of the file system. You just apply these one by one in order. And that's how you get your container image built up. This is how this is all stored up on the container registries themselves. And so probably some detail, some people don't care about so much, but it's going to be important later on. We can start getting into some of the details. You'll see these patterns repeat over and over again. We've already picked different decisions as we were doing things in the future. So with this, I think people will get a little tired about seeing text on the screen sometimes. So I'd like to draw at least somewhat semi-pretty pictures. Maybe not quite. I don't see tomatoes flying yet, but that's probably just because that wasn't on the snack list earlier there. But the V3 is a tag. And that tag, it is mutable. It can point to different manifests over time. So you can have a point to one manifest one day, a different manifest the next day. You can have multiple tags pointing to the same manifest. Think of it as a symbolic link. The manifest itself has its digest. And so you can hash that content. That hash itself has inside of it all the node itself that you were just hashing for the manifest has all the hashes of the config blob, of the layers all in there. So what we get there is a nice little structure where you know that nothing changed in the process. When you pin your images and you're saying, I want to run this image with this specific hash, what you're pinning is that manifest hash. And so if you ever say, I want to pull down this one specific image, that's what you're pulling down is that manifest hash. Then result of this, we get a nice immutability result because what we have is that content addressable store we were first talking about. We've also got a structure that is called a directed acyclic graph. That structure kind of tells you there's no loops in the data structure we're doing. But more related to what you're probably familiar with if you've done CS classes in the past is a merkle tree, which is that we create a node inside that node with the hashes of all the child objects and inside of those child objects, the hashes of all those child objects. And the content of each of those things is itself hashed. The hash of each of those child objects is the hash of the data. So you know that if anybody changed one of those child objects, it's hash would change, which would change this parent node, which would change this parent node. So it gives you that nice guarantee that nothing there has been modified. You get all that immutability promise, which is going to be a problem later on when we're talking about some of this stuff. And so we need to fix this of, you know, how do we add extra data to this? But I'm jumping ahead of myself. Another type of manifest out there that got added a while later was to say we need to have multi-platform images. And to do that, we created a data structure called a manifest list, an OCI we call an index. This is just an array of manifest. Now, when we had the layers, and we started pulling all of those, we didn't really care about which layer was which. They were just in order, and you just applied them all in order. When we're pulling down a manifest, we want to know which manifest we want to download. And so to do that, we need some extra data in here. And I failed to use one of the terms there. Whenever you see one of these fields that has the digest, the me type, and the size, that in OCI is called a descriptor. And we use that as all over the place in OCI. It's just a pointer to another piece of data somewhere in OCI. But we had to add onto that descriptor some extra data to know which manifest to pull. And so, since it's a multi-platform manifest, we add the platform field in there to say, here's the platform, if you're looking at this list, to find the one you're looking for is a runtime. So it gives you some kind of context when you're looking this up. Okay, I know which one I want. And now you can pull that one specifically for your own use case. You don't have to download every single manifest just to find the one you're looking for. The resulting picture is going to now look like this, where we've got our tag which is going to point to the index, which is our list of manifest, which under that, since the list of manifest, is multiple manifest. And then each of those has their own config, each of those has their own set of layers. We have this nice big structure here. And again, this is all a merkle tree. You get that hash of that index. You know that none of the content under there has been changed because those hashes have been pulled up. All right, so that's one half of the background. One other piece of background here is that a lot of people are pushing artifacts up to container registries today already. And so to do that, I'm just going to take a little bit of data. I'm going to say it contains electrons is my data. That's really fancy. I think pretty much all the data we push up to registries contains electrons. This is the simplest build materials ever. And for this, I'd push it up there as a blob. A blob before you saw me pushing a tar, you saw me pushing a JSON. Registries don't care what you put in a blob. Put whatever you want in a blob, they'll take it. Random arbitrary data you want to push up to a registry. But blobs themselves are not really preserved by registries. They need something else. You can't tag a blob. And if a blob is left without a manifest point to it, you don't have a good way for the registry to keep track of it. So it just says on this next garbage collection around that, hey, nobody's using that. Let me go ahead and clean it up. We don't want that. We want to keep these things preserved, right? So to do that, we need to wrap that blob with a manifest. Pretty simple. Same manifest we were using for the image. We used the exact same media type for this thing, the exact same data structure. I put in here the media type of the config is now the media type of whatever our artifact is. And so I said this is the electron bomb config. And then my layer is my actual data that I'm shipping out. And so I shipped out this layer of the electron bomb data. And there's the digest of my blob that I was looking for. The other one, the digest in there was kind of an empty thing. I'll circle back to that later on, because we're talking about more details and what we can do in there. But this is what we're using today for Helm charts. This is what we're using today. People have a flux and they're using GitOps and flux to store their data on container registries. A lot of existing projects, I think Singularity out there, they're using this exact same structure to push their data up to container registries. You don't need any new fancy stuff. This exists right now in OCI today. So you can push all the data you want up there. So now the challenge. The challenge is we've got all this data. We're starting to ship up to registries. We've got S-bombs out there. We've got annotations. We've got vulnerability reports. All this stuff is related to an image. And so when we pull down our app image, we don't necessarily know what is the metadata that's associated to that image. What S-bombs are associated with me? What signatures are associated with me? We need to be able to make this connection between the two. And when we pull down this image, that image is immutable. It's got that digest. People depend on that pen. They don't want to see that thing change. So how can you get that reference to that thing that is pinned without modifying it? That, hence the title of the talk. So how do we do that? There have been multiple suggestions of how we could possibly do that. One of them early on I've seen was to say, well, I'm the image originator. A docker actually came up with this one and said, hey, we're building all these images. We're originating it. Let's just go ahead and throw it right in the index. And so they started throwing extra data in the index. They said, hey, I've got an extra record in here that I'm going to point to my previous record in the index. And that way I know whenever I pull this one list of all the manifest, I've got not only the image manifest, but I've got all the metadata in here along with it and I can just kind of query it. That works good if you pull it from that top-level tag. It doesn't work good if someone accidentally decides or intentionally decides to pull it from that digest of that platform-specific manifest because there's no way to come back up to this. You can have the same single platform manifest listed in a whole bunch of indexes. You don't know which one they came in to download from. And yet there's also the challenge that some people decide to pin on this index itself. And so this index has a digest. People pin on that. And so now you can't change it. You can't mutate this thing without breaking anybody's workflow that depends on that digest, not changing. So how do we do this? We went ahead and spun up a working group in OCI. We decided, what else can we do but basically build a group of people that want to decide on this stuff, design back committee. It's a great way to do solutions, right? We did have a list of requirements in there. Some of them were things like we wanted efficiency. We wanted to be able to not only make this quick and efficient, but we also wanted to say, let's attach and detach content after we've shipped it out. So we're looking at saying when we ship an image out, maybe upstream is attached a whole bunch of S-bombs, and we only care about our one-specific format and our one-specific, you know, whether it's JSON, XML, Cyclone DX, SPDX, pick your favor of those. Some downstream user might only want one of those. Maybe they don't care about any of the upstream signatures because they're going to re-sign this thing downstream in their own organization by their own internal key so they don't even need to pull down those upstream signatures. So they may want to detach all this data that you're assembling upstream, and they want to attach their own data because they're going to do their own vulnerability scan in-house and they're only going to attach their own signatures. So we need to be able to attach and detach data. That was a big key requirement. We also admitted that we didn't know what people are going to use this for later on. Today, people are using this for S-bombs, signatures. You've got to add the cute animal pictures to their images when they ship them out so that it, you know, follows the Get Standard whenever someone submits a PR. You've got to add the cute animal picture, right? We don't know what people are going to use this for. We don't have an idea, so we don't want to try to build, like, a distinct set of things, and you can only use it for X, Y, and Z. We want to keep this very flexible. The good news is we weren't working from an MP box. We had a whole bunch of solutions already out there. We had people that are doing this today. If you look at the ORIS group, they were coming out with an effort of their own manifest for an artifact. So they create a new manifest that Registry is going to work with. And they had a field in that manifest called Subject. That field pointed back to the image that it was extending. And so you would create this new artifact on the right. It would be something like your S-bomb. It would have a subject that says, I'm the S-bomb for this one image, whatever your application image is. Now, when you pull down that application image, you don't necessarily have a pointer in there to go back and find this. You don't have to create a refer API that they could query any time they needed to to say, given this digest of this application image, tell me all the data that points to it. So they created that. There were some downsides to this. One of the downsides was, first off, this artifact manifest. Since it wasn't known by registries, it would not work with a single registry out there a day until that registry got upgraded with this new functionality. That has an adoption challenge for us. And it also means anybody that implements this that registry has also been upgraded. Adoption comes very difficult when you're relying on people to upgrade their registry servers, especially when you talk to someone in the DoD, behind an air gap that never upgrades anything for five years. It becomes a challenge. The other difficult effort in here was that refer's API didn't exist on any registry, so that was another bit of an adoption challenge. We can at least solve the first problem real easily. We can add extra fields to an existing manifest and follow the OCI spec perfectly fine. We can allow that as JSON. You can add additional data to JSON, and we're still within the spec compliance with old spec, and the new spec tells you how to use those new fields when you see it. And the old spec just says, if you don't know what you're doing with it, just kind of ignore it and move on with your day. It's extra data you don't need. So we can do that. That's half the problem. But we still have the first half of the problem where we've got this new API we need to be able to find this. We don't have that API yet on existing registry, so you'd be pushing this data with no way to look it up. That's no good. There's another option out there which was, hey, if anybody out there is familiar with SigStore and Cosign and SigningImages out there, we've got a way already they use today which they make a special tag out there. Tag has a big long SHA-256 checksum digest in there of whatever the images that they're extending, whatever the images they're signing, they've got that digest in there. So they basically leverage that tag API as their way to make that backwards pointer that they needed. There's another option. And so the downside of this is that you do have some race conditions out there. If two people are generating some metadata simultaneously, they're going to look and overwrite each other and potentially have conflicts. And also, people are looking at the tag listing. They don't like seeing a whole big long list of digest in there. They want to see that I've got version one, version two, and version three, and they want to be able to pick whatever the latest one is without having to sort through 500 pages of a bunch of digest that they didn't care about. So given all that, we pick, well, it was designed by committee, so we picked them all. The answer is yes, we picked everything. At least initially. And so initially we said yes, and then we kind of came back and said, you know what, we're going to take a defer on that new artifact manifest. We're going to push that one off until later. So we initially had in there that's been pulled out in the most recent release candidate. But we did go ahead and add the subject field to the image manifest. We also had another field called artifact type in that artifact manifest we thought would be useful. So we added both of those in there. And we wouldn't decided to say, we'll go ahead and make a refers API. But since not every registry today has that refers API, hey, we got a cool backup plan here, which we can just push a tag. And so the result here that we came up with looks a little bit like this, gets a little bit busy. But we're going to have our image on the left-hand side here where we save our images, whatever image spec it is. And we're going to go find from the digest of that image on the left, go look it up and say, is there anything that points to it? Go find this tag with that digest. And we see that we've got this index that is, from that earlier slide, a manifest list. It's a list of manifest. And it's going to point to all of our different artifacts that we had that are extending this image. So we got that. And when we add the new refers API conveniently, it is the exact same picture. We just changed one field in there, we changed the server instead of that tag that we were managing at the client side. But the response is an index, which already exists in the specs, so we didn't have to make any changes there at all. That data was already existing, it just works, it's nice and portable across everything. End users that are parsing it can parse the exact same data, whether they got it from the API or from that tag that they queried. In text, because I can't get too far away from the console before jumping back to it again, what it looks like is I query that tag with that first digest up there. So I saw I had an image with this digest, and so I found a tag that has a very similar format. You swap a colon out with a dash, and you suddenly have a tag instead of an actual content addressable store. And when I query that, I get this index back, and I started putting two extra bits of data in this index. So in that manifest list, I've got the annotations that I pulled up, and I've got the artifact type. So this is very convenient. When you see that whole list of manifests, you need to know which one is the artifact on a DX first, which one is SPDX first, which one is our signature, and anything else we're looking for. And maybe you've got five signatures on there, so you want some annotations in there, say this one was signed by Bob, and maybe you've got three different vulnerabilities out there, and you want to find out which vulnerability scan had the most recent creation date on it. You can put all that in the annotations so you can sort out which one you wanted to pull down. So where do we get that artifact type and those annotations from? Well, it's from the manifest that goes up into the listing when we generate our listing either from the client side building our tag or from the referrers API generating this on the fly for us. We've got the artifact type we can put in there. If the artifact type is not defined, which is not in a lot of the current data, if you look at Helm charts, that sort of stuff, we just only define this the most recent release candidate and nobody's using it yet other than maybe me. We've got this where we fall back to the config media type, and so if you don't have the artifact type, you look at what the config media type is and you work from there. Additionally, all the annotations, we pull those up, and then we have that new field at the bottom of the subject. So that says whenever we run this query for the referrers API, whenever we're looking for that tag, we're looking for everything that has the subject field with that one digest in that subject record. So that's how we found our data. All right. A lot of slides, a lot of talking. Let's actually see some demo and see some type in here. See something happen. So I'm going to spin up two registries and I'm going to set up a couple of variables just to make these pictures a little bit easier to follow, but the first registry is on port 5001. The second registry is going to be on port 5002. In addition, I'm going to spin up a couple of variables here for my media types. They're going to be the OCI manifest for the image and they're going to be the index. So I kept everything with the OCI media types. There are Docker media types out there that you would have for some of your images or other things like that. I kept my examples very simple. I'm going to use that in some demos where they've got some experimental features that are just being worked on now where you can start shipping your stuff with the new Refers API in Cosi as well. So for the first registry that I'm spinning up is going to be the CNCF distribution project that has the original Docker registry out there that people might be familiar with. If you ever said Docker run registry colon 2, that's what I'm pulling up here. And this does not have the new Refers API. So this is mimicking a registry as it is today without any of the new functionality. I'm also going to turn off all the TLS because as fast as I'm typing on these slides here, if you watch me type a bunch of open SSL commands, you will be asleep by the end of this demo and I don't want that to happen, right? The second registry I'm spinning up is a project called Zot. So that one a lot of you probably haven't heard of. They are one of the early adopters to this. They pretty much made an OCI-specific registry that only implements the OCI media types. And they had the new APIs in here for at least one of the earlier release candidates. So I did kind of back some of my demos up to not the most recent release candidate with the artifact type, but I got at least a couple of those. Lastly, I'm copying image over. So I just copied one specific platform over into the first registry and I'll go ahead and grab the digest from this. And from that, we're going to see is this just a regular image? We haven't done anything fancy yet. We just spun up two registry servers and I've got one image copied over into my registry. And if I scroll up here, you'll see I've got a handful of layers. The config is in there like you've seen before and this is just a regular image manifest. Nothing fancy. This is kind of what everybody has today as you have an image that's on a registry. So now we need to attach data to it. So for that, what I've got for this sample is I'm going to go ahead and use sift because they are the nice easy button for me. I promise to determine people when you get me the easy button I will use you in all my demos just going this point forward. But I do have, I'm piping the output of that into this RegCuttle artifact put command. I pass the dash as subject to say I'm going to push this artifact with a subject field point to this other image that's already up there. So I've got some media types. So I'm going to say this is the media type for my first sample of Cyclone DX. So I set that for both the media type of the layer and the config media type. I can pass a couple annotations. Click that command and boom, I now already did the first half of the demo here. I've got the artifact uploaded to my registry with the reverse API and everything. So the second one, if you ever pick one sbomb, you've got to pick the other one because otherwise people will blame you for picking favorites. With the spdx example, again the same idea. I've got the subject field specified in here to say you set my subject to this other image that already exists, set my media types, set an annotation. And now for the third example, I will go ahead and spin up cosine. And so cosine normally when you do the sine here, it spins up an extra tag on the side with a dot sig on the end. In this case I specify an extra flag in there with the experimental environment variable as well that says go ahead and have an artifact in there. With that, I just uploaded three different artifacts into there. Sorry. So it's uploaded. How can we query it? It's no good if you can't query it, right? So for that, I'm going to go ahead and say let's do an artifact listing. And if I say do an artifact listing for our app image that's already up there, I now have three different artifacts associated with my container image. And let me go ahead and show you, you can see the tag did show up. This is the red tree. It doesn't have the new API. But if I scroll back up, you can see that this has got three different descriptors. And the first one's going to be for, let's see, well, first it is an image index as we've already gotten an OCI already. We've got the artifact type. And the first one is going to be cosine it is. And then down below we've got cyclone DX and down at the bottom is going to be SPDX. So there are three different things we've got referencing our image. So just like that, I didn't have to know anything else about my container image. I pulled down and had that one, you know, digest whatever it was, its name, and I was able to find all the data that was associated with it. So now if I do the manifest get on that funny looking tag with the SHA-256- and the rest of the digest in there, you see that exact same content. That's exactly how I got the tag points to the exact metadata you saw earlier. So that's how we're shipping this stuff around the registry. So this is something that the clients have to manage, though. If I run this artifact get command and I say, get me the artifact with the subject field of this app image and I filter and say, get me just the cyclone DX, this is the easy button for the end users. They don't know anything about how we're associating this data. They just say, get me the artifact of this type that points to this image. And just like that, they're going to click enter on this thing and boom, you've got your S-bomb right there. So super easy for end clients to pull this stuff down if they need to. All right, so we were able to query it. I did promise a couple registries. I got the new version in here as well. So if I use Zot, I'm going to copy this image over from my one registry over to Zot. I kicked this off in two phases because Zot currently has a bug where they want to have the refers pushed after I pushed the rest of the layers. I'm working with them on that one. It's an open issue with them, and they're looking at it right now. But I then pass this dash dash refer flag and that just says copy not only image, but all the refers with that image. So just like that, I now have all the content copied over there. And if I do the artifact listing, I'm going to see that I have my image and I'm going to see that I also have all the different content that points to that image. And there's same three artifacts. The SPDX, the Cyclone DX, and the Cosign in there if I go through all those. So the nice thing, though, is that earlier when you were looking at the tag listing, if I run the tag list command down below, you're going to see just app. I don't have that long digest in there. So for the end users, this is nice. For the users pushing this stuff, it's even nicer because this is all managed by the server. That refers API is what I hit. Since it's managed by the server, we got rid of all the race conditions. You don't have to worry about two clients colliding with each other. And we've simplified the workflow a lot for the end users that way. So that's the easy demo right there, right there. You know, we've got this. This is just in tools that are out there today. You can use this. Let me show you that it isn't just these two tools. So I'm going to pick on Oris. They're another one of the projects out there in the ecosystem that has their implementation of this. They had previously done their own version, and then they adopted the new stuff. And so you can see if I pick both 5001 and 5002 for my two repos, I've got those artifacts, so it found it with the tag and with the refers API. If I query Cosign, both on the repo one and repo two, you're going to see it's going to find the sighing data that puts a lot of data in the Cosign output, but it was good. It was a verified output that it saw both of them there. And I can also hit a regular registry in production, GitHub Container Registry out there in the field today. I can query that and see that I have a container image on that registry right now with artifacts attached to it. So this is just in the real world. You don't have to wait for somebody to upload something. This is real right now that it works. That's because a lot of what we design here works with registries as they are with a forward, you know, upgrade ability and they're built into what we made. In addition, I showed you that Reg Cuddle can do it in existing registry for us as well. I don't want to pick all the tooling favorites, so I can show you that as well. And so just like that, yeah, we've got this out there for multiple registries in the real world, multiple tools that's using us. So where are we at? Well, mentioned we are with release candidates. So we've got RC3 for the image spec, RC2 for the distribution spec where we've got this data out there. And so this in OCI, we've got a spec that defines how we define all this data, all those fields we're defining in the image manifest, all the JSON out there. We also have the distribution spec that has all the APIs that we added. So the new referrers API and how that response is supposed to come back, that's defined in the distribution spec. So that is out there in a release candidate, which means we're looking for people to test this out and let us know how it works for them. In addition, the image manifest, we all have this new artifact type field in there, and so I want to give you a little bit of an idea of what's going on with that. What you decided was not everybody has a config that they're pushing up there, and so if you don't have your own dedicated config, say you're just trying to push an S-bomb, it'll be nice if you didn't have to make a config for that S-bomb, and someone upstream like the SPDX crew or someone like that has to define some OCI-specific config data to ship their S-bombs. It's nice to say I'm just shipping this SPDX JSON data and to not have a config, and so you can define that just by specifying the artifact type, and then we have a field in there I think I might have had on earlier slide where the config just has a scratch field in there, which just says it's empty, there's no data in it, and so if you specify that, we've got that in the spec now of how we want to do that. And otherwise, yeah, ready for testing, ready for people to play around with is what we're looking for. In terms of registries, both Zot and Harbor have support for the RC, RC1 slash 2, depending on which of the specs we're looking at. I think Zot is quickly coming up with the release, with support for the latest releases. I think it's all PR getting merged like earlier this week, so they might already be there. Harbor adds support for one of our earlier release candidates, so they'll probably have a little bit of work to get the next version updated. We are blocked on a couple of registries, so if you're using Docker or Hub, if you're using ECR, those are explicitly blocking this kind of data. What they're blocking it for is this very last bullet point on the slide there which says that registries need to go back and retroactively find all the stuff that had been uploaded before they had the new API and include that in their output of the reper's response. And they said, we don't want to retroactively go back through the entirety of Docker Hub. They've got a little bit of data. They don't want to go back and re-scan everything. So what they've said is they don't want to go back and retroactively do that. ECR, they have a different take on things. Anything that isn't explicitly in the spec, they don't support. And so even though the spec says you're allowed to extend stuff, ECR says we want to keep things locked down a little bit. And so because of that, until we hit the GA, they're going to hold that on lockdown and they won't support the field. But as soon as we hit GA, both of those should unlock very quickly. And they'll be supported on both sides. And I've seen Docker Hub do work on their part. They actually had this enabled for a short while before they realized we were bickering back and forth and they turned it back off real quickly. And they just told us that they're ready to go. They're just waiting for us to hit the GA button and they'll turn it on. So we're looking good on both of those. GitLab is on the slide because at last check, and I'd love to work with them to see if we can get some better progress on this, they have an allow list of what other kinds of artifacts they allow on their registry. So if you're not a container image, if you're not a Helm chart, you need to go to them to get them to put you on an allow list of different artifacts they're going to allow on their registry. Anything so that you're just going to be flexible on what they're doing, but that's where we're not there yet today. So other than those, if you're not on this list, either I haven't tested you, which good possibility I haven't. There are a lot of registries out there. But for the most part, registries just work with this because this was all built in the spec already and then we're extending it with other fields. And so this is something you can use today on just about everywhere. The request that we got for registry servers, please don't filter. And then, yeah, that last bullet point, making sure that anybody pushed some of this data before that you support. And the way you can tell if you're a registry operator is that you can go back and look for anybody and push that special tag with that big long digest at the end of it. Then you need to go look for this stuff. So hopefully it wouldn't have to scan every repo, but for safety sakes, some folks just said, no, we're going to wait on it. For the clients, I think we got a couple of people here on the client side in the room. For them, we're looking for more client support. We've got a bunch already. So we've got CoSign out there. So those are up and running with this. I'd love to see more support. I'd love to see if you're running that SIF command and you say I want to scan that, generate the S-bomb for this image, that it just pushes it up there as a refer immediately right away to the side of that image and you don't have to do this actual little pipe but throw it a command that uploads it to the registry. I'd love to just see it automatically shipped. That would be a great place to be in. For clients, they do need to manage that fallback tag. And so that does have the risk of race condition. If you have two different clients updating this tag I'd say that I put that in the spec. I've documented it and then I'd shot myself in the foot the other day because I had a concurrency to my own code and then I'd forgot that I had this issue in there and so all of a sudden my code was concurrently updating the same field and over it itself. I've since fixed that. Other requests, we've got people pick appropriate artifact types. If you've ever done anything in terms of IANA and registered me type with IANA, they have got a way to register them. If you don't want to register it, you can use like some reverse DNS syntax in your .notation in there. They've got ways you can specify that. So with that, we're hoping that you can push stuff up there where it belongs to your project and nobody else you don't have conflicts on artifact type. But also that people can find your stuff. So if you push an SPDXS bomb, it should not matter which tool you use to push it, everybody should be able to find the same S-bomb up there. All right. And the last request is to use annotations responsibly. What do I mean by that? When we generate that request response, we've got all the different descriptors in there. We pulled up all those different annotations up to that request response. And that can get to be very big if everybody starts putting a whole bunch of annotations inside of their different artifacts they're uploading. And so we've heard back from registry operators is we've got a four megabyte limit on that overall index response. And they want to be able to have 100 different responses minimum inside that listing. They kind of want to just be able to cap it. And we have page nation options in there. So if you have more than 100, you can hit the refers API and say give me page two, give me page three. So with that, what they're hoping for is that they're going to cap it. If you start pushing annotations to extend past, I think we're looking at 400 kilobytes, they're just going to reject your manifest upload and say you got too big for us. We don't want to store this because we can't be able to put in the refers response. So we're just going to reject it in some of these registries. If you want portability, don't push a whole lot of annotations up there. Keep it to just the stuff you need to be able to find the data you want and not all the other stuff. So push through the signer is not everything in the world. No kitchen sinks. So there we go. I think I saved a few minutes at the end here for questions, but it exists. These slides are up online. So the QR code takes you to the presentation repo. This should be the first presentation up there as it is now. You can even click through the examples and everything. The demo script, I was running through there. You can run that shell script on your own and try it out on your local machine and see how it works. I think I'm kicking off commands like SIPT and other things. You've got to have some of these tools pre-installed. And yeah, if you have any questions and after the fact from recording something like that, I'm up on Macedon. Issues on GitHub are always welcome. I'm in the slacks. I don't pay attention to Twitter quite as much, but I am up there still a little bit hanging on. But otherwise, people in the room, any questions? I know I went quick. Come on, Phil. You've got to be wondering how this stuff works, right? Go ahead. Questions back. Do I need to get a mic? We need to get a mic to you. Is it back here? Talk like an ice cream cone. Yeah, okay. Got it. Most of all, I love the OCI storage kind of movement. For me, mind-bending. But what happens when like two people upload to the same before, like the same as, but I upload the same as to the same image twice at the same time? If it is completely identical, we've got the content addressable store, the digest match. And so the registry itself sees I've got this same layer being pushed up with the same digest it deduplicates on the registry side. What happens? It's also content addressable. It's got the same digest on the manifest itself. That JSON gets serialized and then that serialization gets digested as well, hashed as well. And so that would match as well. If they do truly push to separate, similar but not completely equal, maybe they aren't completely reproducible, you've got a different time stamp, something like that, you would end up having two different s bombs attached to your image. But maybe you'd have metadata in there saying this was different creation time and you would prefer the newer one or something along those lines. So when two people upload two manifest to the same index, it's not like it won't overcome the same one? If you are using the Refers API, the Refers API is generated on the registry side, it's doing all that concurrency work for you. It's deduplicating, you know, it's doing the lock contention or it's eventually consistent. Either way, it's going to sort that out and you'll have both of those in your response. If you are using that tag, then the client is responsible for that and the client is doing the best effort. And so what the client is going to do is it's going to pull the tag as it is right now and the manifest uploaded. It looks empty or maybe it's got one from something other previously before and if someone else did the exact same thing simultaneously and you both did the same thing, the first person to push would get over it and by the second person to push. And so that's where that race condition assists and that's why you want to get this onto the server side. That's why it's better to have the server doing the Refers API than it is to have clients doing the tag. But yeah, that's a limited race condition so my question is in terms of attestations, I apologize if you covered this already but it's the, for example, SolSatastations, you can upload that to OCI, but what about other ITE6 in total attestations, how does it differ? So if I want to upload SolSatastation on another ITE6 that could be a runtime attestation or something, could I do that and could it differentiate between the two and pull down something? So in response, when they're coming back up to here, they should have a different artifact type for that. And so in the listing here, you would pick a media type for your specific artifact and that artifact type field would be I've got this kind of attestation with this data in there and if they're both the same artifact type, maybe you have some other differentiation that gets included in those annotations as well and so either of those, that's going to be in your artifact that you upload in response. No, no, no. No tough questions, but I'm pretty sure sadly I don't keep up well enough, but on your list of clients, I'm pretty sure the Notary V2 Latest RC has, or the Notation client has refers They do. They use ORS as their backing. So they use ORS Go on the back end so that yes, they do. It is good to have more clients in there though. Okay, I have to ask. What happens when you delete a manifest? Oh, the garbage collection question. Someone's going to ask that, right. Okay, there's debate. OCI is good for having debates and so we haven't gotten to the bottom of that debate. One side of it says that if you have got one of these things uploaded to a registry with the refers and the subject points to a manifest and your artifact itself isn't tagged, there is nothing pointing to that anymore and so the registry should be allowed to garbage collect it. The flip side is that if you're pushing this stuff up there with the fallback tag, that fallback tag itself gives you that persistence so the registry isn't going to clean those up and so if you want to keep consistency with the old, you would not delete them. And the third kind of take on that is that we haven't defined garbage collection anywhere else in the OCI spec so let's stay away from this debate and avoid it entirely. And so we're somewhere in the middle of all three of those right now. It's not the easy, not the good answer, but yeah, there is debate. If there's debate, how do registries treat it today just however they want? It is implementation specific, yeah. And so each registry can say we're going to implement garbage collection this way. For example, if you talk to some registries they say anything that's untagged is now eligible to be deleted and we're going to clean it up but if you talk to Docker Hub they'll say if you have tagged something and it has had a tag at some point in its lifetime and that tag has not been deleted they keep a history of everything that tag has ever pointed to and they don't garbage collect any of those objects. So different registries implement garbage collection differently. There are registries out there that just delete everything after 48 hours. They just have a routine thing, open registry, something like that. So different registries have different implementations and so we've left that up in OCI as an implementation specific detail. So if you needed guarantee that an SBOM would be available in a year or two years then you need to make sure you're picking the right registry. Yeah, don't pick the registry. It automatically deletes everything after one day. That would be the good option. But beyond that, I think the safe option is if you're pointing your artifact to a manifest that still exists I think we're expecting that that artifact is going to stick around. We're hoping so. Okay, one more for Parth in the back. Take the mic. For the recording. What is the timeline for the OCI registries to implement this? Or is this already live? Oh, timelines for things. We're terrible at timelines. We debate too long on this. Whenever we finally get the GA release is, I think, the big thing. But the there you go. The important detail here is that this is a release candidate. So it is out there. Everything that we showed without the whole referrals API is within spec, and so it does work. So I think the thing some people are waiting on is to say, hey, Docker Hub doesn't quite support this yet. We want to see that support. We want to see ECR. That's going to wait for GA. We have another release candidate that just got put out there. We probably have one more after that because we want to get some compliance code and some other stuff update on our side. We got a few loose ends to tidy up. But as long as we don't have any major changes, we're going to be able to do that. So I'd love to see it over the summer go out. Cool. Oh, one more. Yep, you're on. Yeah. What if I want to point like an artifact to an artifact? Can you, like, have a tree? Like under my... Say a little letter. I have an artifact. I connected my S-Bomb to my image. And now I want to connect a scan to my S-Bomb. It's like a CVE scan to my S-Bomb. Mm-hmm. That's somewhere we want to get to. And I think that comes down to the client tooling that needs to be aware that there are these referrers out there. And so the scanners, we'd love to see this get implemented in the scanners to see that there is an S-Bomb in this image. I don't have to re-scan the image. I can just use the S-Bomb to attach the image and run my vulnerability scan based off of that. We want to get there. And so that comes down to the client tooling starting to implement this directly in there. And a lot of scanners these days allow you to input the S-Bomb in there directly instead of inputting the image. And so you can manually do it if you wanted to. I think with that, thank you.