 Wow, that's a lot of people. This is getting more popular every week and right at nine o'clock, too They folks, you know audio check sound good to me. Yeah, we can hear you Everybody's so quiet. All right Slack sorry hack link posted if folks can just check in Do I see Dan here yet Yes, great. Okay, folks a minute for those locked people here already so Hand off to Dan a second. I'll do a quick intro to how this came to be Then is that Google and has been working on some of the secure secure What the hell is that substand for now secure software foundation is that where it is? The open source security foundation open source security foundation. Thank you, and I've been tracking it also with the Notary in registry work just to kind of keep the pieces combined and Dan had reached out on a mailing list And they got forwarded to me and we basically were able to connect On some various efforts and surprise surprise that looks like then You know has also taken a very similar approach than what we were doing with the notary work specifically the MV to client around signing and was actually also hitting up some of the limitations that we're Working on now with the registry part. So I thought it was a great idea for him to come and join because as with many of these efforts There's they just become this Retangling of dependencies that people get blocked on Because they're just in so many places and I'm hoping that the work that we've been focused on recently to get the reference types and art of Registries will kind of unblock this last piece that Dan's working on and we can talk about the synergies and To go from there. So with that, let me hand off to Dan and Take it away Sure. Can everybody hear me? Well, I'll try to share my screen and hopefully that works I'm gonna do entire screen Okay, is this up everybody can see. Yeah, awesome. Thank you. Um, yeah, so I'll just give a little bit of background here So my name is Dan Lawrence engineer at Google I've been in our Cloud and security organizations for a while now doing a whole bunch of containery things Particularly my team is responsible for producing a whole bunch of Container images that are used all over the place and it's been bothering me for a while that I can't sign them In any real way. You got a couple different tools that kind of are trying to sign images With the existing registry API today I mean after talking to Steve I kind of refactored out this one and extracted it into a standalone tool to demo for everybody now Basically just want to sign images with a you know, public private key pair I'm nothing more complicated than that and then have a way to distribute those signatures alongside the images to people that want to use them So I named it cosine. That was a little clever You can kind of show how it works here in a terminal too But the basic flow is similar to a lot of tools like mini sign or signify or some of those other minimal Artifact signing tools if people have seen those and you can generate a key pair and then this is kind of the basic flow you do cosine sign with your private key and it Signs whatever container images at that spot in any registry and uploads a signature there next to it So I can show how this works in Oh, I will sign This is going to decrypt my private key This signature here this should work with any registry today. I wanted something that would work basically the existing APIs So it uploaded that signature here now. We can do a verify Same key and it'll print out any matching signatures on that image So there's a couple of different ones here because you can sign something multiple times I think this morning a few times so each one of these is a payload that got signed and verified with that public key that I specified For the format here. This is if you're familiar with it. This is the red hat simple signing payload format This is how red hat signs container images today and a bunch of our other tools at google use the same payload format like our binary authorization service That's not really Coupled or part of this system though. You can sign kind of any payload you want to and if we jump back to the read me All the commands are decoupled So you can generate a payload sign it with a different algorithm if you don't want to use ed 25519 which i'm using here See where that is Yeah, so if you want to upload something in a different format say not simple signing you can just do cosine sign and skip the upload You can sign a different payload format and then upload that here You can just generate the payload format and sign that with something else and then have cosine upload it and store it in the registry for you So you're going to use open a cell or some kind of cloud kms All of that is possible here We've got some doc showing how to do most of these things Yeah, different public keys, uh, there's no kind of pki or Any policy language or anything like that as part of this tool? This tool just does signing with one private key and verifies with one public key So if you want to build something more complicated to distribute the keys, um, that would be the layer on top of this Um, you can download all the signatures. Yeah with this example if you wanted to verify them with some different more complicated key ring system shows how the signatures Look, um, and the key format if you want to use the Cryptography that's in here And then john johnson here. I think is on the call put together this awesome diagram showing exactly how this works And how we store things in the registry today using the existing apis and specifications But the basic trick to do that today without needing any of the new features you've been discussing Is that we just stick them in something that looks like an existing image or uh, uh, manifest index And stick that next to the image that we signed and give it a predictable name so that way we can find it So that works as if you're signing Image one we take the digest for that image in here And then we turn this digest into a tag So we create a new object that's stored in the registry With a predictable name so that digest gets turned into Um Something like this so it looks like a digest, but it's actually a tag because we dropped the at sign So that's up for any object in the registry we can find where all the signatures should be located That was someone gonna say something sorry No, I was just I was snickering. It's very ingenious to It's just you never never doubt what ways you could hack in something to get it to And I they say that in the most you know a respectful way of just like It's sometimes just get unblocked through brute force and that works well Yep We can do much better, but it's a Approve that it works and going this is obviously something we should fix because everything else works Exactly, this was all kind of like a thought experiment to see how close I could get to a working system without needing to ask for any changes in the specifications So this works, um with that naming convention and those caveats The other kind of hacky thing I guess that would be nice to get changed in the registries It's a little bit racy. Um, like I said before, uh You can store multiple signatures on any image like you can have anybody sign it as many times as you want with a bunch of different keys Um, so as we append into this Object, it's actually like a read append right So if multiple people are doing this at the same time you might lose some and you kind of have a last right wins situation Um, I don't think that that's handled at all by the registry API today Um, I've only tested it with gcp's gcr and um package dot dev Uh registries, but this uses on our you know pretty well tested go container registry client which works across Tons of registries, so it should work because you know, I'm not really using any fancy or special features with other registries But it'd be good to try this out I like your uh, who's using it reference. Yeah Hopefully nobody's using us yet But I do want to start using it on some images soon um Yeah, so any questions here? Um And I know john you had a couple things you wanted to say too about the kind of hacks and everything we put together Do I stop screen sharing? I I can yell about stuff. Um, the the raciness Is fixable uh using existing hdp semantics I brought this up last week in uh, the oci call and everyone seemed amenable to fixing race conditions like that uh but I mean the this the scheme for like storing Via naming convention is also bad. Uh, and so I mean we can fix that racing convention But now we have just a bad thing that works instead of it's racy um, so so there's definitely better ways to do like discovery of signature artifacts, but uh Until we have them. I like this one And the other thing is like dan mentioned You know, we can sign arbitrary things. Um, the red hat simple signature stuff is an existing thing I like um What what I would also propose is just signing a descriptor Uh, no ci descriptor with some annotations on it because that's an existing Thing that the client should know how to handle, uh, and and is effectively just a pointer to an artifact And so if you wanted to sign a tag, that's a pretty reasonable representation of one in the registry Uh I think that's that's all I thought was worth mentioning Well, yeah, as long as the payload when it resign has the digest of the image in it, then it's You know pretty good and you've signed that image You can add in all sorts of other crazy annotations and things like the tag itself that start to get you know, weird Not weird, but you can build up stronger systems on top of that if you kind of put the tag and the digest in there together Um, you can get different types of claims Um, I started doing a quick thought experiment and I think you could actually build something like tough on top of this Primitive as well. Um, you know tough is just collections of things and we're signing arbitrary JSON Which contains collections of things or could contain collections of things So I think you could also publish in a full update framework Snapshots and things like that in the registry for storage and then sign those as well with different sets of keys So I don't think this precludes any of the more complicated Update uh framework stuff But I haven't actually tried it out myself. So there could be something I'm missing still I guess just to end it. Um, is anybody else doing anything like this and tricking registries into storing signatures and using them to find them later Is this useful to anyone? Um, any Comments are you horrified by this? nothing Hey, this is Andy first time joining the meeting. So hey everybody I do a quick question based on what you're just saying Dan. I'm curious about what y'all are thinking longer term for If we need to do changes to any of the OCI specs and registry specs to Not have this be, you know, something that's sitting alongside Yeah, I'll It's probably the best for john and steve and the other people on the OCI calls you tweak, but I guess If I could have one wishlist feature it would be some way to set up these back references So some way to upload something and say that it is attached to this image Without changing that image and without having to do a hacky naming convention And then of course a way to find all of the things attached with an image I think that's the one kind of primitive that doesn't exist today that would have made this a lot less hacky So this is the part where the two efforts kind of really converge. So first welcome, Andy Who are you with? They would be with I work at VMware and I'm the tech lead for our supply chain tools effort Great. So we've been working with a bunch of the VMware folks. In fact, they've recently over the weekend we're asking about how to Integrate some of the scanning api stuff which kind of correlates all of these listing apis and discovery as well So I see a lot of this coming together The back reference is something that we've been you know working on for a bit now the NV2 prototype Actually, which is a a temporary fork. It's a prototype fork of distribution To prove out some of these concepts that we can get that NV2 client experience working And it looks very similar to what dan did We didn't hack in the tag because Let me I'll share my screen because I'll show why that starts to become a problem Share screen This one, okay So this is a PR for the a new artifact manifest that is designed exactly to take care of those references So if you notice here a notary signature, which could just as well be The cosine signature and that's that's another interesting tangent that we've always talked about is having multiple signature types You know, whether it be the red hat signature was what the original one was or now, you know, a google cosine signature I think there's a little bit of a I think it would be a failure if we didn't have a standard that people could you know have better If you will and compete with step But we want to have some standard because our goal is to move things across registries But I digress the idea is that you can store anything whether it be a notary signature or an espom The problem that these things are they're independent artifacts. Sure, I can stuff in a registry, but how do I link them together? In the current NV2 prototype You basically submit them both and then there's a completely separate api that you call that says please link these things together Really just a different hack quite honestly What we've been trying to do is come up with a way a manifest that you could submit to the registry that handles these references Because the let me I'll scroll it forward here a little bit The problem with the tag listing um, there's an problem with using tags for instance is this if you look at a repository And you were to look at all the the things in it If you would expand it completely you would see layers But of course we don't show layers in a repo listing because the layers are known to be part of the image So those kind of go away But the same token would you show a signature or an espom? Maybe you want to show a CNAP or helm chart, you know Whether in the same repo or not is that it's going to take a little liberty here The point is is that there's certain things that you want to show and certain things that you don't Whereas if you look at it pivot it a little bit if we could establish these relationships Then we can say the image which of course we're not showing the layers Has a signature has an espom maybe even has some metadata on it And now you're looking at it as one entity and we don't need tags We don't actually want tags on them because tags are meant for here's the things I deploy And all of that other goo is reference Data and then of course you could kind of do the expando and expand this out And obviously you can kind of see this is kind of hacked together just as an example So the the work we've been doing here is to do just that Manifest for an OCI image knows how to point to layers a notary signature knows how to point to its layers We've called them blobs now The question is how do you do the double direction? So a magnet wordpress chart defines layers. That's fine And now when you have a notary signature It or anything in the registry supports this new artifact It supports its content just like this, but it also can store a back pointer Because this focuses on one of the main goals we've had from Notary v2 is we do not require a changing of the digest Or the tag of the thing you're signing But we need to be able to move it across registries. So there's an interesting double thing there. So This latest proposal and then this just kind of elaborates on more of the references, right? A wordpress image that's got its notary signature an sbomb document But a helm chart that can reference this as a weak reference That gets into a bunch of other stuff. Let's focus on So if we look at an OCI image, that's the standard thing, right? This is just a standard OCI image format And layers and there's your descriptor If we scroll down what we've done is this new OCI artifact manifest So we're suggesting a third manifest get added to the registry Because registries have to support at least two but they can support more There's nothing in the spec that says they couldn't support more And we've for the last couple of years have been jacking in stuff into the OCI manifest and trying index And we just they're just there's a fidelity and compatibility. We keep on creating weirdness about it. So the idea here is let's Let those exist as there's they are and have one that's more generic that can support notary cosine images helm charts Nitis formats and all kinds of things So all we've done here is we've changed layers to blobs. This was always a pet peeve of folks So this is no different right anything in here. This manifest says this is what makes up the content But what we also have is a manifest collection. Let me skip to this example actually I'll talk about the subtle difference between the two So this one says this Notary v2 signature or cosine object doesn't matter Says it's pointing at this digest in the registry So this is that back pointer that we we need so by submitting this to the registry It's one put obviously the blobs get put. That's the standard thing But instead of submitting this artifact and then telling the registry by the way link these things together As you're submitting this thing to the registry you're giving it everything it needs to link together So now we have the back pointer And of course we now we put the registry to have persistence and so forth on that And I'm sure they'll be varied implementations. We've done this with blob storage in the nv2 prototype here. So if I go to Distribution You'll see there's a pr here that has this hacked in implementation that Avril has done But this is just another hack of what den was kind of doing. This is again to prove the concept But we just took it to the next level proved it in the registry And this is the formalization of that Proof And then of course we just need to have the appropriate listing apis that says hey, I need to get this thing out So the idea is that I want I basically it says give me the reference artifacts. Let me scroll all the way back up here for the wordpress five image And it will because you're asking for it, right? You're not asking for the signature because there's lots of signatures. You're saying Let me know all of the reference artifacts to this wordpress image By the way, please only give me the own nv2 for ones because I don't want to look at the s bombs and other things They just want the nv2s and then we're also Trying to work out I also only want the things that were signed by docker or signed by acme rockets or web at networks Because in the current in this nv2 prototype We actually bring back all the signatures and the client has to do this round trip and go I know not that one. Nope. Not that one. Nope that that one. Oh got the one I want and that just becomes you know a chatty api The just the last thought and I want to get more feedback on this The reason we jump to annotations here And here we have a formalization in the manifest Is these we believe apply to all artifacts now not all artifacts will necessarily use manifests or sorry or references They'll use one or the other because then If they don't well, I guess they don't have to use one or the other But that's the idea is you have the ability to say I want to link to something else or I want to Have a week. Well, they're both links to something else. Just one's a weak reference But this the idea is that a particular nv2 prototype wants to pivot on something We couldn't find a good way to generalize this in a manifest that made sense So it made sense to use annotations of being artifact specific information So it's only the nv2 client that would actually do something with this the problem is we want the way the The listing api to be able to filter on that also And it might also be give me all the artifacts that are set to expire today because I want to go delete them So there's a couple of different interesting apis for that Anyway, let me pause there for conversation I'll just say most of that one over my head if As long as there's some way for me to like, you know, we can already store anything you want in a registry. That's how they work As long as there's a way for me to say that one thing that I uploaded is now attached to another thing And then a way to get all of those, you know, that reverse index later, then that would be sufficient Um That definitely sounded well not sounded but that that was definitely way over my head though all the other things you were mentioning But I also haven't been in all the oci calls and Don't understand all the other things going on Yeah, sorry, I'll work on the the walkthrough. This is some things. I'm trying to figure how to present better Steve, I think did we go through the query Apis that you were thinking of because that might help answer some questions Not yet. Honestly, I've been um, I the biggest conversation Has been this a and b conversation that we've been having But that is the next thing and that's kind of merged with john's focus on the listing apis and now The harbor folks are also saying hey, we need to standardize We would like to standardize in a way for scanners and other things to get things out So I think We're seeing a convergence of that Um And honestly that was there's two things I was going to do one just as an update from last week I didn't get as far as I wanted on the status email for notary. So I will work on that today and tomorrow And get that out for people to review and it'll cover just this and maybe I can Show what the current mv2 prototype does for that listing api But I'm nervous that'll make more randomization um and then uh, yeah, uh, the listing api and Conversation is the next one to to pull in the requirements here into that Oh, I'm branded. Sorry. Where I was going with that is you guys have raised a bunch of questions over the weekend Then I didn't get a chance to respond to that's what I was trying to say Oh, no worries. Um, I was just thinking for the Since we were using the tag interface and hijacking that with the shahs there There isn't a lot of work that you've been doing steve to say there is a way that we can query these So you could say show me all the artifacts that are currently pointing to something like the The manifest you're looking at so you do have that back reference there And a way to do the tag you're saying um, I've just been I'm thinking of your Your solution to the stuff that they were working on On the cosine side that there is a way we can do what cosine was doing without having to hijack the tag interface Oh, yeah, yeah, and that was the piece that dan and I talked about the most is he's basically There's a friend of my work with john gossman. He has this term gossman's rule of any good ideas implemented five times But you only know about two or three of them So it's you know, a lot of the artifact work we were doing Coincidentally the quay folks went along the same approach, right? The stuff we did with the nv2 client dan's basically taken a very very similar approach With the cosine work. So to me that's like what that tells us that we're really honing in On a good experience. Of course, we need to continue to churn on the security and you know We shouldn't give up on all those things But when you've got multiple people coming at the same direction that didn't know about each other That starts to have a good feeling that we're heading the right direction And i'm hoping the work here will unblock the stuff that dan's doing To me, I still have the question of whether we want to have tags that can point to the artifacts And i'm thinking along the lines of say your helm chart You're going to want to be able to pull that by tag You don't want to say find the image that is using this helm chart and then find the helm chart that way So I feel like absolutely If maybe this is again ready to continue churning on this example And maybe I just need to record a talk or something with the slower pace to it But if you look closely at these examples, let me just get in the right section here Let me go back up here We've done exactly that the the helm chart does have a tag The helm chart does and whether it's called dash chart doesn't matter it could be a completely different repo But you have a wordpress chart with a tag You have um the CNAP has a tag If we look at these display listings, what we're saying is There's two categories Some are and haven't found the best words. So please help me with better words here. Um I've been lately calling them enhancements. There's certain things. They're just enhancing Another artifact a signature and an espom really have no value of their own Their whole purpose in life is to provide additional information to another artifact But because these are sealed we can't crack the manifest on that without Breaking the seal So there's a category of these things and I really want to get eventually to the metadata stuff as well That these in fact the idea is if you delete the image the thing it signs these get deleted with it And the same and the CNAP is just another one of those a helm chart I have to look closer at the night of stuff I don't completely understand the implementation to to comment on it, but a singularity chart, you know sorry a singularity image Just there's a bunch of things like that that absolutely have tags and have to reference But there's other things that don't have any tags Or you actually don't want to give it a tag because you don't want it to surface like this And it's not just the visualization It's the delete and I've put I haven't finished that part of it Um, but in here Where is that there? I've started covering the delete Scenarios as well Where there's some things like you want to delete the signature and espom You might if you delete the helm chart you might Want to delete the images it references But you might not so the weak reference collection Just as a way to say here's the things that i'm referring to and the client can use that information from the registry and decide I do want to delete because there's a extra deleted, you know Delete what did I call it? Delete references or something like that But it's really up to the client to do Does that make sense? It does the part that i'm Undecided on right now is whether you want to be able to have The same tagged two different artifacts and references by artifact type Or do we want to push that onto the user of saying the user needs to specify That the tag includes whether there's a chart or a cnab in the tag name So putting two different things in the same rebuttal It's basically whether your namespace your tags according to artifact type or whether you put them all on one namespace that um I think that chip has sailed uh with the Platform stuff I don't think any registries would support that currently and it would be kind of a Large undertaking to somehow have tags namespace by type within the same repository We had this question came up with multi uh multi platform images as well And a lot of people suggested. Oh, well, you know this windows image should Have the same tag as this other image and the way that was solved was by having One artifact that references the others and And the client does the resolution Depending on which artifact it wants That makes any sense It does but even within that same example you query a registry and say that I want the The image manifest for this tag and it will go back on the registry itself It will do some of that resolution and pull back. Okay. Here's the first manifest from that manifest list Just because it knows how those things work together So you can specify which artifact type or the which content type you want to get back from this tag query I I think everyone hates that Um, yeah, and that that is a concession for old clients and not a feature to be Exploded, I think I mean I've implemented it twice. So I I know what you're saying but um I don't think anyone wants to use that going forward. Unless I'm wrong It's been painful. I mean the way the the way you're doing So here's the original the original proposal from the artifacts approach Which just so happens it sounds like it mimics some of the stuff that was happening with the multi-platform stuff If I like when I'm working on stuff and working on proposals I do my visualizations in powerpoint and I write my stuff in a word doc, you know, just as an example I'll name the two things the same thing. There's a teleport dot ppt x and there's a teleport dot doc x I can save those in the same folder in my you know on my computer. So why can't I do the same thing in a registry? So we did propose that where the artifact type was a pivot, which would be very similar to the platform I I couldn't have gotten the more, you know Negative response from people and I'm and I'm sure it was wrapped up in all kinds of other historical stuff So I I kind of realized that let me give up on that for now Because it wasn't making any progress and was stalling all kinds of the other work Whether that's something we eventually get to or not. I don't know. I mean I the multi-platform stuff or the multi-pivot In the same repo has definitely been An orally one. I think people are chewing on As The multi-platform stuff is continuing to be problematic It just hasn't gotten bad enough for anybody's really put in a good enough investment to do anything about it yet I think most registries have assumed that a tag points to one thing and To change that is TARD Yeah, I don't think worth the effort when there is alternative ways to accomplish the same thing well the The two downsizes that if you ever get into a situation like this You need to either tell that that the client is going to have to maintain this naming scheme To be able to keep things different, but I'm also thinking along the lines of Like integrating something like tough if we try to pull tough into the registry concept They want to have some kind of consistent name that says Here's the object that points to the snapshot for everything in this repo And so they will push something up there and they want an easy way to query that and so we need to Need to give them a query or we need to give them A tag name that's unique that's not going to get stepped on by something else or Some other solution. I'm just kind of thinking of the different options there for that It could be possible to do something similar to the helm chart example Or it's like wordpress dash chart where it's like snapshot dash tough or something That um, maybe isn't explicitly saved but wouldn't be taken by another image It's just the risk those tags could be ever written by any way that want to do something unwise Yeah, I guess so Basically a tag is a the thing that the human and interacts with and tells the system to interact with All this other stuff whether it be tough or signatures or helms. Sorry tough or signature or S bombs or gpl license packaging those are all supportive information to the main thing And the more you surface the supportive information the more users go delete delete Copy move do let me mess with and It just becomes a nightmare. So I think we that's part of what this new artifact manifest is about is supporting the extensibility we need With and telling the let in the registry do the garbage collection and the reverse reference stuff But not expose it to the user And you know, so again, I applaud Dan for what he's done because there's a there's an article I've been meaning to write if you ever seen those videos where or pictures where there's a highway it goes through And there's a house right in the middle of the highway It's you know, the obvious thing is the house has got to go right that they lost But it just didn't happen yet And this is kind of the obvious thing like Dan's created a great experience back and forth And it's just there's this one really there's this house in the middle of the road that we just have to get rid of so I think the question really here is I'm hoping Like I'd like to get a little more feedback here. So I got a bunch of feedback And I was talking to Derek about it. So we were debating on this The fact that there's now three collections So if I can take this meeting for a second and get some feedback here because this is the Having to maintain an a and b it makes a little harder for us to move forward So I definitely would like some feedback here In this version Sorry in this version We actually have three collections. There's a blobs This you know the layers nothing's changed there Manifests which says this is the optional thing. Sorry No, this is the reverse pointer at the things I want to point to so in the example up here That was the helm chart, right? So in this example here the notary v2 signature Is pointing at the mysql image. That's what this digest this descriptor is, right? the And that's pretty straightforward from a registry. We know how to read, you know the manifest We know how to read this being the manifest in the manifest collection. We know how to read that We know how to do it. It's a very strict. It's a very structured thing In this option what we're doing is we're also using manifest but We have this Additional annotation That says I depend on the relation the ocea distribution relationship says This manifest depends on this descriptor And so now a registry would need to not only parse this collection, which is fine But we're also now digging into annotations for something specific to figure out. How do we manage garbage collection? In and the reason for that is if you look at a helm chart The helm chart has weak references to images We want to store this information in a registry so we can do deep copies optional deep copies And we can optionally do deletes and the deletes are obviously ref counted But you don't have to do it, right? This is like this is the beauty of it is we're now introducing linkages between things So in this case what we're storing is I'll use this example first A helm chart doesn't have any upward references, right? It's not pointing up. It's pointing down So what it's saying is here's the blob that makes up the helm chart And then the helm chart references the wordpress 5 image And the mysql image But I've got it in this references section and that's how I'm making the distinction between a loose reference and a hard reference In option a you'll notice that there's an Excuse me, you know ci distribution relationship that says references So if we go with the two collection approach Now to do garbage collection, you've always got to look into this annotation Where in the three collection approach It's very straightforward. There either is a collection or not and I don't have to go any deeper into it The the acr engineering team kind of was like Not at all interested in this because garbage collection is a you know, the amount of and we all have this right I'm not trying to make point of one or the other Any production registry has a lot of volume But we really have to be super fast in how we process and do garbage collection and process inbound and delete references into well So this was not well accepted Um, and this was we kind of we've been pivoting back and forth here I'm curious what others think specifically like people that are running registries that have to deal with that I can see if you don't have to deal with this problem. Why it doesn't matter If you are kind of is important I'm fortunate to not have to deal with the problem, but I also prefer the Three type rather than the two type get that out of the annotation I am curious specifically on the example you got right now You're saying like wordpress 5 7 and the annotation down there And is that digest point to a digest that's in the current repository with the helm chart? Keep Or is that digest? Because I'm trying to mute my mic back and forth then I'm having a Um, no, so this is another area. I need to explain more because this is I think this is also some of the questions You guys had over the weekend what this is saying is This is giving a tag to this digest, but it is not assumed to be the same repo The reality is a digest is globally unique and we can find a digest anywhere in the registry Whether it has permissions to it. It's a different question But you have that problem in the kubernetes deployment regardless, right? A kubernetes deployment is always potentially referencing or helm chart or anything referencing multiple images There's never a guarantee that auth is going to find all the things However, if we put this in a manifest and if we want to deploy based on tag I have another proposal about doing the mappings That says this tag is located in this repo Because all of this is defined here a client can say The scope of which that i'm operating right now Because the idea of the person pushing this thing into the chart is probably not the idea of the The kubernetes know that's doing the the deployment Does the scope of that thing do a deployment right now have read access to all of the artifacts in the graph? And now you would be able to figure that out So This split like this with the mapping concept Allows me to either deploy by digest you know pinned to a very specific digest Or float on a stable tag So you kind of get the best of both do I want to get the latest version of 5 7 in the current registry? Or do I say I don't know what the latest is if that's compatible with my deployment so I can deploy on digest Yeah, I'm thinking through the case. So if someone said let me copy this entire helm chart over And helm's a bad example, but I'll use it anyway And I would need to go through each one of these references and if that reference is not in the same Repository that copy would fail if I was doing a recursive copy through all the different components Sorry, okay, so I was trying to find the copy command because I do have that copy here the copy scenario in here copy here we go So there's that's the standard push and then you have the option of saying Oh, where do I have this example? Oh here ocirag copy. I just totally hacked up an ocirag I haven't looked at all the other ones. I'm purposely trying to not pick one Source target and then you would have the ability Did I put that example? All right, might have I might have scrunched Taking too much out. There was a an example in this that said do deep copy So you could basically say copy with references now the approach Assumes that you've got This is a silly place to have this discussion I recognize But this is this model here that says you can Where did I put that? Yeah, so you basically now have a way to map where This where this repo is in the current registry um And this one I'm not as prepared to talk about yet because the redhead folks are really interested and we're trying to reconcile the short name stuff that they've done Which was a great concept but turned out to be a big security hole because you have to search things fall through This is intended to eliminate that Is it is a very deterministic resolution? So I can say I have two registries I might have a base images registry and I have my apps registry And now I can say the wordpress is in my base registries and charts So it very explicitly can find it and what this allows us to do is um If you think about the way we have Images right we build the concept is you build the image once Tested in dev you don't change it you test it in staging and deployed in production. You're just changing the config you pass to it Why is it that a chart has to also be changed as it goes through because it's making references now if I have A chart that says I depend a wordpress image and it depends on these two Sorry wordpress chart that depends on the wordpress image and the mysql image I should be able to very easily move that chart through the environment The problem is is that we've kind of left that to the helm team to go do with their Internals and there's no way to generalize that in a registry So we wind up with a very fractured experience Well, and from an azure, but I could tell you this hit us directly We went around and cleaned up all the references to Our services were dependent on docker hub and gcr and quay and stuff And then so we got all of that into mcrs and our own services are independent externals But we saw that our internal service teams and certainly our customers will go take the engine x helm chart And they put the engine x helm chart in their registry But it referenced the engine x image from some other place So we really want to make this super simple so that you can define this and it just it just works Um, so this is still a work in progress. I actually have it to do to follow up with Mila I think was mila that connected me with everybody and I just haven't had a chance to follow up with everybody yet Mila, do you want to chime in here? throw them on Not much I think I'm very very much on board with the idea of mapping images a deployment type I just don't think that it needs a separate syntax and the reference is just should be fully qualified with some default location What we've been through that many times So this is this is the you know, I am sure this idea will evolve And that's why I'm not as ready to present it yet, but Back to the question that you were asking Brandon was this Let me scroll back up to the helm chart example Getting lost in my own document Table of contents would be super helpful here Um The helm chart here we go The idea here is that instead of having a baked in fully qualified path Is that now I have a named reference that this manifest Doesn't change no matter what registry it's in The only thing it changes is You know a client mapping that says by the way that wordpress chart doesn't even come from this registry Comes from something completely different because I'm managing two different registries in my environment And so to me what this saying is if I was to implement a copy that wasn't built into a registry server something else some external command that in addition to Going through that there's listed digest I also need to Expand those annotations and look for that one annotation to see if I need to query a different repository for the same digest Yeah, so this is um, if you looked at a previous version of the OCR OCI artifact manifest I actually had expanded descriptor to have some other properties in it And again, you know descriptors have done as well so far. So this is where I move these two annotations The I I think I don't have this fully baked yet. This is why I want more feedback. So the This is one and I think at least I heard from one or two people here that they like the Option b model of this. So there's a separate collection The question here is what do we do around this name? And I think this one is We just need more time to bake on it. I think to your point Until I get this other conversation happening Where to go this one It's hard to really visualize how that gets solved so I I don't have a great answer So right now I'm just making sure that we this already is as part of the Descriptor the digest is already part of the descriptor. So that's already there This is an additional piece of information that can be used Whether we formalize that and move this up into an enhanced descriptor I'm not ready to commit. I don't have enough feedback from people to say One way or the other yeah, my own knee jerk is to Say we wouldn't allow references to stuff that's outside of the current repository, but I'll leave that up people that actually have to run the registries So I hear that a lot. Can you elaborate on that? it it just kind of Takes your entire security concept to A single reference. You don't have to think about are you having to re-authenticate to another Registry you might not have originally had authentication to and I have two different user IDs pulling this image around But also I'm just thinking of all your garbage collection scenarios Now you have to keep track of garbage collection between two different repositories. It just seems like it gets a lot more complicated I can't agree anymore I agree so so much I'm just I'm trying to get some space for some others because Like if you look at a deployment a deployment in any kind of environment always deploys images from multiple repos That's not true But it definitely does in many cases if i'm deploying a set of services I'm deploying the mysql image the help the wordpress image. Those aren't in the same repo Those are in separate repos and you know whether it be my marketing app is going to have a set of services from multiple teams So I feel like helm is a bad example because you can template those image names and everything gets Kind of crazy when you mix a template of value in addition to putting it in this format. That's not templated So changing examples in the CNAP spec today. Isn't that all pushed into one big object? That's all in the same Repository you just push all of your images up together in the same repository It is but I would say it's actually along the lines of what dan's trying to do It's a work around around what exists today Like even they're even stuffing things in an image because doc rev doesn't support anything other than that yet So they're definitely in the I'm going to plow through what exists today because I don't want to take a dependency on southern doesn't exist you But I can tell you from a you know customer deployments whether it be helm charts or kube deploy People are deploying things for multiple repos and it doesn't matter whether you want to get two repo name spaces like the org Thing or with acr and some others that we can go all the way up to 256 characters. It doesn't matter The people deploy across multiple repos now and they also deploy across multiple registries We see the multiple registries less. I think we'll see those more over time so it's not a matter of Convenience because to your point the other point if you try to stuff everything in the same repo Now that we're supporting different artifact types. Now you got to start playing this game with tag names also Um, and there's no formality to it because it's just a string So I think I guess my opinion is I don't know if we can force people to keep things in the same repo because we think it's convenient because the reality is They're already doing this If we try to fight it, we're just we're blocking the we're forcing them to try to stuff things in the same repo or force them to Put them across multiples Honestly, we even have I'm looking to see if he's here We have this argument with the multi arc stuff We're seeing a lot of the library images and even our dot net team has been fighting this a bit They want to put the different architectures in different repos because it's just too hard to manage multi arc images the Image index is it I I think Having multiple repo artifacts is easier than I don't know that How does this solve that I guess is my question Well, it's a slightly different thing because you're not pulling the windows and linux image at the same time So to be fair, it's not exactly the same problem. My point is that People are seeing the multiple repos of being a better way to separate things even for things that shouldn't have to Like multi arc. I'd argue you're right. We should just fix the multi multi The index collection tooling so it's easier to deal with I think people get confused by multi arch stuff because It's slightly complicated where you have One extra layer pointing to multiple things But it's all in the same repo. I worry that A single artifact that can reference like multiple repos and multiple kinds of things Uh, I mean that's just making a problem worse. I think Why does it make if if people are already doing that today? How are we making it worse? And and the I don't think they're already doing that. So the manifests Just to be really clear the manifest collection like a signature. Let me go back up to signature example Uh, the signature example where this is a notary v2 signature Enhancements to an artifact. I have said absolutely have to be in the same repo for all the security part. It's not just a security. It's It doesn't make sense for a thing that is that is a thing It's split across multiple locations And I'm quite frankly i'm throwing the windows foreign layers under the bus on this one Right, the fact that the layers come from someplace else is just ridiculous And we always knew it was ridiculous. It was a legal requirement as I mentioned in another call We're we're really close to getting rid of that requirement but That's actually the reason we're trying to get rid of it in a way. We want everything that holistically makes up a thing Must be in the same repo The idea with references is these are weak references loose references. Whatever the term you want to call The helm chart can exist on its own The images can exist on its own The fact that I have a weak reference that says by the way these things do make up each other and I can find On is an enhancement, but it's not a blocker because the image might be in another registry altogether And you resolve it through the mappings So I'm trying to at least address the concern that says Yes, if you don't have off to both things Fine, you don't get the two things that you don't get the brother and sister, right? But they do exist on their own. It's just there's Experiences where you need them both If you need them both, you're going to need them both anyway and the role that has to pull them both Because we have it defined in a manifest. You can do a pre-check And says, hey, do I have access to all these things? And if no The deployment Will fail because before it even starts my inclination Trying to find the right words for this stuff is to say that the artifact Maybe it just shouldn't even include those at all That if it's something that the helm chart is saying, hey, I've got reference to external stuff Then maybe that shouldn't be part of the artifact itself The artifact should say here's a helm chart Here's how you look up all the details of the helm chart and then go into that to figure out everything else But when you author the helm chart You're saying and I just keep on using the word press images Maybe some other ones, but if I have It's two images The mysql and the helm chart. I just say that these are the things that make it up I'm not saying when I author the helm chart how you might want to store them I'm giving you the flexibility to say by the way of my in my environment I store My helm charts at one place I store my base images in one registry because those don't change And then my app code Is in The the engineering teams registry So you kind of like if you have 20 divisions at a company They're all referencing the engine x image from the same one But their app is specific to their team Yeah, okay, I'm not selling it yet. It's okay I'm just thinking how I'd write both the registry code and client code for this and I don't want to Well, the beauty of the registry the registry doesn't care about references I think it has to why Because those are the down references You don't want to garbage collect the down reference when there's some above this point to it registry never that's the beauty of this one The registry would never do garbage collection on it. The client Can say by the way when I'm deleting the helm chart I actually do want to delete the mysql and Helm chart that mysql and wordpress image And then the registry will just treat those as independent deletes and do its normal garbage collection Your reference counting and so forth The go-ups and manifest those follow all this the registry Haps to know about blobs and manifests because these do get deleted and to be fair It's not you don't delete the signature that deletes the manifest. You're deleting the manifest You do a reverse look up to know you delete the signature So with all these is there a way just by looking at the schema To know that when an image gets deleted that you would delete the artifact or vice versa Or do you have to go a layer deeper and know which artifact type and have custom code per artifact type? Awesome. No, that's exactly why we've got this thing. That's why I have that option b Is if it's a blob if it's a manifest. Well, remember the manifest of reverse reference When this manifest is being asked to be deleted this thing always gets deleted But you wouldn't actually delete this first what you're going to is when the mysql image gets deleted This because I've added this enhancement to the mysql image The mysql image is keeping a reference to this And I don't know I need to elaborate this more There's this reverse look up thing So the mysql image picked up this manifest because it's referenced here And now when the mysql image is deleted This manifest is deleted when this manifest is deleted this blob in this config is deleted It doesn't do anything to the references It's totally up to the client to the side if it wants to walk their references tree It independently asked for that thing to be deleted because those are independent objects And we're at time. We already lost John too All right. Well, uh, we we lose Dan also. So anyway, so thanks folks. This is this this is the conversation that You know, we need it helps to be in person because there's an iterative if you've been watching this This is a bit evolving a lot because of this feedback Some of it is just collaborating a little bit more. Some of it is just changing the names like Manifests is the newest name because it just makes sense now Um, I will keep iterating on this this week respond to all the questions They came up in the listing API get the harbor folks involved so that we can get them into that list of requirements And we'll continue to make progress With that we'll wrap up for the week And thank you here