 Good morning Morning Yeah, I'm gonna drop out in half an hour and join again Yeah, it's a one hour It was all supposed to be it's all complicated because it was supposed to be a keep kind of cuts It's all really weird that we should be able to keep we should all be in Amsterdam We have to pretend to be an Amsterdam That's true. I should find some answer that someone's got to find an Amsterdam background Yeah, I was struggling I realized last Friday like wow, we still have this meeting because it kind of snuck up on me So I scrambled to make sure we still have a structure of an agenda To get things going so glad Joe Yeah, I don't know if you changed your microphone or what but it sounds like No, it helps to actually have it in front of my face. Is that better? Yes, sir Thank you Somebody noticed it yesterday when it was like or the last week was behind my head. That's really cool But it might help the other way He's just not used to this kind of stuff so all right So here is our structure for today we've got we put the HackMD in our chat here and It is a loose schedule Shuffled it last minute a little bit because of Justin wasn't going to be available till later and I wanted to try to just cover a couple of things. I've also asked William from one of the supply chain efforts to come and kind of give us an update of what they're doing There was just a question of that on the Slack channel that you know, we're are we expanding our scope and no we're not in fact, it's matter of Trying to find the scope to kind of componentize things So that's what the supply chain efforts are, but I don't think it's going to be a half an hour, but you know, we'll see for timing Other than that if anybody's got other agenda items, please put it in there if you want to put in fact here I'll put another one fill If people want to put something here Your topic Then we can you know fill in for other things so yes, it'd be nice if we were in Amsterdam Although I don't know if the weather would be any nicer there so hopefully this summer which I Yeah, it's I won't get into that rant of conversations All right with that Why don't we Get started Oh, I see somehow I missed The there it is it's reformatted out for the distribution getting this from distribution spec to a 1L All right, so let's anybody have any other items they want to talk before I kick off have a raise hand What's the raise hand equivalent on zoom? Vincent Zurich People just drop in chat. Okay Yeah Okay There's if you click on the participants list There's actually a raise hand option Okay, cool. And how would I see it? Can you raise your hand? What does that look like? I'm actively raising my hand. Oh, it just shows a blue hand, which is four or five people so I don't Okay, then look like a notification. It's literally just a blue hand On the participant list Do you see that? Oh, I see. Yeah. Now I see it. Okay, so I'll just keep the participants list open Yeah, but is everybody just saying hi? I guess All right, cool. All right. Um, I'll try the screen sharing here Just to try to do a keep things going here So the first one was the latest on artifacts So where at is, you know, we've Just as a quick update on what artifacts are, you know, so let me see. I think I've got it up here the idea is to take The Mechanism by which we store things in a distribution registry and the fact that there's these layers Which are blobs and there's a manifest that describes a collection of the blobs The idea that these things can be downloaded concurrently And the fact that all of our major clouds and major vendors have created An implementation of the distribution spec that runs in pretty much any modern production environment With vnets and security and all kinds of other capabilities that are required to be in that production environment It'd be nice to be able to leverage that So that when new artifact types, whether it be helm or singularity were the two that impacted us and acr That we didn't have to create a bunch of one ops for them as an interesting side story to this In azure, we have this acr tasks feature, which is where we do cloud builds in the in the cloud We've been keeping logs In the storage for the registry as well Recently, we've had a support customer managed keys so that they get double encrypted by the individual customer I And we had a struggle like what do we do because that cost we hadn't done that work with the the streaming of logs for this thing so we were able to completely change that interaction model and We basically just going to store it in the registry and because we're now storing those logs just as another artifact type Then it could benefit from the new cmk feature with vnets and you know Try not get into a the advertisement of acr But I'm just saying the point is is that we all have these hardened capabilities We're putting in registries And by storing things as just an artifact in the registry with a different media type to differentiate them Then all the other features we have in the registry, you know get benefit including the auto purge that deleting the stuff that we've been talking about recently Um, so that's an example of something. It's not just what I'm calling a well-known type like such as these But things that might be cloud specific That we then don't have to create all these additional ways to handle The standard capabilities we need on storage in various clouds so the the conversation that we had last year on how to Understand what the different artifact types in the registry are are using the config media type Not the manifest media type, but the actual config media type and that Was the least impactful way that we were able to leverage the Schema of the manifest that exists And give us the differentiation and the best example I have is we all you know as we look at our file systems hunt disc They have extensions And by having the extension on it We know that if we double click at what app will run virus scanners know what how to process those files And that's effectively what gets done here. So we can differentiate the different media the different types of things that are stored in registry Anyway, so with that There's some content in here and what I'm looking at is I'm actually viewing here the last pr in just view format to get a sense of it There's how to define a unique media type and one of the latest conversations we've been having is How do we define? You know, how do we create a clearing house for these media types? Is it up to the artifacts or t.o.b? maintainers, you know to approve these as a first come first serve basis And then it was kind of reminded that these media types were already set up in Ayanna Ayanna Ayanna Anna whatever And there's a clearing house. It's already made for those There's a well known format forum They've got a really good structure for the root trees So that's the work I've been doing in the latest round of updates is to account for a Ayanna registration And the various forms. So if you notice here in this latest Instead of just saying application slash i'm referring to their registration tree And then an org a company an entity Would be the name The next name and then there's object types sub-object type and you can go with many sub-object types as you want config and then Version and an optional config format. This was some of the other feedback that we had So the Ayanna registration we've been going back and forth on that for those of you that kind of watch some of this this The fund of formatting email has been an interesting challenge. So in the last round I shifted to Just putting in as a markdown document for to get feedback here I believe Oh, I know if I look at the pr you'll see the actual commentary And so far they I just posted this the other day So I just kind of put my own notes, which was their feedback these quotes were the feedback from Ayanna And how to to do this So I'll just I'll finish up the sign the thing that I'll pause. So the Ayanna registration. I'm doing here is just for the Where does it actually even say this? Let me just go back to the clito formatting For the image manifest. This is not a specific media type, but this is the manifest format itself That we use at the well the schema of the manifest as opposed to the config object and I just picked the Simplest one if you will to kind of see what this registration process is like to go back and forth If you've been watching some of the emails, I've been letting them know that we anticipate a bunch of things coming in I don't know. I was trying to do some guessing of numbers somewhere between 50 and 200 over the next year or two For people that do media types the the artifact types and then their layers which Could be another registration as well if they choose So it's been a back and forth. Of course, we've had other stuff happening in the world that make this a little more slow I was suggesting maybe just a conference call or something at one point just to to talk it through But once we get this round through so this one seems to be the most concrete of The feedback they've given and by all means if others have feedback Please provide it to help me with this because this definitely gets into a level of details that I hadn't really considered before But assuming this one gets approved then we'll actually start doing like whether it be index or the image the actual Image config object, and then we can you know let the helm team go in their submission process and so forth So with that, I'm going to pause for a moment You think into the hundreds of mine types Well, the if you think about it and it might be less with the latest thought process that I kind of realized if you How many people will create artifacts? You know, maybe 10 to 20 well known well known that need to be registered Because like the one that we're doing for a CR task for instance doesn't make sense to run another Cloud so it's perfectly makes sense that it doesn't make sense to register as a well known type Just the way we leverage that so I imagine a lot of clouds and even customers might have their own But I would think I don't know 20 or so and then the original thought process is People would be creating their own layers Because they need To differentiate one layer from another But again some of the feedback we had in the commentary was why don't we just leverage You know well known standard not needed to be versioned things like application text So for instance for task logs, it's just a text file. It's a lock. There's no versioning even to it So for the layer we decided to just use application slash text So it might not go that high It really depends on how many people have different layer types that they feel they need to register as well What one quick logistical thing do you mind? Are you sharing your whole screen or just a window? I was sharing just my screen Then maybe just zoom in the font a little bit Okay. Yeah, is the registration a lot enough resolution you yeah How's that Perhaps better I could just change the res would it help to change the resolution. I don't know just like that's Seems bigger on on mine, but somebody else in the channel says that Let me um Where is the other window that I can Post the I was trying to look for the chat window. Oh, there it is. It gets moved up here when you share Where's the chat window? That's interesting All right Logistical challenges aside Okay, I was going to try to paste this URL and for the for others but okay So, you know, we'll see in the numbers. Um, I don't think it's thousands. I don't think it's two So, uh, it was more of for them to think about all right. Can you help us with the feedback? Because if you look at the end of this Um, so in this section, there's this optional defining artifact publisher manifests and the idea here is for Things like helm things like singularity things like opa that Want to be equally shared across all clouds Uh, we want to be able to publish these well known types so that we as registry operators can create A view similar to this like what's the icon that we'd want to show What is the string in a localized format that we'd want to show? Um, and then for computer processing what makes it actually unique? so that's what in a Schema that somebody would put this information in And the idea is to be committed to this artifacts repo And then we as registry operators can occasionally pull from this And have these well known types published Um, it is a beginning of it. Uh, here's I guess here's an example of something that I had made for an unknown Um, as just an example of what one of these well knowns would look like But the latest things that I've been working on is this media type would get registered with iana And as I get the feedback on how we do this process Which is add iana registration requirements here, uh, that others can follow the same So there is other feedback in uh, the repo that I in the pr feedback that I need to address But that's the latest round of stuff I focused on so And joey, I saw your feedback. I haven't had a chance to address it yet No worries I do have a question, but it's possibly slightly off topic. Um Where does manifest lists Uh, yeah index. No, it's a great great great conversation. Um, because we've had that conversation with senab was the The most prominent one about that The last conversation we have and I had to go back to look at the notes We had general consensus amongst people that the same way we do a manifest config dot media type That we would have an index config media type um and Right now the index doesn't support that. Uh, we talked about how we'd version it and so forth, but that is the the Working plan if you will and the only reason we haven't pursued it yet was one we felt like it wasn't necessarily And maybe we can discuss this in the the 1o spec Cut off if you will get into a 1o we felt let's finish this with the manifest one because we literally only had senab that That Theoretically needs it and they're actually doing some changes on some things lately anyway with how they handle Non Instance, I figure what they call them the basically the the root Image instead of putting the content in the image that there won't actually be in a runtime image. There'll be this You can just bundle a set of artifacts with it So I expect that will evolve more, but I wanted to get more concrete through this And then once we get that done, we would come back and address the config object Yeah, um I mean the reason why i'm asking this slightly out of but I was playing with manifest lists with you know to try implementation and You know, I got it to accept the advert tree The advert tree I got to set the manifest lists media type but Actually, he used the doc and manifest command. I still couldn't push To the registry and I wasn't quite sure what was going on and I didn't know it was oci distribution spec Doesn't say anything about manifest lists. So possibly it comes in at this spec. They they got uh Probably i mean derrick's on my call I don't remember all the reasons why but we push the the name got changed to an image index as if there were a list of images which At this point manifest list is a more appropriate name for them Okay, cool. I didn't know that So if you look at the image spec slash image index dot md They're they're effectively identical Okay Yeah, I I was implementing our experimental support for oci for us last week And I literally just took our manifest list implementation and renamed it Um, it works functionally identical I've you tried working with the docker client and the docker manifest commands With an image index or Well, that would be you know, that won't work image in the index without I assume that's a different media type. So yeah This is definitely a different media type, but it is structurally Yeah, I assume I was missing something all there's a bug in the docker manifest command because I couldn't get it to work Yeah, there are some bugs in there. I think Phil's tools actually probably more up-to-date is I had a rewriting of the that manifest sub command at some point, but It never finished because we were focused on There's there's a lot of stuff in the back end that would need to get corrected Which we mostly did in container D. So I won't get into that but Yeah, I would suggest using probably try that other tool Okay, thanks And one of the things we discussed Was kind of like this refactoring zoom out of this one a little bit is we do have some and you know the image spec and the distribution spec were built when there was this The images right there was just basically the container runtime images and now that we're we're using that for other examples It does make things the the factoring of them a little interesting So we do have an open issue for refactoring this nobody's had time to go look at it Where the idea is those the manifest themselves so both For manifest the schemas for both manifest and index would in theory come out of those two specs or move up to distribution spec or something The location is actually an interesting challenge as well And then the idea is that the image spec actually is an implementation Of the runtime image of as one artifact type Where in fact you can see that here where helm and singularity would also be of that type Where image spec actually uses index as well And if you drew cnab over here, it would you know in theory be Both of those as well So that's that's one idea. It's an interesting question In the layers list or it would it would replace the image that image manifests image manifest box Sorry say that again, Vincent Are you saying it would take the place of that image manifest box or it would be listed in Like the layers as a layer um, what i'm saying is that the image index and the image manifest Those two schemas are things that we're reusing for all artifact types and In the image spec happens to be a type of artifact that sits within these two Okay Right, so if we all three of these types use these well, they use the manifest schema And they have their own config Media type to define how their layers Are used as opposed to that these these two they're not ordinal layers In fact, I think both these happen to only use one There was discussion at one point of how i'm using two possibly, but I think there's still one I thought I saw Josh yeah Josh what's the latest on home latest in terms of Is it one layer? Yeah, it's a single layer um What's it called compressed tarball And then you're not using the config object either the No, the the content in the config is actually um There's a there's a manifest in the chart package called chart that yaml That's j sonified And that's the big okay So if and it's a total if right, but if somebody wanted to have Uh two layers for an artifact and I'll use helm as an example Where some of the information the chart itself is one layer and other Config parameter information maybe default values of something might actually be another file that somebody could overlay Then you would have two different layer types Um, and that you know, they're not overlays. They're actually different files used for different things So but it still fits within our whole registry distribution implementation so um And then you know the config object is optional as well You can actually pass null into it and it works fine or you can Pass a custom object. It sounds like the helm team is using that It doesn't even need to be json was one of the last calls we had All you need to do is just change the extension. In fact, if you look at the latest update I've got here Um, I talked just uh specifically about that Uh config There's the config Well, um Here optional config format. So it could just say config dot v1 And there if there's nothing in it then there wouldn't be a dot json I wouldn't you know, but if if helm wanted to do it as yaml They could certainly do a dot yaml and put content in there as well as the idea But back to sorry, I got a little bit off the question of refactoring So the the point here is this these two schemas that we already have already support all this capability It just so happens image spec uses it one way helm uses a slightly different way singularity uses a different way And and that's the beauty is these don't have to change Um, there is the one change that we'd like to do here is put the config object on index I think an interesting question for Uh getting to a 1o distribution spec might be hey, do we Maybe I don't even think it's feasible, but does image index maybe become not image index It becomes manifest index, which is I think what I heard you guys discussing that would be uh Maybe not changing it, but You know Because that's that's that's a different spec. Yeah, that's a different spec And that would be a breaking change for the image spec, which is already v1 If it incorporated a new mind being mapped to the new montype name and We could figure that out later, but Yeah, so that's there's been a couple like I had some feedback like why are we using layers in the artifact spec And in fact, I rewrote it We wrote the feedback with using blobs and then when I try to reference it back to the specs that do refer to Them as layers. It just became too confusing So I think this is the problem that a lot of people have especially people have been so invested it for a long time Is they come up and they don't really understand what we try to do with the artifact stuff because they're We're so focused on how image uses them But if we can take a step back and generalize and it makes sense and just some of the names aren't the most Generic of names, but the usage works perfectly well generically Okay, so we've got two minutes left for this section. Is there any other? Discussions or thoughts we certainly want more feedback in the prs. There is this this spec that we're looking here is a pr That we're looking for feedback so we can merge what's there with this the newer feedback here Steve really quick question When we're talking about iana registrations Did we did we determine that the actual layer media types would fall back on Normal already known types or is that just depending on the case? because with helm what we ended up doing we originally had a Custom type, but it was just a charcoal. So we ended up using just application slash tar Yeah, so you can in fact, I don't know if iana has application slash tar if it is I'll call it out here, but I did write this section. I updated this section to reflect that conversation So defining your layer types. There's and this just the You know, how do you want to think about the format? I and do you split the layers? So here i'm referring to kind of the way oci images are split. They're split due to size Amongst other things and there's stuff in here around deduping But you may split things because they're just you want reuse and optimizations for other reasons So and then there's a concept of layer versioning And then there's the actual media types. So this is where this background of how you might split them could come into play So if you're only doing one layer and if you're not looking to version it and it fits in one of the existing formats Then by all means use this like why why does it need to sorry? Why does it need to be only one layer? it doesn't oh a great question so um if it's If you're trying to have multiple layers and they're not just ordinal and because you're trying to use let's say a Configure that the default values and a manifest itself Then how do you differentiate? Is it you just know to look layer zero is this or layer one whichever you want to do? Instead of trying to just do ordinal base. You can actually use different names So where do I have an example of this? Well, here's how much I guess isn't isn't that more appropriate for like an annotation then um creating a brand new media type if it's just a tar or json like I thought that that was kind of the Conclusion we came to a few months ago But well the ideas doesn't have to be a tar like for instance and then you might decide to do this with helm or not But for the task logs, they're just fairly small log streams. They're not very big The idea to have to tar them and compress them was overkill. So we're just storing them as text Yeah, we're just doing application text. No, I I understand. I'm just like let's say you have 10 layers And they're all text and you're asking now. How can I differentiate between the 10? if you If you need to Sorry I was gonna say like I just don't um The way I'm kind of thinking about this now is that the only Media type that we really need to register is that for the content for the actual high level of the new type itself the individual layers I think we should avoid at all costs being custom types. And if you do need to differentiate Um, you see like for example between 10 different text files. I think the solution is by using um some known Annotation like open containers image name or something like that and then you could just put different names on the text files instead of Coming up with custom layer media types that that's my only comment No, it's fair. And that's something that I've been Contemplating about I think to your point most of them will probably fit in a single layer or an existing type Um, my general struggle with annotations is I find them as kind of optional And not very definitive From our registry like I expect that we will index them And I think joey does this more than anybody more than either registries Is uh index them to make rich metadata for people to query on them To as a registry for dedupping and garbage collection and others having a more precise understanding of what the types are Becomes much more meaningful But again, this is something I think we have to experiment more If you have 10 10 layers to your conversation and they're literally Well, it depends on what they're used for the one of the examples that we had was One of the layers is is a doc, you know, like I here's the information on how to use this particular instance It might be the source if you decide to put the source inside of as a layer as a separate Instead of another artifact type by having that as known that this thing is The source layer as opposed to the runnable layer that might be interesting to somebody Um, but I definitely wanted to tease out that if these fit your existing generic formats fit Then by all means use them So I wanted to actually bring this up as well because it's a meta question that I have on the spec which is Given a set of layers with particular mime types How is client how is a client tool supposed to know what to do with those layers? And it seems like it's a very artifact type specific um, and so I tend to believe two things one is Um, as long as we say that client tooling is only going to be able to handle layers on a per artifact type basis Then you can kind of have the freedom to put the set the layer types to kind of whatever you want Right, if you want to multiple layers of text and then it's up to the client tooling to understand that if it sees You know an even number it needs to merge them and an odd number it needs to you know do a little dance fine The second thing I I would tend to prefer to say is if you're going to be putting supplemental information Into an artifact I would prefer we did that via an index or a manifest list as opposed to Other layers that aren't used as part of the primary artifact only because It means that we get into this nice ability on the registry side to be able to say okay Here's the actual artifact and here's the stuff that goes along with it So as a canonical example, we've been having conversations about you know, how would you include source code? with a container image and I like the idea of having a manifest list that has two entries one that's the container image itself and one that's the source Reference and I would prefer it that way as opposed to having the source reference be layers inside of the Container image manifest most notably because then tooling that handles container images today would have to be changed and it would also Unless it was very intelligent would have to download all those layers and replicate all those layers when Honestly, most of the times when you're operating container image You don't need the source associated with it. You just need a reference to it So that that seems to be from my perspective why a manifest list makes more sense For two distinct artifacts as opposed to mixing them together but At a high level if we if we just say, you know, as long as it's very clear like explicitly called out in the standard client tools It's up to the client tools and the particular artifact type definition to explain how to process the layers after downloading them if this thing is actually to be used then Generic tooling can just move the blobs along without any real insight, which is good Yeah, so that's a great point. So on the latter thing you're just mentioning so that's exactly what I've captured here in the scenarios for the Software supply chain conversation Is I'm actually using an index for the image the s-bomb and the source and each of them be signed So that's I completely agree in that example It probably makes perfect sense to have it as an index because you want them separately because if you look all the way to the Right, you'll notice over here in the deployment area. I no longer care about the source. It's still in the registry I'm just not using it anymore and I might have implemented some deploy configuration that needs to be persisted for instance So I think this is And to your other point It is up to tools so up in here defining unique artifact type. It is expected that An image something that runs the runnable image, you know container d docker Are not meant to deal with helm charts That's not that's not the format that they were designed for and it gives them to do what they want in their format And that's what singularity was able to benefit from as well But I definitely wanted to make sure that the layer types Had these as options And I think that's the thing that we'll watch is that if most people can just use these then I think we're fine And that's just a sample of this right you're talking to any Yeah Okay I Strangely, I didn't find that many in iana for generic So I was kind of I was actually a little surprised that I didn't find more of them and For things like, you know, I think even text I couldn't find an application slash text It was text slash and there's various other values in there json xml were in there Um, but I couldn't find as many as I thought I would find there Well, okay. I mean we can talk later, but I thought like um Like text slash plain would be considered Those are other great examples as well. I did not find very many I think people use those generically like if you do a search on the internet for various Uh media types their people have some standards. They've created they just never got registered with iana So if you look here and we're coming up to the half an hour. So I was trying to but if you just look in this list Like text or slash text like in fact, there's nothing here Well, here's text slash. Is there a text slash? Yeah, that's that's kind of what I was talking about like css is there Yeah, see these I found but I just didn't find like plain old text And there was I can't really that's I think that's text plain properly There is no text plain Interesting Yeah, I that was kind of where I was a little, you know Lost a little bit and then individual vendors went off and did things Uh, and there's obviously lots of other ones in the section. So You know for the sake of time by all means, please people have been giving me great feedback by looking digging into some of these That's how we stumbled across this It took me a while to go through and read the various specs and get my head wrapped around it So that and that came from this feedback process Anything else before we shift off to Vincent? Vincent it's all yours I don't know I should have actually prepared something when you emailed me this morning So the the the biggest part how much of a time slot did you allocate? Half an hour, but uh, what's after you uh, the supply chain thing unless william pops on I haven't seen him yet Um, it'll be I probably have 15 minutes depending on the conversation because it as we put in the slack channel It's really a scoping conversation just to let people know where we're at on some of the other conversations So go as much as you want Sure, um, it's it's really pretty straightforward. Let me see if I can Just get a single So we'll be talking about as the distribution spec 1.0, which Is a lot of what you just touched on might have made not a lot, but most of what you're touching on is I'll simply the Relates to the image spec For the distribution spec most of this is already Hopefully stabilized So um, we do have a A couple of Open open issues on the the distribution spec and the milestone that I don't think has really been tidied up The the thing about the distribution spec that makes it slightly different than the Image and runtime spec is that we were there was more iteration On the image and runtime spec back in 2015 to 2017 As we were developing the tools that were using it as the this registry api that All the different folks that are dialing into the calls from You know the docker hub and from amazon's registry and quay and as your gcr Um and harbor is as we can get the times on right This is already like in place and so even though this is somewhat in a release candidate march at this point Towards a 1.0 It's practically already in production. So we're just trying to smooth off a lot of the rough edges that folks have already found You know certain end points for otherwise that everybody wishes to get rid of because they already disable them where they can or We're never good behaviors to begin with so if we do make effectively a breaking change so that's one that we could All make together and then have some idea for deprecating it like you know that getting rid of the catalog api for those that pay close attention But Most notably there's just a couple of things outstanding for um smoothing out this March to a 1.0 there that for notary Uh, particularly there's thoughts about how to use the registry effectively fold back in what's needed for the registry Using what we've already done. Um, and so there there's the idea that Everybody's very familiar with push pull Otherwise my basic container image workflows for a registry um But practically speaking it's just a content addressable storage with some kind of a tagging mechanism. Um, Which helps for garbage collection or acls and otherwise and so There could be behaviors like for notary storing and otherwise signatures um That just reuse this content addressable storage Almost like a look-aside cache um, and so these things could be done Additively to the registry api the distribution spec Without breaking existing functionality. And so how do we before 1.0? at least stub out some kind of Additive extensions so that you can see oh this registry supports The things that I need to be able to store signatures in it or something like that Um, and so we've started walking through that um In that same kind of at you know additive extension proposal The the big other one is effectively the the successor to the catalog api and Joey Schor that's on the line here Has a proposal up for a events hub sub type model um These things are kind of like signatures for notary and then other stuff for like discovering the the number of images you have um or new images that come out would be Some kind of an extension to that registry api um So I don't think that anything particular with the notary v2 is going to be Any kind of a blocker, but we're just at a kind of a Suspension moment where as some of the dust settling and everybody's like trying to agree upon the approach that we can Do that before this 1.0 settles or at least fold it back in early enough So now's the time for that kind of like request for comment in the registry api and how best to use it um And then as we Are now moving from the use case Use case and requirements model for notary b2 that it becomes like a very very concrete example of How some of these would get used and how it would integrate back into the registry api Derrick McGowan who's on the line also here. He and I had a a decent call this week um talking about how to reuse the registry api and just starting to stub out some basic routes that Don't exist for notary but like playing through the use case And this is where we're gonna have to obviously get much more On the notary side of what's the workflow that's needed and what are the kind of endpoints and you know security contexts and whatnot For actual, you know pushing pulling fetching seeing deleting updating whatever revoking Signatures that are attached to assigned blob Or assigned to what label tag Um This is a straw man just to have a conversation, but this is the kind of Uh, these two worlds combining are kind of overlapping in the Venn diagram of What endpoints would be useful for notary and needed for notary like a notary signature extension to use the registry Um, so that we know that what we're doing and going to stamp as a distribution 1.0 um Enables that so we don't have to Bolt on solutions to too terribly I'm gonna pause for a second because there's Probably a lot of clarification that I could provide that I don't know that I've just skipped over of like how the OCI distribution spec and Notary overlap But please Help Is the idea here that um, because I'm just looking at the things where the Using signatures as an example, but it was this because it's the extension proposal Is the idea here that I could put some extensions on a repo or on the registry as a whole Because I'm just having a little bit of reaction to the no, it'd be a right It would be a registry the registry owner would know would I mean that's a clever idea But no, it's not talking about namespacing it to where you'd say my my repo only can respond to the same Uh, Derek do you Derek or Joey do you have particular thoughts on faults enabling extensions per namespace? so We would like for anything new that we implemented we would likely whitelist particular namespaces and or repositories for testing But from my perspective the way we would do that is if you hit the api for an unsupported namespace or registry We would just simply return a unsupported HTTP code So tooling would still be able to discover that we support it as as a product that particular extension and then They tried to operate against that particular namespace or repository. We would just return a code and likely that would be a temporary a temporary State which time as we've fully implemented it and or fully tested and or Added some new feature or something. So that's my gut instinct um, I could I I could see a scenario where you know an administrator may want to have certain extensions enabled for certain namespaces maybe even repositories, but I haven't heard a concrete answer yet where Not having the discovery mechanism work is Required for that versus just reporting an unsupported code if it's disabled explicitly for whatever reason Yeah, and then in the repo space I wasn't necessarily talking about like root repos where we use those to disambiguate different customers right the multi-tenant nature I was more thinking there and I and I wasn't it's an interesting idea that maybe a production repo might have or not have a Certain capability, but the if you scroll down a little bit repo. Are you talking about like container registry at a whole No, that's sorry A little bit or move the screen up So to speak because I've seen mac and windows and Linux people get have different opinions of what up and down means I was looking at the line where it says get and post put the second grouping where the signatures are on a repo This was just us trying to Derek and I was trying to work through In my mind extensions would be the difference between these two groups here was just that Within an extension directory Whatever extension that you register or describe be it Events or signatures or whatever Would then be its name and then that's the new prefix for any like effectively sandboxed extensions that you describe like any end points That the the thing that the Derek was saying is that for most registries the existing like top level and then name Already has a lot of like acl logic built in So you'd you'd register an endpoint or you know, there'd be some something over here Ignore this part. There'd be something over here where you could discover the extensions that a registry might support But that the actual mechanisms would integrate back across Existing end points So this is this is basically trying to think through Something that's wholly outside of the known end points and would be sandboxed or relegated to a prefix And this this is the approach thinking through it Integrating it, you know, whatever extensions the registry does support get integrated back across or back through known end points And there's those there's more discussion about that But and this is the kind of thing that just using something like signatures as a discussion point how best for registry operators to handle the acl's and the garbage collection and Uh, whatever else and some of this even gets into like Should be a delimiter a delimiter here so that we know, you know, if we have more than just two-part, you know bbats my app repo name if it goes like multi stages deep how to delimit towards all these other things that are happening here Yeah, I mean just being out loud about this. Yeah, I put that proposal with the underscore I mean I would like I see us having a need for both There's going to be like so the pub's a proposal that I I put out. Um, most of the end points are Are kind of universal right because you're subscribing at different levels, but those the level is part of the subscription So it's not under it's not home done. They're a particular repository but No very signatures and other kinds of extensions like that could absolutely be home done to particular repository. So I That's even where something that that uh, you know, I went back and forth on because you might you might sign You know a tough level image like I might push and sign bbats my app but you know particular Creator of like a layer I'll pick on red hat my employer Might push out a base image layer, you know, and they're not signing the config Manifest and all that jazz. They might actually be signing like an object that is the entire layer of something Um, and so you might want to recurse back through and see like actually, you know, one of the layers in my stack Has a signature that exists for it, you know, so like if you wanted to sit and check in something like this like show me the uh signatures on This object and it might not have a name on it for say, but it might be You know an object in the stack that somebody has attested to And the acl is here get tricky. Um Because what if somebody's just pushing arbitrary signatures anyhow, that's getting in the weeds But this this is the kind of thing of like where should it go back across the stack Is it only at the top level named object or is it any object at all? Yeah I agree with you and I think that's I think we need to unfortunately leave the flexibility in place for both those options And then it's going to really come down to the very specific extension as to How we're going to handle acl it's going to have to be part of the design of the extension and how it's going to affect these other paths, but there's I can easily think of multiple scenarios where you need both a Extension specific endpoint that's not tied to a particular namespace or repository and ones that should be tied to the existing acl And they could very well and likely well exist within the same extension So as long as we reserve both sets of paths and say, you know, if you define it as sigs you get it both here and there Then at least there's flexibility for the future Yeah, yeah somebody Yeah, yeah Yeah, that's well. That's why I added that, you know, let's have a the limiter in there Something between the repo name and the extension name. It doesn't have to be underscored. I just Yeah, that's pretty much the same thing of like if you Otherwise you have to kind of Do a head and a tail? Yeah, um so This this this is probably one Having having at least any kind of an idea on this It even this could be added after a 1.0 of the of the registry But it's all in the the dust settling kind of phase Along with even really really simple stuff that I think nobody's just Cared to think about but like the read me is just pretty bare minimum and Getting it out the door, but it is for all intense purposes already in In production as it as it is and the conformance worth it Josh and Peter have been working on is I think really driving some related conversations finding that even when this was the registry API documentation over on the the docker distribution project That um, it was kind of a skew of conformance as as it was um, and so we're having really really productive conversations about What even exists in the wild versus what's documentation versus You know all the different places so Uh, if any of that's of interest to you feel free to reach out and get involved But that's a quick question here Someone can hear me Okay, that's super. Um, uh, do you have a timeline for when this goes like fully 1.0? Sorry I'm asking the hard question. That's I know but like you can say like we don't have one you can say six months like I just want to know like the I can I can only set aspirations on this Same kind of same kind of comments and back and forth happened with the image and runtime spec that All of us set what we thought was The conservative estimates and they ended up coming and going Even with the distribution spec everybody was like that seems to already be like in use and this was before all the stuff started happening We thought it would be done by the end of 2019 That has come in past already so There's been probably more activity on this in the last three months then In the prior three or six months Um And it is getting closer. I'm not I have no magic eight ball to say it'll happen in some time frame I would love for it to be Done now, but that's it. It's not quite ready for that Okay, it sounds like I should take my question offline and we'll work out what timing actually looks like That's fine I'll probably get a I mentioned on the oci call last wednesday that I'd like to get an rc to vote up And the the the things that spun off out of that So basically it was it was less of a timeline and more of just a nature of the beast Tom, you know kind of an estimation When we were doing the other ones was that we'd get an rc out and then look for what kind of things And people are like, oh this needs to be in and as that tone down Then we're like, okay. There's not really really been any changes since the last rc We'll do one more rc and the next after that will be the final That's kind of what I always I want to get an rc alpha rc2 alpha door Release kingdom two and then see what's next and from me kind of surfacing that not from but In in in the same moment of me representing that long we've got rc2 out the door Josh and peter were like we found so many inconsistencies. It would be great to just do a big Pass over the spec. So we'll probably have a working session where we just Kind of reorganize some of the spec There will be fallout from that. I'm sure So what are we doing rc before after that and then just kind of tidy it up because It's one certain thing. They're just unclear in the spec Not a lot of that's just a cruft of time Hey, really quick on that point. Um, so me and peter sat down this morning um and kind of look through the spec and Uh, I put a link in the chat, but we put together a hack md or peter. I should say but um Of the possible things that might make everything a bit more clear um, and then also on that point the So I sent the email out last week to do a working session and I got quite a bit of People interested in that so I don't know if we want to make that more of an official Um session, but sure Do you want to do it? How soon do you think it is? Not soon enough tomorrow wednesday uh Yeah, I mean I was kind of thinking We do wednesday prior to the actual call, but I'm not sure I don't know if we have an agenda this wednesday and since we're not all announced We we could just stomp on this wednesdays make that the call. Okay All right, I'll respond on the mailing list with that Yeah I'll second it because we use that's in the march towards 1.0 for image and runtime that that ended up being some of it just like We're taking over this hour and just make it clear on like the weekly meaning, you know weekly email out that like This is all we intend to talk about or work on In the hour. Yeah, that sounds good cool. Let's just call that now. I mean we don't have an agenda yet everybody, you know So I think it's fine just to call it now my problem is I haven't had time to dial into some of these because That that hour is blocked out and I've been busy with some other stuff. So I think that'd be great Okay Consider it called um Is there any other questions folks have about this right now Is your idea that in we wouldn't necessarily hold for notary, but if we do the extensions That's the placeholder that we can stick lots of things in is that kind of the assumption? um Lots is maybe being Starting That It's just a place to have a kind of Additive conversation Additive is the term that keeps coming to mind that It wouldn't be breaking and there'd probably be peer reviews. Once it they're not probably there's definitely peer reviews So you get merged into distribution spec pages about this um, but others could have a framework very similar to You know bit torrent extensions the bet bet BEP process or python pep otherwise That folks could try things out, you know, and even if they're they're trying out some new thing in their home local registry that They can discover it and try it out and that doesn't exist upstream But if you want it to be Generally reviewed they could So it's it's more just getting the framework in place for that And just using something that's not That's the that's the danger with specs is you know, just arbitrarily contriving things that don't exist, you know for your use case as opposed to saying Like with the events endpoint or the signature endpoint Like pretty tangible. You can start discussing the you know the the types of workflows or whatever that's needed How does that affect the existing API? How does it integrate through? That really really really helps in describing that kind of endpoint so Like I said, it shouldn't be blocked by the signatures or one shouldn't block the other They just make for this is the time to have that discussion and it's good to have a very tangible use case Um to when describing something like this extensions endpoint or extensions aspect That's all No, no, I look if you was going to get anything into the one aspect to have something that says here's how you add Extensibility to it that seems like the right great one, but to your point not having a concrete use case Makes it kind of hard. So maybe between notary or the pub sub thing that joey's working on um I don't know of others that we've been discussing I don't know of others that have actually had some concrete Execution like we've talked about a search api and some other things, but there's we haven't had any traction Yeah, I was about to say the other ones that kind of loosely bang around there are search apis or some other kind of like endpoint to This is where it gets effectively into search api, but some way to like query for you know, like What architectures are you know something that gets down into the Configs that right now has to be built on outside indexing kind of thing and if somebody wanted to do that and you know Propose it and you know Kev like if you wanted to expose this index endpoint like okay sure that but those get a little Long in the tooth and this is just like what's a what's a very common critical use case that everybody could discuss Great I don't I do not expect it to be Many I expect it to be a bare minimum and then a place for people to experiment Have you thought about how extensions register what apis they get called on so that there's some uniqueness Is that was a part of what was in there or How extensions if I have a full extension Or was that kind of what you were getting at is they would hang off the root or it hang off the A repo like what would be the api name Of my foo api that I want to add And how do we Avoid collisions Like is that that's pretty much the the discussion there is like I wouldn't mind those things being sandboxed Because if they're But as long as whatever name that you Register so at first I was trying to make this thing almost recursive that anything describing The types of extensions Are nested under some name initially but In the sig signatures example that like dollar sign name would be Sigs if they wanted to put something that might it might be prudent to go ahead and make that like sigs v One or v zero as we iterate on that So that that extension itself takes on the oneness of Versioning it Have not gone so far as to Propose something like kubernetes style Versioning, you know v1 alpha one v1 beta one v1 But for the like they do for the crd's um, but the Name the name Would carry that so if they changed it then it would have to change the extension name and that would subsequently affect all the endpoints that that that extension Provides or does Um That's really it it would it would have to be worked into the path like those two Conversations whether they're sandboxed or whether they're integrated back across the registry endpoints, but the name registered for an extension Would be worked into the path. So if it ever needed to change due to versioning or otherwise that That would affect everywhere that that path is integrated It was the crd's that was making me wonder if there's a Prior so to speak that we could leverage there because but I love the the acl conversation. So it's It's going to have to see if I get sajean others to look at it. Yep. Yep. Cool Okay, there's any other questions. Uh, you can direct one to me or somewhere in this cut in this chat, but uh We do have some milestones that folks are trying to tag and if there's an issue that you see needs to be raised issue it and say please tag this for a v1 milestone piece But otherwise I'll yield the floor to the next person Anybody else up for the spec 1o conversation? Okay, so Let me second get my screen. I mean I saw the comments. Let me try changing the resolution of my screen here To something more reasonable for sharing I'll go all the way down by eight by six, but Okay, it's really strange But okay, um, so the next one was just where we're at on some of the the Secure supply chain work. So here there we go screen two Um So I and I saw some of the conversation and some of the slack channel on that so the The conversation here on some of the supply chain effort because remember this this call or this meaning that we were going to have In kubcon was more a matter of a bunch of various efforts that various people are working on It wasn't necessarily all grounded in oci such as notary v2 is actually in cncf. It's just a a collaborative effort the There's been a couple of things that we've discussed as part of notary v2 such as pedigree and What's the other pedigree and I even wrote it in here pedigree and providence And the the thought process is that there's a certain amount of things that we should be doing with notary such as the the ability to sign something But it's not necessarily meant to be everything to everybody. So In the scenarios doc that we've got under requirements I did call out specifically this kind of Working flow. Um, so if we look at this imagery and this is joey was kind of referencing this before I was using the stuff that joey and i'm not sure if joey and vince have been working on this together Or just they've been talking about it. Sure. But the idea that I might have source That I need to ship alongside My runnable artifact my artifact in this case a runnable image Or I have a generic s bomb that I want to have in this thing The idea is that from a notary perspective for it's it's not very Sensitive to colorblind people. So I apologize. I realize that as I Wrote did this but the blues are meant to be what we're trying to do from a notary perspective Meaning we're signing things don't care what we're signing we're signing things And it can move across registries One of the things that we could sign is in s bomb And there is a couple different efforts that are going on. One of them is this 3t S bomb conversation and there's conversations even what they're going to continue to call themselves But the idea is that at the end of the day There's a whole bunch of things that happen to the left of What happens before it gets into a registry how images content is built how it's At a station and compiled and so forth And the idea is that At the end or one of the ends Is that there can be a document that says here is what makes up this thing? And it can be everything from the packages that were used to the compiler that was used In all kinds of other information. So That work has been going pretty well As all open source projects go there's lots of back and forth and trying to scope things And the only part that I've been trying to make sure from our effort is that if they're you know Is there going to be a single document at the end that accompanies the things that are done? So that we can say look we we can sign this thing We can group them together in some way that we know that this s bomb that Categorizes what the content is is associated with another artifact such as an image or a source And that we can move this from the vendor's source where they the vendors Use Published location and I want to confuse source with source code But location of where that vendor comes up with gets it into a public registry moves it over to that customer's registry And then they can have that content And this right here is where you see the conversation of generic imagery came up I was trying to look for a generic policy management icon and all it was left with the opa shield without opa on it The idea here is that Some policy management system can look at this s bomb again. We don't care But the idea is that we can at least attest that it is valid. It has a signature. It is what it claims to be And as long as the keys are continued to be you know valid that the policy manager can act on that very efficiently One of the things that I've been trying to Keep an eye on is making sure that these supply chain Efforts that as they go through from development to build to publishing to production Is that the closer we get to production the faster we want the thing to be Evaluated and allowed through or not We got in a lot of these conversations on image scanning Um Do you rescan something every time it goes to deployment or can you look at the digest and know that hey I've already seen this thing before so until somebody tells me it's no longer valid I'm gonna know that I've seen it before I saw it within a safe period of time and I'll very quickly let it through The idea is the work is done up front To scan the content and know that uh, you're you're happy with that thing before it goes through And the analogy is you know you go to the airport you show them your passport And they look at it for the 3d Uh, you know imagery to make sure that the the document is valid They scan and the number is quickly evaluated to know that you're not on some Government watch list or otherwise as long as those two things are good And you actually match the picture to whatever evaluation that person's doing That you're quickly allowed through you're not asked to Represent all the the information that gave you that passport in the first place So that's kind of all I'm trying to scope things here from what we've been doing And at least from the the working group that's been with uh, and I see Santiago here With the notary and the in toro stuff that are tough that There's a document coming together with spdx is also that are trying to get some format put to that that that document's done We could sign it That's I was curious about that You mentioned that there's a relationship work being done here As in how to relate artifacts to each other, which I think it's a crucial for supply chain integrity Uh, could you elaborate a little bit more on that or? Yeah, so basically if we look at the gray boxes the the idea is that uh, this is you know Why it's good to have all these different groups working together because we see them as separate problems That cumulably, uh makes for a better and then solution that each area is Managed so from a registry perspective if we just think of think of these things as generic artifacts And then we can store an artifact in a registry and sign it So for instance the sbom document that uh, you and kate they've been working on That could be put into a registry signed and it has some This is the part that we haven't figured out yet, but I assume with the notary signatures, it'll happen as well Uh group these two together or it might just be grouped because they're both in an index the oci index So now I can have an index that says here's the image. Here's the sbom potentially here's the source Because when you get all the way to the right When I deploy this to my cluster, I don't want to deploy the sbom. I don't want to deploy the source All I really need is the image So we don't want the image for instance payload to have the sbom in it. We don't even this is I don't know how to handle the sick think about the signature yet But the idea is we want the each element to be as small as possible to just define its space so that That that's all that needs and you can go on So the idea is that artifacts are things such as an sbom And it's signed and there's a correlation between the two at the worst case. There's an index that pulls the two together I don't know yet what we'll do on a better Better example than that and hope in the notary stuff. We'll start to surface in that Okay, thank you Yeah, I would like to take a look at this because I mean this is something we've done a lot of research And there's many ways you can get it slightly wrong and uh Open a big gaping hole in the supply chain theory front That's also partly why I think it would be important for the I mean an sbom can refer to an image as far as I'm aware And the linking should be on Can So what I think it would be crucial for Like a registry say notary to do the snapshotting and the linking of the At station on an sbom And the image itself. Uh, I think at least that's what tough is doing in the case of data. For example You get a you get your artifact and then you get the sbom on it and then you get all of the underlying at stations that you can Quickly chain together I wouldn't be too worried about the About the performance on that. It's basically a couple of string comparisons And a couple of if you want to make it very efficient ed25519 signature checks Yeah, so this is why I've been watching and just trying to keep up with what's going on to make sure that they do Kind of meet some of the the goals that we've kind of outlined here is that we can achieve this Yep, technically, this shouldn't be terribly difficult to do you can in fact to do this with notary we won right now As rado has showed building a proof of concept for the cnap security stuff. You can already do this Uh, the suggestions that there are some security improvements that that would be great But we can we can already do a minimal working example of this and that's why I follow the key management issues some of the security issues that related to that Uh, the second thing that Santiago is talking about is that you may not want to sign everything Like the snapshot metadata, for example, and that's something that can be done by the registry shouldn't be forced But we should allow for Yeah, I think similar to the artifact conversation earlier there's no There's very few musts per se. It's more amount of what can I do? And the idea here just purely from a noted perspective as long as we can sign contact because if I have an sbomb But there's no way to prove that that sbomb didn't change from point a to point b Then it doesn't really carry a lot of weight the same way that an image might change from point a to point b or any artifact So the main thing we want to do is just be able to say that as these artifacts move Within a registry or across registries that we know or and then to an endpoint such as all the way over here That we know that they're still in the same state that they were intended to be then You know that unlocks the possibility that I can this this policy manager, for instance Can look at this document and make decisions on it because it can trust that that document is still has its integrity associated with it Uh, so for some context, this is just a funny like notion In in total verifying an sbomb creates an attestation So if you want to do this type of uploading of the computation down the line Then you can on the public registry verify the sbomb and create the sign up station from the public registry That says I verify this sbomb and then pass that attestation down Uh, I think this type of semantics are what's important because it allows you to do a lot of interesting things and also Uh, make things more efficient later down the line Yeah, exactly. I mean this is why you know me sitting kind of outside of these and hearing Vincent and others talk about stuff that a red hat You guys are doing like the reason I purposely did the sbomb with the lower case. I always to be more generic because Red hat might decide not to use the 3ts bomb format They have a different one that they decided it's valuable to their customers from a registry signature perspective We shouldn't care right it should just work and they might have a different policy manager But if somebody wants to use opa and a 3ts bomb That should work as well. Obviously, there's a coupling this thing that's reading this document has to know what it's reading so to speak, but We really just want to facilitate as many different Projects that want to move content through so yeah, I'm looking I'm really looking forward to the work that you guys are doing I was really hoping william was going to be able to join although I didn't ask him last minute to be fair Uh to kind of give an update there So, uh, hi, this is nisha One thing that I'm having a hard time wrapping my head around is the relationship between the the work that's going on in the distribution spec and the artifact spec required If there were any considerations there for how these Different artifacts are linked together So the the scenario that comes to mind is you have you have an existing image That has That's from a supplier that has You know the ability to build your golang App So you use the image you add your app you you probably You know use a client to go and make Uh your resulting image So at that point you would need to Generate an s bomb Uh You might want to include sources Uh, and you might want to sign all those, uh, you might want to sign the those resulting artifacts Um, how would you how would you account for? Any additions that come up? You know, uh on top of existing images Or is that something that? Um, we're not thinking about right now I'm so it's a good question. Let me clarify to make sure I understand so if I assume you can see my screen here. So Um, we've got an image And there's an s bomb as a separate artifact in the registry is what i'm showing here And I think what you're saying is if the s bomb from here To the next step has some added metadata that gets added to it. Is that what you're asking about? Yeah, so, I mean that this is how folks use container images, right? They'll Say from and they'll Uh, they'll put stuff on top of it. So this is at this is at build time so, um, I mean It's all this is all well and good if you're like building everything from scratch But what happens when you start augmenting on top of it? No, that's fair. Um, so that's two aspects. So there is one I have built artifacts That i'm not changing the content But I want to add some other information to it. I might say that in my company I have some other out of station or something else that I might want to add to the s bomb So I'll come to your rebuild scenario in a second But in that in this in this case that I was just going into I might have a second s bomb because you're right I can't change this s bomb because it's got a signature that says this Intentionally say it's not changed I can add another s bomb that references this one is my understanding how the team is doing it so that they're more hierarchy Um, and the new one gets signed, but the old s bomb is still there and it's still signed So I could kind of see the lineage if you will Okay, so just being more generic here So the that's kind of the the generalization there and uh trishank is kind of referring I'll turn it over to him in a second here from So that's one model the model that you're getting at is another conversation We've been having is if I take an ubuntu based image And it's signed or maybe even a rail a rail image because there's you know, there's certainly a backing on that And I now add my application to it Is there some concept that when I've Rebuild it with a from statement from a rail image. What do the signatures look like? Um At this point what we're saying is there's a signature on the newly built image Uh, we had an open conversation around how should we think about? Uh base layers and should base layers and base layers are meaning any base, right? It's like second layer anytime. There's a from statement which it itself might have three from underneath it But the point is is any Layers that I'm referencing in the thing I built is there something about the signature chain that we might be able to leverage I still see that as an open issue. We haven't figured that one out yet Uh, just because the way layers kind of get moved within a registry not all registries do de-duping It's when the de-duping is done Um, but it's an interesting conversation. So at this point at least for now uh, the current working thought and actually, uh Vincent and others that spent more time on this is The thing that gets built gets signed And I'm seeing Derek's face here. Is there any work or Vincent? Is there anything's been thought about on the layer signature stuff being able to carry to another image that gets built? That was what I was alluding to earlier. It's just like concrete use cases, but the acl's of that get a little tricky I feel like there should be a way that we sign layers, but At the point at which you ask for like an arbitrary list of what all issuers have Attested to a particular object and that's to imply that anybody could push a signature regardless of acl's And then registry maintainers have a you know, registry owners operators have a situation at their hands If anybody can just push arbitrary signatures And have an unbounded list of these objects being shoved into the registry With no acl's that gets a little funky. So how do you actually say? No, this is exactly my object or I attest something about this object That that that that discussion has to be teased out further It is definitely something we're thinking about but it's I don't Have any particular epiphany on how to handle it yet Yeah, we talked about having the signatures cover Not just the manifest but everything under it So the problem right now is like you sign a manifest the manifest links to all the layers underneath it But then if you build something from that manifest that original manifest is gone So unless you also push that new manifest up to your new repository There wouldn't be a good way to link Kind of the those layers back to that signature So you know the possibility is that the let me make sure the signatures cover all the layers and everything that's part of an image and then You as a builder when you build something from some base image and you take those signatures you can just push those back up And while that repository may not have that manifest If it's also covering those layers you'd be able to make use of that So that that was just one kind of idea, but yeah, we need to figure out what the best way to do that is Yeah, it's kind of linked with the sbomb work. I'm doing too because at this point Um, like I have a tool that can generate an sbomb for a whole container image But it gets pretty large depending upon what kind of image you're using. I'm thinking about the The use case of cloud native build packs for example So if you were using build packs, you would end up with the container image that has many many many layers um Generating an sbomb for that ends up being really huge the sbomb becomes very huge So it it it seems to me that um, it makes more sense if You know that the sbomb was generated on a per layer basis and stored with Uh I mean in that scenario it would Be easier if the sbomb was stored with each layer in which case How does signing work? um and if you if you're thinking about that kind of ecosystem then you know that that is That's just one concrete scenario that I can think of Hi. Yeah, um So this is interesting because this is at least three issues that we have seen before in practice So so i'm glad you brought it out The first thing is let me see if I remember this How do you make sure okay? So you have sbomb for one step for your package? Let's say it's uh, it's a container image and then now you want to build other images off of it And that has its own sbomb materials the good news is that at least in doto it has solved this problem There's ways to do it so you have rules for your supply chain And your own supply chain can refer to the rules for the other supply chain that makes sense at all So for example, you say I trust the base alpine image for example, and here's the sbomb rules I'm just going to refer to that and then I'm adding rules on top of that so you can sort of inherit supply chains that makes sense The second thing you brought up is The second thing that actually winson brought up was how do I make sure that I know Which signature signatures are being pushed all over the place, right? How do I know that this are the correct signatures and it's sort of the planning depending on just acl for example Because if the registry gets compromised, how do you know that the signature is legit? Uh, one way you do that is to use this concept of delegations where the registry can tell you Hey, look, I'm using my master key and I'm telling you that this is winsons key And this is the key you should trust with this image, right? And no one else can if they tried to tamper with the signature you would know because it wouldn't match uh and the third issue Is that that nisha brought up was Sometimes the as bomb gets huge and in fact, I have seen this in practice Integrating toughening dotto. So I know exactly what she's talking about and the way we fix this was to separate You don't want to download it all at once necessarily You may want to download it on demand and the simplest way to do this is to just use different files different pointers For example to different as bomb materials Yeah, that last bit is where I was thinking this kind of dovetails into the distribution spec and the image spec which is that as um As uh, Steve has shown in his diagram over here You you want this you want this data to exist in the background but not necessarily when you're spinning up a container um But when you're ready to go and distribute at that point you should be able to I don't know like query I and thinking about this from a client tool side for example, so the client tool should be able to query the registry to see if this data exists and if it exists then What are all the what are the properties of it that it can use? so like for example, uh you know if There is if there is a signature and if it is a signature is it like What kind of signature is it am I capable of? You know doing something with it. Yes. No Uh, that sort of thing um So I don't know whether the distribution spec allows for that I'm not sure about this It this this sounds like part of what would flow into the kind of extensions we're talking about it's like if you have basic push and pull of objects um And tagging them with something that was his tradition had been an image name um This this kind of workflow of like bolting all this other stuff on might be its own like particular extension so that It would because it would introduce new and different workflows and certainly new and different ACLs like permissions of who has the ability to attach So that this is this this very much would add into the kind of like extension proposal we're talking about of like Above and beyond the basic current push and pull blob workflow, how does how does this how do you Describe the workflow you're you're you're saying in a way that works into a workflow of end points And I don't have a good answer for that right now, but that's the kind of feedback feedback that i'm Interested to hear before the distribution spec because v1 Well, there are some sunny parts to this so the actual like trust and permissions model and things for this In total does actually handle this with the way it deals with layouts and functionaries and other things of that sort However The notion of how you collect and send and sign metadata Is done differently by all sorts of different adopters who have their own workflow models For this so like trechan can talk about what they do With their way of doing it and you know, we can get several other Folks to talk about how why they made why you know different groups made different choices um So that's an area where intentionally there's some flexibility left open because I think reasonable people may choose to collect and upload this Information differently. I don't know if notary v2 needs to have like one prescribed doubt shall do it exactly this way Sort of thing or whether um some of the exemplars for the right approach Well, this is actually coming to a good transition point to hand off to the next topic was the notary work that uh, justin and derrick and think Other justin and others were involved as well. So If there's no other supply chain stuff or the supply chain focus conversations, I'll Hand off to justin for the next topic So justin I guess that's all you What did you just sorry for reference? What did you just cover now because I was not on the call that this So just so Wonderful everything you're going to do and you're going to solve the world's uh hunger problems And just any starting any sentence with that as a joke is not even funny anymore. So, um Anyway, we were just I was trying to put some context of the sbom Some of the sbom works and how we can just sign things and there was just some background on Some of the problem cases that were being solved were Outlining here have been accounted for some of the work that you guys have already been doing with tough and in total So I think that kind of brings in the the next conversation. Just you don't necessarily have to bridge on to this one per se Um, but just if you just give an update on where you guys have been at for the last uh week or so um I The probably the the maybe the nearest bridge might be if um Derek and Vincent wants to talk about What they've been discussing around signatures in oci directly because that is obviously Perhaps the most oci relevant piece could maybe start with that because I know you have been talking about that Yeah, we covered it a little bit earlier I know Vincent wants to share what uh, we talked about earlier um, most of the discussion is just about how we're going to Design the endpoints how they're going to fit into The existing oci distribution spec. Um, and how we're going to make it Basically, we're planning on having this is extension design and try to get that into the 1.0 so that we can add Other things to the the api without it having to explicitly be part of 1.0 So we can add there's a few other things, but we want to tease out basically the whole api for How this flow is going to work for adding notary to or just signatures in general to The distribution spec and once we kind of feel confident about that and we can move forward With adding that to the 1.0 spec um So I don't have the thing to show up that the Vincent had up earlier that had The endpoints that we're talking about Were you looking for Vincent to bring up the other doc you want me to bring up something or Yeah, if you can bring up that doc so I'm not on my I'm not at my laptop right now to share a screen But the doc your notes from uh the last meeting I think we have it in the notes. I don't know if we have it in the issue as well um But essentially like what we have here is we have the extension route Where we can define what extensions are supported by a given registry So this would be kind of a global level what the registry is actually supporting and then We need some way to add these signatures to the individual repositories So just as blobs are repository specific the same The same thing would be for signatures So essentially when you go to do a push You're pushing up all the blobs and then you could push up all the signatures that go along with it Uh using the same type of push flow and Then for pull it would be the same thing you push them up then you can just pull down those signatures for the individual blobs So the so Derek I'm sharing the notes. Is that what you want me to share? Was there some other Or just if you have it, maybe you can Um Oh, I think it's this one here. Um, yeah, oh then I can Yeah Um, it's this Just one I think say Um So the the signatures extension This dark hair Derek Yeah, yeah, that's where we scribbled down some possibilities. Uh so I mean we we discussed kind of the different layouts earlier for how we could do it whether it's kind of under a sandbox prefix Or something that's kind of overlaid with the existing distribution spec um I think I mean the reason why we would overlay it over the existing distribution spec is the way ACLs are performed so Yeah, our thought here is that signatures From the perspective of how the registry stores blobs and how the registry handles permissions Signatures aren't really any different than blobs. They're just objects that are Pushed up and pulled back down So even permission wise it maybe it's the same. Maybe there's permissions added That's up to the individual registries, but at least the way they handled Wouldn't be something that's that's unique Two signatures Yeah, so I will we just want to have It's it's really hard to say like Uh, so you can imagine cases where there's abuse of just people pushing up that signature Or just irrelevant signatures to blobs and then you pull down. How do you know? Now you now it's not just querying for an individual or querying What are the signatures for a blob that then wind to limit it by specific keys or something like that? um So you really want to have it be on a per repository basis anyways uh, so We could have the name at the front just like all the other apis Or if we had it sandbox under kind of just underscore extension extension name You have Kind of some duplication of how the apis is defined But either way like we have to have the name in there in order to do those sort of permission checks and like Putting each of the signatures into the bucket But yeah, I mean what other stuff do you want to talk about for like from how it relates to like OCI? I mean That that's kind of our thought of just making it Ensure that it flows into the existing like OCI flows Uh, at least in a way like push pull The way it gets into our like content The content hashing uh per repository and stuff like that yeah um I don't think we we came up with a I don't think we've We got to the point where we figured out how individual signatures are referenced in the api Um, that's what we need to figure out like is it is the signature going to have its own identifier kind of Almost like tagging. There's the signature going to have its own hash like how do you delete individual signatures? Uh, I think that was one thing we wanted or Permission wise whether it's a different permission for deleting signatures versus adding signatures Uh stuff like that Or could you add signatures but not push content, you know stuff stuff like that That part would be nice good because we've had these scenarios where to to sign something You have this problem with the notary public loan signing stuff where or docker content trust I don't know which way to call it but where I could have If I asked for something with docker contract docker content trust turned on I could actually get completely different content using the same exact tag So that's that's a slightly different issue Um in the sense that Notary v1 has a separate This would avoid that because it would be attached to the same content But this does come up to some of the issues that we started discussing on our meeting on friday about um The fact that there's no There's some differences with Tough in that there's no No natural way in oci to sign a collection of objects because Um, there isn't really Um You know, it's not it's it's no it's a pressed api in effect. There's no concept of a collection of objects in a registry And it's a distributed system and there might not be a consistent set of objects at any time so any kind of collection based signatures are problematic so Signing individual Objects make sense. There's also the question about whether Which has come up about whether signing In what in what way signing names is meaningful in the sense of or the equipment of signing get tags If you took it as a get repo Um, but yeah, signing collections is definitely problematic, but this this but that's partly where the that these semantic differences that we had as a discussion about them on friday about um semantic differences between a package A traditional package repo and a registry and the kind of issues that Um, potentially mean that the model Um potentially differs from Um from the tough model just because of those differences so, uh Sorry for the maybe dumb question, but why would I query a signature? And just the signature a single signature Yes, like I get I do v2 x6 and uh, and I get a 128 bytes Uh, are you talking about what's up on the screen right now? Uh, yeah, or in general like why would we need a specific endpoint for signatures rather than There this this part of this discussion right here. And this is obviously not comprehensive for like Signatures, this is just trying to think through one workflow As far as it relates to registries this would be Um Show me the List of signatures that have attested this blob and since you're pushing a blob It's not particularly descriptive whether or not that's a Say tar layer or if that's a manifest or a manifest list Um, it's just you've you've signed a thing and so to get as close to what justin was just saying It's like I signed the digest of the manifest list that I just pushed and that manifest list might have You know arm and you know AMD 64 and each of them have a nested list of objects and it's kind of a merkle tree that you just signed the the Digest of that top level manifest list And so one of these Endpoints right here. You're not querying them the signature per se is just to say Show me all the signatures that have been attached to that top level digest because it might be that um Just the build infrastructure itself You know, whatever build thing built it and pushed it signed it and then later they came through and they Also signed it with their production key that like this. This has been run through some ci system You might end up having more than one signature attached to it Um, this is just to get a list back. So, you know It's not querying per se per se a signature. It's just a way of discovering The signatures that are attached to some other digest that exists in the registry Okay. Yeah, I just wanted to make sure of that Would be like signatures for Yeah, because fetching the signature itself should be reusing the existing registry apis for like Okay, so here's You just gave me the digest of a signature for another object now on now I have the You know digest of the signature that I need to go fetch from the content address the store So part of the reason why I asked this and it's probably been considered but uh We would have to not trust that the service giving us a bunch of signatures would have to verify them ourselves anyway And this type of matching is probably also a vector of problems Sorry, could you repeat that? I missed a little bit of it. Well, so, uh, we're having detached signatures live somewhere on a database And uh, they are testing for another blob that lives somewhere else in the database Uh, and we are somewhat trusting the server service the signatures Uh for this blob with the server if it's uh And it has happened server can be hacked and it can be forced to do Like not the right thing Then uh, it can either service Signatures that are not relevant to this blob or it can serve as a subset of the signatures that are relevant to this blob For any sort of nefarious reason Uh, and that's because we're giving the server the power to say this is uh This are the signatures for this blob So you still have to fetch the if you got the list of List if you said who all has signed this blob And you get back a list of descriptors of other objects you'd have to get any signatures that you have to go fetch You still have to fetch them To validate that you trust the issuer and or the content that they signed which would be presumably to check some of That original object you've queried for right, so say that I say that to the u.s. government and uh Iranian government both signed a blob uh, and I query for it and I only trust the I only trusted when both of them agree on it a server can still serve me Perfectly nice and dandy signature over only one of the only one of the two signatures And that is a trust assumption that i'm giving to the server now to give me a correct series of signatures Uh over this blob So I can only get the signature from the iranian government or the u.s. government depending on how the Uh, this server wants to uh, misrepresent the facts But yeah, I do but is there any alternative to that? Because you can all I Never that's what the snatching role does in tough and that's why I think uh, we should all just not throw it away because We think that registers registers are different from I don't I am I would imagine if you need to attest that that you would download that blob and somehow, you know effectively carry it detached Or cache it and detached locally so that you could have the blobs and these Two disparate signatures that prove the story that you're saying that you've checked later But the registry is effectively done to that Well, the registry can selectively serve you confidence to trick you into thinking that somebody didn't sign for something Right, it's it's hard to prove the that A repository is omitting data that it should be providing you That's the real challenge and effectively what the snapshot role does is it makes the repository attest to What metadata it has to users and then it's not possible to go back and say You know, oh, I actually didn't have this because Before hand when the repository was, you know, honest and doing things it had already told you that it had this version so maybe I'm missing something. But are you saying are you proposing a particular logic piece here that this is missing that you know of or are you just saying that Yes, I just wanted to make sure that the I mean to me, this is a common A common pitfall when designing this type of endpoints and again I know that we're right in the time of like considering these things That's why I bring it up and I think having a snapshot capability Can prevent things like this from happening Is the is is the question whether we all client should always do snapshotting or it's a Thing that is a capability when I want to do an extra verification of trust All clients that don't want to be hacked should use it. Which should be all clients Who would sign the the snapshot? The repository registry whatever you want to call it Is the usual party that does it mirrors and others like that typically do not Is this solving for the the mirror case or the like I don't trust the registry server That i'm talking to case It's it's more the second case um because you you may say well, maybe they were good in the past but You know you want you what you ideally want is you don't want them to be able to go and tell you Hey, the latest version of this important piece of software is this old version of this thing that is 18 security patches behind or something you want them once Once people have sort of said like yeah, you know, this is um You know, I'm a testing that that this is the thing that you should use You shouldn't be able to like remove their signatures and remove their trust in things Which is what snapshot prevent I think that the case i'm trying to understand is as I think director said this is there's the untrusted source and Registry.com I won't pick on any particular one here That is a place where people get content from it But then I bring that and I bring that content into my environment because we all use upstream content And I do this enhanced checks that we're talking about Uh later on when i'm in my production environment, which is you know, we're still checking things But we don't necessarily check them as detailed how there's bound the levels of verification that are done at a certain point uh because it just becomes suboptimal on a point and I think that's part of this verification. So I'm trying to make sure we can achieve both Without having to do the deep verification checks every time The deep verification is very little like as you call it that it's very little additional work especially in your context of Having a server that's serving like some subset of content It's actually harder to do it across a large amount of content than a small amount So, I mean it's it's an extra You know couple signature That that even like in the automotive case where People are using this in little embedded devices The they're doing these checks and these verifications. So I'm I don't it's this isn't like a You know, you don't have to mine a like a a winning block in bitcoin or something like that to verify This happening. It's it's not It's certainly not on or especially if you're redistributing a subset So I guess what I'm trying to figure out here is um, so notary v1 it kind of turned the uh The key ownership model back to the publisher right and The problem with getting rid of that is the the registry itself isn't necessarily trusted um So it's a question here. How do you get those same guarantees? For example being able to push up as the publisher specific types of signatures that would be snapshots um that Would represent kind of all the All the uh All the signatures that could be expected for that blob Um, like trying to get all the same tough guarantees That's that's what I'm trying to figure out here I'm not as involved in the notary b2 on on that side of it, but that is something I'm curious about as well So I think this is the conversation that we kind of have in the threat modeling I'm sorry. I'm just looking at time people dropping off. So, um This is the exact conversation we want to have in the threat modeling and then the key management aspects To some extent the experience as well. So This is the ones where I guess I'm looking at you Justin Cormack. Yeah, I think also there's a there's a there's a bunch of questions around what um I mean, I think that we could um snapshot signatures for a particular um Piece of content but piece of across multiple pieces of content is more problematic. So we could have assigned um, I think we've very straightforward to have a signed um version of the um Keys sort of the signatures for a particular piece of content um Because that would just be another um Another signature on top of this. We have to work out the API thing. I still think that the Um a snapshot is rolled across multiple pieces of content is more difficult to implement um But we should um We should have that conversation um Because yeah, I think because there's as yeah, we started discussing this issue about what's What's actually feasible to do in a registry architecture in a way that's integrated versus what we actually need In the threat model, we've got there's a bunch of outstanding issues from our meeting on friday in terms of just writing down some of the Um scenarios which are going to work on but I think these conversations are very useful from that point of view in terms of um Putting together what we need in the registry versus what we what we need from the threat model point of view And how they relate to each other. So I think continuing to do that makes sense Well, I think you've got all the right people in your working group to continue these. It's just a matter of us finding time to spend more detail on this so um You you'll probably start seeing in uh join some of these also I just call him out at the at least the bottom of my list here and this list of people from He's in uh in an our azure group that's also interested in a lot of this work So trust trying to ramp him up to speed and the continue container aspects of it And uh, just looking forward to more of these conversations. So first, I'll just say thank you for everybody for joining with being flexible And I encourage everybody to continue to engage in the working groups One of the things that I noticed is that we need to figure out how to have better notetakers for people that aren't speaking Or to balance the person that's speaking and whatever um We have the recording and I just wanted to thanks to everyone and we'll continue to keep the notes going and Keep the work going We'll see you all next week Thank you