 Hey folks, Brandon who's been pretty active trying to keep up with you, dude Didn't mean to mess up your nice long weekend there. Oh, well, no, you didn't I ignore it for the weekend to be fair I opened it alive looked long enough to make sure that I was being responsive for the things for this morning But I did have an early morning. So I was trying to catch up Most all that came out of me just looking at a lot of the stuff thinking back to the notary v1 and tough and I've Had a client that's been to one notary v1 lately. So it's fresh on the mind Well, it's good to make sure we're keeping the things that were good and addressing the things that were gaps Yep Yeah, it's from an agenda. Were you able to do you have anything for us today? I wanted to go over some of the feedback on the Tough document. I don't know if that was discussed in the last meeting I'm not sure which tough document there Brandon's got an agenda and for tough implementation discussions. Is that overlap? partially I think Looking back at the doc that Marina had shared. I think it's in the I noticed later that it's in the Git right now and I provided feedback on the original Google doc, but I was gonna push some to the GitHub as well Yeah, I don't think we covered it in the last meeting Sorry, Maria. Oh, sorry. I'm just gonna mention that. I think I just some of the feedback in the translation So hopefully at least some of that shows up there It does I think there's a Larger question I had around tying in Well, I can actually add this to the agenda rather than kind of hijacking the call here me and Steve Thank you. It was just a question to make sure we have the right things on there to try to time box stuff More so for myself To give enough of what the progress we make it on the persistence and make sure there's room for the other topics Because I brand didn't highlight it basically three it can almost say it's one agenda item with three specific things Hopefully we can crank through those Yeah, well, there's a lot there My main goal of this was just to throw them out there and say hey they exist This is where to find them and drive some of that conversation back to get in case other people are following along with this But also to handle any interactive if we need to discuss it here Yeah, for the three that I saw just for others that I think I maybe I'm oversimplified but it boils down to the Digest and tag combinations and how do we account for those? Would that be fair? That was definitely one of the big ones in there But I had three different discussions out there and one of them was like, you know, what is the root key cover? And I think we've been chatting about this back and forth of is that like a registry concept or as an organization concept? Is it a repo concept? You know, we've been looking at a lot of different ways and so that might be more just generic discussion And the other one that wasn't about tags so much was about the time stamping because I think that's something that From registry maintainers a lot of us. I don't think we would be comfortable putting a time stamp service combining that on a registry necessarily so Does it make sense to push that into the registry or does it make sense to have as an external service? Make sense. Okay. All right, so while I do this, let me try to give an update On where out in a couple of things. I was looking to see if Omar made it because he had some questions also on progress At the high level I would say this and I'll give some context on what the first agenda item as we make forward progress on the prototype and I'll share my screen here There it is Okay, so if we look, I thought I had it up here. Yeah, Sonic series. So we look at the flow. We've been trying to make sure we can validate Actually, hold on. Finish hitting the share button. There we go. Everybody see my screen. Yeah, just get Alright, I'm trying to get this visuals. All right. Anyway, so we have, you know, the promotion of content through registries with multiple signatures. This implies a couple of things the ability to define a signature, which we've got a prototype of that today. And there's a bunch of good discussions around that the ability to push discover and find and pull push discover, which is fine and pull a signature in a way that is not let's Hack on another very specific thing to a registry that we kind of did with the sidecar military be one that can we do it in a generic way. So that we can put other artifacts in there that we want to sign as well. You have a signature obviously is one thing, but there's other things like S bombs and wasms and Artifacts that we don't even know people thinking about at this point. So there's some stuff there. And then there's the key management aspect of it as well. Those are the three critical parts key management being an interesting conversation how critical people are going to do that. And then there's another part of great. I've got the signature, but what can I do with it. And that's the policy manager portions of it. And I, and I would suggest and we'll talk more about this in Brandon's example that maybe some of that is part of a policy management that somebody can apply to things. So we'll play with that idea. But the main piece here is as we've been discussing with these prototype, you know this this model that we've been going after and I talked model meeting the, the gaudy model that before we go build this Sagrada familiar with very detailed design plans let's create a couple of models to have something Everybody of the various trades can look at the subject matter expertise can look at and going, I see where my thing fits in there. So we're finally after the holidays and everything going on, getting some folks from open gatekeeper to help do a prototype around this, so that we can see how would I apply something to something like Kubernetes. Because remember this is not our notary scenarios is not dependent or singularly a focused on Kubernetes Kubernetes just one of the many places we run containers. It just happens to be an easy way for us to prototype some of the policy management. So those are the kind of big things I the big rocks I think of that and Omar was kind of bringing up some of this and how do we land this. If we can get the policy management validated with the envy to client that we've got today. And I can get an update which I'll show in a second here on the progress and how do I persist and discover artifacts and registry. I think we're at a point where we can actually say here is something that registry operators can implement. So that we can start testing this with actual customers. It's great to do PowerPoints and pictures but until we start playing with this in a, in an interactive fashion it's really hard to know where the rough edges are. So the questions, you know, we can just focus on some of the brand and straight questions of, we can flush out some of those because there's, there's with many of these there's no clear black and white answer it's, is it good enough is always the challenge. So, with that, any questions before I jump to the first agenda and does that make sense for progress and kind of how we've been thinking about it. Silence kind of means your mic doesn't work or we're okay. So, in the concept of that persistence so one of the things that we've been struggling with and this is one of the overlaps between notary and OCI meetings that we have on Wednesday for the OCA distributions back in OCI artifacts and so forth. The concept here is, you know, as covered here is we need to move content between registries registries are run by various companies, including competitive companies, right we all recognize the value of collaborating because we all have content that needs to move between these. So, the, we can be comfortable with the kite with the way we put these in registries so there's a couple elements in here and for those that try to read up over the weekend I did have a bunch of notes in there that was noise so I did clean up things this morning so this is a new push. Once I get a little bit cleaner I'll actually push it as a PR to the artifacts, the OCI artifacts repo so it's not just buried in mine. There's a couple of things that are design points that I've tried to put in here and the main one is content will move between within in between registries. And kind of the idea here is that you might with had a single registry promote the same content across different repos from dev staging and production, or you might put it with across registries might have a dev registry and a prod registry because of vNets or what other reasons you might have different registries. So you can put your content to some public registry from a dev registry to another one so products for web and networks and you can see they've got various artifacts there. And the idea is that there's something the whole way we're trying to reference these things from signing to even copying content. One of the things I'm trying to tease out is what I started to cover here is basically is it time to change how we reference container images and other things. If you think of any other package manager. The domain of where you're getting the npm is not specified in the npm reference you make. That's a separate configuration there's defaults and you can alter those. So I'm trying to start teasing that concept in here and so we have the basis base in place for that. The concepts are is deferred resolution. I will reference something and it may may not be in the registry that I'm using. I may pull a chart from one registry and I might pull the images from another registry. So that's those are valid scenarios. The layers or the blobs that make up a given artifact have to be with the artifact right like a family might be dispersed across different locations. But the legs need to be attached to the body kind of thing right what makes up an artifact needs to be complete. So that's the, the general design points that I was trying to reference here. I'm messing with some examples of this where and I'm playing with the idea this is not a spec yet it's just to show an example. Let's say there was an OCR registry CLIs that you can actually copy content from one registry to another within or across. And you might set the default registry to be acrockets, for instance, you might set the namespace to be dev, because I don't want to have to specify namespaces in things because in dev. I want to reference the web and networks image and it comes from the dev namespace but I want to reference that same image without changing anything. And actually comes from a different namespace or a different registry. So there's some ideas in here. I don't know if anybody wants to riff on this right now as opposed to let me just make more progress on where this picture of things that we come back to if we want. Yeah, I would say be extremely careful about such things. I heard we've had such features and we mostly regret them. First of all, once you have a default registry people will ask to have a search list and that gives you ambiguity and gets into impossible situations. Second, if you make the registry optional, know that namespaces within registries can look like host names so you have syntactic ambiguities in there. I would like dots in the namespace. Yeah. Yeah. Oh, sorry. Second is that we basically require everything to start with a host name. So we have a worldwide namespace of image names. And then we have a separate mapping configuration that allows you to map it to physical registries. So you can have an image that is called registry dot redhead dot IO and have a configuration that says oh this is on my local mirror over here. Can we talk more about that because I like to get more of that input. And then the other thing I'd say which I think is somewhat related is, you want to make sure it's very clear what priority order, the different repositories or registries are in because that way if, for example, you know a package is deleted from where you actually want to get it from you don't want to download somebody else's version of that from somewhere else because that might not be what you expect. Exactly. Those two parts one. And as because an attacker can just take the first make the first register inaccessible. Yeah, all of this are like that's the part of the next piece I want to drill into his details the ordering obviously is configurable the client can configure it the belief here is one things will be in different locations depending on what environment but to the ideas the signature is still valid. The point is is that, if to your point Marina, if I was in, you know fell through to a secondary registry. And, but if I can fake I'm not sure why it's a bad thing let's just say it's for some reason it's a bad thing. If that secondary registry got hacked. The signature in theory would not be valid. And if the signature is not valid, then my validation would catch that so that would be one of the validation scenarios. Well I think the, my microphone working. I, it is quiet. Okay, I think I think the problem is more so it makes it impossible to unpublish things. How is that so, because well if you have multiple registries. People can own the namespaces on different registries and copy things between them and preserve the signatures but the very top of the tree can unpublish something and someone that's copying that can not honor that on publishing it. I think that's the problem with sort of a search path, even if the signatures are valid. That's how you determine if the signatures are valid. If you're, if you're getting the keys to the signatures and their paths, somehow originally from that source then I don't think they would still be invalid if they would be booked. I mean, we should get some public registries now, that's a foregone conclusion. So it's just a matter of, and sometimes, you know you want the content to be copied into a place that you control because you, you trust it and if somebody does unpublish it, there's a couple of node examples of node yarn or left pad, where I actually want to make sure I don't get the upstream stuff. So far this is, this is the challenge we're trying to talk with too many concepts in one so what I, there's a great conversation there, it actually wasn't the main point so I'll come, I want to come back to that one. The main thing I wanted to show was this directionality that we've been supporting adding to registries to be able to support adding artifacts, like a signature to an existing artifact in a registry, as well as referencing the actual artifacts in it. So what I'm getting, let me walk through a couple of examples and see if this makes sense. If you look at an OCI image, the manifest that you submit has direct pointers to the layers, right, so I can look at a manifest and see it points to layers. And that's the thing that makes this up and of course it points to the blob as well but it's a directional, it's a downward directional mercury kind of model where I can see from the top down everything that's there. From a notary signature when I submit the signature of itself, of course there's a manifest that says I am a signature, and there's the blob that makes up that signature. And that's very much the same structure as an image. The problem is that I want to be able to take the WordPress image that I have here with its tag and digest, and I want to add a signature to it, and not have to change the digest or tag of the WordPress image right that's, that's a fundamental design point of the NV2, notary v2 what they were doing. So what we've been doing is saying that's fine, you can push a signature to a registry and it references the WordPress image. There is a, if you notice the directionality of the pointers is changing. This was an interesting thing that we had to think about from registries because registries have always been a top down looking indexing thing. And now we're saying, I want to ask the registry hey what are the signatures for this WordPress image. And the registry can go back and look at hey what other things were pushed in that referenced the chart. Sorry, the WordPress image so there's a bidirectionality of those references that we've added. You know that meets the requirement of the digest and tag don't change for WordPress. And I can add any collection of signatures over time so I can go to do web and networks adds a signature Docker hub says I test to this content. And it says after they've imported it into their environment they test it, and it meets their requirements. So now when they do their deployment, they can put a rule a policy that says they would only allow stuff that signed by active rockets. And over here somewhere in the environment is where they say well only pulling content that's signed by Docker hub or the web and networks. And that's the model we've been going through, you know as a design point, but it's been a little challenging is to make sure we can do this in a scalable way. And when we start looking at other artifact types, it becomes a little interesting and how we can do this. So I've been playing a little bit with some of the ones that I know of, I, there's some other ones that I want to look more at for like single well singularity I sort of know but they're fairly simplistic and, and the kind of references they make. So here's some other so here's the ones that I do know of. So imagine the one of the challenges we have today with WordPress, sorry with helm charts is a chart has references to images. But they're all buried inside the helm chart from a registry perspective it has no knowledge of that unless it was to parse the helm chart itself. I'm trying to come up with a pattern that doesn't need to know the intricacies of every artifact type. And the analogy I make is, you know, as we say files to our disk, you know there's file system API's. It knows, you know how to link in, you know, multiple file segments especially as they get fragmented across the disk. There's no word office word API for saving files right there's a file system API there's no, you know, go language file API is for managing files on desk there's, you know, files happens to have a dot go extension or whatever and the file system handles that. So I can copy something from one folder in a, in my local storage to another one or USB stick or whatever and it knows how to find all those elements and copies and with them. We want to be able to have that same generic API's and registries. So the scenarios that we want to be able to support is, you know, the downward links which registries are to support right the layers, I've just called them blobs now. The reverse lookup I can enhance an artifact in a registry with additional information. The, the third is I want to be able to take something and put it in a registry says, hey I'm referencing this other thing. It may or may not be in the registry right now. And that's okay. Like if you look at pi pi, you know the ways this package references like it could be evaluated later. If the WordPress chart can be submitted with a manifest a new OCI artifact manifest that says hey I'm referencing this thing by its digest and by a tag. And now I could say I could pull it out of the registry is just the home chart. And now on the client, I could say, Hey, do I have everything I need. Well I already pulled the MySQL and the WordPress image from Docker hub and I pulled the home chart from home hub, whatever. Now I can look locally and say yep, I've got everything valid to put into, you know, to do my deployment. I could also say I want to copy it from into the registry as well. So there's a couple of interesting scenarios. And I talked about the copy and delete below. But I can take this even one step further. A CNAB, which is this idea that I can, you know, stand up and run a set of installations of other things. And everything being embedded in this, you know, invocation image that's referred to, I can have a CNAB that references a Helm CLI, and it also externally references a chart, which is just what I've done up here. It has references to the images. So I started to get an interesting graph of content that defines the things I'm absolutely dependent on the layers or blobs that make up the artifact, the things that I'm enhancing a notary signature enhances any other artifact type. And the third being loose references to other collections that may or may not be in the same registry. And so that's kind of the piece that I've got here. So that's one of the supported scenarios, discovery, copying, deletion, you know, the visualization, which you can kind of see here. And I'll walk through this visualizations and I'll pause for some feedback. The reason that we're doing this, these three collections to reference that is if I push just loose artifacts, the way we've got an artifacts be one in the registry today. And I, I've been a little bit more, you know, abusive of what we've done. We don't show layers separately when we do a registry listing right, we know that these things are part of this image. In this case, then why would a signature show up separately or an S bomb or a CNAB or helm. The idea is we want to be able to have that same kind of information to be able to make those enhancements better. So today what I can, well not today, sorry. What I'm going to do is say, yeah, this is the image. It's got a signature, it may even have an S bomb. And over time we're hoping we can have a generic metadata API over that artifact as well. From an discovery perspective, I really this is the thing I want to see I want to see I've got an image a CNAB a helm, and I've started playing with little expand those here. If I really want to dig in, then I can say, well, what is in the CNAB. So I can click the little expand. Oh, I say, oh, the CNAB actually references the helm COI and helm chart. And oh, I can expand the helm chart to see it references, you know, these container images. So you start to see we've got, you know, the core pieces to be able to represent these things. So if you squint at this you'll notice the only thing that knows that this is a helm and image and other signatures the only reason I need that is for the for the icons. We've got a separate way to map media types to icons, all the other representations here you took the icons away. That could be done with this new manifest that we're proposing here. I'll probably copy semantic structure to cover that as well so I'll probably stop there for discussion. So this is Marco speaking. Steve I have one question regarding discovery and storage and so let's say you have this generic community signature, which was used to sign the artifact first. I put it into my own registry and I had my own signature there. Is there any need to synchronize these signatures between the registries and what about storage. So I can imagine if you are depending on a lot of community registries out there in the public. And you're copying a lot of images. So that more and more signatures there is more and more need to synchronize these as well. So, did you have any thoughts on how, how this will be resolved. Let me make sure I understand the questions so I'll come over to our example here. So, the networks is a software vendor, they sign their content for each build, they're going to sign that particular build right because we signed digest and tags. So it's, there is one signature per artifact that they published, they then publish at the Docker hub, and then Docker hub would sign again that same digest and tag that they would put so at this point. And it is located on Docker hub because somebody opposed from Docker hub might have a network rule that says I trust Docker hub but I don't know who these people are so the content has to be here. Likewise, when I go into acrockets acrockets is adding a third signature to that same net monitor software image. So now there's three signatures signatures are pretty small that probably is bigger the manifest itself is probably bigger than the actual content of the signature but there is three. Over time, when web at networks is making new builds they're creating new signatures but it's for that new piece of software there. There is probably reasonably numbered amounts of signatures for any given artifact. They're more related to the workflow. So there's the two points is we definitely want to have the ability to have the content move with it, because acrockets operates in a private network here they don't want any access to anything outside of their environment so the content has to be there. They don't have a dependency on it from is the internet available, and a security wise they don't want to go out. Yeah, so for my question. Earlier there was also the question what of the upstream artifact is deleted. In those cases, we personally also want to have the private registry so at least our builds do not fail, because the thing disappeared. But what about signatures let's say if web at networks revokes a signature because they figured out that was done by a person, which we shouldn't have put the signature. What about synchronizing those kind of changes should that also mean the docker hub signatures should invalidate or is that just something where we need to notify docker hub that signature was revoked and then still have them decide if they want to revoke their own signature. Yeah, great point. So the content, the content may disappear which is not necessarily a good thing and I want to be protected from that but I also want to protect it if it was discovered that the solar winds package that was put up there just to play with the latest, you know, changing discussion point that I can revoke that. So that falls into our key revocation scenarios and our key management that I am casting over to Niaz, who's been cheerleading that with with the end is, you know, trying to support him as well so yes we definitely want to be able to do that. There's two sides of it one in an air gap, which is also a network isolated environment. I want to make sure that I'm not dependent on external keys, but I also absolutely want to make sure I can bring the key revocations into my environment to say that key is no longer valid and just trust anything associated with that. So you can also revoke a signature. So revoke the signature is also possible. So that means the existing key is still there, but I'm revoking the signature. Yeah, I think that's an interesting one. I, I think we have to think further and I hadn't put a ton of thought about it I certainly can understand why I might want to revoke a signature. I think we should spend more time thinking about that because I can see that also being abused very quickly and I'm not sure if that's the kind of noise we when I say noise, the amount of traffic that would have to go around to try to synchronize all that I think that's I don't know. Are we really going to think about what invalidates the signature because is it just a vulnerability because there's always going to be vulnerabilities. So it could also be that you just release a new release with a new signature and don't revoke the old one. So, more immutable. In terms of like if you're resigning artifacts it boils down to a question of where do you want the controls to be. So if for example you're taking a dependency on Wabbit networks then you could essentially just go look at their signature directly you don't necessarily need to look at the Docker or the ACME signatures. In that scenario you're obviously dependent on like Wabbit networks if they if they make a call that they're evoking a signature then you're essentially saying that you know I'm going to go into the revocation path. If as ACME rockets let's say you decide that hey you know I want to do my own verification and rely on that instead. You could, you know with the second signature you could have your your validation look at two things one look to verify an ACME rocket signature and also look to verify a Wabbit network signature. In which case if either one is revoked you could fail or the third scenario where you look at just the ACME rocket signature in which case whoever is responsible for signing as ACME rockets is also responsible for checking for revocation. So it really depends on where you're delegating your, your trust for revocation and for signing to. Also, if you think revocation is an essential part of your workflow, consider whether you want signatures or just an online service. Like if you want was this approved or is this free of known vulnerabilities that's a bulletin that should be delivered over to ELS that's not a signature. There's the middle ground of those moving tags like colon latest or colon one which means the latest version of one X. And in that case, some revocation might make sense. It's difficult to square this way to disconnect the scenarios. Yeah. The signatures just have a short expiration date and for disconnected you would add an extra tolerance for recently expired signatures. Yeah, I put a good top I put a note in there for us to have a good discussion because I, there's definitely some interesting points that we should dig into there. I think from the, from a point of what I was trying to just cover here is, as long as we can persist these things, you get the right document here. By persisting these things in this model, then we have what I'm trying to do is we have the model in the registry to do these kind of behavioral things and I in fact I noticed some comments in the chat session around local host and other things like those are all great things that we need to be concerned with. Those are things that you can add, in theory, part of the policy management that says well don't allow local host it must come from particular domain it must come from particular registry. From this portion of the content so we can get to the next phase of testing the end to end. I'm trying to figure out. So we represent these things as in the schema and I'll just pop to the last, there's some examples in here of the schemas here's like a standard image with layers. What you'll see here is now I've replaced layers with blobs because people there's a lot of debate around that because we're introducing a new schema of an OCR artifact manifest, which here the signature has a dependency on the MySQL image. Now that reverse pointer is saying I as a signature, signature stored here is pointing to the MySQL image there's the tag, and then here is the digest. But you'll notice, I don't care whether the MySQL image is in the dev branch dev repo that's a prod repo, or whatever, or even which registry that's in. These are lazy deferred resolutions, but the signature signs the digest of this. So, wherever it comes from that, you know should be able to be valid. And you can I'm scrolling through this quickly because I do want to get on to the other ones and I only get to set up for PR for people to read this and provide just tear it apart with feedback. And then there's just again some more examples here so the helm chart is stored its blobs and and Josh I'm just I'm exploding here just for the point of clarity not trying to say this is how it actually works. But the helm chart this part definitely doesn't work because you don't have any way to do it today. Here is where the helm chart could declare it depends on these two images. I'm going to copy a helm chart from one registry to another. I have the ability to actually pull this content from one registry to another. So that's part of the nice is the that we have there. And then lastly the CNM reference for those. So, I do have two thoughts on this. The first is you kind of started showing some recursive stuff in there and do you envision the registry server itself doing that recursive search or do you see that being a client side thing. It's a little bit of both I've been playing with the both sides of it. That's why I kind of was messing with an OCI reg API to kind of visualize what that experience would be. And that's why you're also seeing some of this config stuff come in at the top here for how it resolves. Here we go, Howard resolve things like well where should I look for the word press chart. Well, look in the charts path. So be careful not to have like the 24,000 depth resolution thing. There'd be, you know, we have to think about where is Sasha had a term for a deterministic resolution, something about closure. You know how do you get the closure to all of those. So definitely need some more thoughts and people that have been through these pain points in the past like is it an arbitrary and 24 for is a configurable depth of resolution. Don't know yet. Yeah, my, my personal meetings on this one not being a person that's run the massive registry server would be to have that client side just because of those issues I'm just imagining someone linking is something that goes recursive a couple layers down and then basically a denial service against your registry when they say query page, not her 99 now query page 1000 assuming we're paginating this response just because it has so many details. You're going to have to regenerate that result every single time is going to be painful. So that that was the first thought. That is one of the benefits by having registry operators. Part of this is, yeah, my junior team will let me never let this be shipped if we don't have a way to avoid a denial of service attack so well the balance of usability and practicality. Yeah, the other thought when we come specifically to talking about signatures is that it'll be nice with that query response actually included the content and not just the pointer to it. I'm thinking, you know, you get back 10 signatures. These are all going to be tiny. It'll be nice. We just actually got the 10 signatures in there. Maybe you have the reference. Yeah, here's a request. No, that's not the one I want. No, let me ask that. No, that's not the one I want. Yeah, there's, um, there's a healthy balance that we've been working through. In fact, the initial prototype, you could argue was overly optimized. What I'm trying to, and that's why I'm experimenting with some other artifact types is I want to make sure that we didn't recreate no to review one in the sense that great, we solved one problem but it actually didn't really solve enough for the problem. So I'm trying to find that balance of how many API calls you have to make without it knowing about a specific artifact type. So yeah, that's that's part of the next piece of this once I get this thing figured out. Great. Great point. Was there a third one. Nope, just those two. Okay. So we're a little bit over what I would help to cover in this timeframe. I definitely want to leave room for the other agenda items. I will get this PR, hopefully this week to the artifacts repo but I'm, it'll be PR. I'll probably do it in our notary. Well, we've been staging everything here. Yeah, there is one here. So I'll probably PR it here. So we have all of the things we need around notary staged in one place and when we got all the iteration between them should be figured out then I'll send the PR is upstream to where they, you know, if you notice we have distribution for changes that we're looking at artifacts or as the distribution spec and be to so on and so forth. So, just as a reminder, we're, we're staging everything here, making sure we're happy across everything and then once we as a group are happy with how all these things interact, then we'll send the PR is back to their originating repos. So with that, I want to hand off to Brendan, Brendan, sorry. Okay. Sorry. And this I'm imagining we might end up kind of going off in discussions on this is this spiffy spire discussion a quick one that we want to get in before we get derailed. I mean, I'm not like an expert in it or anything. I don't know if anyone else wants to take this one. But I've done some, you know, some reading on it. So I put the agenda item there. The first one was what if we could utilize it to not have to do the key management. So my, my thought was initially what if we have this spire server to define our route certificate. The protocol from what I understood so far is rotating keys basically automatically every now and then. And if you would register a workload, let's say a signal service within spire, then. Yeah, you can use the at the, at the station. Yeah, to verify that this this signal service is trusted by the by the network. And maybe that's already good enough to, to use that as a signing solution. The thing I'm doubting though is, yeah, how that works with with signatures and keys that are continuously rotating and things like that. So it was a wall thought which I got when I saw this spiffy spire presentation last week. I was thinking what if we could use this not have to do key management at all. I think that they really have thought a lot about that like initial source of trust problem and how to kind of solve trust on first use. And I feel like it might be good for us to take advantage of that as like how to initially distribute it. But like you mentioned I think for the more day to day key management keys that are used and changed more often. I think it's nice to have something else which is why I feel like it's best suited for that kind of, you know, just just distribution of root keys and kind of initial root of trust that you that you can then build the rest of your system off of. But I think that's a really great way to do that. I love the connection for for a bit so I didn't catch it completely, but I'll just probably watch back the recording of this. This is something we might want to move to get a GitHub discussion going on pros and cons. I was just saying we can move it to the news has the key management working group if we can put it in that you know not that we shouldn't use a get up discussions that by any means but there's a group people kind of wrestling that for ball would be good to think about. I don't think we should take too much time over here but just want to drop the ball here and see what are the thoughts and maybe my idea is just crazy. Maybe it's also something which which is feasible. I just don't know myself. Yeah, I think a bunch of us are guessing it looks interesting that's why I want to if we can put it in the key management group then we have the key management experts could say yeah this thing is just doesn't have anything to do with this or yeah this actually would be great if we did x, y and z. So this Wednesday I have a meeting with call Kennedy. He had been doing some spiffy spiral integration within total so hopefully I can get some more insights from him. And then I keep you guys posted. Full disclosure he's my coworker. Okay, so we want to jump in to a couple of scenarios and stuff going on here. Go ahead and share my yeah that'd be great. Can I ask one thing though. What's really bad about keeping notes. Can I ask folks that instead of put things in the chat session to please put it in the note the note section of the hack empty for those that joined later I'll repaste it in the chat session. So I put the first one I want to chat about real quick here was adding to the notary requirements I threw a poor request out there, and I think we might have gotten a little derailed on this one the original focus of scenario 10 was to kind of just that clients might reference things different ways and so clients may actually query and say I want to pull this image with a shot or I might want to pull it with a tag or something like that and I think we all know that happens I'm just defining it here and the reason was kind of thinking through a lot of the tough requirements. I wanted to make sure that tough didn't go down one path of saying everything is based off of tags when there are users out there they're basing their decisions off of just shots. And so I want to make sure we covered all the different use cases also covering that sometimes we might be pulling multi platform images different kinds of scenarios like that and that maybe one day we want to consider that notary could also handle other stuff like home charts or other things like that so this isn't getting into the discussion yet of do we want to actually sign the tag it's just saying that users may want to do a query based off of an image they're running that is defined with the tag and then the client might need to do some dereferencing or something based on that any discussion on number 10 thumbs up thumbs down I would say throw it over in the GitHub side if it looks good or bad to anybody there. The other part of this one I was starting with just 11 and then I realized I need to break off and do 10 before then so how to renumber things. But I wanted to get into the delay verification because I think this gets into different scenarios that some of us might face where we have a bill to see I pipeline something like that that signs an artifact and then we go along and do more builds and more signing of other artifacts and then we finally get to the process of actually deploying that first artifact be signed way back and so I just wanted to specify there might be some space in there and that gets back to if we're thinking through some of the tough requirements where we build a whole list of targets, and we refresh the target for what version two might be or something like that, that we don't want to automatically unsign something that was previously signed, and then therefore no longer trust it because that could get us that combined especially if you've already deployed this out in production. Now you have a node failure and the workload fails over to another node. You don't want to implicitly unsign something that you previously trusted. I think that makes more sense to have it as something that's more explicit where someone says I specifically don't want to trust this image anymore because something went wrong with it. So I just wanted to throw that one out there is something I think makes sense for us and I'm sure the tough people are looking at going on or that gives us headaches but I just want to at least see if that's something that we agree as from a group standpoint or not. So Brandon are you basically touching the thing that just because I got a newer version of something doesn't mean the old one is bad. Correct. And I think it depends on what we're signing if we're signing tags this becomes much more visible. If you're signing the shot digest like you're doing Steve I think it's something that just, it comes out of the box it's just the way it is. Yeah I think it also comes into the concept of stable and unique tags like if you're deploying something should you be generating a unique tag every time you do a deployment as opposed to base images or to some extent software that I do deploy that I do reference, you know a stable tag. So if I'm referencing the node nine image in my front statement, I want to be able to get updates to know nine without having to say no nine point one point six point three point four point whatever. To throw a scenario out there. If you said that okay we're going to do our build and our build is version nine, and I want to deploy that and trust it. But I want to actually be very specific on you might be references part of your build trip you may say okay I'm going to trust this shot with this massive digest so I know that is unique and I go through all my steps and I know that it's not changing anywhere. But if you're signing system says okay let me go out and sign all these images now and do a time stamp against it and keep the snapshots and everything think through the tough model. And I only look at all the stuff that's currently tagged, then we may lose that we had something that was previously signed, because it's no longer referenced by tag. Yeah, I think that this is, I think this is mostly a matter of making sure that the yeah the snapshot includes old metadata as well. And it's mostly about making sure that you know the metadata that's there is, is current so that if you if you do decide for some reason to remove something. It's no longer there but we should definitely make sure that anything that's not removed sticks around until otherwise. And that gets to the implication is just saying that implicitly we shouldn't implicitly unsigned something I'm thinking it should be something that's more explicit where you actually say I specifically don't want to trust this digest anymore. Right, I think it kind of comes into maybe even the signature replication scenarios kind of thing. And maybe you have a policy and again might come into the policy management that you know the older version of my sequel is still perfectly valid. Yeah, has vulnerabilities most software does to those vulnerabilities matter to me, and if I see a my sequel eight should that point to my sequel 8.1, you know as a policy, I mean, those are interesting problems. Yep. So hopefully this was just throwing these easy ones out I think a lot of discussion that the group will want to have is more around some of the things I didn't turn specifically into this but left as an issue so I would say, thumbs up, thumbs down this issue because you think it deserves or doesn't deserve to be approved and I don't know what your policy is on approving some of these PR Steve but I just wanted to get that out there. You know, it's, I'm not the gatekeeper per se everybody should, you know, give a thumbs up thumbs down voting. I'm not sure where I've looked at the maintainers but we'll, you know, there's a, we're not just arbitrary committing things we want people to be comfortable with them. Yeah. The one that I think is probably going to get the most discussion is whether we should be signing tags in addition to the digest. So this isn't saying that we don't want to sign the digest. I think we've got a very clear understanding from the group that there are people that are deploying things by digest and we want to verify images they're referenced by digest. But the other question is, do we also want to sign tags and what does that look like. And the example I threw out here was just saying, you know, maybe you've already deployed something that is version one dot five. And should that be something you can check and make sure hey, was this really one dot five that I'm pulling or did the registry have some man in the middle that jumped in there and just returned a zero dot nine dash debug image that we don't want to trust we don't want to run that production. But something man in the middle that gave us that and we checked the digest to see if it was signed. And the digest says yeah it was signed by that person so you can trust it. Is that a case that we want to solve with this. And then the question of what do we want to sound with that tag comes into what if you have multiple repositories that same person is signing all kinds of stuff that digest come back and say it's verified by the same acne rockets in this case. If you only sign the one dot five you could have a different one dot five for a different image that gets thrown in there. Maybe they've got a different debug repo with other more exposed images so through that out there just kind of think through the logic of that. Do we want that and does it make sense and I've seen a few people have said no we probably don't want to do this and so you know if there are people that kind of agree with me that that makes sense as a concern I would say jump on this issue because I'm seeing a lot of people saying it's not that big of a deal. I mean, I think I such a tree way kind of jump in here we definitely are currently the current prototype when I say definitely current prototype has the digest and the tag, the repo tag signature included. I think actually has the whole registry which we're wrestling with that that's part of what I'm teasing in this latest spec. The thing that I was trying to play with is again a policy decision that you may want to get new versions of the tag. And as long as they're signed by web networks in this case, that will be valid. I think your man in the middle attack comes interesting is, since the digest is always the thing that the digest is always part of the signature. So I always know very specifically the, the absolute thing that I'm trying to pull is assigned by the original entity. So your man in the middle attack is unless they've got the actual signing key, which of course is always, you know, a concern which is the key then gets revoked, because we don't know what they're doing with it. Then if I were to update the MySQL 8 image with, you know, the bad version of it because I hacked it, that even though the tag was redirected to something else, because the digest is what gets signed that MySQL knew the bad version of MySQL 8 image will not be signed by MySQL. So it would be considered invalid. And I think, looking at this, I think we're always going to know that this was signed hopefully by someone we trust, but it's just it might not be the same image that we thought we were getting. And so, verification of this tag. This is something called the Resolution. So that's what I was looking for. The, the verification of that tag resolution is one of the problems that Notary V1 actually solves. And I think I would hesitate to, to basically remove features in V2 by making it so that we don't verify that resolution here. And I think it might require a couple extra signatures, especially if, you know, different, the tag is used a lot. But I think hopefully people aren't using tags so frequently that it would be a, a burden to re-sign those and push them as they change. At the same time they're signing the digests. Yeah, my vision was that you would probably have whatever the current tag is signed. And then maybe if there was an old version 1.5, you know, longer is the 1.5 tags. I don't care as much as that's no longer sitting out there because you just pull the newer version of the image. So from that perspective, this model is very closely to what we have with Notary V1. I think that the allergy some of us have to this idea is that signing the tags ends up being really close to signing the names and that's one of the things that's been a big usability challenge with Notary V1. So what I'm interested in is are there other ways that we can achieve the same goal, basically a rollback protection without having to resort to signing the tags. I think that we could look at things that are different from that might have different trade-offs instead. Well, it's not rollback protection. It's if I want version 1.5, I need to get 1.5, not 1.4 and not 1.6. Yeah, and the challenge is if you reference it as one, you know, the way some of these stable tags are done is one currently pointing at 1.4 or 1.5 or 1.6. And what does one represent is often the question there, because that, in theory, one rolls forward, and it's the stable tag for whether it's 1.4 or 1.5 or 1.6. One would always be the most stable release of that versioning scheme. When I think about it, when I'm doing deployments, one of the things we see is people want to reference tags because they're human readable, you know, they give you the break-glass scenario so you can redeploy something. But some people really, really don't trust anything and they just want to deploy by digests and too bad, read this really long key when it gets to be 512, read that longer one. And I think those are actually supported with the way we've been going about this, right? You can, because we sign both at deployment time, you can decide whether I want to reference it by tag or digest, because the challenges we definitely don't want to sign just tags. I think we get the flexibility from the naming that Sam's talking about, and we're referencing here. I think part of the naming, and again, part of what's what I'm trying to pull out of this other artifact spec scenario, is the name just the last element and the registry and path, isn't anything to do with the signature. I'm signing this thing and if I want to promote it from a dev to a prod repo, it's still the same entity, still the same sign, still the same name. If I want to put it across different registries, whether it be WAPNN, Docker, or Acme, or even across my own environment, that shouldn't matter. The thing we've been wrestling with is, do we consider the name, even the repo name and tag to be like the sign thing? And I'm still struggling on that one. So at this point, I'm thinking of proposing it. That's just a client CLI experience that you can decide when you go to validate the thing if the repo tag is different than what's in the signature, because the signature has both. Then you can say, I don't trust this thing, because it's not what originally was. I think there also could be some advantage to actually having separate signatures for the digest in the tags, where you have the digest that sign that says, okay, this is the digest that we made this digest, here it is. And then the tags are, okay, so this is this release and we're attaching it to that. And having those kind of these separate processes, because the tags might change and then you can update those signatures, whereas the digest obviously will never change. And so those signatures won't ever need to have anything different in them. Yeah, I definitely see it as being two different signatures, one for the digest and one for the tag. My thought coming in here is does it make sense when you start to mirror these, you know, a lot of our concern is as we start mirroring these images across different places. Does it make sense to make a new tag signature on that mirror that would be signed by whoever's running that registry whoever's making that mirror. And then you decide if you want to trust that signature. You can just tell the client to accept images with a different name that can be client side policy. In most cases, the administrator of the mirror and administrator of the clients accessing that mirror are going to be very close. I think mirrors there's an interesting conversation are we talking mirrors generically are we talking really a mirror implementation, because if it's more of a, if it's a mirror that I don't know how you add content to a mirror, as opposed to, unless it's Snapchat where you can add things to pictures already but is opposed to the gated import stuff that we've been talking about is I selectively choose content and bring it into an environment, and then I can do whatever I want to it because it's really it's a copy. Because those definitely make sense for the additional signature scenarios and that's part of our, you know, web and network to act me rocket scenarios. Yep. So real quick being sensitive on time. Just wanted to point out there are a few others out here that throughout another one of the issues was do we want to limit the trust of a sign key and this kind of gets that. If you have someone like Microsoft saying all the Microsoft images, should you trust it if they suddenly started signing the Google images. Not to pick on you Steve but just kind of thrown out there. Yeah, who do we want to have trusting each image do we want to start to have a client side validation that I trust this key for this namespace. What does a namespace mean in that case if content moves between, because like when we, when we publish stuff on MCR we fully expect customers to copy that content into their registry. So the idea that when they run it, the MCR domain will not be anything to do with the image they pull because it would come from their GCR registry for your example. I think similar tags that would be a client side configuration if they could say, this is going to be trusted on whatever mirror that I copy it to or wherever I move it to. One thing I think we'd covered as a requirement in the trust or configuration on the client devices for you to be able to scope what the key is being used to trust whether it's a repository registry or tag. And it could go either way. Yeah. So that one there were the three other tough discussions as well that throughout there but I think we've kind of hit our time limits so I don't want to take up time here but I guess people to be aware of this. Feel free to go out there and chat on those. Yeah, I just noticed we actually are over so when we wrap up and first of all to say thanks for everybody joining for the new year. We finally got this thing kicked off in the new year we're obviously have a lot of progress going on a lot of great projects and we'll kick off next week and with the next round but I love the new discussion stuff that we have in GitHub so it's definitely utilize those so we don't have to squeeze everything until one hour conversation. Thanks folks. We'll see you next week. Thanks. Bye. Thanks. Goodbye.