 Hey folks, we could start in a couple of minutes here. Just get people a chance to connect. That's a really nice background. It's not even the same background as you had in the last meeting I was in with them. I was joking. Like everybody's doing their virtual backgrounds, but what if we didn't have virtual backgrounds like well, I don't want this is my virtual this is my non virtual virtual background less boring more interesting than my bedroom background. All right, let's see here. The notes there. So on the agenda last week we had a couple of things around the target scenarios for the prototype and this reverse look up proposal that could that one of a chance review. If there's any other agenda items. Um, we were having a discussion about the tough document. I gave Marina some sizing figures for estimate. Okay, that kind of model from that proposal. I don't know. Yeah, I also had a conversation with Samuel car offline and went through a bunch of things. I don't think he said a chance to go back through but I think we resolved many of the issues in the document that he raised which I think were points of confusion. And also he helped me to understand one of the things that one thing that I have, I think been assuming that that the registry was doing differently that was actually extremely helpful which I'd like to talk about in this meeting. Because it's a big security issue. If, if we retain the same behavior I think. Okay, so what what is your topic that we want to add to the agenda can you just add it in there and we can. These meetings have gotten a little bit of just kind of going off and then we run out of time. So I just wanted to put some structure back to him because we originally had the hour slot so that we could have these open conversations we were supposed to these breakouts we haven't really been. Everybody's been super busy the world's been crazy so we haven't really been as good about doing the breakouts. So these ones. It may be five minutes to discuss this depending on how long the conversation goes after but I'd like to just raise the point so that we're all talking the same language and that hopefully that, you know, people that maybe didn't understand it worked that way which would have been me and I think at least Marina last week. And maybe we're the only ones but maybe there are others. Now would understand that, although Maria and I both I think understand it now, and that others that we can hopefully convey why this is a big problem although I don't think it's that hard to see, but we'll see. Okay. So did you want to go. Do you want to propose something that we talked about for like a time box amount or what do you want to do. What are you proposing here. I'd like to talk about that. So I am fine with however that works on the agenda. I would like to have more. I don't think Marina's put her numbers into the document yet we're still, she's run a bunch of sort of like simulations and things and her and I are so iterating on that. We'd hope to have it in but I would much rather give you things a week late than give them now and have them be wrong. So we're, we want to be sure they're right before we present them and right now we're not sure they're right. Oh, do you want to just do this next week do you want to, you know, get get time to do that and do it next week. Yeah, we'll leave that we'll leave the reporting about the what the numbers actually mean and how some of this. Like what what the overheads would be for the different design options and things like that later. I think next week would be a good time to put that on the agenda. I think also it's the one of the conversations was not only is the performance a concern which if you guys are doing some stuff to investigate that's great I think it's also a question of is not that it's not a important problem. Is it a important problem related to private registries or public registries like where is the problem really scoped. I think that would actually help with some context as well. That's actually I had a nice conversation. Once again with Sam about that and I think I got him to come around a little bit so even in a private registry. I assume you're not going to be using like TFTP to just download images because hey you know it's like the private registry and we kind of like block the network and we don't need any security or signing a verification will just like dump the data on the network, you wouldn't do that you'd use HTTPS because it's effectively free and sort of, you know who cares if you've got a, you know, small overhead or whatever else. So the thing is that what we think we're going to end up here with here is going to be something that's going to have small enough overhead that this will just be the way people use it and if not then we'll know. And then they'll there will be able, you know, people will be able to make a decision to say in this environment, you might want to not use TFTP and in this environment, you might want to use TFTP to use that analogy but I think a priori trying to say Oh, this thing we shouldn't, you know, it's messy we shouldn't have to use it here I think we should wait and see. Let's put it on the agenda and let's have the right conversation about it because I feel like we're just kind of pulling on threads without really knowing what we're talking about. So one of the things I wanted to talk about was this conversation that I was hoping Sam was going to be here. Let me just see. He's not going to be here. Okay. He won't make this to me. So make it next week. So one of the things that, sorry. I'm going to say, and so like comments he put in the dock and things a lot of those are misunderstandings that I think have been resolved, but that he and I talked about individually but we, you know, whatever you'd like to talk about is fine. Go ahead. Okay, so one of the things let me share my screen so it's the link that's in there and let me put it in the session as well chat session and try not to use the chat session as notes because they're not nobody's figure find them afterwards. So we're keeping the notes in the slack conversation in the hack and D rather. So let me share my screen here. We'll see how screen sharing is working for folks, but the link to it is in the doc. Sorry, in the zoom chat. So what I've got here, where's the one here. One of the things we've been talking about is how do we find the content and part of it is, you know, driven by some of the stuff that capos is, sorry, Justin, I'm going to get capos as opposed to Cormac just to separate. I hope that's okay. That's fine. Yeah, go ahead. Okay, so the next is the ability to find the most recent content. I know there's a larger scenario that you guys are focused on. But the reference to Sam was Sam had written one of the design docs design proposals not designed doc to be fair. And we've been evolving the conversation since then so I wanted to kind of take that same thing but it inverted a little bit to try to address some of the more recent conversations. And that's that we don't seem to be sticking very well with the Google Doc stuff so I did copy it into here. And this is the PR that's mentioned in the notes, what is it add signature verification look up. So the, the overview is basically, you know, we're trying to find a set of signatures and the verification objects that could go with it. So I'm kind of outlining that for the, I kind of did identify some of the goals and non goals and a workflow, which comes from scenario six in the signature requirements or the scenarios under the requirements repo. And basically one of them and this is something we were discussing early on is the artifact and its digest shall not change as the result of something being pushed. One of the things that I know we debated a little bit but I think the conversations we've been having around the registry operators that have been seeing this in AWS and Azure and in other places as well. As we see that customers actually follow and deploy based on a digest directly in some cases. We don't like them to be doing for what we call unique tag but because unique tags are not a physically enforced thing is a spec tag locking is a number of feature number the registry support but it's not really in the spec that some customers get nervous and actually want to deploy by digest. Developers are putting in their deployment artifacts in their cube deploy files or compose files, whatever the helm charts is a referencing referencing a specific image and tag, they can also reference a digest. The point is is that as the developer puts that in early on some of the proposals we were thinking about is we would update the index and then that digest and or tag would change and that would be the deploy animal would have to change as well. So this goes with the assumption that you cannot change that as part of breaking the developers workflow. A collection of verific object very collection of verification objects may also be associated with a single artifact. So I can have multiple signatures on an image. I could have a tough verification graph whatever we want to call that metadata tough metadata that's what the terms you guys been using that can also be associated with that. And then we want to be able to leverage the garbage collection because if you think of the things that we can change easily on registries and things that we can't. The garbage collection is one of the biggest monsters that we all implement because just because you can automate things customers do and we wind up with massive amounts of content and registries. And then the shared layered stuff we invest a lot to make sure that we can clean these things up and reduce the size over time. And then minimize the requirements to change the persistent object stores because that's another thing each of us implement on our unique infrastructure, you know AWS has their storage, we have ours, Google, so on and so forth on prem. So that's another expensive part of the stack that we would have to change some non goals just to put things out there is, you know work within the existing OCI distribution spec API, not that we're breaking it. But the idea is that we would add an API is possible like that. I don't think adding an API is relatively cheap. It's the garbage collection and storage that become expensive. So that's all this is outlining. I mean, sorry, good. Yeah, I mean, I think yeah, I mean adding API is expensive and and some in some sense and it does depend on people's data models and things how easy it is. So this is quite I mean this is a, this is not trivial API change. I think what the API is but my point was is that just a concept of adding an API to go find something implementing the API might be expensive depending on what the API is but if we just said, we want to add one of an API on something to the distribution spec to find these things because in this case what I'm suggesting is you look up things that reference an index and we'll see that in a second. And you can argue the point. I think we just, we need to, we need to have some discussion with implementers about how difficult they think this would be. And then compatibility with notary V1 we've said all along that V2 is not a compatibility bar. We want to leverage the work, the thoughts, the efforts, but it's not that anybody that the few people that have implemented notary V1 in registries to be fair, not notary all together but notary V1 in registries docker content trust. It's not expected that that work would be just transparently move forward. So then the workflow, like I said, we is from the scenario six is a dev team builds a container image. I've just put a tag on there for reference. They signed that image and which is the verification thing. And then some artifact scanning solution comes by and says, hey, I've looked at it today. And as of today, you know, here's what I assess is the vulnerabilities and it could push that assessment to that same tag. Again, it's referencing the same tag but we'll see in a second the tag and a digest for that actually don't change. It's another reference to it. The test environment goes through the test environment says yep, I've the image was built securely that's great it was scanned secure that's great but it actually does work in this environment. So I will stick another verification object on it and says this thing is now good to be deployed. Actually, I guess this is a unit test I think I did a second one here which might be overkill. Let me remember it's been a couple weeks. Yeah, so basically in the staging environment that's right. The staging environment is now verifying that the right signatures are on this thing before going to production. So, another verification object is pushed to it. Now as the thing moves into production, the production environment will only deploy things that have the right signatures on it is the design here. So it has to have the build. Now, I didn't get into the detail here it could also have the Ubuntu signature it might must be signed by Ubuntu as well. I think it's more focused on the company specific signatures because they could say that that signature itself says they will only sign things that come from vendors that they approve. But those are additive there's nothing exclusive of it the whole idea is you have multiple verification objects on it. Now, I'll ask for a bit of patience here for one we can talk about this but because I'll this is the part there's, this is the one that discusses what I kind of referred to as the downward reference. So we as we tend to dig in quickly I was and I'm debating whether I should even include this, because we start evaluating this what I was trying to do is provide this section as staging to why I was doing the reverse lookup model. So, before I do that any questions on the workflow or the goals and non goals. Just a point of pause. What for the requirement that any signature should not require a change in references to images. As for the limitation or how that in integrates with stuff that's to be resolved, but it's good to have the requirement recorded. I'm sorry I missed part of that. As for the specific implementation or how this would integrate with stuff that's to be decided. It's good to have the requirement recorded. Okay. So, good to have requirement to integrate I read that to the PR everybody. Okay. I think I don't want to derail so I'll just kind of say this like quickly but I think also you should list under non goals. Something about like protection from attack if the registry is compromised because you can be more specific about it and say that you know an attacker will be able to freely replay things or move tags or do other stuff like this in the design. But there's a big kind of I think someone reading this might get the impression that they're that they're getting certain security properties out especially if you're doing things like if you're transporting tough metadata the expectation is sort of well hey if I'm using tough then tough is both for compromise resilience of my infrastructure therefore I bet I'm getting compromise resilience of my infrastructure. Okay, I mean it's that's a good one like I hadn't explicitly thought about excluding that this was more of a base that we could add and that's a good conversation for us to have either if we have time today or in the agenda item for next week. Because yeah the the the that they're yeah if they're a security compromises for not including signatures in the content which then you know then I think that's you know a significant issue I mean it's it's clear that signatures via pointers versus signature by content are not equivalent in security sense. Fair so I think I understand what you're saying so let's get down to the some of the scheme on it that might help on some of this because I don't know if it solves it or doesn't honestly. I mean it. If, if the content does not refer to this, you know it does not include the content of the signature, then I wouldn't know. Well, no. So let me let me jump ahead because unless you've read it I don't think you've had a chance to read through this have you Justin. Sorry, Korma. Can I reference the person by first name. It's like using you, as long as the context is clear it's fine to say it. Okay. All right, so let me let me go forward because I definitely want to give time for debate but I wanted to at least get some context out. So in the where I refer to is the downward reference and is that you push a thing. And like a manifestor index so in this case it is an index, and I just showed the multiplatform start participant as well. And there's actually two of them, right there's one it's AMD 64 one ppc so just copy paste thing. And then this thing is pointing to a manifest and the manifest is what points to the layers right this is the standard OCI index and manifest model. This is kind of a downward reference right an index references a manifest, both of them can have signature content by the way, but there's this downward reference that's the way we think about with registries. So largely focusing on this downward reference that I can push a new index, and you know that could have my new signatures. The problem is is if you reference this index the index, even if the tag doesn't change the digest changes. So that's been the concern that is, even though I don't believe in doing digest based deployments as an opinion. That's the point it's an opinion and some people do and we need to be able to support that. And based on Sam's example. You would pull it as you know this web a to be to, and you would have, you know this manifest. And then you can add a signature with into an index. And this is the point here is that the index of course is changed from what it was originally or changing from a manifest to an index, you know, depending on when the signature was sent. And even if you do did the initial push, if I wanted to add an additional signature for my devs staging test environment scenarios. There's no way to do that without these changing in the quote downward reference. So let's look up model. And we can talk about that's the right name or whatever but the concept is a what if. So, here's the original thing that's pushed in, it's an OCI manifest it's web a to be to and the digest is whatever. But it's a specific digest is the point. And then I can push additional signatures and verification objects so I could say this thing is signed by test. And so so calm. And this thing has a digest and I could. Oh, I can sign another one that. Oh, see scan.com I forgot what I had here myself. So I've got two signatures that are verification options that were pushed later that can reference this. And that could further be exact. I'm going to skip ahead just for a moment and I'll come back is the thing that it references could also store additional information such an S bomb or tough metadata, as well as scan results as well. And it's not just the scan but some scan results. So you can see that there's this model here. So this is the part that I just wanted to get through and then obviously open for discussion. So in this case what we're doing is the index is versioned to include the config object the same thing we've got on manifest. So now I can see an OCI index is of type CNCF notary or whatever we decide to call it. The OCI that lets us find index objects artifacts that match this type associated with this. So this object didn't change because it didn't reference this but this does reference this one all of them. So now what happens is the manifest is the same as above. Nothing's changed here standard manifest it's an image config, and it's got layers. And the staging verification object now this is an index, excuse me, with a new config object. This is the stuff we've been talking about we haven't committed to this yet. But the media type of that is now a CNCF notary verification object. So this is how this index is known to be of its type, and it directly references this image so it's this got pushed referencing this. Sorry, am I going too fast for screen refresh. I forget I'm not presenting directly but I have this internet in the middle. A couple people across some ponds over there. It's been working quite well. Surprisingly. There's no trust is like surprisingly. So basically that's that's kind of the main thing I have a detail that again might be too much detail and it might distract from it that the idea is that the actual signature could be put in the config object so let's just skip that for now because I don't want to I don't want to do well too much. But then it was to stop the signature being garbage collected because it's not got a tag pointing at it. Ah, okay so that's been an image I saw some people talking about that. So let's talk more to Joey and Sam and others and how much I have Docker hub handset as well. What we've seen is customers actually because they reference by digest don't want things are to automatically garbage collected if there's no tag in a CR we support both you can do. It's not automatic garbage collection if the tag has been deleted, but we actually have a delineation that we won't. If you choose we don't delete anything by default obviously a customer can choose whether they want to delete digest that have no tag references, or they save them and actually deploying by tags. So the garbage collection the way we've always thought about garbage collection as the core that is not really a questionable thing is if I have two images that reference the same base layers that we don't delete the base layers until all manifests are deleted. We don't the connection between a tag and a digest is not assumed to automatically garbage collected. Okay. Docker hub is moving to the model where if it's not tagged is G seed. So I can certainly understand that for something like hub. But that's not something our customers would accept in private registries. Because they do deployments on digest only, because they feel that's the only secure way to do it now. If we're successful. And maybe if we were able to get tag locking as a spec conform thing that customers would accept it as a standard, then maybe we could get them to agree to unique tagging as a model. But even though I would like to get there we can't we couldn't do that today on ACR and keep our customers happy. This part is not an inherent unavailable part of the way it is of the design. If we are talking about a registry that can do reverse lookups that already requires the registry to have an index to have a new lookup API. So we could also add a new upload API that makes the association explicit. This is a signature for an image with that digest. And that also automatically allows the registry to model the dependencies for garbage collection purpose directly. And it would not have to have to specifically recognize every specialized conflict types to say, oh, this is a signature. So I need to mark the reverse dependency. Then that's a, well, because that's a really weird model because I mean the nice thing about it, apart from that is it's just, you know, effectively, you could point to the signature object if you want to but you don't have to in this model. You've got a choice you can point to either thing because you can retrieve the signature. Anyway, but it's just a normal object having a list of types of object that are not garbage collected normally. Because they are often points to in the other direction is really weird kind of conflict. Like how would you, well, it's definitely like the reverse, how would you add, how would you add the set of types for which they don't get garbage collected. I'm curious what you're saying about gargoyle signature type that is not recognized by the registry. The registry is just going to accept it as a valid OCI index, but it is not going to tell you by the way I'm going to garbage collect this and your signature will disappear. Justin is your concern that you can push these CNCF notary objects I'll just call them for simplicity because that's what the red text says. They're without a tag and it would automatically get garbage collected in Docker up. Yeah. Okay, so there's two parts one does nothing says you couldn't push it with a tag as well. Yeah, that completely you could push things with a tag. In this case I just don't say think it's useful because there really isn't the tag isn't anything useful on it. You could say like obviously there is some API changes here and I appreciate that you guys are go into a tag, what we call orphaned. What do we call them orphaned images or something. I forget if it's not tagged we call them orphans and I'm not sure if that's should be not called them because orphans aren't bad. The, I recently watched in the orphaned blacks series, which is got my head all twisted too much detail. So the, what you could do is in this API you could say that these things that don't have tags I have digest. If they reference this, you know something with a tag that's still valid, that you don't garbage collect them. So you could, I mean, obviously there is work to do in a registry support this concept. So it would be an additive thing that you would have to put in your exclusion is don't delete orphaned artifacts without that don't have tags. If they are referencing something that is valid. So it does fit into the garbage collection model but it is a change, especially if you guys are just starting to go down that path. I think there's something else in here. What was in the present con section. Yeah, I was so in the process of basically multiple signatures can be added. So that was, you know, one of the things there with is no content change to the thing you originally pushed so all your deployment artifacts, you know your compose files your helm charts your, you know, could deploy they change don't change whatsoever doesn't matter whether you reference them by tag or digest they stay perfectly the same. In fact, the scanners that are trying to do index information on it will also be able to cash that information. So that's the point there. The con is it is a new API, which we said wasn't really a con but it is, you know, I still want to call it out. The reverse look up does actually did reference it does have an impact on garbage collection because there is some new logic that does have to get added to it. We do tend to traverse down. But in this case there is an awareness that has to be done of all the things that are, you know, linked after the fact. This is actually going into what I think you're calling out Justin Cormack. Yeah, I there's this really weird inconsistency between what you're saying that people want to refer to something by hash because they don't trust it might change and yet they actively are also saying that they do want some things about it to change, like the signatures, but not other things about it. And so there's a weird characterization of what things about it exactly which they want to change or not change. And is this behavior actually is this kind of requirement actually reasonable. So what they're saying is they don't want the thing they're actually deploying to change they want to know that the image that's going to run. Do they want you know what else about it are they okay with a K of the S bomb changes are they okay if like, but there's a lot of things that we might end up pointing like this if we go through this model and I'm, I'm not 100% sure that actually everyone will be happy with all of these things changing what if like a signature was removed. And so it was no longer signed by my by the author because the author invalidated the signature say by deleting it because the key had been compromised or something say like with that still be okay if the signature has been removed for it to an unchanged object that looks still kind of looks okay I mean I mean, I think there's there's just a bunch of questions that immediately comes to mind about what what changes are okay and what or not and and by making this decision you're you're effectively making a decision about that for people about content whereas previously the definition of content hash was everything about the thing and obviously nothing can change. Everything can change and like that. We're kind of asking for intermediate stage. And I think it requires some thoughts about whether that's can be concrete, you know, is there a good definition of that all users can understand about something in between those two things. So it's fair. And let's tease those apart because there's a couple of them that I can think of three parts of it so one, the thing that they like about the digest or an immutable tag, either one is the actual content they're going to deploy is locked. It will not change right we're not adding new content to the Docker image we're not adding or just not changing anything about the layers or the image itself everything about that is sealed. So what I'm getting is additional metadata and I'm air quoting because that is a loaded term. They get additional information about that image that is added after. So that's the, it didn't change its additive is the thing that ties the two together, because I can get these new signatures. I think that part still meets what the requirements of the people that want either tag digest deployment or an immutable tag deployment. The interesting one that you're mentioning around the delete is one. So if there was a verification object that was there then somehow it gets deleted. But you know, what do you deal with that and that's kind of I just think about as the registry security models is that we support. Yeah, just because you can push doesn't mean you can delete is something that the private registries tend to support. Yeah, but that's where as Justin capital is pointed out. Actually, I don't have to send you Justin stuff. Justin pointed out that that is something that a tough security model is protecting you against and therefore is considered security relevant. So it's interesting you think sorry somebody else. The idea that the images identified by the manifest digest is a complete unit and it either nothing changes about it if the digest doesn't change. It's good to have that property but we are anytime we go outside of the realm of one image we are losing that anyway. Like if you have a Kubernetes deployment for that image and you change the security configuration of the product definition or you change to volume mounts, you can break a fully tested image with exactly the same manifest digest the same way. There is always some higher level metadata that changes the behavior of the image and keeping the manifest digest exactly the same can't help you with that. So, in that sense, I don't think this would be introducing anything new. And if we are in the model of a signature as an approval or something that can be edited or removed. Then that's necessarily a metadata that is pointing at a digest at an image digest, but conceptually should not be part of it. It's uncomfortable conceptually messy but I don't think we can just avoid it or pretend that it never happens. So one of the things that you mentioned something interesting Justin that the tough metadata, you know, tracks that things got, you know, deleted or not of what the original state was. So the thing that I was trying to get as a minimum viable product, you know, MVP, I don't know if that's a term we use anymore but anyway the minimum base to build upon is that if we can just get things that are signed that and including additive collections that one of the collections could be the tough metadata. So, if you ask for something you can get the tough metadata for that thing. And whether this and as often it says that this tough metadata couldn't reference this directly that well, I guess depends on the for tough metadata to reference another thing it has to be an index so maybe tough we want to be an index instead of a manifest. The tough data will need to be pointed out by the tag effectively because that's that is you have to be able to discover it. Well, I mean, technically it will because as long as the object references this, I can add tough metadata and not change the digest of the thing that was originally pushed. For tough you need separate storage apis for the root keys and timestamp keys and so on. Those can just be well known to access not so much of an issue. Yeah, but it isn't going to be associated with a single image like this anyway. I think tough can deliver the lookup by digest just by looking up the digest in the in the tough sign that data. So it's not inconsistent but it doesn't require this mechanism in particular. I've got a drop, but I need to think about this a bit more. Yeah, sorry, I forgot that we were back well we were half an hour I didn't realize we went that fast so that's what slacks for. Yeah, that's notes. There is some notes in the Google Doc originally that Joey hasn't had a chance to copy over yet but let me know and we'll pick up next week. Thanks folks. Thanks.