 Hey, folks. Hey, Steve. Hey, Sturt. Hey, Trishank. Can somebody speak to make sure I've got audio working? Sure, you can, Steve. How are you? There we go. Okay. Nothing like a British accent to come in the morning. Is it truly British or did I offend? No, no, it really is, yeah. Okay. The North, the South, the East, the West, it's like, you got to say it just right, or it's like, no, that's not who I am. No, I'm talking to that. Hack doc, if I can get my mouse working here. There we go. All right, we'll give folks a couple of minutes here. Cool, we have Steve, Milo Slav and Samuel here. We've been probably the three most active people in writing comments in the document. So that's really good to see because, as I'm sure you all have seen, we've just gone and tried to resolve a bunch of those. And I also want to thank others. I see a lot of other folks that are on this call is also having helped out and mentioned things here and there that we took a look at. But I think the three of you have at different points mentioned things that required a lot of back and forth and required, you know, maybe some text additions that in some ways were worth talking about. Hey, Justin Cormack's here as well. Good to see you. Hello. Yes, I'm here. So I've had a brief review of your document, but I really need to read it more closely. Okay. There is one, I know we're still kind of waiting for people to show up. So I'm, you know, before kind of like launching in with a lot of detail, I will say that there is one change that's been on the surface moderately substantive, but isn't particularly, which is that we were discussing a mechanism to provide a property we needed. And when we went back to look at that mechanism, we decided that we just wanted a simpler mechanism that for the purposes of notary to do exactly the same thing. So we used to have references to something called tap five and internally when talking with some of the folks at PI PI and also some of the other use cases we were thinking about originally for this. We realized that those use cases were not as pressing or important and that really the case that was needed for notary v2 and PI PI together was really the only case that mattered. And so as a result, we were able to do a much simpler design. And I'm really glad to see Evan joining because I think he was involved with the original tap five proposal that we ended up. At least the current thinking is that we're going to do something simpler, which is the tap 13 that I've linked to in the document and posted in the in the notary v2 channel and in the tough channel and a bunch of other places. Okay, we're now five after we have Marina and Trishank and myself. I guess should we maybe start a little bit. Yeah, I think the question is what exactly are we starting with here. That's been the biggest question of how do we work through this in a structured way so we understand what it is we're working on. Right. So we tried to look through the scenarios and understand how a design would fit in to that and I I'll start to talk through some of those cases, some of the problematic scenarios that people have brought up in a moment. I do have one more like very minor kind of like housekeeping thing I want to mention, which is that in a few places in the text you'll notice there's a little bit of orange text. This is text that has been added in lieu of something that where if you read it last week you might have had a different impression. In one situation, there was some text that had been inadvertently copied from, I think the tough spec that wasn't accurate, because it was like not expecting top tap five or now tap 13 to work in the way it was and so was misleading and I think that if I remember right, Milo slot very correctly flagged this as being wrong. And so we stepped in and and I fixed that text but I only fixed it. I don't know, maybe three hours ago or something. So I wanted to to call that out. Another case the the options we discussed about for keeping registry metadata private had for some reason missed one of our design options, like the way that it worked so I went and added text in for this. And as a last point here, I would say that the appendices you should not view as complete or finished the appendices that have metadata and things I've called out at least one place where it's not quite right. And we will need to depending on what we end up with from picking design option one or design option to or supporting both a little part of the appendix will also appendix a or and appendix B will need to change to support those. So those should not be taken as as correct depending on which design options chosen. Okay, so now I think as Steve has suggested we should talk a little bit about scenarios here. There were a few scenarios that so I, all right, I don't believe we need to go through the sort of like obvious everything works this that's basically the same way as it did in notary B one sort of scenarios. I think the real questions that people have had have evolved around a couple of different cases. One of which has been this idea of or this way of copying metadata from one repository to another specifically targets metadata from one repository to another. And this works just fine in notary B two. In fact, in a tough repo you can literally just take all the targets metadata out of tough repo a and stick it in tough repo B with a sign it with a new root key sign it with new snapshot and timestamp. And that all works perfectly fine with one per proviso here that was pointed out I forget someone pointed it out, but the, the thing that was that was question about is when the targets metadata points to a specific location for an image like if you imagine that the thing the target refers to isn't isn't just like the name of a piece of content, but is a URL to a piece of content. So it's not like a file like it's not like a it's like a fully qualified domain name rather than like a relative path to data that's going to be retrieved. Then there are potentially issues with the way that you would retrieve images there if you move to an offline case. So in other words, if everywhere in the targets metadata it starts referring to go to this, you know, this URL, but when you copy things to a private repo, because now you have something airgapped, you can't go to those URLs, there has to be a way to address this. And there's a couple of ways to address this. I think this is a problem both and it's a problem today notary v one. And the reason why I think that this typically isn't a problem for other tough deployments is I don't believe other tough deployments use URLs when they refer to things. I think fully qualified domain name style URLs. I think everything is always done in a way that's that's like absolute to data storage that's on the system that you're going in like retrieving from or alternatively there's the client is aware of like a known sort of prefix or path to go to to retrieve the actual content that it is requesting and the repository is effectively just the metadata store over some sort of like backed storage. So with all of that in mind, I wanted to kind of open the floor for others to chime in and give their thoughts on how important it is for targets metadata to be able to refer to images that aren't stored that are stored like elsewhere and it's some distant URL because like the more typical way would be be to just mirror like mirrors the wrong word but to copy all the content over and and by the way and also just for reference if anybody's trying to like like look at this and wants to read something other than what I'm you know the words coming out of my mouth there's a section here copying images to a new perhaps private registry or mirror that's about two thirds or maybe halfway through the document that also describes this process. So one of the things that continue to come up and is the fact that the an artifact in a registry is referenced fully qualified. So and part of this is a little bit of interesting to talk about word legacy here but it's again we're in container year so it legacy is like six months ago. The if we look at you know if I look at when tough is designed and you know focuses on package management and so forth you're right the packages that are referenced are decoupled from the package manager at references and we see this more and more as more of these package managers are being stood up for private package management with npm and maybe and so forth. So what it allows is it allows me to reference something as a deployment and then as a separate configuration say by the way I want you to get that package from this other package manager or registry so you can interchange those words. Because that is the root of the problem that we've been avoiding. It comes up in a couple different ways when we talk about everything from layers to signatures to even things like helm charts and so forth because the fully qualified names are baked in and what I'm just wondering you know it's it's totally bubbling up because the problem I'm trying to totally bubble it up because we seem to be continuing to try to design without understanding that problem or or just ignoring it or assuming it as a problem and it's the elephant in the room I don't know if we can work around. So I kind of wonder if that's a question we should start to think more about. There are ways to address this. I mean one one way to think about it is that you know you have your fully qualified name for something but view that as as a string for the you know that references the thing you're trying to retrieve and plus you have a secure hash that you know of inside of top. And so there's there's sort of no reason why you can't go and effectively make a request like give me this thing that has this name which you can once again treat it as a string that has a secure hash. And to be honest when you get something that matches that secure hash. I think you care a lot less about where it actually came from and how you got it because it's it's no longer. I mean the real reason why you care about the fully qualified domain name is two reasons. So one in the normal case one is is that you need to know where to actually go to get the damn thing. And two is usually you're relying on HTTPS to go and give you the thing that in a way that's trustworthy. But you don't need the second property here because you already have a secure way of knowing what the secure hash of the thing that you're retrieving is you already have a like a signature over over the hash of the thing that you're going to get. So you don't need to worry about have I you know am I being given the wrong thing. Then it just becomes a problem of given this name how do I know where to get it. And so for the offline mirror case this concept where you know you could go and someone could say hey to get these target names I have these target names. Like so if you're trying to request this this string come here and I will give it to you and it's you know because you know the size you know the hash you know all the other things you would need to know for it. So you're you're sort of. I think it's use cases that makes a lot more sense than in the mirror case. I will want to want to be careful of the word mirror but the ability to copy content to another location is the thing as I think there's a couple aspects of where what you're referring to is important but I think it's applied in a different way in that where a content actually came from actually isn't where content came from from a registry perspective isn't as interesting as the entity that it represents it. And we've talked about earlier like even the MCR images when they were initially pushed their push to some back back of the house ACR that we as part of our workflow publish it to an MCR domain and then customers move it into theirs and IOT continue to bury it down. What at the end of the day they don't really care it came from MCR they care that it came from Microsoft or they came from Ubuntu or it came from you know Adobe whoever the and then you have the other aspect of it that inside of a deployed environment. The environments locked down their network to be the to the sources they trust. So the registry winds up being in that private vNet, whether it be on prem with IOT or in a cloud. So the idea that they can reference the thing by unnamed disconnected from the route registry it pulls it from is something we don't deal with today in registries customers have hacks that they kind of make up their own design around it. But and then have a third entity that says by the way regardless of where you got this thing from here's a collection of the people that have attested to this content. So it could be Microsoft. It could be my within my Contoso company. It's the test environment signed off to this content that came from Microsoft has also been certified for my environment. So I just being able to do those kind of operations is the things that I think we have to kind of address. Yeah, I mean so all the things you describe here can be done like are done basically assuming once again you have this you know this whatever way this is name mapping you want to have work it works just fine here. The the you know one thing I would say though is just to be clear so in general the way you need this to work for security is you need to figure out for my name space. What are the who like what are the you know should this thing be in my name space based on who trust in what I said is supposed to happen. Right and if there are other signatures and things that aren't related. Maybe someone at some point cares about this or you need a way to look this up which is all very possible to do, but it's it's sort of not as prevalent for you verifying a file if random party X who you've never seen and never heard of signs and image so we're not kind of, you know, we wouldn't be providing all the signatures on an image basis because this what we're proposing here doesn't store signatures on a per image basis. It's, you know the way that you and other parties, you know, basically the you know, like the you or your organization sets up your name space for tags and everything like this, and then you only use the metadata the targets metadata and things, you know, for those things that are in your name space that you're going to be using. That's the part that I'm not sure I fully understand that I understand the analogy because I think one of the things I continue to hear is and we've been trying to solve this impedance mismatch of conversations at times so it's like I'm trying to stand more about tough, you're trying to stand more about how the registry stuff works and the the there seems to be an over a big focus on package managers which are all public and they're slowly learning to they're not slowly learn they're slowly becoming more and more private copies. I haven't seen more and more investments being made and how companies can have private npm's mavens and so forth. But the the general pattern people use are these are public and I kind of have a mirror concept in container registries because they are we've shifted from package managers that are part of a development resource that are more upstream and it's not as critical that they're as reliable as trusted and so forth. As opposed to production where they absolutely must be critical I will not have them being anywhere else that's but that's public and if I'm still referencing it's because somebody hasn't realized it yet. So in the container registry space, it is much more focused on signing that individual artifact that says I trusted for my environment now I also want to know it came from a trusted entity. So there's two aspects of it. So I think it is important for us by the image and tag. There's a questions of, you know, I think the thing that that has been the stumbling block here is is that you could separately upload every single signature you make for every single image in some way, which is basically what what you're proposing is you sign some piece of metadata that you upload that you sign us. Or you could sign, you could send a piece of metadata says these are the signatures that I've made. And what we describe early benefit me get though that's a partner right. So you said that we thought about it the way the big example the big motivating example we have very early in the introduction that talks about all the issues with tag references being out of date and this leading to potential issues when people refer to latest. So that paragraph talks through several of the problems several of the security problems you have, if you break the things you signed down individually. And I think maybe we should do because I noticed that Samuel, in fact has already mentioned a comment on one of the options in the keeping registry metadata private and I know other people probably have other things. I think, you know, I think we would be happy to talk more with you about that case and for others that are interested in and have that same, that same kind of question, and we could talk about it but we've also maybe should, you know, actually, like talk through a broader set of the questions that people have. So Samuel I think your comment, you're indicating that option one is something that like Amazon wouldn't use, which I think makes a lot of sense. This is this option one under keeping registry metadata private is probably not the, not the preferred way because it leaks information through the metadata. Do you do you feel that something that no one would want to have or do you feel that I mean you feel it's like something we should just remove here from an option. Yeah, I think I think it's totally infeasible for anyone unless they have totally public data, which is as far as I don't know of any registries that are full is that a public only. Okay, so what I'm going to do is I'm just going to remove this paragraph and I'll renumber the options if there's no objections. I'm hearing no objections. Okay. Okay. Okay. We're through here for there were a couple of other comments here. So, Evan, you had a like a comment that I didn't really understand that I wonder if we want to discuss here. I think you yeah you are on the call. Okay, great. So, you would discuss through, you said I'm curious if, if we've thought, I think thought through options for having multiple sources. No, I actually can't parse that sentence. Okay, so sorry he has zoom issues. See here, currently listen only okay. Basically, you have you have the scenario where your third party image scanner you have your own root of trust. And this is really where I get lost here, a little bit here so you could be generating tough metadata. But are you in this case are you trying to say you want like some other type of metadata like in total metadata or like S bomb or something like this that is used as part of the trust that isn't tough, or are you I'm, or maybe I'm just confused in general like do you mean a root file or something like this or like what's the, I'm just slightly confused about the point you're making. Cool. Yeah, typing. But even if that helps, what I hear is that customers want or imagine they want to make a copy of a public image signed by the entity who created the image. And in their local mirror to add an extra signature to indicate that this image is approved for for private use for a within the company. I'm not 100% sold on whether this is a valid use of signatures because you have difficulties with revocation but that's what I hear that customers imagine using. Okay, yeah, we definitely support that use case. That's, that's not difficult to do. Maybe we need to make this much clearer in the document. I'm not sure whether I mentioned the original signature should be preserved as well. But yeah, I imagine that's not to be able to do. Yeah, that's not difficult. Like the in in tough you would never get rid of the original signature. Like, it's not like there's a place where you have to put a signature and you have to pick Alice or Bob, it's Alice has a metadata file where she signs things, and Bob has a metadata file where he signs things. And so in what you would do in the tough cases you would have it where you would, you would go and you would say well both Alice and Bob have to sign it. Right. And that's, like I said, that's, that's like trivial to do it's like baked into tough. So I think we can make this clearer. Let me think about maybe that belongs. This is a client side configuration. It's not appropriate at this native to the repository. Like if you are under a tough delegations and multi signatures, you would have the mirror user target signatures delegation or something like that that requires two signatures, but that allows the person operating the mirror to kick out the upstream signature require not any time. I, it's interesting. I'm not sure if mirror, it'd be interesting for us to think about if mirror is something we really want to do, like we, we, we tend to move, we recommend away from mirrors, because of the stability issues, but if, if you were didn't use the word mirror and said it would be that Ubuntu image that signed by Ubuntu and still valid, but the new version of it, I have not verified works in my environment, my being my company. So the Contoso company. So inside Contoso I want to know that that updated image that I copy to my private registry, or my even might be my corporate private registry. I make sure that that Ubuntu image passes all the certifications inside the Contoso company. If it does, I put a Contoso, you know, corporate signature on it that says passes, you know, whatever. And then because that happens to be the Ubuntu image, you know, the assumption is somebody's going to put something on it. The Java team inside of our company build the Java runtime, they put on it, they sign it. So now it's got a Ubuntu signature, it's got a Contoso security signature. It's got a Java runtime signature. And then the individual teams that write apps that deploy on top of that Java runtime say they get I've tested this in this environment. So that by the time it gets to the AKS or the Kubernetes environment, there's a configuration error that says in this Kubernetes environment, it must be signed by Ubuntu, Contoso corp, and the web analytics team that's deploying that particular image. Right. And you can do all of that in several ways with with tough. You can literally if you want to put things in separate tough repos, you can say like this tough repo and that tough repo the metadata there has to agree. Or you can go in and have your targets metadata inside of a single tough repo where you keep that all that metadata together. So there's, there's multiple different ways to do it depending on exactly what you want to do. It's, it's very, very, very much supported. And there are options. Yeah, so we can make any of any of those cases clear we've been trying to address some of the other issues and the TAP 13 stuff has been more pressing for the PI PI community so we've been rushing to get that out. But what since we're over time maybe what we should do now is if people can, you know, continue to add comments or things like that, we'll put some mention of, for instance, the use case that the scenario that that Steve Lasker posted in the zoom group chat, and others like that and try to make those clear in the document to make sure we haven't missed anything. And, you know, I think that, you know, I've been encouraged though from what I've seen from people's comments here there's been questions about like well how do we do this or how do we do that, but there really hasn't been any kind of like out of left field requirement that is something that we feel like we couldn't handle. So we'll work on that and one other thing we'll try to do in the meantime is, we can do some measurements and try to get some estimations about efficiency and things like this for the different the two different design options that we talk about for we can handle snapshot later in the document. And we may be asking folks here if they can give us some order of magnitude ish numbers or some numbers that aren't wildly off so that we can help us, you know, kind of produce something that says yes you know this would actually be better or that would be better. I'll step back into what, again, it's just which problem it's trying to solve because I think there's that's part of it like we would. Many of us have millions upon millions of content in the registry is that you know have no correlation with each other. So, the idea that I have millions of signatures isn't a big deal because it's there associated with their things but to have some kind of uber graph across all of them. It's not clear how how that would scale or even why it's needed. So it would be inside of a Merkle tree with design option to, which would mean that what problems itself. I'm not worried about the design I'm sure we can design very efficient things it's just why. Yeah, so, so we'll, I think, you know, we should probably let people that need to go go. We'll talk for a few minutes but this is exactly the thing in the third paragraph of the introduction the big example there that we we've talked through. So, yeah, but I don't know, does anybody have anything else we should discuss or should have discussed for this meeting. Thanks I just wanted to be cognizant of everybody's time and not just try to, you know, not not hold people over while we we dig into things unless that's something everybody wants to hear. Okay, so I guess what we should do now is you want to go through the scenario in the third paragraph again. I. Okay, I mean if you want to what, what, yeah, if you want to bubble it up for what scenario it is that'd be great. Okay, so the problem here is is that you have a party that's going and what they want to do is they're trying to have a tag that is something latest that they're referring to. Marina in the text here I think did a very nice job. I think there was feedback from others that you probably want to have like a one point latest or something to refer to whatever the latest version is that isn't a 2.0. So, and in the example here, we talk about how you can have all sorts of problems if people are able to go and cause your one point latest tag to refer to old versions that were at one point, a one point latest, but are no longer and maybe, you know, they have been replaced by version, you know, effectively, the reason why is because there was a security release for some other issue. Okay, so keep in mind here that that we're not saying that if you say I want 1.3.1 that that 1.3.1 as a tag is going to point somewhere different because there probably is only ever one 1.3.1. Right. We're really worried about the case one point latest here. Okay, and there is definitely a potential security problem for people that if you're able to make one point latest point to anything that one point latest ever pointed to. If you compromise a repo because you can pick and choose an outdated version of something that is an outdated thing that the tag pointed to that that may have vulnerabilities or have other big problems. So is the scenario there that the registry got hacked and it's indexing got hacked or is there some other man in the middle attack that you're referring to because I'm just trying to bucket the two because if a registry gets hacked there are so many ways that I can jerk around with stuff that I I'm just trying to put a context to this. Not necessarily strictly hack but maybe a CDN good hack or something like that. Yeah, you're it's your to the to the extent that the less is close to good enough but there are cases when it makes a different. There's actually a really nice conversation that Milo Slav and Trishank had in the document that really captures a lot of this to like the longest comment the comment that's like the length of the document at least the way I view it is is really awesome and we should probably move to an FAQ and I think answers at least parts of this but really anywhere where someone has an opportunity to tamper with it whether it's them. There's a party that somehow like copying some targets metadata over or CDN or yes maybe they have compromised the registry. Those are all different scenarios where you want to have this type of protection. Another version of that long thread is that if you have snapshots per repository, then you only detect rollbacks when you fetch from the same repository again. And if there is a snapshot per the whole registry server, then every time you pull any image from the from the registry. The snapshot and then you can't rollback to in any repository on that registry. So if you are pulling from registry once a month, but you are pulling something from that server every day, you get a one day rollback protection instead of month one month rollback protection. That's all there is to it. I think if you don't know how frequent you are playing, you don't know how much improvement you get. Yeah, I'm trying to wrap my head around that problem. And if it's the answer is it solves that problem and the other things maybe but I like the hacking the CDN had really thought about. It's an interesting one. Hacking a registry is this the only thing that solves it because if once you once you've depending on how you've hacked the registry because that's part of the idea is that if you have the keys. If you don't have the keys, then hacking can only do so much. But to your point, if I don't have the keys, but I can retarget a tag reference to an old valid image because it's still the same key still signed by Ubuntu. And we're not invalidating Ubuntu because we're saying no Ubuntu is still good at that image is still good it happens to have some vulnerability in it but that's the nature of all content. So I could kind of get that, but I guess I'm still trying to understand the how you go. Well, maybe how you're going about this problem. So I can only I can hack a registry but the only but I can't hack the keys so I can change a tag reference. So the thing the part of this design what it does is the deposit the registry itself, unlike the vanilla tough spec. The registry itself doesn't indicate trust intact it doesn't indicate tags. But what happens here this whole tap 13 thing is talking about how the user ends up being the, the, what's effectively the, the namespaces root of trust, while the information they get all comes from the repo that's where this whole tap 13 is all about. So it means that even if an attacker goes and breaks into your registry. All they can do is the same sorts of things they could do with. Like rollback for, you know, in milislaw's example, like a day worth of information. If you update every day, you have, you know, you have a very small window where they could say, Oh, you know, actually three hours ago somebody pushed a bug fix, like a new version of this tag but you won't you'll never see it, because the attacker did it but they can't roll you back more than 24 hours ago if you update every 24 hours. They can't make a tag do that. So it's, you know, and they can't make a tag points and where they didn't point before that wasn't going to you. So it really limits the damage that can be caused. So if I look at package managers and registries we all have the concept of versioning, right like package managers have a versioning thing and they, you know, they have master majors and minors and so forth. So we have to be versioning without structure. And there's lots of debates around this including the latest thing but the point is effectively a tag is a version to the artifact in that particular repo. So, and there are places where we should be updating tags for base images, and we should always have unique tags for deployments. So the idea then is I have this window I don't want it to be rolled back to what was valid it's still valid but it's an old implication. I guess I kind of buy that. I guess but it helped me with why I need a whole registry copy of something to for metadata rather than the metadata associated with that tag. If it's associated with the tag then you don't have the, like the timeliness information you don't know when things change. If you knew the only thing I will ever install is exactly this one thing here. If my entire namespace will only ever be this thing and I'll never need to think of anything else. And I only, and it's only what this person tells me are these five people tell me about this thing that's all I will ever ever ever care about. Then you could literally put that metadata from those five people into a single tougher repository and keep that like have the snapshot contain that information and keep that up to date and be protected from rollback because what you'd need to know is every time you talk to the repo you need to know what's the latest version of metadata from these like five people that I that I that I need to get the latest version of what they tell me tags are. You're not just worried about the reference to repo another artifact in that same repo, like the Ubuntu versions from yesterday you're worried about the Debbie somehow that they're pointed at the Debian image and a whole note that Debbie and Ubuntu are too, too difficult to compare. If I had a different Ubuntu image in a different repo. How would could I point my latest tag at that is that where you're trying to progress because I am this is where I get confused on whether it's the terminology issue of registry and repo. Having an index on a particular repo that I buy that you know that's a fairly significant thing. Having any kind of metadata that spans multiple repos is the part that I don't really buy into. We're not. What we're saying here is that from a namespacing standpoint you get better protection. Okay, there's like this, this, um, we back up a moment. There's this continuum. So the most secure thing would be from a from a timeliness standpoint would be to for anything you might ever want to install anything anywhere in your potential namespace to always know the version of all metadata. Because what that would do is that would give you like the ultimate protection against like rollback and these other types of attacks. Right. And then the other end of the spectrum is you only know what you've downloaded. It's basically you're blind to everything else. And so this is like the per the per like image metadata scenario. And the problem there is exactly the problem that we've been talking about that that basically people can play all sorts of games with tags and you don't have a way of knowing that. So what what you want is you want something where the you actually from a security standpoint you want a tough repository to be as large as possible, like in the case of PI PI, their new version warehouse, it will be one tough repository, every single piece of software on PI PI will exist in a single tough repository. Right. I think that that, you know, that it may be possible and efficient to do that as well here to have a registry, have a single repository. You know, but once again, then we have to start to get into the side conversation of privacy and all these other things which one of our design options addresses quite well. And the other one pumps to the side by saying, well, what will happen is is that private data needs to be isolated data that needs to be private from other data needs to be contained. Each, each subset of that needs to be contained in a separate repo, just to hide the metadata aspect but not actually from an efficiency, like from, you know, it's just a way of hiding the metadata effectively is the way in which that that works. But ideally you want that, you know, from a securities point on that spectrum you want as much content as possible that someone might install in really everything to all end up in the same repo. Okay, I just I don't buy how that's going to scale even in PI PI, but you know it's certainly from a from a PI PI from a public red public distribution, you know, I could kind of understand some of that like all the things across all of PI PI and I get that's the difference between this and PI PI like here with registries, even Docker hub being the public one still has private registries because that's how they make money. So the problem when you get into privates is they intentionally don't want to share content and have any crossing of information because you risk leaking any information across that are that's private and even within a company, we have multiple teams that share the same registry that don't want their information shared with the other. So problem is we're trying to make sure that the latest of something is the latest. Yeah, and, you know, and as we talked about in the document there's there's two ways that can happen one is to just have a tough report be separate so there's clearly no leakage. The other is is that you use a Merkel tree, and your Merkel tree. So the two things scalability first of all, if you have a Merkel tree. The depth of the Merkel tree is logarithmic in the number of items in the thing. So even if you have a billion items like in this would be a billion people that use a single registry and have their own metadata file. Then you have to download like 30 hashes in order to verify something 30 like you know shot 256 hashes or whatever else. That's small. It just, it just doesn't matter. It's the, it's the tracking of it because they're a constant state of change. There's a constant cleanup happening. There's a constant set of garbage collection that's trying to be done. Yeah. At some point when you need to generate a new one which you do periodically you just take all the content that's in the repo as of that moment. And then you it's basically, you know, as though you're putting it in a file to sign it. And generate that metadata and then you have it. You have it for the next period. It's like the same process you would go through to generate like a snapshot file only instead of generating one file, you're creating what are basically either a whole bunch of small files, or a single file that you can then pull the right hashes out to reconstruct the thing and give. But let me, let me kind of. Yeah, so let's just say like I think we can convince you the convince everyone that it both scales and there's, there's multiple solutions and part of what we want to do is provide information about what the cost would be what the efficiency is and so on of those and have people look at that but that's one area that we unfortunately didn't get to in the in the past week or two was kind of an apples to apples comparison, but you know we may be asking you guys for some real actual data, or you know, I can give you some numbers I've got a I've got to go now but just bring me on back I'll get you sizes for. Thank you so much we appreciate it right thanks Justin. I think I got to drop off because I got 1130 also but I got a couple of minutes, I think that by scoping this to saying you're trying to make sure there there's you know like I said of the two scenarios there's updating of stable tags and there's unique tagging. Unique tagging isn't a problem because it's always unique and you know it doesn't change the, and although there's no tag locking as part of the specs so it's hard to say that you know that that's anything other than a per registry implementation. So that makes it hard as well so even in unique tagging you still need to make sure that the digest behind it didn't change. Scoping it to a particular repo totally makes sense and that becomes scalable it becomes within the scope of security that you know customers think about with registries. And that would alleviate the concerns that you're hearing back and then you are scoping it to a scenario that does make sense. Because it's what's really interesting about it is the signature is still valid right the key is still valid rather it didn't get revoked because they went to have last week that wound up this week having a vulnerability is still a valid reference. It's just that there's an updated version of it we want to make sure gets deployed. That's an interesting scenario that I think we've captured I think it's one of the ones that I added. It might be in the PR versus what got merged because I do remember this one. So that is that helps scope it. Okay well I think a bunch of us may have to drop off. Is there anything else that we should talk about in the meantime. So others had comments that did anyone else who had a comment not have a chance to speak. Okay and if you look over the comments that got resolved and you feel like something wasn't actually resolved please please please flag it. We never, you know, we didn't close any of these just trying to get rid of them. We really did. We really do think we've addressed all the concerns. If you think something could be improved then let us know and we will continue to post information about this and also, and after a couple days I'll probably actually turn this orange text black so please take a look over those, those parts that have changed more substantially, but we'll continue to try to revise this and try to get information on like overhead and other types of projection and stuff like that for the meeting next week. All right, thank you everybody. Please Justin. Thank you. Thank you. Thanks.