 All right, I'm talking to myself Are we gonna have a meeting today? I guess is the question Hey, dude. Hey Steve I'm doing well. How are you? I am good trying to wrap up the end of the year. So But I'm not sure we can talk about whatever you want I could talk about what I was just mentioning on the slack channel It doesn't look we have a lot of attendees probably because I did not get an agenda out and I'm guessing people are starting to Yeah, it's a It's the start of occasion season, I guess. Yeah, and as much as people can go anywhere. I You know, I'm just gonna walk around my yard and go around the neighborhood. I'm just I really Just you gotta take a mental break. We actually a friend of ours Has a house up on the beach and she just like rotates anybody that can take it just for them And she's got all these rules about cleaning up after you leave kind of thing. So It's getting out and doing something Yeah, it's important to get out We can do a rough Q&A if people want to discuss in such a small group The work that I've got that I was gonna present is really just kind of work in flight that is not directly It's enabling for notary, but it's not a notary. It's it's an underlying infrastructure to make notary work and registries I'm not sure what people are interested in discussing You're interested in So that's I'd mentioned I'm I'm still trying to figure out exactly I'm Container and build tax program. So it this has been like discussed for a while I think particularly on like the VMware side There's like a strong interest in having some notary and then we're very excited to hear about Notary v2 not requiring a new, you know a new server, etc So I'm just trying to sign in so I can bless myself on the thing You shouldn't have to sign it. It should be anonymous The case See I had to but it's fine No, okay, you can sign it, but you're not required. That's what it is. I didn't do that. Okay Yeah, but you know, I just wanted to I'm not gonna be able to you know come necessarily too often I just wanted to pop in from time to time and use myself and kind of see what the community is up to Yeah, we've been trying to be good about the recordings to because we recognize that we've had struggle with time zones also So we try to use the recordings to get people a chance to get on Yeah, I actually I appreciate the time the switch because I'm I'm in Israel So this is 7 p.m. She's better than 8 p.m. Which is what it was previously Does they have time zone daylight savings times kind of things? Yes Okay, and they're probably not I actually did a time zone time Feature years ago and I was amazed at not only do different countries do daylight saving times at different times during the year But it has changed over the decades. So if you wanted a change like five years ago US has changed and Europe has changed So if you try to store something in a database and you want to reference a time, you know Over some period of time you have to make all of these adaptations based on all these rules So it wasn't as simple as just doing math. We actually had a lookup database that you had to go and look up to do the conversion on and a team in place to be able to maintain this as geopolitical things change time zones. It was pretty amazing feature work Yeah, that's why I I guess import time zone libraries and I don't do it myself. There you go So Andy, what is it that you're doing? Yeah, I'm actually can you hear me? Yeah, no, okay Yeah, so I'm actually on Nia's team At AWS so just sitting in learning a little bit I'm gonna have to eventually work with this and code some stuff up against it So trying to learn a little bit as it gets developed and Keep myself informed Gotcha. Yeah, so Nia's our agenda for today kind of fell apart. Nia's was not feeling well. So he has to It was a migraine. So it's probably more have a box yelling at him. It's probably not the best. So He'll have work that he'll post online. I think is what he was saying Um Was it you would like and Maria What was it that we Maria and I know each other? Is there anything that people would like to talk about today? I could we can make it more open and free for him. I think particularly today. I'm just gonna check in see everything's doing Yeah, yeah, I also didn't die in I watched the Presentation at QCon and I think I'd seen another presentation you did a bit ago. So I just no one stepped it on this Okay Hey, mostly so that likes that the work we've been trying to do we've got the prototype In a place where we want to do more end-to-end and I was just in fact, I think I just saw Gareth or Rita respond So if let me pull up I can share my screens Just Screen share so we've been You know working on this, you know, it's end-to-end experience where a company web at networks is developing software They have a registry where they sign their content. They publish it to a public registry If you want to pull it from the public registry based on that, that's fine You know, so app at Acme Rockets doesn't trust web at networks, but they trust Docker hub So Docker puts the second signature and then actually puts their signature as they move it to their registry for their validation So we feel pretty good about these overall The things that we this is you know conceptual we know to do here But that's always the the challenge is conceptually we know a lot of these different things It's putting it into practice And if you remember one of the I think the coupon talk we did at the beginning of the pandemic Just and I were joking about we did a blueprint for a bathroom and we're ready to go build it and we're all ready to have contractors show up and We kind of had one more sketch just before we did it and his joking comment was where's the bidet? You know, so it's not You know from a US perspective, you know pre-covid pre-lock of toilet paper We didn't really worry about bidets, but now we have a better perspective on that So the it's all a batter of like it's great in theory until you start putting it into into practice And then you start to find all the rough edges so I've been working with the Opa folks the open policy agent to see if they could implement the notary vtubes You know prototype to see if this would work right put the put the car on the track See if it actually does what we think it's going to be So that's kind of the next step is what we want to do from a validation perspective is see Does this flow as we're thinking work we know it's very minimalistic we don't even think it's minimally viable yet But what are the things that make it more viable? One of which obviously has been how do you get the key on the ephemeral client? So that's part of what Nias has been doing in the key management with Ian and some others and trying to figure out What is the right experience there? So we'll hopefully make some more progress and I'm hopeful we'll get something because Over the holidays some people really like to even under normal circumstances like to do work over the holidays on this stuff So we'll see if anybody's signed up to do that I really wanted to have the opa folks work on this because they are the experts They are the ones driving the project. It could be opa contributors. I don't really care who But I really wanted to have somebody in the opa community working on this so that they could say hey, here's here's what works today Here's what doesn't work, but here's this thing we were thinking about anyway because they're you know kind of in that incubation phase as well So we'll hopefully get some progress going there At the same token the other thing that we've been working on is the the whole idea here is that These are three different registries and possibly three different clouds that have been run here So obviously there needs to be a solid standard that this stuff can move between these different registries and projects if you will So there is some work that we've been uh, the other work I've been doing is what are the changes needed to distribution to be able to support this And uh, I'll go through this relatively quick. Oh, hey, Ian I just cut in for a question. Yeah. Yeah Uh particular I guess um The assumption with the keys is that they would be kind of a standalone oca artifact or that they would be Pieces of metadata that are attached to the image to the you know, the actual image that you'd be signing Right. Um, and you're saying or is it like part of it, right? They are signature, etc Or would it be you know based on I don't know a docker manifest or something like how it Yeah, so those are important and they come to get interchangeable sometimes So we actually are storing signatures in the registry as an oca artifact is the is the goal The separate oca artifact. Okay. Yes, and that's all and what's really important about that is if you notice We're doing additive signatures here so Web networks might have you know, they have a stuff this the key the signature Sorry, not the key the signature right in the manifest because they originated it and that'll be fine But the problem is is that if I reference this artifact here with a digest or a tag When docker hub wants to add a signature to it If they're adding it to the manifest then the manifest digest changes and that breaks the core requirement that we have I just say that makes sense So part and that's part of why we have all this distribution work. We're doing is we want to make sure That we can do this we what I call reverse lookup Registries are really good about creating workable trees and looking down Right, they can trade create a tree and there's lots of you know routes that go off And we index when I say we the registry is index in that direction There is no indexing that goes the reverse route for all intensive purposes There's some garbage collection index thing that's going to get done to make sure of that But it's really a one directional kind of thing Generally speaking you can't go to a registry and say What uses this layer? Right, it's just into some there's some of that because of garbage collection, but it's you also can't It's just not optimized in that way Where so what we want to do here is I do want to have an optimization. I want to say of the Of the 10 signatures or in this case three of this artifact Well, actually, it's the other way I take that back. I want to be able to come to It's easier if I just go to hold on. Let me just go to the slide that shows this So Here's which one is this? Oh, okay. This is the same chair. Okay so an image Is an artifact and it's the you know or the artifact itself is just the thing it's in the registry and it's got a manifest and it's got layers When you push a signature You're actually pushing a signature that is referencing this artifact the artifact doesn't actually know about it So I push the first one by wabbit networks. I push the next one from docker hub and I push the last one by acme lockets When I want to validate this what I'm doing is I'm going to the registry and says hey, by the way Can you give me all the signatures for the net monitor v1 image? Registries don't have a way conceptually to do this reverse lookup So that's part of the first thing that we want to change And if you take it helm it kind of takes this the same way like a helm chart In theory should reference the images that it has in the registry Today a helm chart is opaque object and the registry doesn't actually know without cracking the helm chart Which is something we want to avoid doing there is no metadata that says this but we'd like to be able to enable this So basically the registries are a thing that reference another thing and it might be one to one So one one artif one manifest references one config. That's the one to one One manifest references multiple layers. That's fine What I also want is multiple signatures to be able to reference a single artifact. So that's the many to one So that's the evolution that we need to provide and not only do is the signature is something that need to point back But we also want To be able to sign all these things. So granted. I'm not going to sign a signature per se But I do want to have these things that point in multiple directions each one of these artifacts be signed as well So that's a the key design goal To some degree there still is I mean There is there already is an inherent many to one relationship where you can have multiple layers You can have layers in different images that reference another, you know stored layer. That's No Am I understanding at least No, you you can share layers across manifest. There's a deduping process that most registries do But that's more of when the manifest is when the layer is pushed up We say oh, we already have that in a scope that we feel is secure So I always use the coke and Pepsi scenarios and I haven't had a coke and Pepsi come back and yell at me So I'll continue to use those You know coke pushes images into a registry including the Ubuntu image when Pepsi pushes it in we actually don't share Across customers because there are vectors where you can actually Hack that shared layer and we don't want coke to kind of screw up Pepsi or somebody else to screw up on both So we actually don't dedupe across customers Storage is not the highest cost and is concerned compared to security. So there is some of that deduping that happens But it's not really the the many and many look up the kind of thing Interesting so There's other collection types. This is kind of there's this interesting conversation actually I might as well just do this. We have some different people here in the call Um CNAP is another interesting model and so both helm and CNAP have both done work based on what exists in registries today What we're trying to figure out is the things that they're blocked on what do we need to add to enable some more richer scenarios So today a CNAP has an invocation image and that's how it you know It brings the environment to run whatever process installer it wants to do And it can reference the wordpress, you know image itself externally But what's interesting because they built this as we were building the OC artifacts approach Um, the indication image in CNAP actually has the helm chart embedded in So it's not external. So if I have all of the helm binaries or whatever else I need to run If all I want to do is add an updated helm chart to do wordpress deployment I actually have to rebuild this entire image Which has all kinds of security questions as well because as a company I might trust A base image that has helm and azure cli or awc li or whatever in it Whenever I get an update for a new deployment. I don't want to have to revalidate what binaries are in that I want to say i've already certified this thing that's got binaries Now all I want is uh a decorative piece of data that I can evaluate quickly and see there's no binaries in it So I can't do any harm. So that's the model that we really want to be able to get to So the theory here is that if a registry can store these multiple references that Now the CNAP can have the indication image as a single thing that I would sign and verify in my environment I don't have to crack it ever again, but I can have a wordpress chart that is external from that And then the chart could also declare what not just embedded in the chart But if it could actually declare to a registry, hey, here's the two images that I'm referencing Or and whatever references Because now what I can do is when I want to move well one one I want to sign this from a registry perspective. We actually can see the whole graph But more importantly if I want to move content from one registry to another I don't need to know All the details of what that particular thing is. I don't need to know it's a CNAP. I don't need to know It's a helm Think of it as you go to the file system today and you copy a directory from one From your from your computer and you want to put it on usb drive or you want to put it in a cloud storage You use the file system apis to copy that you don't use the powerpoint File apis you don't use the word You know file apis or the you know my thing File apis there's a standard way to copy content from one Storage system to another That's the concept that we want to be able to provide in registries is I want to be able to say I don't care what the artifact type is There is a way to tell the registry file system Here's the graph of information I care about so when you want to copy it from one place to another Replicate it from one place to another move it into an air gap environment delete it actually as well That there's enough information that the registry doesn't need to know anything about the specific artifact type So that's that's our our underlying goal here So again the collection types like I'm kind of being repetitive and I missed part of what I was referring So the point here is that we don't I'm Shifting this from collection types to being more reference types and we'll we'll see kind of a little bit more of that come up so The persistences persistent types are kind of interesting because what exactly is a signature Is a signature a full-fledged artifact or is it? You know additional metadata on a registry artifact So I kind of was playing with the idea that if I go to a registry and I want to see all the things that are in it What is it that I want to see and you know if you look at you know The mac file system was kind of known for this if they had all these Additional things you could embed into a file both good and bad But you didn't see that you only saw the one file where in windows you would see a you know binaries and just a Huge foray of information that really was much more detail than you really needed to see So here I've got a whole bunch of um, this is the repository's blade. This is all the things that the tagged Sorry, these are the repos that are in this registry So there's lots of tags under each one of these and this was just the hey What is a 256 character limit look like for a repo? So that was just an interesting one If I look at a particular repo like hello world I can see the individual tags that I push to it And then there's metadata on each tag and so forth But this is the kind of thing that I think that I'm really most interested in I don't want to see I want to see the tags the individual artifacts I don't know if I really want to see The different artifacts being the signature and is it really just an individual signature? No, it's the acne projects acne rocket signature the wabbitz network signature Then there is you know a tag for the artifact and I'm repeating that again Like this is not This is the windows way of doing things is that it was my little self-joke What I really want to see and if I can put an annotation on what this thing is that great I can see that this is an image and these are signatures I don't really know that these things are related That I just did an arbitrary PowerPoint pasting of ordering here. There's just this information. This is a signature But what is the signature of? So if I shift this around what I really want is the artifacts in a registry In this case, they are container images. I want to know that they're signed I want to kind of change the pivot and in fact, I don't even want to tag on a signature I really want to know that this artifact is signed Likewise, I might want to say that an S-bomb is part of this artifact I don't know if I really even want to see an S-bomb as a unique Row here. I want to say that there is an S-bomb for this artifact And likewise additional metadata. What is the git digest of this thing? Who pushed it? Who pulled it? How many are there? How many poles are there rather? Sorry, I went fast on that so the the idea is that we want to basically change the Basically kind of give some hierarchy to the artifacts that are pushed to a registry of different types and how they're related to each other Make sense So the signature is wouldn't necessarily be seen and when OCI I guess the image prepositions are more just being met somehow attached to the image Yeah, that's the idea. I mean obviously depending on the tools It's like what is the mainline scenario that you're trying to do? Do you really want to stick? Yes, of course you can get signatures out but Is the signature by itself interesting as opposed to no signature is actually on something How do you think about it that way? And more importantly, it's not just signatures It's all these things like we're not trying to build a signature only solution We're trying to enhance our registry to support things including signing So that's kind of I purposely wanted to take real durable artifacts that I really want to interact with as first party first First concept kind of things like an image or a home chart And then I want to say what are the other things that I would do to that? Well, I could sign it I could put an s bomb or I can just put a set of metadata on So we're just trying to break these things down a little bit more that there's artifacts. There's metadata, you know, and the what and I want to give Derek credit for this He was kind of pushing on why are we keep on putting More stuffing more stuff into image manifest an image index, which we could In fact, we probably do want to put the media type on index. It probably is helpful But every time we have these conversations we're basically debating with the Container image folks that are trying to build the natural image and trying to iterate there There's this oci v2 thing, which is a generic term, but it's really they were focused on How to do a v2 of images? So how would we Make this more, you know Flexible for what we want to do in artifacts that aren't a container images are a type of artifact So rather than trying to jam or in here what we're really saying is maybe we just need another media another manifest schema And here one we can change from layers to blobs. We talked about that, but maybe it's actually just references In fact, let me just get something I don't know if I have it here. I'll find it So and I've had gone back and forth with two collections versus one collection, but let's just say there's a ref So now we can let the image spec continue to use those two media types But we could allow helm Singularity wasm CNAB, you know opa and all these others to say look, I don't have to You know weirdly fit into these two manifests Because a registry spec doesn't specify you shall only support two manifests. It says it supports manifests Just so how's the image specs defines these two a registry could have another one as well What we'd like to avoid though is having a specific manifest for CNAB a different one for helm a different for singularity Because as registries we have to put in manifest processing because that's how we do our garbage collection and indexing and that listing So we really want to hopefully get one more and get it right Um, and then enable a larger ecosystem Oh, I see where I was going questions on that before I go to the next monster slide So in that you would still have The signature or the the other metadata or relevant metadata is a separate image. It would just be that Using the artifact manifest you wouldn't have to worry about the many to one Like about having the reverse tracks, but instead you could make make it very obvious Okay, and that artifact manifest I guess wouldn't be incorporated in the um in the image id or in the shop because otherwise Right Because otherwise you can't if you change it then that would change the image id Exactly Exactly So let me try this next slide. Um, I haven't finished it yet, but so we'll see how the story goes So you have two images you push into a registry And then you have the helm chart that I did the dotted line here because a helm chart technically does not reference From a registry perspective the registry has no idea that it's referencing these two images We don't crack open helm charts. We just say it there's this blob That's got a media type of helm that's all registered really care about So I don't really get the benefits of being able to help make that copy right the usb copy thing we talked about But if we look at the way these things would work is each one of these things would have a signature Um, and notice this signature points back to the artifact and those are An artifact of their own that points to an artifact If I look at the way registries store this information They and this I kind of did this in a little bit of a minimized Ordering so when you push an image, excuse me The first thing that we do is or the the tooling does is it pushes the blobs So it pushes blobs to the registry doesn't puts does the debbie input does the wordpress image layer put And then it'll come back and post a manifest and the manifest There's validation on the manifest that says hey, by the way this manifests references these two blobs Yes, they are in the registry life is good Um Put complete and then you can also apply a tag and the tag is interesting because the tag actually does point back But tags are a kind of a nebulous things that are in registries There actually isn't much definition of the spec because this is part of very much of the implementation detail But there is a little bit of this like it can look at a tag and I can see what it references And then the manifest does the the directional piece And then the signature is the same thing right the signature. I push the blob I realize my animations and backwards I push the blob of the signature And then there's a manifest that says hey this manifest points at this wordpress chart That's the the current prototype that we back on And then we just repeat the same thing for the mysql image and you know It's very repetitive here, but you kind of get the idea This is this is how registries store massive amounts of information I haven't even called out the devian image is actually several layers and the mysql is several layers and so on and so forth So I'll take all the noise out of the way But now when I push the wordpress chart What I want to be able to what I do today is it pushes the chart as a blob And then it pushes a manifest that says by the way this blob is a Wordpress chart and it's got the config media type to help you figure it out And and there is a tag for that as well. It's this link across the top What we really want to be able to do to support that file copy scenario is we want to come We want the helm chart to tell the registry By the way, I'm referencing these two images that you possibly have in the registry already And notice they're not referencing the layers. That's a detail that the wordpress chart shouldn't have to worry about The wordpress chart should say I'm referencing these tags, which are rep points to digest So actually it's interesting. I pointed to write the manifest So it could point at the digest I Don't like pointing at digest because digest give you no flexibility. There's no break glass scenario It's great that it it's locked and you don't have any drift If you want to reference a particular wordpress Chart, but if the mysql image has a security update I can't just ship the mysql security update as an updated tag I have to actually change the chart if the charts referencing a digest So we believe the right philosophy here is it references tags and tags can be locked Um, and that's really a registry and a customer choice So now a wordpress chart can be published and as mysql revs their security updates or the wordpress revs their security updates They're doing the proper discipline on tags and not just digest and now I get a balance of security and usability Question about that one, but the locking tags. So who does the locking of the tags? Like what's the what's the mechanism there? Unfortunately today the customer has it has to do it because Most registries have to feature as individual registry features. There's no oci standard for tag locking All right, so if anyone sort of like, you know in the case of something going wrong, that's kind of not a guaranteed thing that the locks tag Well, it depends on what you're trying to do. All right If you're and this is part of what we're trying to round out with the signature scenario So the the the big open vector is if there's no signatures Then if I update the mysql image with the mysql evil image There's no way to know because evil company doesn't need to do anything other than have ability to push to the registry Which in itself is not a is a pretty high bar to be fair But if there's now a signature on that as well that it is the mysql image signature Then you've got that at least that second line of defense if not third So the the idea is that not only should there should be a signature Um, but the tag can be locked as well this way that you shouldn't be able well to be fair You have to be careful where you're trying to lock the tag because you This goes into the whole gated import thing And this is the balance between public registries and private registries So where I get the wordpress chart as a public chart In fact, the way these charts are published today The charts are published in one registry and they reference images in another registry So charts come out of helm.sh or various other locations And they reference images that are in docker hub or quay or what and eventually get up probably as well So already kind of syndicated across different locations, which is kind of weird What we really want to be able to do is And that's part of what the ocr effects supports is I don't have to have helm charts stored in one place That knows how to store helm charts and references another place that knows how to store images Registries can store all these things. So now they can be all kept together So that's that's really what the main goal is about And if you have that now I have the ability again to reference the tags And the if they come from let's just say we'll just use doc rub and let's say you know doc rub can eventually support Helm charts being pushed to it as well Now when I reference those I can reference the helm chart I can reference which references the sequel image that references a tag And that's all signed and I feel really good about it Because Helm the helm chart is not the wordpress helm chart is not owned by my sequel the My sequel team can push security updates to that and as long as still signed Then the helm chart is fine. It's referencing the particular digest As I bring it into my environment the acme rocket scenario Then I can validate that a those things work for me in my environment You know, maybe my sequel did a change to something that broke my breaks my logging system And now I can't see any of the the sequel diagnostics logs I want to catch that before it gets deployed. So I'll bring that in I'll validate the my sequel image is signed by my sequel or helm whoever's signing it Um, I'll test it and I'll then put another signature on it for acme rocket to my environment And then I'll deploy it to my environment And you know vulnerability scans and whatever else you want to do as well Is that your does that answer your question room? Um, yeah, I think so Okay, it's it's nuanced because it's not just any one part. That's kind of the detail here Is tag locking gives you the ability to not let somebody else go update it But what happens when you do want to update it? So my sequel does want to update their images for security So you don't really don't want docker hub up tag locking it The real tag locking scenario is more in a personal not personal but a corporate environment or customer environment I don't know what they will all customer environments corporate Um You know, I'll pick on David and Andy because they're first in my list David and Andy are the developers on the team, you know, they want to be able to push images We want them to be able to push images But uh, Ian's our security guy and he doesn't want anybody to be updating anything existing So he's going to put a flag on the registry that you can't push A new content to existing tags in that environment Um This way he knows that everything's immutable and everything it gets deployed is unique And if you really want to do a security update that you should do a new deployment Um for a tag one two three five Uh, and that's the stroma way Ian because he's also the security person realizes that one two three one actually has a security patch problem And for some reason they can't do a deployment So he can break glass patch that image update that tag And now have all the nodes that for some reason you can't do a an updated deployment He could at least do An update in place not an update in place per se But he can basically tell the node to rotate and it'll pull the same image because it's referencing that tag And but he'll get updated content I just don't get when a deployment wouldn't be possible, but it could just be i'm not thinking of you know It's not a deployment could be possible if there there are scenarios where The deployment definition Was is fixed by somebody that they don't have access to and we've seen this in more surprising scenarios than you could imagine I mean look even in apps even in azure App service doesn't have a way to float tags You let you have to either Change the app service deployment Or update the existing tag and then tell app service. Hey that tag you're referencing pull it again Because I changed it. So it's actually set up as an anti pattern to be fair So the only way that would work is if you're referencing it as a tag Because now I can update tags if it was references a digest There's nothing they can do So if it's could you not change that digest if because you know In a hopefully rare situation where you have to update the tag you could just also update this one other Field right or is it more complicated? Well, that's what I'm getting at is I'm using I'm throwing rocks at our own app service team But it could be I have a helm chart that I can't change for whatever because it's owned by another team How do you get an update rolled out when you don't own all of the layers of the things get defined and because digests are fixed are computed, you know for You know computed you can't Push new content to the same digest and if I can't change the deployment definition Then I have no way to to get through that In fact, the perfect example there is that wordpress chart. We have Thousands of people deploying wordpress charts That are not updating the image references So we're doing all this great work to make sure that they're not Pulling content from docker up for their own images and they bring copies into their own registry and they own them They deploy them so they're not Limited by throttles or more importantly the failures of the internet connectivity. That's really the bigger problem Yeah, but yeah, they do a deploy they do a helm chart deploy and what is the helm chart to it deploys engine x from docker hub or quay or gcr I guess my big concern is just that you lose some of the security properties of the references If you're referencing the tag and not an immutable digest of the object itself Like I just add some overhead to if the tags ever needed to change But I just don't see why you couldn't this couldn't just be one other step for this great glass scenario is okay You update the tag and you update this, you know the references And then you've updated the you know the situation So take the helm charts. Yeah, let's say engine x has a new update Engine x puts a security update out, but everybody's using their helm charts to deploy it Who the engine x went and did the update Who's doing the update to the chart that everybody's deploying? Because in your what you're suggesting is this does happen is the helm chart references the digest now Now I have to go find the helm community owner that owns that helm chart and update that as well So it's it's an additional step. Even if I have control over it Okay, but it's an additional step that makes the helm chart immutable so like an attacker couldn't Say change the tag and make the helm chart Invalid or how would they change it? What is it that they're changing? Well, if someone were to somehow compromise the registry or something else and point this tag to the wrong digest How would they do it? Well, if they if they had if they were compromising it so like this wouldn't be like a normal situation This would be if an attacker got into the system But you would lose kind of the properties of saying okay This has been additionally verified by this third party, but this third party You know can't prove that they verified this actual Object I feel like you just lose the We say this but I actually don't understand the flow because what we're saying is the helm chart is Referencing a digest to another registry. That's fine but what or and actually digest isn't tied to specific registers tied to a specific reference the Actually, it is tied to a registry which is part of what we're trying to change because it's got the the registry referencing as well So they have to actually Break that registry if they break that registry fine, then you know if they're referencing a tag then they could have updated it But if it's if we also have the signing the notary signature on it They would also have to have the key that signed the nginx image So what we're already doing is with the notary v2 work. What we're saying is signature is the digest though, right the signature The signature references the digest of the might will keep on using the mysql image in this case So if the helm chart this wordpress helm chart is referencing this mysql image And if it references it by tag Then when it's pulled even I have the ability for the mysql team to update the mysql image But because we're doing signature validation The chart references a tag. So now I have the floating the ability to float to a new security update My double check is that the signature on the mysql image is signed by the mysql org Okay, so I guess it just depends on their level of Like what their threat model is what they're worried about that kind of thing because I feel like in In most cases that probably should be enough to signature across the The tag but like maybe they still have the option to reference the digest right if they for some reason wanted an extra level of Guarantee, but that's my point. I think by providing an extra level of guarantee You actually lock them in you lock them out from being able to do security updates So you've locked them to it's almost the definition of what you've been Your team has been referring to of the rollback scenario Is you're blocking me from being able to roll forward Yeah, I guess yeah, especially if people can't do that release. I guess that's the That's that's the question. Yeah Because the big picture that we're it's happening here is there's no one entity that owns all of this This is just a scattered ecosystem of people trying to contribute collectively The helm chart owners don't own many of the images In fact, a lot of these help charts don't even have constant maintainers, which is a different problem Okay, but they you know that they trust whoever pushed the tag to maintain it correctly and not like You know, there's there's a lot of challenges there I won't go into the detail there, but the point is that's what we're trying to do is provide Secure with rollback and roll forward, you know capabilities So if the if the signature is verified is and you can multiple signatures That's also why we want to have multiple signatures I that I could also say, you know back into our all the way up To In this case, I might say that my deployment doesn't just depend on acme rockets It might say I also want to take a validation on the software vendors. I Choose from so in this case, it might say I have to have from acme rockets and lab at networks I may not care about docker up in this case because docker up is more of an aggregator just putting a stamp on it and it's a good stamp but I might want to say that You know, um, I care about the originating author Their signature and my company's validation signature. Those are the the double check sums that I could put in place Yeah, so I guess then if someone wanted to change the tag and you had all those check sums in place You could check you both of those signers would have to agree up to this Tag change which then gives you that that guarantee that that kind of answers the question because I feel like, you know If someone even if they just messed up on a tag push, they could break a lot of images Any attacker involved But I guess that's that kind of situation with the additional signatures would also Kind of deal with that Yeah, I mean that's a perfect point is this is exactly why we feel the gated workflow is the is the best security model because most failures Start from well-intended situations, right? It's this very very few of the things that break workloads are actually malicious Most of them are the flawed humans in the system And the computers just deal with the flawed humans tell them to So all right, so, uh, let me come back to this other slide here So what um, all right, so now we've pushed the two which Repeated what a what a docker image is pushed and we got the helm chart that's in So now what we want to do is we want to say this helm chart is referencing those manifest not the layers That's really up to the manifest to decide, you know, it's up to those image authors What was I going with this? Oh, I guess that was it. Okay. That was the conversation So the the idea is that if the helm chart can actually if the registry can know what the helm charts pointing at Then there's a bunch of enablers that happen. Uh, like I said, it's the copy scenario. Um, it's also the Uh security scenarios that we can see the graph of what's being secured So a vulnerability scanner can now look at this and go like, oh this thing's referencing this other thing that that It turns out we have, uh, a Debian layer that's bad I can now see not only the other images that reference that same that same devian image But I also could see the helm charts that are impacted by that as well Um, because often the challenge is the security issue is not other vulnerabilities. There's always vulnerabilities There's always fluffiness The question is what is the blast radius of that vulnerability? And if I know where that image is being referenced and being used and being deployed Now I have meaningful actionable information that I could do about it So having this graph is is really important to us And then it's just I filled in the rest of them there. Um, so if I let me find I have a A mock of what I've been thinking about for The sorry, let me just find it. I wasn't as prepared Where do I have Mock of the graph let's see Okay, so here's a very early thing that we've kind of been thinking about So I was originally going with A flat list of references So this is a completely new schema. Imagine. This is the artifact schema So in fact, it's of a media type artifact collection. So it's not image index. Whoops. It's not Image manifest it is artifact collection. That's the new third Manifest type that we're suggesting would give this generic use We were playing with putting the config in the references, but it's a one-to-one. So we left it here So azure's got arm templates azures Azure resource manager templates. There's a new thing called bicep that they've been experimenting with so the idea is that this thing is a bicep template and The layers here have nothing to do with bicep. I was copy pasting. So ignore that. So the idea here Is this is not a great example. So I'll switch to another example But the the mixture is it could have actual layers So you see three layers here that are referenced And I can also reference a home chart so I can intermix The layer the blobs the references to be a manifest that collection at the top But I could also reference other things as well. I might want to reference layers directly I might want to reference config objects. I might want to have a soft reference to something So let's see if this was the more Look at which one I had it was better Let's use this one. So In what the later one is we're trying to figure out the hard references and soft references And the hard references are more a matter of, you know, this is what it takes to actually make this up A docker image makes no sense unless it has layers Right, there's there's definitely you have to have some layers for a manifest to be useful A home chart might reference It has to have the Yeah, so and I've also broken this out. This is not the way home charts actually work, but it's a conceptual way So here this is a helm chart the helm json for this manifest notice. It's still an artifact collection but and I've actually Since we're now have a new schema instead of trying to Put it into the config and having to read this. I've actually surfaced an actual artifact type So let's just say this is The extension the file extension as you would see on disc could say this is a helm chart Here's the helm config object if they want to have that But now I have a blob that actually is the actual helm chart. There's the digest. There's the sauce I have a separate blob that is the helm values file because I might want to have You know 50 helm charts that are only differentiated by the values file So I have a way to de-do these if I wanted again Here's the blob the digest and the size that's because it's a physical reference this thing when this helm chart is deleted It will delete these layers and the config object Reference counting aside for any other helm charts the reference is as well However, when it's referencing The wordpress chart and notice I've kind of just stuffed something here. It says wordpress 5.7 This is saying it's an image manifest And here's the other image manifest There's no digest. There's no size because this is kind of a soft reference to it And this is modeled a little bit after python where If you look at like node and npm and and some of the uh and new get They actually have references that are stored inside the same package manager registry But python is kind of interesting is that it has it makes a reference, but it doesn't actually force it to be there Um, it assumes that you're going to get it. You could possibly get it from the same registry But you might get it from somewhere else I might get the wordpress chart. Sorry the wordpress image from a different registry It might actually already be on the node So I want to say that this chart references this But I don't have it as a hard reference to say that it must be in this registry In fact, when I delete this chart, I don't want to delete the wordpress chart. Sorry When I delete this chart, I don't want to delete the wordpress image Right, so and but if I delete the wordpress image, it might be interesting to know Who's referencing it in here so I can maybe generate a warning on delete to you know, let the user decide Do I really want to delete that thing because it's referenced by other things in the registry? And then of course, there's just the the collection of annotations that you can have on things So this is like a couple of hours worth of work of just trying to figure out how do we make sense of the Uh, the scenarios that we were kind of put in the power points of how we want to try to track these things And then the idea here is that if this was a signature and I don't think I actually have a real signature here that Taking time to finish it Let me look real quick. Yeah, so here's here's the uh, and so this one should actually Possibly not have this information. I'll delete it for a second In this case the signature the notary v2 signature is from Wabbit networks And here's the actual blob for the signature and it references this in image That's kind of the reverse pointer But like I said, you've noticed that it says references and it's not separate collections yet So this is just you're starting to see how a signature can get put in as well So I'm gonna Stop talking for a minute. Let people digest that and give questions I was waiting for your kid there David to pop in Uh, she's she's she's here. Don't worry. She's just hopefully occupied by a youtube video on my lap Sometimes I uh, yeah, that's uh the benefits and the challenges of working it out. Yes So, um, to put so what sorry go ahead So you don't have any or this is a signature this this this isn't what a signature object would look like not yet It'll in the current the second hour of conversations It would probably the reference would be more here the blob of the signature would be here And it would point to the artifact. It's signing here Um, in this case, we probably do want the digest of it. Well, that's the problem. We're trying to figure out Actually, you know, the signature does need the digest. I take that back. So that's a good example where I need to evolve this a little bit more Because you do want to say it's not you're not we're not doing tag signing So signatures actually do sign digests to be very clear So back and that kind of helps with marina's, uh, conversation is While we want the helm chart to have a semi-loose coupling to the image it references So that we can update the mysql image with a security fix We don't want the signature to float the signatures are Absolutely pinned to digest those the the correlation between the signature and the thing it's signed are immutable So that's that's a good point to clarify marina. I hadn't thought about that sense, thanks And then the big challenge is for ian and neos and their working groups to figure out where's the key management fit in all of this so That's a big piece of this next questions Heading in the right direction Kind of confused Doesn't make sense I don't want to take quiet as being an endorsement They're looking good from my point of view Really cool the way this kind of fits in with the the registry scheme I still think I still have a couple of like, you know, larger questions about finding tag finding signatures and Making sure you have, you know, the right signatures on the right object and that kind of thing But I think in general the format wise, I think it makes sense about, you know Way to get this stuff onto the registry and reference each other and and all that so Yeah, pretty cool Yeah, I'm definitely seeing it as the right direction Compared to where we're at and what we were looking at before with v1 This is definitely headed down the path that we wanted to see Yeah, you know, like this is the the critical point here is like these the whole movement is really important like from the mcr to a customer Or into air gapped environments, whether it be physical clouds that are air gapped or even just network air gapped environments This is the pieces that we need to figure out how to reference these things and then obviously You know the keys need to figure out how to flow with these as well as the next That's right Cool. I also appreciate this is helpful for me to kind of contextualize where uh, I guess the build backs project saw that it would be helpful like it will be uh It's gonna be we would have to slot it in a few places both I guess when producing images and when accepting images, but It could be very valuable to do that All right, um With that I will Call it the weekly meeting turn. Hopefully this was helpful for folks and we'll get the recording up And uh, as far as next i'm taking off the next three weeks. Um, so uh, I don't know if Niaz or anybody was able to get some stuff done That he wanted to report the next couple weeks. I just wanted to reiterate We are making good progress. It is a little slower than we would like I'm not blaming covet in this case. It's we're just at a point of year for our planning For the next next portion of the year. So that's thrown a wrench in my time schedules But I do feel good about the planning that we're doing and Just it's how important is it we have as Ian will you know mention it We have critical dependencies on this in azure. So we obviously need to do it and It has to work across different registries So I feel good about our commitment to making sure we will deliver this in time And not just some here some interesting side project. We'd like to do, you know, just out of our board time sitting at home What we do need to figure out how to do the distribution part So that's why you're seeing more progress happen there, but i'm hoping that we'll get The opa folks to be able to help us validate. Do we have the right components and deployment and marina? I'm sure you'll have a bunch of questions there With how are you actually testing some of this stuff that will surface some interesting questions? Yeah You said we will deliver this in time. What is the expiry time? I don't know. It's whatever We to be fair I would probably say by next summer I would hope by first quarter We'll have something that we can prototype that we feel comfortable enough that even we would roll out in acr in various regions We're at a point now where you can take the prototype and stand up your own registry Like we happen to run one in azure websites, but it's not at all scalable. It's based on docker distribution, right? So Um, I would hope that we feel stable enough to ask the cloud that the cloud providers would be willing To roll out the code and the design patterns that we're talking about in their production registries By first quarter of next year. I'm hoping we'll be at that kind of stable place and the spec will evolve and probably by summer We could probably be taking customer production workloads on by next summer That would be kind of an aspirational goal is what I would say Cool kind of quicker than I thought you might say Good so I can over their promise over deliver Or something like that. I'm not quite sure. I hope that they opposite there So anyway, I do have to run to another call. Um, thanks again for everybody joining and david your daughter's absolutely Yeah, she is. Thank you for that. See ya congratulations. Good connection. Bye