 Hello, welcome to kubecon We're here today to talk about Naturey v2 redesigning the security supply chain for containers It's me and Steve here and I'm not not unfortunately, but hey Well, it's a virtual conference. So Omar is even more virtually actually here while you're doing this, but not Anyway, it's it's great to be here and Let's let's get let's get started Who are you? Who are we indeed? I'm Justin Cormack. I'm an engineer at Docker I'm also a naturey maintainer and a member of the CNCF technical oversight committee I work a lot with them security and CNC. So I'm really interested in all the bits of how Security fits in with containers, but I'm also I've been working for you know for five years now at Docker I've been very much embedded in the whole How do we do containers? How do we make them better space? Every container has a bit of Justin in it As I'm Steve Lasker. I work at Microsoft on the Azure Container Registries for ACR and MCR I work a because of that because registries are at this source of all, you know cloud computing at this point I get involved a lot of the end-to-end workflows Including security and as part of that I work a lot with the community in the OCI technical oversight board for Things like notary as well as artifacts and other things that we do under OCI So it's very fun to be able to continue to work on this with Justin and others and I'm I'm also probably much here with Registry ECR registry at Amazon. So he's also really heavily involved in the whole registry and container ecosystem So it's a it's a good team. We've got Just to frame this this is really about supply chain security and why it's important The way I like to tell the supply chain security story is back in in the good old days before the whole Cloud thing happened you had things like hardware firewalls and Actual cables you plugged in and you knew that if the computer was plugged into this cable It would get this network and straightforward things like that, but everything is software now And if an attacker can just modify your software You can change absolutely anything to do with your infrastructure and connect the computer to a different computer Rewire things we rewind networks rewind connections Rewire TLS certificates Rewire, you know change absolutely everything and so Your code suddenly starts to really matter and it really matters that the right code gets into production And someone hasn't come along and made any modifications to it and that your code's got, you know security checks in it And that you understand how everything gets from source code into production and make sure the right bits get into production Supply change and tax have been growing Not Petro is a famous one from 2018 cause billions in damage You know brought down all of companies a supply chain attack indirectly via a you know a Ukrainian a piece of accounting software, I mean Probably a state-level attack, you know these things are Important and so we want these we want protections against these in the container ecosystem because like more and more people are shipping more Millions of things with containers and this becomes really important I mean containers are effectively becoming the next virtualization stack We keep on moving up the stack from you know physical machines to virtual machines And this is just the next way to get the best first better virtualization It's got to be secure and and Actually in many ways we have opportunities to build better practices than with VMs I think that's what's one of the exciting things about not only is it in a way I mean like there are potential risks because everything's software, but actually the fact that everything's software means it's more mutable There's a lot we can do the artifacts are more manageable We've got a better ecosystem around them than VMs ever ever did So I think there's there's a real opportunity to do this really really well with containers And that's what's really exciting about the container is not the fact as a container It's the fact that we can build better infrastructure around these things And a bit of history about no, this is notary v2. How did notary v1 start? Where did where were we? Notary v1 was an implementation of the update framework, which is a kind of cool name Usually known as tough to its friends As the update framework adapted to containers It is appropriate it came out of NYU. You know, it's New York tough CMD It was based originally around the security problem of Linux package repose such as apt or More generally language package repose such as NPM But it was originally and the work came out of studying what went wrong with them, you know your package manager Turned out there was a bunch of security issues Serving up fake packages and claiming that your packages engine X when it's not Serving out replay to accept me up an old package and saying it's a new one Just deleting packages from mirrors and saying that they aren't any updates this package to keep people on an insecure version changing dependencies between packages Mixing imagine different and and so on. So there's a whole set of papers around what these threats were and so the idea was to kind of Take these take these ideas and models and apply them to containers was the original idea it was originally a docker project and Back in 2015 and We worked at docker with the NYU people who had invented tough and Let's join CNCF in 2017 as one of the one of the quite early projects in CNCF along with the tough specification itself Which is also a CNCF project But there was a kind of a bunch of issues I think 2015 was an awful long time ago in the in the world of containers That was back when basically when I started working in this space. I actually was not involved with this originally It had just been launched when I joined Docker. I think My first docker con was in Barcelona and I think that's where the launch was if I remember I think it's that one We gave everyone Yubi keys for signing their containers Under the chair prize, I think I still have my But back then containers in production were really new we didn't really know what all the issues were We didn't have a kind of field for exactly what the workflows were going to be Like you know the whole way that people actually work with containers was still in flux I think there was a bunch of design mistakes. We'll talk about some in a minute But now the security needs are really really important and getting this absolutely right is important And you know, we've had time to reflect on what the best practices are and where we're going with containers And where we want to be in the next five years Bunch of issues to fix The first one really is the No, true was added as a side car, you know, not not a side car in the Exactly container sense, but it runs on the side of a registry. It basically has its own database its own server and that was quite good for just shipping it without Modifying stuff and we're training stuff, but it's actually We've really developed since then a kind of whole registry native ecosystem I'm just Steve you can talk more about that you've been involved a lot in that recently as well Yeah, it's the the there was so much value in the work that was done in registries and the abstraction models that When we started looking at other artifacts that we needed to manage in deployment such as helm and you know We started this project as to how to kind of stuff it in That the more we were doing that, you know singularity came up say hey, we'd also like to use a registry CNAB was starting There were some questions around terraform and others and we're like Can we just leverage this common infrastructure so that it works across all clouds because we can't really ask the helm team to Build a helm CLI that's unique to Azure. It needs to work across all registries So we went down this OCI artifacts approach and what was really nice about that is is just there's so many things that we can know leverage that you know global production resilient infrastructure including putting signatures in and You know where we talk about the side Carly who cares about the implementation details The problem is is that was leaking out and if I pulled an image without notary enabled I would get one set of you know artifacts images at the time and if I turned it on to get a different one So it just created This very end user poor experience and one of the biggest problems of you couldn't actually move the signatures from one registry to another That's kind of one of the key scenarios We see is I have a pump something. It's on some public registry I need to bring it in my private registry so I have control over you know that content whether it be vNets or making sure that I nothing changes on me If the signature doesn't move how do I prove that this thing that I've got in my private registry is the same thing It wasn't originally so those are some of the key problems that we just had to overcome to make this thing usable It just so happens that the artifacts approach allows us to do that because if every major registry Supports artifacts then we'll be able to support signatures as well And it is so in addition to the that being I mean I think in a way That's being our number one requirement the way through but I think There's a whole bunch of requirements that are really important about usability and usage We've that we haven't had a lot of people using nursery v1. We've had a lot of complaints about usability issues with it And I think I put a separate thing about observability and understandability and debugging these are kind of related though It's difficult to see What the state of the world is what the the clients are not very good at telling you What the signatures that there are there are and why for example why it's complaining you can't sign something and If you know if things go wrong It's very difficult to debug the state of the world because the toolings really not Not very good from this point of view in in particular There's a kind of there's a tendency to fail if it to fail closed which is the right thing to do when some signature check fails But that actually means it's hard to see it doesn't give you a good explanation of why that happened Which is really makes it very difficult for people to see what's going on because if something Something has expired so it doesn't check something else which you're expecting it to check and you don't know what's gone wrong So you don't know how to remediate and the security model is not widely understood which I think is is a partly an explanation problem and partly I Think people's model in their heads of what actually Signing is for or not very clear and I think we need to explain that clearly more clearly But there's also some fundamental security issues that Really not addressing this this tofu one. It's not about the food stuff. It's basically trust on first use and Part of the model with notary V1 as used. I'm just it's not actually requirement of notary It's a requirement of all bit of the docker and docker-dressed implementation in particular But it's also the way the only effectively the only implementation. So it's kind of The way it is used is that When you pull a container, it'll pull the signatures and check them the first time in the same way as SSH does and it says do you want effectively SSH says do you want to use this host key, but for Containers this doesn't work very well because a lot of times you're pulling container. It's basically it's a it's an ephemeral server It's always first use and so And that is something that's changed a lot right since when when we talk about first start of this project years ago We didn't really think about this serverless kind of model where every time a container is pulled It's a clean pristine environment that the clouds kind of hand you so you only pay for what you use We were very focused on you know VMs would come up and they'd have lots of cached images and that was okay You kind of build up some state, but we were now building to this ephemeral client model where every time Wherever the containers run. It's like brand new Yeah, yeah becomes a challenge and nature and people are actually doing this for security reasons because having clean starts Is actually a good way of keep making sure that everything's refreshed attackers around all those things that there's good I mean, I think the ephemeral thing is actually one of the great things We've actually managed to do and that's good But it basically does affect security models of things that assume long-term state and we have to adapt to to that model So How we're getting there with V2 We started really at Kubecon a year ago Originally we had a meeting with Docker on Amazon and Microsoft on the on the side and San Diego on one of those rainy days rainy San Diego But and then we had a series of meetings back when you could have meetings in Seattle Over in December and January. I think February even I think we had we had a few meetings in person, which was actually Really great Lots of people involved NYU with the people who've been working on tough IBM Red Hat VM where let's more J. J frog All sorts of other people have been involved. So it's a very broad Harbor yes, yes, well, yeah, yeah, multiple parts of being where So it's been a really great Community thing coming together to To really work out as a community what everyone wants We had hoped it would finish sooner I think growing consensus is actually really difficult for especially for I mean things like signing and supply chain security are complicated and trying to Get everyone aligned with understanding What each other's problems are and what the kinds of solutions that we want to Explore are and why things are not working now was actually really complicated and I think that That that to take a long time, I think we were a bit slowed down by the COVID thing and And just you know generally And I think we are we are making progress now I'm kind of pleased with the progress we've made in the last few months, but it's I think it's um It's always you can always be optimistic and think that things will happen fast, but it's just difficult sometimes, you know, it's just And we just blame everything in COVID in 2020. We just want to get past 2020 as fast as possible We don't want to get I mean we do want to get this right We don't and we want to lay down a framework of it Everyone's happy with and we don't want the community to fragment because we don't want people saying this doesn't work for us We're just going to do something else we want one solution that works for everyone's use cases And so we've been very inclusive and try to get as many people in as possible who are doing things now or Got interest because it really I think having everyone doing the same thing and having interoperable tools is really really important I mean if an image can't move between clouds and between registries on-prem different clouds and so forth then we failed So, you know, we have to make sure everybody's on board everybody agrees it meets them the scenarios everybody needs and Then we're good. Yeah, no big deal And you know this kind of highlights that what we've also some of the problems We've talked about is you know multiple the multiple signature problem Which has an effect in a number of different ways So we'd like to try to have fun is some things can get way too serious So we have you know a company we just you know fictitious web at networks that creates this net monitor software It's an ISV kind of product Where they're building it in their environment and they create a key that they're not to create key They create a signature with their key and they sign it They could then post that content on Docker hub as a curated location But Acme Rockets is this company that wants to consume the software. They don't know who web at networks are but they trust Docker So Docker certifies that content. So great. Okay. I will now use that So if I go to Docker, I'll actually see two signatures one from Docker of one for web at networks And then the Acme rocket team can bring that software in as they bring the net monitor software into their environment They'll only bring in stuff they trust from Docker hub. So that's a first check for the signature They'll run some functional tests to make sure it works in their environment because it's not just the first time They wouldn't be able to get updates and then if it passed those tests They will add a third signature, which is the Acme rocket signature So now when they try to move it to their production environment, they actually don't care about the other signatures They only will put things in production that has an Acme rockets, you know validation on it so you can kind of see this progression of flow where the Software starts all the way over in you know Way upstream if you will moves into another environment the signatures move with it the signatures are added the add additional attestation But the digest and the tag of that original software never change So then when it moves into Acme rockets, we you know can continue to add signatures to it as we go So that's like the major model of what we're trying to do And support this you know workflow, but there's yeah, I mean there's a lot of detail That doesn't cover like how does you know, how does the end users have policies for they trust who? But which you know which bits of You know which keys attest to what how do we deal with one of the problems with nature v1 has is there were too many keys And there was a huge proliferation of them. And so we need to arrange keys and Manageable but not too extreme way. And so there's all sorts of Issues that follow on from from these scenarios that we're going through We've made quite a lot of progress with storing signatures in registries. I think I mean we've we One of the aims here has been to have a very generic signature model. We don't want to Just constrain signatures to being something that's specific for nature v2. It's like signatures are something that could be used for other security applications And so we want to have a generic solution, but there's we had one requirement that really Is one of the things that's actually meant that we've looked at actually making registry changes. So This is question about inline versus detached signatures really the way to understand that is like is the signature part of an object and therefore As registries a content addressable part of its content hash because either the signature is You know actually part of the Jason manifest of the object or it's referenced from the Jason manifest of the objects And so if you want to change the signature, you actually have to change The object and then in the registry means you have to re-tag the object now a lot of people don't like the re-tagging model in Registries lots of registries are now implementing models where tags are immutable and can't actually be changed once they've been once the content has been shipped to the registry and so There's there was a big we don't want we don't want re-tagging. We want to be able to add another signature for a different purpose like You know I bring in an object and it's signed by it's a bunch of it's signed by Canonical that's great, but I also want to add a signature as part of my workflows, but I don't want to Do any you know, I don't want to chain I want any code that is looking for Genuine Ubuntu 14.04 from the Ubuntu source not to have any issues that oh actually this content doesn't look like genuine Ubuntu It's got a different hash. Oh, I have to look at it. I see that the hash is actually like it's actually an additional signature It's not a problem. It's still genuine Ubuntu. So these kinds of concerns meant that we we've Been actually proposing a major change in registries to add extra API's Which where you can basically have a what a detached signature, which basically just means a signature It's in a separate document, but not just detected as in a separate document, but as in there isn't a pointer to it from the Registry object instead. There's a pointer from the signature to the registry object and we want an API that can actually traverse that Backwards and that's a real Significant registry change because registries don't do things backwards right now. They're forward, don't they? There's no reverse very much a That's very forward like if you you traverse a tree downwards and they're not Not upwards. There's actually I think potentially other interesting use cases for this type of model But anyway, so at the moment we're we're doing work on actually putting that into the The Docker distribution registry reference client and the spec And so because I think that there was enough demand for this But it was seen as important to actually design these make these registry changes around that So that's a kind of substantive change I mean one of the One of the interesting ones about that is you know people have these deployment manifests whether it be a kubyaml You know a helm chart or just even a docker file that references something with a digest and a Tag and there's you know tag locking you know semantics and so forth and even docker lock is some projects that are out there the challenges if we To add another signature in the Acry rocket scenario for instance if that digest has to change for that tag That kind of breaks the workflow. So that was you know a big piece to this So for the demo we can show what we're at on the current prototype. Yeah, so just to talk about We've started prototyping a couple months ago now. These are not in any way meant to be like final things do not use and But these are Kind of worked examples to show how things how things work How are we putting things together how we're experimenting with things and the kinds of things so that people can see them And but they're they're not take they're not just totally fake. These actually implement the actual Work behind the scenes in the there's actual things in registries and their signatures and registries and all the formats And so on are documented and they're all in the in the PRs that these correspond to so it's some It's Pretty it's it's very real price typing not just mock up So they are nothing up our sleeves will take a look. We have nothing locally on the machine So what I want to be able to do is you know, son build and sign So what I'm going to do is just make things a little bit simpler And I'm going to set an environment variable for our image We've got this registry and v2.azurewebsites.net and it's hosting the changes that we've been proposing to the distribution Docker distribution for the distribution spec So now what I could do is I can say Docker build dash t and our image And what that's going to do is just take this Docker file from hello world and just add something to it really super simple and It's not helpful Oh, thank you minor detail um So now we can see that I've got my image and that's great And there's that image now what I want to do is when I sign it I you know, I need to enable the notary workflow. So I'll say Docker notary dash dash enabled And what we're doing here is where And you know saving this configuration and now you'll see we've got this for you know our initial prototype where we have just enabled trip So now when I want to sign this thing I'm going to say docker notary sign And I give it the key for the azure websites netkey And the cert for these two and then I'm signing this image So we'll generate the manifest because we're we're signing a manifest and we output This JWT token. That's what's the the OCI artifact that is the signature that we want to push to the registry But we don't want users to have to think about pushing and pulling signatures independently We want to build it into the experience So I'm going to log into the registry And when I do docker push What we'll see is because we're using this docker plugin, right? We didn't actually change the docker api docker cli yet. We're using a docker plugin that we're routing this to So now we're just going to be able to do is push the actual image And we're going to push the signature. It's just an additional component And then when I want to do a docker pull, but let's get rid of our images first Is dash q we'll get rid of those And now when I do a docker pull I have not configured anything yet So it's going to go look because I have notary enabled says hey, I found the signature But you don't have any of those keys. So by the fact that I don't have any keys local that match it It is by definition going to fail So I don't have anything To Enable this What I do is I put the reference to the key The azure webs notary v2 azure website cert and save that And now because I have that key locally It's going to look up signatures found the signature And it's a valid signature So now I am going to be able See I have now found our image So that's the prototype right we're talking about prototypes from end to end And we'll continue to experiment with this and decide what experiences we like How do we validate this with things like opa and then figure out the next steps So Going back Yeah, so What's interesting and just to kind of talk about this before is you know everything related to How do I want to store things in registries? It's more than just images and helm charts and Scenabs and opas and wasms and other things I want to be able to store signatures and have a movement. In fact, I want to be able to put verification objects signatures on any of those things Well, I also need to know well who signed the signature. What get commit was associated with it. What was the When was it added and so forth? so We and once we're doing this reverse lookup on Hey, I'm looking for this thing. What artifacts are impacted by this thing Then we can actually start to create more of this generic api for finding these things And that's been kind of a lot of the feedback as well as like if you're going to make a change to our registries Let's make sure that I don't have to make yet another change for the next thing and the next thing the next thing Yeah, so there's the typical problem of yeah generic tools that will enable us to do more More good things in future Yeah, the generic apis and the generic tools that will do that. So We are definitely making sure that we're doing that, you know canonical balancing of the boiling the ocean, which will never happen and Versus a small pond So we're trying to make we feel good about the progress thus far And these prototypes help us figure out where our rough judges for instance How many signatures are on there? I only care about this one signature So when I'm looking up Signatures for an artifact don't make me download all of them I want to download only the signatures for that match a certain criteria So that's the kind of things that we're looking at. So next steps really We're carrying on with prototyping. We're doing a lot of work on key management There's a separate working group on that. We need to consolidate recommendations that enter the requirements then Get agreement on the directions from these prototypes and make sure there's there's actually other prototypes of the one We're showing that are going on as well. So And so and then we have to check everything against our threat modeling that we did earlier on Um, and then get production next year, which would be great for fun. Um, so if you're we just it's just let's just get past Kill 2020 and 2021 will be nice and secure. Uh, so come and join us. Um Uh, we have weekly meetings get hub Uh, naturally cncf cncf slack, uh, is the sort of central source of um, Where everything is so thanks very much and See you soon