 So, you know, last couple of months, a couple of months back there was this news about Docker Hub, which is the most popular repository of content images, getting hacked. And plenty of data was compromised. And that's, if this kind of situation arises in a registry, which is popular regularly, what could happen is an attacker can change the images that you trust while that you download. And you may end up downloading the images that are actually compromised. So, and this does not mean that it only happens in the case of Docker Hub. You could have your own registry and this could happen in anyone's case. So, but you might argue that there are ways to prevent that if anyone gets, let's say, an access to your registry, you could have a Docker Notary or a Portary setup in a way that even if your registry server gets compromised, your signed images will not match and you still can't protect yourself in such scenario. However, if your registry holds images which have a sensitive content in it, it does not prevent you, prevent it from the code getting compromised. In that sense, let's see how we can envision a scenario where we can mitigate such an attacks. So all, what we are looking for here is, we have, we need to be able to create container images where we should be able to, during the build process, we should be able to encrypt them and then upload it to the registry and then decrypt them while you run them using, let's say Kubernetes. That way, the advantage of that process is you do not depend on registry being highly privileged. So, as I was talking, so if you start, if you think about encrypting your Docker images, you do not need to have extremely high privilege registry. That means you do not have to trust your administrator, either malicious or the intentional or unintentional compromise of your registry, as well as it brings in image confidentiality. So you can have extremely confidential algorithms, like say, you might have a trained TensorFlow model, for example, which you do not want to be exposed to the outside world. You can argue that you can keep the registry password protected, but even in that case, the administrator, or the way the registry is hosted, storing the data has access to the contents of the store. So, and once you bring in encryption, it allows you to do a lot of interesting stuff. Like for example, you can sign the container images with the target systems public key in such a way that you can enforce that the container images that I'm building will run only on specific geographics. For example, you can say image, a particular image will only run inside the European Union's geographic borders. So, let's say how we can envision this. Now, I'd like to tell you that the work has been happening all the way from OCI spec and Kubernetes and anything in between. And we are talking to all these communities and the final output that you may say when everything falls in line, may or may not actually overlap with what I'm saying now, but it will be very similar to this. So, let's see how we were able to encrypt the container image. So, you have a Docker build command, let's say imagine and you should be able to pass a public key. So, let's say I want to be able to create a container image in such a way that only Brandon should be able to run it. So, I take his public key. During the Docker build process, I should be able to take his public key, encrypt it. And then this encrypted key, I will push it to registry. Once inside the registry, it gets interesting how you're going to use it or consume it within the context of Kubernetes. So, we all know Kubernetes has this concept of secrets, right, Kubernetes secrets is a way to store sensitive data within the confines of Kubernetes. So, and there's a very particular type of secret called image pool secret. This secret is used for fetching images that are behind the username password in the registry. So, we thought that we have a very similar use case here where instead of pulling the image, we want to decrypt the image on the host, right? So, we started modeling the secret that holds the private keys required to decrypt the images and we are calling them image decrypt secret. For this work, we already have submitted KEP and it's under discussion. As I mentioned, we already talked into OCH spec. There is a PR in container D to support it because you need to have the support all the way throughout the stack. So, once you have the secret, what the flow will look like, you know, if there's a request by the user to create an encrypted part, the image secret, image decrypt secret will basically provide the keys required and the part will create it with the keys that are required and those keys will be pushed down to the Kubernetes worker node. And then the worker node, the cubelet will receive those keys. That will be passed to CRI from CRI to container D and the container D will, while pulling the image, will also decrypt the image and then it follows the standard container D workflow. Let me quickly show a demo. Yeah, this is strange. So, let me just say it clear and then we go to container D. So, although I talked about Docker build, to begin with, we started playing with container D directly because that was easier to deal with and Docker build end of the day will end up in working something similar. So, let me see. As you can see, you only have one image right now, Nginx and then on this system, I have a sample GPG key pair and I'm gonna use that to encrypt the image first. So, what I'm gonna do is, I'm gonna say, take this image, encrypt it, take this Nginx image and encrypt it with the key that I'm asking to. So, it's gonna encrypt that image. All right, then let's list it and there you go. You have this encrypted image. So, just to understand things better, we added a small command in container D which helps you to see the images. So, let's zoom out a little bit. Yeah, so you see there is the image here which says encryption and recipients none. So, this is what you'll see for a regular container image right now that everyone is used to. But, if you try to use the layer info against the encrypted image, you have a GPG and the target system, target recipients address in that. So, it basically means that this image can only be decrypted with this target key that is showing. Let me just clear this. Let me just push this image, we lost it. Oh, okay, we have it. Let me just push the image to the local register data and now I'm gonna change it and this is the standard Kubernetes secret. It's a default token that we have here and then now, what happens, the image that we just encrypted, if we try to deploy it, let's say you are running this register, you upload it to the public register your encrypted image and someone tries to download and run it. So, let's see how it reacts to that. So, I do not have the private key at this point and I'm trying to run an encrypted image. Is this created? Let me just wait for a second. Yeah, so I got an error. Let's see what the error is. As you can see, you're not authorized to run this image and the missing private keys needed for decryption. So, think of it this way that if you had an encrypted image in a public repository like this, whether it is protected by password or not, if someone tries to run that image without having the private keys or without having the target system that is encrypted for, you are supposed to get, yeah, I think we're doing fine. We figured it out. So, you will get this error. So, let's see how we can run an encrypted image. I'll just delete this part. Okay. Remember that key, that key pair that I had? I had already extracted the private key from this GBPK key pair and I have converted into base 64. I'll create a secret. So, this is a secret that we are introducing in Kubernetes. So, I'm saying create a new secret type of image decrypt with this private key. So, when you get to secret, there you go. Now, you have a key secret. Let's go deeper in that. Yeah. So, it holds the decryption key, the base 64 key that we had. Clear. And you get parts. Let's see. So, we will modify our part YAML slightly. So, instead of having this, when you want to run this Nginus encrypted image, we'll have to provide the key that Kubernetes should pass down to the worker node. So, we'll say image decrypt secrets named key secret. The key secret is the one which we just created. Let's try to create this part. Okay. Clear. Get part. And it works. When I tried to run this image without the secret, you remember you got an error saying you're not authorized to run this image. But now I'm able to because I associated this part with the right secret. And that secret was passed to KubeLate and KubeLate passed to CRI, and CRI eventually gave it to Continuity. And that Continuity was able to extract, not only extract the entire image, but able to decrypt as well. Let's delete this. However, those of us know how to use the secrets a little in an elegant way. We know that we can use service accounts in Kubernetes to pass secrets. So we plan to support service accounts for image decrypt secrets as well. So let's see how we can achieve that. So we'll first patch our service account with the secret that we already created. And we'll say describe the service account. And if you see, this is a new thing that we're trying to add here. So now this service account has the image decrypt secrets. That means every pod that gets created with this service account will also have an access to the secret. Now this time if you see, I'm trying to start the same pod or deploy the same pod, but without any image decrypt secrets. And this time I do not have to have because this pod is gonna access a default service account. So let's see how it works. Create, and it's working, right? So I think that completes my demo quickly. And I jump back to the screen. So if you want to, thanks, Brandon. So if you want to know more, understand more about the work going on, please visit these links. We have a very good and detailed discussion going on on all topics. And we hope to get this merge very soon. And with that, I will dive deeper in that and I'll hand it over to my friend in. All right, thanks, Arshal. So, wait a minute, sorry. So what I'm gonna talk a little bit about is about our design in the encrypted content images and how it relates to deduplication because the conventional knowledge is that whenever you have encryption, you're gonna suffer in terms of deduplication because you can't, every time you encrypt something, it's gonna be different, right? But in the process of designing this, we also consider that this is one of the reasons why people like containers, it's a layer deduplication, the layering. And we're gonna show that in the way that we design it, deduplication is not necessarily thrown away once you have encryption. So I'm gonna go through a deep dive. I'm gonna start with a little bit of crypto background. So just bear with me a little bit. So we're gonna go through a short primal encryption. So we're using two kinds of encryption. We're using symmetric encryption and asymmetric encryption. So for those that may not be as familiar with encryption, what symmetric encryption is, is you have a piece of plain text. So a message you wanna encrypt and you have a key, which is a symmetric key. Now, in symmetric encryption, the key that you use to perform the encryption and decryption is the same. So this is good because this is, asymmetric encryption is generally really fast. It's good for really large blocks of data. Unfortunately, the issue with symmetric encryption is sharing this key over here proves difficult, right? So we also introduced asymmetric encryption. So asymmetric encryption has pretty much the same setup as that. Then instead of having one key, you have a key pair. So you have the public and private key pair. And so in this case, to encrypt the message, you would use the public key and the encrypted message will be decrypted by the private key. So the great thing about asymmetric encryption is the public key, as the name suggests, can be public. So this makes being able to authorize and encrypt the image to someone or to be able to share keys, it will be much simpler. Unfortunately, it's so. So what we do is very like most type of encrypted messages we think about encrypted email and things like that. We're kind of following the same kind of design concepts. So what we're doing is we're taking symmetric encryption and asymmetric encryption and getting the best of both worlds. So this will maybe familiar. We have our favorite people in crypto, Bob and Alice. And so in this case, Bob wants to send an encrypted image to Alice. No, we haven't forgotten about the duplication. We're getting there soon. All right, so Bob does a build and what it does is it generates a symmetric key. And this symmetric key is used to encrypt the image. So now we have an encrypted image. And what we wanna do is we wanna say that this is the image I want to encrypt for Alice, right? So we somehow have to get that yellow symmetric key over to Alice. And the way we do this is to take Alice's public key and create what we call a rep key. So this rep key you can think about as a message that contains the symmetric key as well as all the cryptographic metadata required to understand how to decrypt the image. So this rep key can only be decrypted by Alice and what Bob does to expose that image is to upload these two things into the registry. So on Alice's side, to be able to decrypt the image is pretty much the opposite of what we saw earlier. She takes the rep key that only she can open and decrypts it with her own private key which only she has access to. And from there she gets the symmetric key as well as the parameters to perform the decryption. And she successfully decrypts the image. So how does this relate to what the images actually look like? So if we dive deep into what changes we are looking at in the OCI spec, so to start off with what we have here is what the OCI spec looks, image spec looks like today, right? So the image spec at least in here you see the manifest is really some metadata about the image itself and each layer is basically a blob of a collection of files. And in this case, the files are just Tata, right? So it's a blob that's represented by the shark. So what we're doing and the design decision that we've made is to perform encryption on the layer level. So in this case, on top of performing the time, the gzip of the collection of files, we're doing the additional step in generating the symmetric key and performing the encryption on every layer. So the changes here to the spec are pretty straightforward. One is we've added a new media type. So we've added the plus ENC media type standing abbreviated for encryption. And what this means is it tells the runtime that this image is encrypted and it's gonna try and decrypt it. And we've also added additional field in the annotations here called the keys. And what these keys are are the wrap keys that we saw earlier. So the keys here could be a wrap key for Addis, a wrap key for Bob, and so on. So one interesting thing, so now we're back to deduplication, right? So the general thought of this is that by doing encryption on the layers, we are able to take advantage of a lot of container like design behaviors, right? So generally, if you have a container, your container may not have a couple layers. So you have like the operating system, you have the middleware like Python. And then usually if you have a sensitive piece of code, it's a very small piece of code that may be like trading algorithm or maybe your machine learning model or something, right? And so what it means when we do encryption on the layers is that we can benefit from this because the bottom layers which end up being the larger layers actually don't have to be encrypted at all. And so these layers can still be shared. You can encrypt just the topmost layer for a couple of images. So that's one. The second thing, which is why we talked a little bit about how we ended up doing the crypto is that let's say I wanna encrypt an image for myself on the encrypt image for Harshal and five other people, right? So traditionally what you would do is I would encrypt once for Harshal, I would encrypt once for myself, I would encrypt once for the five other people. But if we look at the spec, we can see here that the way we've done it is we split up the actual encrypted blob separate from the wrap key itself. So what this means is that if I wanna create an encrypted image for multiple people, I can have the same encrypted blob and just upload five different wrap keys for the five different people. So in this way, if I wanna be able to give authorization to additional parties to decrypt my image, I can do so without actually changing the hash of the blob in the registry. All right, so this is work that's currently ongoing. I think it's Harshal mentioned before. This is something that we started off in the OCI spec and it's made its way through the entire stack between OCI, container DE, Kubla and so on. So one thing is we're still trying to get this, we have a PR open for the OCI spec. And if you have any views or if you like this feature, do definitely chime in on that. We have ongoing work like the KAP that we just showed. We're also looking to integrate this to basically everything within the stack. So the build tools and create support as well. And finally, one thing that we're looking at also in the longterm is when we're talking about this encryption stuff, when we want features like geo-fencing execution, we want to really be able to tie this to cryptographic keys that we have higher assurance on. So this would be, how do I interface with TPMs? How do I interface with HSMs? Is that FIPS compliance within my crypto libraries within the entire process? So that is something that is still in the pipeline. All right, so yeah, that's all we had for today. So yeah, feel free to ask any questions. Okay, it might be a stupid one, but my question is we are solving one problem, but how do we manage those keys? What's the plan to manage? Are we integrating it with any key wall solutions or what kind of solutions we are integrating it with? So right now, the way that we've set this up is that the key management solutions not necessarily tied to the technology. So you can still be able to bootstrap your keys through any key management solution that you have, but with features that you're talking about. So you could use like vault or key protect or Azure key vault. But when you talk about being able to tie it more tightly with the keys, and this comes into, I think one of this here is like key management key wrapping services. So some key management systems actually allow you to perform key wrapping and unwrapping of the server. So the key never actually leaves the key management service. So in that sense that there's still a bit of work there to have a very tightly integrated solution. But right now, the way that we have it is that we still rely on the external process for that. We do have, so there are two models of this, right? One of the models that we just saw is that you're taking a secret and you're putting it in the Kubernetes secrets, right? So you will manage that secret the same way you will manage your image pool secret. There is another model that we have looked at and we have some prototypes on, which is that really my key management and authorization is tied to the host. So in this case, what we're doing is we have a prototype that actually injects this in the runtime. So there's validation within a key management service to the node, so it does attestation of the node to ensure that it's authorized to obtain the key. And this ties back to a rule of trust like TPM or something. And for that, we deliver the keys through the runtime. So yeah, there are a couple of models. Yeah, and to add into his, you can use KMS services with existing Kubernetes secrets. And this is another type of secret. So if you can tie up your Kubernetes to your key service provider, all your keys get, instead of they staying in the master, they get pushed to your choice of KMS. So similarly, this is not any different. This is just another type of secret. So this will transparently get attached to the KMS. Thank you. Can we decide which level of layer has to be encrypted and which does not need encryption? Do you mean, can you selectly do encryption? Yeah, you can. Okay. It doesn't have to be the topmost. It can be anything in between either. Yeah, one more question. How much more resource do you need to deal with the encryption or the encryption images? Or maybe the percentage? So we talked about overhead, right? Yeah, we're dealing with encryption and the description of the image and how much more resource you need. So this is something that we haven't measured in a while, but theoretically it shouldn't take up that much resource. So the way that with the implementation that we have uses a streaming type of cypher. So in terms of the encryption process, it's part of the pipeline. So although there can be like, it's the latency is very minimal, but the bandwidth is still gonna be about the same. Yeah. Maybe if you deploy the more images, it depends on, it will be using more of your source to deploy it, right? Yeah, but I... You need to do this only once when the first time image comes in because the continuity will extract it or any runtime will extract it and keep it on a host. So it's not something which gonna get executed every time you run the container. This is only first time the images has to be pulled on the host. And second thing is the way we have implemented is we next time there's a request comes to deploy an encrypted image, we do not download again, we can check against the existing extracted image and we got the image authorization. So there are two concerns. One is like the latency concern, which I don't think we're that worried about because the pool latency is still a lot more than the encryption latency. The other resource concern you may be worried about is maybe, you know, how much actual computation there is. And that's why like in our design, we, a bulk of the encryption is symmetric encryption, which isn't really that, it's pretty efficient. Yeah, thank you. Have one question is that, suppose you have this encrypted image and you put it in a registry, how are tools like, you know, vulnerability scanning going to work if it's encrypted? That's a good question. So we actually, we left out the slide. So there are two ways to do it, right? Well, actually there are three ways to do it. So, so there are two main ways that we kind of recommend. One of which is, you could actually, so you know, when you saw the rap keys earlier, you know, I created a rap key for others, I created a rap key for Harshal. So what you can do in the creation of the DevOps pipeline is that you can say that if I trust a certain service, for example, I have a vulnerability scanning service, right? So I say that this vulnerability scanning service has a particular key pair. So I can say that I'm going to encrypt a rap key for this service. So if I trust the service, the service that pulls the image should be able to decrypt it. The other way to do it is we're trying to see whether people are accepting to move the vulnerability scans before it goes to the registry, which there are advantages and disadvantages like it's trade off, yeah. But no matter both ways, we have models that would be able to do solve the issue. Just a quick question. So let's say I'm going to encrypt the image twice. Is it going to generate the same symmetric key or different symmetric key? Yes, okay. So if you're going to generate the same image twice, it's going to be a different symmetric key. Even for the same user? So okay, so what you can do so, let me... So what we do in this is that generally when you encrypt the image, when you create the rap keys, you're going to create one for yourself. So if you're going to be able to, if you want to add a recipient, let's say you want to add a new recipient, you don't have to re-encrypt the image. Basically what you're going to say is I want to add this recipient. So what it's going to do is it's going to decrypt your own rap key and then re-wrap a rap key for the new person. So if it detects that the layer is the same, I don't think we do probably to the extent of figuring out that the layer is the same and doing it. But if you have knowledge of that, you can specifically say I just want to re-wrap the key. Okay, so just following question. So is there any exploration for the key, for the rap key? The exploration for the key. Yeah, so I think that's something that's going to differ to the key management solutions. So yeah, that will have to include the process of your DevOps pipeline. Yeah. Thanks. Any more questions? All right, thank you very much.