 Next up, we have Christophe and Jakob speaking about confidential computing with Katka container. So, Vestegi, is yours ever fun? Hello, so I'm Christophe de Dinachin, I'm working for Red Hat on OpenShift Sandbox containers, which is the integration of Katka containers inside OpenShift, and we are just going to release the product with OpenShift container platform for 10. I'm also a grandfather for about one week now, so that's why I have a smile on my face. Jakob. Yeah, hi, my name is Jakob Nauke. I work in cloud development for Linux and OpenShift on IBM Z and Linux 1 at IBM. I'm also active in the Katka containers and confidential containers communities and Christophe is going to kick it off. So, let me wait for the slides to show. Yeah, when are we going to cover today? The first question, the problem statement is why should a host in the cloud be able to see your data? And that's from in particular for data in flight being processed in memory. So, a solution to that problem that is emerging is called confidential computing, and that includes encrypted virtualization that encrypts your memory on the fly. One aspect that impacts the way we have to deal with containers is in particular securing image downloads and attesting the workloads, which means essentially checking that you run what you think you are running and where you think you are running it. Now, this talk is a continuation of a talk I gave at defconn.us last summer, and so I'm going to give a bit of a progress update since then. And Jakob is going to give us, so he's going to talk about attestation and other technical aspects, but he's also going to give us a demo of confidential computing running on IBM Z. So, let me start with the problem statement. Can we trust the host? The answer, of course, is no. So, your containers run on a host, which is often managed by a third party, like, for instance, a cloud provider in the public cloud. And there are a number of sandboxing technologies that are used to isolate, for instance, the kernel from user space or create name spaces inside user space to isolate processes from one another. But all these sandboxing technologies only go one way. They protect the host from the containers. They don't go the other way around. So, the resources like CPU, memory, disks, networking, et cetera, belong to the host, which owns them and has practically free access. Your containers are really carved out of these both resources. Now, what do you need to do if you start considering the host as hostile? The problems that you can run into include, first, data exposure. That means the information held in the container can be seen by someone else. That makes it difficult to have multiple tenants on the same host because they may not want to share information or have confidentiality risk with other containers. At the limit, this can pose legal concerns that preclude the use of containers if you cannot legally guarantee your confidentiality. So, the first enabling technology to address that is called confidential computing. This is a sort of umbrella term that covers many vendor-specific technologies from AMD, Intel, IBM, which has them both on S390 and Power, ARM, et cetera. All these technologies really rely on virtualization, but they work in a different way. And so, a product like KALA containers has to really take these differences into account. Now, confidential computing is more than just encryption. Of course, there is memory encryption that prevents the host from getting valid data out of guest memory. In other words, if you read your guest, you'll get only junk. But that's not enough. Integrity production is another important aspect that guarantees the host cannot correct the guest state. It cannot override the registers or it cannot inject malicious interrupts. And finally, an important aspect is attestation, which lets the guest owner, that we also call tenants, validate what runs in the guest and where it runs. So, it knows that it's running on a trusted platform and it knows that the content it runs is the right one. The second enabling technology is KALA containers. So, I don't know who is familiar with KALA containers and I'm sorry for those of you who are. I'm waiting for the slide to show up. Okay. Now. And so, what KALA containers does is essentially run containers that are described the usual way. For instance, the same kind of manifests, the same kind of container images, the same storage setup, the same networking. But now these containers will run in virtual machines. There is one virtual machine per pod. And so that means they have their own independent kernel. And on top of that kernel, there is very little user space, essentially just something we call the KALA agents. So, why do you want to do that? The idea is to get the ecosystem of containers and that means rapid application development and deployment, while at the same time having the sandboxing of virtualization. Now, this was made possible because the original design for Kubernetes is quite modular. And so we have something called the container runtime interface that lets Kubernetes dispatch to various runtimes. And so we essentially added a runtime for KALA. And then you have other components that matter to us like CNI, which is a container networking interface and CSI, the container storage interface. So, we also need to deal with that. We're going to see that in a minute. Now, let me talk a bit about the basic architecture, which is essentially running KALA container VMs on top of a confidential host. Sorry, I'm waiting for the slides to show up each time. So, what we have to do is to separate the various trust realms into platform, tenant, and host. So, the trusted platform that is shown in red on the diagram offers confidentiality guarantees using typically hardware-level cryptographic enforcement. So, you can create trusted execution environments in it or the host environment will offer and manages all the resources that you need to run the containers that includes CPU, memory, IO, and so on. And finally, the tenancy includes everything in green on the diagram. And these are the confidential areas carved out of the host and that are hidden from the host or not visible nor accessible nor can be modified by it. But it also includes the tenancy also includes things that may run outside of the host that we generally call relying party and that include services such as key brokering or attestation. I'm sorry, I'm waiting for the slide again. So, in order to enable confidential computing for catapults there are a number of impacted components. The runtime itself, the virtual machine monitor, the kernel, the firmware, and the hardware. And all of these pieces currently are moving. There is still work going on, for instance, at the firmware or kernel level for many of the platforms that I described earlier. So, for the catapult runtime, of course, it starts with passing the right options to the virtual machine monitor. Then the virtual machine monitor will enable memory encryption and set up productive VM. That requires some services from the kernel in particular in the KVM component with low-level hardware support for technologies such as AMD, STV, or Intel TDX. And both guest and host firmware need to provide new special services, for instance, to validate the pages that you hand over to the test, these kind of things. And all that relies on hardware support like having encryption in a memory controller or having a separate processor to deal with cryptographic setup. So, well, the point here is that we have a lot of moving parts, and we have to deal with all of them at the same time. So, this is a relatively complicated and long transition, and that has been in progress for quite a while. And I'm still waiting for the slides. Sorry, so I'm speaking, okay. So, the transition to confidential containers started with enabling confidential computing when running virtual machines and doing a number of small API changes, for example, to make sure that the host cannot see the guest log or execute dangerous commands like Qubectl exec. So, we started that in a phase that we called CCV0, and I talked about this at the Defconn.us talk. So, I'll refer you to these slides if you're interested. Now, we entered another phase called CCV1 where we are going to focus on pulling images that needs to be done by the guest to use encrypted images and make sure that the host doesn't see the images themselves. So, that's small as in progress at the moment. We also started working on the build NCI because those need to be adjusted quite significantly to be able to exercise code. We'll see in a moment why. And we are also working on really defining the threat model and making sure that this is solid and documented. And there are a number of future topics that we're already working on or thinking about. I'm going to quickly brush over hot plug-in for CPU and memory being incompatible with attestation and also with a model for administration where the tenant and host have to be really separated. So, the step that matters here at the moment is image pulling. So, in order to secure the image download, instead of downloading the images on the host, which is the way it's done traditionally, we need to put the images from inside the guest and to store them on some kind of encrypted volume where only the guest has the encrypted keys. And so, this is possible because the kubelet in Kubernetes delegates the pool image operation to a service in the CI called image service. And during CCB-0, what we had is we had some sort of ad hoc image pool from within the guest, but that used something that was similar to TMPFS where we were essentially using or consuming guest memory, which is an expensive resource to store container images. So, the effort in CCB-1 is to streamline that aspect of things by developing a new component instead of re-leveraging existing ones called image rs, where we are going to implement encrypted image storage locally on a transient volume that we are going to use for that location. Now, in order to use the images that we are going to download here follow a standard for encrypted an OCI standard for encrypted images, and we need to adjust to this new build. I'm sorry, I'm filling up there because the time it takes to go to the next slide is longer than I anticipated. So, in order to build encrypted signed images, the first step that you see on the diagram here is more or less standard where your developer will build some OCI image and that image, you want to transform it in such a way that it cannot be used without a secret that is going to be provided later with mechanisms that Jacob is going to explain. Now, that OCI standard is still under development for encrypted image formats, so we are leveraging code that is a bit moving, but the important part here is that the build pipeline now really has two outputs. One that is shown in red here goes to an image registry for encrypted images and that's on entrusted infrastructure. So, you're producted by encryption essentially and that's the thing that will be downloaded by your pod later. The other side goes in green following the green rows here and it goes on entrusted infrastructure and that includes things like a key broker service, etc. So, the diagram is simplified here. I'm just showing that we blindly hand over the keys. In reality, the exchange is much more complicated and Jacob is going to explain that in a minute. Now, one aspect of confidential computing that I mentioned is integrity production and in order to explain the problem, you have to know that Calaclinus today relies on hot plug-in quite a bit. So, I'm still waiting for the slide to show up. I apologize for this delay. How long does it take for the slide to show up? Okay, now you have it. So, how do you configure your virtual machine? Well, today in Calac containers, we use hot plug-in to add memory, CPU or devices to the pod and this is the reason for that is illustrated on the diagram on the right where Kubernetes essentially sends us a pod creation request that tells us little about what is going to run inside the pod. And so, we start the virtual machine with a fixed configuration, for instance, two gigabytes of memory and one CPU. And then when we get the description of the containers that you want to run inside, then we'll basically grow that VM adding, for instance, 16 extra gigs of memory and eight more CPUs. Now, the problem is that integrity is quite hard to guarantee if you can change the configuration of your virtual machine at runtime. So, if you stick with existing APIs today, you would validate the small VM, but when you try to extend it, it would fail because it would change the hardware configuration that's not acceptable in a confidential computing setup. So, we are shifting towards something called immutable pods where we get extensions to the pod API so that we have a fully defined pod ahead of time before building the virtual machine and our containers have enough space and resources to run. Another, so, that's a bit of a future aspect. The API is being worked on at the moment, but that's a discussion with higher levels in the stack like Kubernetes, CIO, etc. So, another aspect, it's incredible how long it takes for the slides to show up. I clicked. Okay. So, the need for a shadow control plane is a future topic. It's something we started working on, but it's not going to be there in CCD1 at all. The problem is essentially that the tenants need to have some sort of isolated administrative realm to talk to their containers. For instance, to get the logs, to get container metrics, to execute processes inside the container, etc. All these kind of things are really happening on the green side there. And so, you have this sort of control plane in green that uses different credentials than the one on the left in red, which the same person also needs. So, these are host credentials. And you also need that to manage physical resources. So, like creating your pod and allocating things like raw disks or getting hardware-level metrics for the pod. So, this is essentially the long-term model we are trying to shift towards. But that, of course, requires a lot of implication from the whole ecosystem around us, not just Kata containers. That's not something we can do in Kata containers or confidential containers alone. And just to give you an example, when you use a Qtcatl command, depending on the kind of command that you run, you will need to go through the red or the green one. If you create a container, then you would have some steps going on the left. If you Qtcdl exec, then you would go on the right. Okay. So, the next part of the talk is going to be given by Jacob. While we shift to him presenting, I suggest that we take a few questions. If we don't have any questions, I think Jacob can take over right away. Okay. Then I'll take over right now, I guess. We can have some questions later. Maybe some more will come up. So, 18 minutes, I got plenty of time. Wow. So, yeah. One, again, my name is Jacob Nauke. I work in cloud development for Linux and OpenShift on IBM Z and Linux one at IBM. And I want to dig a little deeper on a testing that a workload is secure, which is a very central problem in confidential containers or confidential computing in general. I'm going to give a demo with a real world trust execution environment. And I'll talk about some more things that we have on the future outlook. So, the central problem really that you have to solve in confidential computing is making sure that the workload that you run is really running in a secure context. Like these platforms, they all encrypt or protect memory and CPU state in some way, and that's cool. But it's really only half the story because when you as a tenant, as a workload owner, want to run something, you have to make sure that it was really started under that context. So, if you were connecting to some allegedly secure domain at a cloud provider, for instance, you can't just take their word for it that they're running it with the trust execution environment. And that is where various methods of attestation come in. So, there are pretty much two ways that we do this. One is encrypting the boot image such that only the firmware can decrypt it in that secure context. And another way is remote attestation that happens at runtime or just before it. So, these are these two workflows. In attestation, critical parts of the workloads are measured. And after a successful measurement, the relying party which has to be active at runtime can establish a secure channel into that confidential guest and provide any secrets that could then be used to, I don't know, connect to some other endpoint or whatever, but any secret that you can't put into the boot image unencrypted. And the other approach for ensuring that everything is running on tampered is image encryption, where usually the direct boot components like kernel and dndrd in command line are encrypted asymmetrically such that only the firmware has the the private key to decrypt and launch that boot image and such that it can only be read from in a secure context. So, that's where the security derives from with the image encryption workflow that simply the firmware keys can only be read by firmware. And if we were, for instance, running attestation with the CataRagent, you would have the environment host firmware and kernel and VMM such as QEMU, and you'd have your confidential VM with its firmware kernel and an init rd that contains the CataRagent. And the stack below the confidential VM would be measured by firmware and the relying party could attest that this setup hasn't been tampered with and then it can inject further secrets. For instance, when an image is pulled in then an encrypted image, it can decrypt that image and provide the key for it. There are more or less two ways of running attestation. One is pre-attestation where the VM is measured before starting it and the other is attestation that happens completely at runtime where the VM is measured not only at start, but can also be measured down the road depending on what exactly you can do with these measurements. Pre-attestation is used, for instance, by first and second generation AMD SEV and SEV ES. Complete runtime attestation is used by Intel's upcoming TDX or the third generation AMD SEV SNP. And one nice thing about remote runtime attestation is that a vulnerable image can be invalidated if there were some known vulnerability down the road, at least as far as that measurement goes. In the boot image encryption based workflow, there would be some secret pre-provisioned into the init.rd which can then be used to bootstrap other secrets like it might be a key for decrypting container images with Cata containers. But yes, this is pretty much the alternative to attestation where the guest image is encrypted asymmetrically and it's ensured by firmware that can only be decrypted in these trusted execution environments. One nice property of the encryption based workflow is that the relying party does not need to be active at runtime, so you could run this basically fully offline if you wanted. And this encryption based workflow is, for instance, what IBM's secure execution or power protected execution facility do. And now I've introduced a couple of workflows. They differ in more fundamental and less fundamental ways. They're all sort of vendor dependent and obviously we didn't want to ride confidential containers from start to finish for each of these technologies, which is why we went with a modularized approach with a confidential container's own component that we call the attestation agent, which is another binary that exists next to the Cata agent or other agents hypothetically inside the guest init.rd. And it, like I said, it uses modules that support these varying methods of attestation and encryption to provide some secrets at runtime, for instance, for decrypting container images. So the Cata agent and the attestation agent with varying modules, depending on the confidential computing technologies. So from pretty much from start to finish, what we have now, that would be you start a confidential container with Cubelet. So yeah, these are all standard LCI components pretty much. And you would then use a container runtime interface, runtime like container D. We are currently running some custom modifications with container D, especially because of the guest image pool. That's something that derives from the regular workflow, which is why we're using some modifications there. But there are efforts to bring those modifications or at least functionally similar modifications back to upstream. And container D would then talk to Shim, a runtime host binary, if you will. So this is similar in functionality to Run C or C run, if you're aware of those, but it's the Cata Shim. And that will launch a confidential VM with the boot image it has, for instance, via QEMU. And in the inside the confidential VM, the guest component, the Cata agent, is started. And the Shim, the runtime Shim, can then talk to the agent over a restricted API that Kristoff alluded to, or at least in confidential containers, it's restricted. And yeah, the confidential containers compatible Cata agent can then pull an image from a container image registry, which for confidentiality would have to be encrypted. So this is a sort of standardized way of encrypting container images. And it would then retrieve a key from the attestation agent that I mentioned, and the attestation agent would get that key from the relying party, depending on technology at runtime or ahead of time. But if such technologies are used, the guest is measured and the secret key is provisioned into the guest. And then you can run a container. It's unpacked, we're currently using Umochi for repacking. And then you're running a confidential VM that functions as a container part, basically. And since I'd like to show a bit of what is already working, I'd like to demo this with a real trusted execution environment, which is IBM Secure Execution for Linux, incidentally the platform that I work for, which is for IBM Z and Linux one. In kernel terms, that's the S3 NEX architecture, traditionally known as the main frame. But yeah, we can use Linux and OpenShift and such technologies on IBM Z. And like I alluded to before, Secure Execution uses the encrypted image workflow. I'd like to present a little how that works in general before we get to confidential containers. So in a traditional Secure Execution-based workflow, you would have some existing VM on trusted hardware with QEMU and KVM. You would have InitRD kernel command line and a normal Linux root FS encrypted with LUX. And the InitRD would have the key to that LUX so it can launch without any interaction. And what you would then do as a secure guest, workload owner, is you would generate a protected, an encrypted image out of that InitRD and kernel command line. And that gives you a secure execution compatible image. And that is encrypted with a public key and the private key is serial number specific to the host and can only be read in a context that's already secure. And that's how that image is decrypted and then you can also decrypt the root FS and everything. And yeah, the header is also integrity protected and supports several public keys if you wanted to run one image on several machines, for instance. These things are all possible. And so yeah, Secure Execution is basically one platform that sort of sidesteps the need for runtime at a station. There is no component required at runtime here, no external component. So the direct boot components are just encrypted asymmetrically and the private keys for that are in firmware. One challenge that this imposes obviously is that you can't just make these host keys, these private host keys, readable to the hypervisor or really to software in general because if it were readable to them, then it would be readable to a rogue cloud provider, for instance, and then you haven't really gained anything. And that is where in Secure Execution, the so-called ultravisor comes in. The ultravisor is implemented in hardware and firmware and manages the guest memory pages. So the guest memory cannot be read by the hypervisor, cannot be read by KVM, cannot be read by other processes, or also one confidential guest to the next. That would of course also be important if you were running multi-tenant. So to get a little into what ultravisor and hypervisor do, hypervisor will reboot to the ultravisor once it detects a Secure Execution kernel. And from then on out, the ultravisor will ensure protection of memory in state, which means that memory pages are encrypted when they're swapped out or dumped, and when they're swapped back in, they're also checked for integrity. And IO and scheduling are still hypervisor managed, so especially IO should still be encrypted, you should still use address encryption with the LUX that I showed, for instance, or Secure Network connections, obviously. Memory is in the current generation, not physically encrypted, so it's not encrypted on the die, but it is encrypted when read through software. But for the next generation, CPU for Xeon Linux 1, memory encryption has been announced. And one challenge that we face in Secure Execution that is also partially phased by other technologies, which Kristoff alluded to, is that we don't really have runtime memory updates as of now, so S390X is not a DIM-based architecture, so no DIM-hot swapping. VertIMM support is incomplete, and Secure Execution does also not have ballooning support yet. But anyway, to get back to confidential containers, how we integrate this with the attestation agents of this component that supports various technologies, we currently use what I like to call the bake-in approach, which is pretty much akin to the LUX workflow that I showed earlier. So for encrypted container image, there are just offline keys inside the guest encrypted by the host key, which can then be used to decrypt container images. This is pretty simple, because it doesn't require any external components, and runtime would also work air-gapped. Because that's what's working right now, I'd like to demo that in a quick recording, because I'm not taking any risks. So let me switch the sharing here to that video here. So what we've got here is we're on a Secure Execution capable node, and we have a normal Kubernetes deployment YAML file, so yeah, pretty much standard components. And we're using Cata as a runtime, and we're specifying this Docker IO Cata Docker CCV0 SSH image for S390X to be used as an image, the confidential containers version 0 SSH demo. And this image is encrypted, which is important because it contains a private host key. And that private SSH host key, and if you can't launch it with Docker or anything, you can also check that that image is really encrypted. So in container D, we configure Cata as a runtime, and we configure a configuration for Cata, which among other things has that Secure Boot image, that protected image that I mentioned before specified as kernel, so it doesn't only contain the kernel, it also contains the init.rd with Cata. And now as we apply this YAML file and start the pod, it takes a short while because the image has to be pulled, but once it's there, now it's there. When we run a kubectl exec, for instance, this is where the restricted API comes in. If I were rogue admin, I could do some evil things here perhaps, but as you can see, this request is blocked, which is important for ensuring confidentiality. And what you would do instead to connect to this guest is you would use SSH with the private key. And now as I set a secret in there, I'm going to set it to very secret data. And I said I'm going to run a grep on the confidential data that I just set on a pipe. And we're going to run a Socat to QMP, the Kimu socket. And we're going to run dump guest memory to that pipe. And what will happen now is that the ultravisor will give us all these pages, but they're going to be encrypted. And that is why the confidential data will not be seen from that grep. So that is basically a small demo of the confidentiality, because if I did this with a regular VM-based container, this data would show up here. So to get back to the slides, I'd like to say a few things about ongoing and future work, because we are still at the zero with confidential containers. First of all, secure execution and the attestation agent, I'd like to increase flexibility a little by using a TLS certificate inside the guest and bootstrap a secure connection to the relying party at runtime. So you lose the air gap functionality, but you can, for instance, provide more guests, keys, more content in their decryption keys or certificates at runtime as you please. I'd also like to find some alternatives to native end-to-end tenant builds, especially for secure execution. So on the left side here, you have to build the guest end-to-end right now, and this might especially be slightly annoying if you need trusted IBM ZE or Linux 1 hardware just to do that. So cross compilation would be nicer there, so that a native instance is not required, and I've begun work on this and some things work, but that's still under construction. And then on the right side, it's a more general, probably business-focused topic that would be getting provisioned builds. So you might not trust your cloud provider, for instance, to never tamper at runtime, but you might trust them to build the correct image for you, or you might trust a distribution to build the correct image for you, and if you could provide secrets throughout East Asia, for instance, at runtime, then that would also be an option so that you as a workload owner wouldn't have to build so many things, and we could also probably increase trust with reproducible builds there. So just to reiterate a little on the attestation process that I showed earlier, the enhancements that we want to do when I'm going to switch back and forth here are supporting container D and cryo on the left. So cryo, we're not supporting at this time in container D has some custom modifications. Those are still to be rolled back upstream. Then the image services inside the CatterAgent should be enhanced. We want to skip using Umochi for repacking because we're not really using a huge part of it, and we're going to use a fully rustified image workflow there. The container images that we're running right now are not authenticated yet, so authentication and certificates are still an area that we're discussing right now, which is going to be especially interesting for running unencrypted sidecars that are merely signed, and we would also like to enhance confidential containers with ephemeral block devices that would also have to be encrypted, of course, so that, for instance, this might expand into ways of using less memory or to reusing downloaded images so they don't have to be downloaded every time. So yeah, these are the things we're looking into. We're on GitHub, we're on Slack, you can join us, you can test this if you have a TE, you can test this with secure execution, AMD, SCV first gen, EAA from Inclaver containers, Alibaba. We also have a generic demo that you can use just to try it out. It won't be confidential if you don't have a real TE, but just to take an idea of the workflow. But yeah, otherwise, thank you for the attention and we're now open for questions. Oh, yes, thank you. Yes, besides the Slack and GitHub, we also have regular meetings right now with confidential containers. These are on Thursdays and I'm going to post a link. François, great presentation. Is there a dependency in the architecture on the use of TPM? No, generally not. So TPM is another trusted computing technology that doesn't really work in the same way that we discussed. The runtime attestation methods depend on having secure platforms inside the guest, like the AMD platform security processor or the Intel management engine, as far as I'm aware. The TPM is more or less orthogonal. You can also achieve specific guarantees about remote attestation. There is the KeyLime project, for instance, but that's not what we cover in confidential containers, which is specifically confidential VMs. A good question. There are some similarities in the, now I have the code game. There are quite some similarities regarding the way measurements are performed and the general principles. And also, it's quite likely that if you want to ensure some kind of confidentiality when you deploy that on real hardware, you probably use the TPM on the host, but that's completely something that the hardware owner would have to deal with. I think you talked about TMA, right? Yes. So the device mapper also has an attestation process being put in place. This solves a different problem. However, we are discussing with the developers to see if there is an opportunity, for instance, to create an image download mechanism that would take advantage of that. And for instance, could allow us to have nice properties in terms of having an image that is compressed, can be downloaded in a sparse way. In other words, you only download the PC that you actually use and where each block is attested and signed. So these are properties that you can get from the device mapper. However, this is completely independent and actually incompatible with the OCI image format. So whether we actually take advantage of that in the long run remains to be seen, but at least there are discussions ongoing. So there's a question about if you encrypt the image, is there still value in signing the image? So yes, there is still value in verifying the integrity of the whole image. The actually happens layer by layer and per layer key. So you also need to validate the integrity of the whole package, making sure that, for instance, there's not a missing layer somewhere in the middle. And the other reason is that we may want to include in the container image a few layers that are not encrypted because that completely stand out. So if you raise on top of something that is just a basic distro or something like that, these layers are not encrypted but you want to sign them. Yeah, sidecars are another use case for this. Like maybe you even have just a single layer encrypted image that contains some secret and that's cool. But you might want to run a sidecar that is just signed but not encrypted because it's a standard component. And then you would want to check that for validity so that no one can launch an unauthorized sidecar in your TE that would be not so great. And that's something that we're working on. Sorry, there is a question. I mentioned that we are discussing the whole threat model. And one of the discussions that came up is whether we can actually provide sufficient security if we have a signed container image when none of the layers is encrypted. And the reason is because we set some part of our security model rely on actually having to use some secrets. And if you don't use any threat, maybe there is a weakness that we're still exploring that. Thank you very questions, François. We're officially at time. If there is a quick question, maybe we can still fit that in. Otherwise, I'd like to thank you very questions and thank you for your attention. Thanks a lot. Thank you from our side too. This is a very interesting topic and I think really up to date with the security, especially in the container world. Thank you, Christophe. Thank you, Jacob. I see that you already shared some contact if someone would like to keep moving forward with you, your teams. And just in case, I think you are going to move to Discord if someone wants to move forward. So I wish you a good rest of the day, a great weekend, and see you soon.