 So I'm very happy to be here today this morning to talk about Confidential computing and containers So my name is Magnus. I've been doing working with containers for many years and Confidential computing is something I started roughly a year and a half ago and In my current role at Microsoft as software engineer, I'm working in the Azure core Linux team and we Are working on Reconciling open source projects with the solutions that Azure provides in this space So the first question I think We need to discuss is why confidential computing Essentially confidential computing is a CPU aided encryption technology that enables Us to run applications that process Sensitive data with trusted execution environment will talk later about what this means a term that often is associated with this is Enclave so it's pretty much The idea that you isolate something from a larger computing context and Confidential computing in this sense is relying on a physical root of trust That is an authoritative source Which guarantees the confidentiality and the integrity of a Given workloads that you apply in a trusted execution environment and when we talk about integrity This means no one has tempered with your data and confidentiality basically means that no one Is eavesdropping on your computations? The whole thing is motivated by a couple of use cases, I think obvious one is that it enhances data privacy and Gives us guarantees about integrity. I think especially in Europe We're very familiar with regulations like GDPR. So to be compliant with those regulations in the cloud It's not trivial and confidential computing is a technology that aims to enable this Furthermore, there's a couple of specific use cases that Involve for example, I don't shipping large language models in a kind of black box for a customer Multi-party computations you think about Supply chains that involves several parties and to Perform computations on this data. You don't want to necessarily share with your partners Everything in your supply chain and for me personally, there's an example I've been working in the mobility industry for quite some time and In the electronic ticketing systems and information systems. There's a huge amount of data that is produced and This is potentially very useful in planning for utilization for vehicles and bus routes and Enhancing services for the customers in general, but it's very hard In Germany at least to use this data because this is from the nature of this data is very sensitive. You essentially have a be Travel patterns from the individual users of the mobility services So in essence this meant often this data would be left in data lakes and Perched After a while because it was very cumbersome to anonymize them or pseudonymize and nice the data in a way that you can retrieve insights and Confidential enclaves could be a way to leverage this kind of Data treasure to say we are actually not interested who the actual user is and we are able to perform some joints on tables but it's not we as The ones who deploy the software don't necessarily have access To those joint information, but only the insights that are derived from this The confidential computing principles. I think I mean you can slice it in different ways what I would point out as key Elements are essentially the encryption of data and use the the principle of measurements and remote attestation and I think we Probably want to go into detail on each of those So I think the first one is a graph that folks that Have seen talks in this space probably have seen a lot of Slides that look like this essentially The point is that we have like three pillars that would be a data address data entranted and data in use and Data address is for example, I know that the data that is encrypted in your Database Data entranted is TLS connections between our services and those two have been today Put into practice and pretty much all important deployments and are pretty Comfortable to use these days what remains is a data and use That is a harder problem. This is pretty much the The CPU is processing our data and in this process if you think about the cloud provider An administrator who has access to the hypervisor could extract data from the Guest virtual machines that are spawned and this is like not just memory. There's also CPU registers. There's caches and essentially this whole block of Computation is not guarded today. It's it's a very tricky problem and Confidential computing aims to close this gap. So again, yes Trivial chart of what is meant by securing data and use So it's not pinned to a certain CPU or certain cache It's really has to build into the processor to I know when doing interrupts evict certain data, so it's not accessible to neighbors or cloud hosters and as I said like it could be Extracted by platform owners, but also just malicious actors that hack the platform And the mitigation is to isolate The memory and all the CPU states from the rest of the platform and for this there has to be a piece of hardware That performs this and this we can call a secure processor So the next pillar Would be the measurements so measurements is essentially A term that means we Are hashing a bunch of code because hashing is a One-way function so it's virtually impossible to to invert this and produce a piece of software that is matching this hash and if we look at a Simplified Linux boot model we can see How a chain of events can be measured so if we start at the hardware We are loading the firmware and we perform a measurement for this one And the firmware then as an expectation about the next stage Measures the bootloader The bootloader again measures the kernel and the kernel measures the initial RAM disk and eventually we are at the operating system at process ID one and Instead of measuring all those components individually There's this concept of a measurement extension Because if you see in this chart Here we produce a measurement for a confidential virtual machine. It's just one long Hash and this is an extended measurement. This means we take an old hash of the prior step and hashes with a new hash and together this is Yeah, the next hash stage so we get an Event list more or less That you cannot reproduce if you I don't change the order or anything Then there's remote attestation As the last concept to introduce and This is also a key concept of confidential computing and I think this one also has Received a bit of press recently not the good kind Because essentially The idea is that You have a piece of hardware on your machine that you basically don't control and this machine is able to or this piece of hardware is able to make statements about your system that can be transferred to remote service and the remote service can then Yeah release or allow a certain workload to proceed and this is I think if you have a laptop this could be I know and an ad blocker installation that you cannot visit the website and unless you have your Hardware claims that you are you have an unmodified installation of of a browser and this I think is a Pretty valid concern been raised that this is potentially removing freedom from users Arguably in this case we are not the owners of those machines So remote attestation confidential computing is not that scary or it's actually Makes a lot of sense because essentially I think like people say it's not Your VM that's running is essentially someone else's computer in the cloud that you're running on and you don't necessarily Can trust it So if we cut out the platform provider and say we trust the CPU vendor and we trust Remote party Then we can establish trust without having to trust all the Components in the middle and this one has also been standardized in an RFC. That is Surprisingly readable. It's called the rats remote attestation RFC and In this chart we can see How in general this principle works so it's this piece of hardware and the TPM or secure co-processor that can be used to retrieve evidence from the platform and This contains the measurements we've been talking about and The measurements extend also not just the kernel and the in it are D and firmware But also the whole trusted computing base and the trusted computing base is a concept that basically means Whatever we need to run our workload And Our workload is considered the trusted computing base and this one we want to keep Rather small because if you have add more components, we have to do more measurements. We have to Trust a bigger surface So in this chart we quickly see how this trust establishment Could work with remote attestation. So it's a bit simplified For brevity, but in principle it works like this A user program requests a report about the state of the system The secure processor returns a cryptographically signed evidence The user program then passes this evidence to a relying party Which is holding secrets And the relying party in the background passes it on To a verifier which makes sure that the report is actually the signatures on the report are actually genuine and It also compares the evidence Claims like operating system version kernel version etc With reference values it keeps And then if all this validation passes the trust is established and The relying party can provide a secret to our user program. So that's essentially Confidential computing so I hope that made a bit of sense and now I'm gonna Talk about Confidential containers so confidential containers We think is interesting because the technology is not trivial to deploy and The aim of the project is to democratize the use of confidential computing Essentially, we hope to leverage the The platform and abstractions that exist in the cloud native space By building on top of them and making it possible for customers to Take their workloads and deploy them Into a Te with very minimal or even no modifications to their existing code base So if you look at this This is pretty much what we aim for and what exists today. So you have essentially at a runtime class to your deployment definition or pod definition and This should be more or less the interface for the user So if I want to deploy something in a confidential enclave, I'll add the given runtime class and ideally The rest is covered by administrator and cloud service providers and How we do a reconciled Containers with Confidential computing It's essentially that we have to deal with confidential virtual machines because the technology that I've Briefly shown before it essentially these days is a technology that's based on virtual machine Technology Before there were or there still exists a process-based confidential computing Implementations, but they're not exactly Or they don't work without Modifications to your code base and the adoption hasn't been that great so far. So what? the CPU vendors essentially adopted is virtualization as a kind of Isolation layer to build confidential computing on With a premise to take your existing applications and to minimal or no adjustments So pretty much what we Haven't mined for confidential containers Examples of this would be in these SEV technologies Intel came out with TDX Recently arm realms are also something like this and IBM secure execution is One specific implementation on the IBM site It has some implications that are tricky because this means we have discrete trust domains for CVM hosts and guests so The CVM guest is actually more like a tenant on on the host now because the host is being Deprivalaged so to prevent accessing or tempering with a guest's memory We have to introduce certain boundaries and that's tricky to consider in your Architecture as we will see I think in the upcoming slide. Yeah So we'll have The problem that we have to reconcile virtual machines and containers first and If we take a brief look at a very simplified version of what Kubernetes is it's essentially a Cluster that hosts a control plane that has multiple nodes and unit of Computation is a set of containers that are co-located you could say so this is more or less the Birds-eye view of Kubernetes and To reconciles with confidential VMs we have the following options so the first option that is pretty Obvious is we can just consider Kubernetes as a kind of confidential application in a single TE so we have multiple nodes we Established trust between them and then then we can bootstrap a cluster and just Consolidate everything In a confidential cluster this is I think has the upside of being very simple model The downside I would say is that you cannot get this managed from your Cloud service provider as many customers want and you also have a rather large TCB so a lot of software Surface is in your trusted computing base there's the other option of We can just create nodes on CVMs like have confidential node pools This looks nice at start, but it also has a problem that the control plane Pretty much also controls our notes. So we have the cube let deployed we have demon sets and those are all items that are not That we don't consider Automatically to be trusted There's the other option of saying okay discard Pots and nodes and just deploy container groups in CVM. So solutions that are outside of Kubernetes That works pretty well the problem is that you miss out on all the Kubernetes integrations and the ecosystem and It might work for smaller deployments for larger deployments is often a requirement to have a Kubernetes interface and Finally the last option is pretty much to consider pots as the Kind of units that we want to wrap a confident into a confidential layer and This is what confidential containers is doing. So we want to have pots in Confidential VMs and this the project of confidential containers It implements confidential pots. It's a CNCF project. It's pretty Collaborative at this point There's multiple CPU vendors CSPs Linux vendors and security companies involved to all kind of create this experience and The project is mostly implemented in Golang and Russ Golang on the more Kubernetes heavy side rust more on the upper levels and Cata side I would say on the virtual machine layer, but there's I think in an Equal amount of Golang and Russ in the project if we go through the individual components We can first maybe cover Cata containers Because Cata containers has been picked as a solution that has been used to isolate pots in VMs for quite some time and It's rather mature and proven so there's multiple major versions In released it's been tested in larger deployments. It supports multiple VMM backends and It's a natural pick pretty much to do this with Cata containers and We have a Kubernetes Component called the operator which Equips the nodes with necessary Cata container and binaries and configuration bits And if you quickly look at the architecture of Cata It's not trivial. I think that's due to the general container stack, but essentially Cata is an implementation of the Shim Runtime v2 API from container D so it means that We can more or less transparently deploy container sandboxes with container D and Have Cata spawn VMs For us So like we see here, we have a pot sandbox. It's actually in a VM and those VMs are really lightweight so the simplest case is really you have a kernel and you have the Cata agent and the Cata agent is the initial RAM disk program pretty much now, so there's not a big OS or anything and Cata is facilitating a lot of resources from the host to this guest like the container images Devices that need to be accessed shared volumes from the host and This is a challenge for confidential VMs because This part is untrusted While only this part is trusted We have another challenge with Cata that Cata essentially as a creates VMs. We need to Have the option to Either run on bare metal nodes, which are quite costly often or not even available Or we have To use nested virtualization, so this means like VMs that are able to host VMs. This is possible was for example AMD's Technology in principle, but it's very very hard to get things like device pass through and It also has certain performance costs. So this as an option is definitely Attractive and viable, but it's it's rather hard and not exactly available today What we can do, however, is there's a Cata implementation of a remote hypervisor, which instead of spawning a VM locally you can also Use a remote VM so instead of spawning VM via QEMO you spawn VM via your cloud API and this sits next to your Pot VM in quotes And this one In Cocoa we implement via cloud API adapter So the principle is as I described we have a Cloud API adapter as an application that is managing spawning and removal of VMs that host the pots we don't have the nested virtualization texts in terms of performance, but definitely the startup time is at least the startup time plus the Time it takes to spawn a cloud VM, which can be around 30 seconds or 45 seconds depending on circumstances So we see here if we start a such a pot we pretty much have a virtual pot on our deployment on our Kubernetes deployment and then CVM gets spawned that hosts the Cata agent and the forwarder service as system services A communication channel is established and the pot sandbox is started In the CVM and from the user side it should ideally be transparent The next component want to cover is the attestation agent This is the one that's performing the measurements that we talked about earlier so it's a key broker client In the project lingo, it's an extension to Cata agent in the trusted execution environment And we gather platform evidence in a kind of unified fashion because this is very different from How it works in different CPU architectures and even cloud providers Do it differently So it's a unified interface To facilitate remote attestation and trigger a secure key release to a workload we have the Attestation service, which is a component that essentially Performs the verification that we seen earlier. So the evidence that we gathered needs to be verified and compared against reference values and for this at the moment there's Regal policies and open policy agent to Make sure values are in a certain threshold if you think maybe of a container image That's you're only supposed to use I know version upwards 2.0. So this would be an example for an SEV S&P attestation report verification Which involves usually a lot of signatures and crypto operations so first we make sure that hardware key in the secure processor has actually signed the report Then we have to make sure that AMD signing key has signed the hardware key and Finally, we have to make sure that AMD root key has actually signed the signing key. So there's a lot of those Verifications in there that essentially and give us some give us confidence that Whoever produced this evidence is running in a trusted execution environment, so You cannot get this report By other means and as the last component we have KBS which is a key broker service. That's essentially A service that guards secrets. It's called a relying party in the reds terminology It could be integrated with the KMS but for simplification we could just consider it to be like as a secret store and An attestor so our user workload provides it with evidence in this Chart we see the relying party hands it off to the verifier and depending on the result. It will release a secret and maybe decrypt container image or give a provide a runtime secret and In the attestation agent has a module that implements the interactions with the KBS and You can at the moment retrieve certain secrets. So as I mentioned image layer decryption keys Are something that Makes a lot of sense also application secrets at pot runtime. So you maybe want to get a pytorch image that Has no secrets in it, but you want to Process secret data with it so to download it from I don't know storage account you need Pre-signed HTTP URL or something like this then there's credentials for the image registry and image signatures and the last two are a bit Controversial because they're not exactly secrets that are Really part of the confidential enclave but some argue that it's pretty much a function of the infrastructure to provide access to image Registry and image signatures essentially are no secret and this has certain Security implications and so they probably might be removed in future revisions the attestation protocol That's the cckbc client implements. We can see here. It's The workload is via some REST interface is talking to a KBS and Asked for a secret or asked to be authenticated and the KBS answers with the challenge so the challenge is also an important part because this means We are guarded against replay attacks. So it's really just for this session the attestation is valid because this challenge that we received from the server we have to put into a the report request when we go to the secure processor we can add a certain nonce or report data and this will be Put into the report and part of it and part of the verification would be is this actually Only valid for this session So in this way we can establish a secure channel between the KBS and the workload and We now have established trust so and I think We could have time for a demo so let's Quickly so what this looks like If you have a KBS deployment that's running in cluster essentially it could be under your desk or at a different cloud provider but For demo sake it's running in our cluster we have the Encrypted container image Pardon that unfortunate there Okay, so Not much missed. I think here. So we have essentially what we've seen before There's a cut a remote one-time class. There's an encrypted image and And We should Yeah, so we've seen the cloud API adapter logs Now a virtual machine like a pure pod VM has been started we can Look into the image that we have deployed and if you look down here, we have certain annotations on the layers That give us details So if you there's a certain key identifier that's associated with the image and Metadata and this means for our Confidential containers deployment it encounters this image and it tries to go to KBS performs a remote attestation these am I actually in a Confidential VM and only if this is the case then provide you with the symmetric key to Decrypt the image So and we've seen it's been successful. So it's running And what we can do is We can look at the logs So and We pretty much See a lot of data That is the exchange that we had before so the The attestation agent triggered and attestation Requesting the the resource and then it provided the evidence so the evidence here is Probably raw bytes. There's a bunch of certificates that are included And then This attestation is successful and we are Yeah able to retrieve this key and this way we We know that the Our deployment has actually passed the stages of attestation that we've seen before Exactly, so if you go back to the Presentation I have to quickly check the time There's also a longer recording how you encrypt images on the slides if you're interested. It's actually not Too hard But there's definitely some remaining Challenges that we have in the project and the first one I think is Secure image pulling on the host That's what I mentioned before If the host is kind of putting data into the TE we have Issues and we have to do it in a secure fashion and this means we have to use certain Features of for example container D like remote snap shutters that We still are able to share data from the host to the guest Maybe via blocks for example, so not to directly Sharing Directories or unpacked Image layers This means we can also Get a lot of speed up because if you Imagine like a spawn to spawn like 10 Replicas of a of a pot each one of those have to Pull an image at the moment and this is quite costly and Quite wasteful and also adds to the startup time and also we need some sort of scratch space secure scratch space like memory or Encrypted disk where actually those contents are Can be buffered so it's not pretty But there are solutions or I think in the next release even there will be a solution for this We have the Harrier problem of a Kubernetes control plane which we have to do privilege and Because the cluster admin is not supposed to Interfere with confidential pots, which makes certain assumptions the Kubernetes code base I'm not trivial to to convert to the confidential setup, but it's also a solvable problem. I think There's like dynamic properties for example config maps that are auto propagating In Kubernetes so this is very convenient if you want to update your engine X config for a confidential Pots this means we have a state that has been measured and then suddenly from outside It gets overwritten so it's actually not valid anymore Do we have to do re at the station? Do we have to do or stop the pot? Do we have to block this so this has to be figured out properly and And finally all those things have to be measured properly this is not trivial and For my experience there's not a lot of resources on how to do this trivial like parsing TPM event logs This really an area where there's not a lot of Pages on stack overflow where you can simply Figure out how this works, but often you have to read the spec and look at implementations and more That would be it from my side and if there's questions, I'd be happy to take them There's one so when you use a cloud VM with get back containers for the network side Is that completely handled by the pod for the pod for water? Yeah, that's essentially a protocol I think it's the excellent tunnel is secure one that is negotiated between The forwarder and the remote Yeah, the Qatar remote Hypervisor, so it means that the cluster could be on premises or in one cloud and the VM could be somewhere else entirely Yeah, I mean that's there's certainly also networking Questions about this like do you want to have performant network? Do you want to have? What is for example the the metadata service you want to associate with this VM so there's subtleties that are hard, but I Think in general This would work. Yeah Thanks I have a little question is what is the the performance Using these Segregation on the containers is an impact on that performance because I think Yeah, this is I think Interesting question because the CPU vendors I think are pretty shy about Publishing so if you ask certain people they will tell you it's 2% Essentially of what yeah, so all of this for example means we have assumptions like To trust the the the disk for example So we have to do is like verification online verification the invariative file systems on the nooks those have a performance impact if you just look at the CPU, maybe it's rather cheap, but Security has a certain price if you have encrypted layers. It's more costly than using non encrypted layers. So There's definitely a performance it but in general It's not prohibitive Like like other encryption methods. Okay. Thank you. I mean that's a question that is implicitly. I think Even now you have to trust your CPU vendor like if you do a computation You have to trust your CPU vendor what we're doing is essentially this confidential Computing we add another layer on top of it and and this closes certain holes but it's definitely You still there's the trust in the hardware. I think in general the the CPU vendors are pushed to release the firmware to those CPUs and essentially those Protocols as open source so you can read up on them and I don't make a review of the the PSP firmwares Those things are possible, but probably also for small amount of knowledgeable folks Can it be applied to more scenario than just computing or can can it be the default? Default configuration for general containers. So how much cost for this so if I understood the question It's more like about costs. Yeah, so essentially what it also is with Kubernetes. It's a bin packing problem Yeah, so you will probably have some waste there. What is for example in the There's certain facilities in Kubernetes where you have Can configure ports in a way that there's hints to which VM needs to be spawned For example, and then we can have certain certain profiles and say I need a memory intensive application needs this kind of instance and it's not so important that it has like a lot of CPU cores and this is the domain of the cloud API adapter which For managers the allocation of those resources, but it's definitely like always I've been packing problem. Yeah monitoring Yeah, actually this is for me a very interesting topic because I've been working observability for a long time and this is something that I mean observability and confidentiality Don't mix very well, so we have to We have to pretty much I Think figure out the model for those concepts at the moment. I don't have a good answer. I would say like just expose Metric endpoints is maybe not problematic, but I Know I it's definitely an issue like with APM monitoring and you get a lot of insights if you deploy I know a data doc New Relic agent into applications. This would definitely subword your TE So I Think yeah, those models have to be figured out for confidential Computing in general not just confidential parts, but I think it's an interesting problem to solve Okay, I think that's it. So thanks a lot for listening and the questions