 Hey everyone, welcome to this session with a big Buzzy word title confidential containers Explained demystified whatever. I'm Samuel Ortiz. I work for Apple And I'm James McGowan and I work for IBM and Yeah, we're gonna talk about confidential computing Cloud native containers Kubernetes and how we kind of all mix together So before going there, this was to be this was supposed to be a panel But it's not going to be a panel. It was supposed to be a panel with Both of us plus three other people are China from Intel and junk from Alibaba and predictor from Red Hat But yeah, it was basically Five speakers five time zones. We were supposed to be to do live pre recording Yeah, what could possibly go wrong, right and try to find the right time for Pre-recording something with five different times on we couldn't make it. So we asked for a change And we're gonna give this presentation here, but it's gonna be a kind of an hybrid way of presenting We're gonna Basically ask some questions and try to answer those questions. So what you will learn today are answers to those questions What is the project? How does it relate to confidential computing? How does it protect Kubernetes workloads more so than other architectures? How can you use it and deploy it? What are the use cases for this project and what's next? But I guess before we start that and some questions Interactive nature to start with wake everyone up Who in the audience by show of hands or some other means is aware of the confidential containers project? Okay, and who's aware of confidential computing? Ooh, and then I guess the last question is who's aware of what a trusted execution environment is Okay, well the good thing is we've got a fair mix and as I went down that we had more people aware of what we were talking about but when we go through the questions and answer them we're hoping to Satisfy everyone's prior knowledge So to facilitate that normally in a panel we'd have five people slightly differing views So as we go through it Samuel's gonna lead you in and draw you in technically and then I'll bring you back gently To a less technical view you as a recap as we go along So hopefully we will cater at a level that kind of suits everyone's prior knowledge But obviously we'll take questions towards the end and we'll be here afterwards and well all week. So thanks Okay, so we'll start with the definition of the project or goals and definitions So what is confidential containers? First of all? We've recently been Admitted as a CNCF sandbox project. So yay for us. It's taken a long time So we are a CNCF sandbox project. We have a logo. So we're legit. That's it That's what we are but not only this to and sorry we didn't fall out over the choosing of the logo either which was good So the the Confidential Continuous Project is a set of open source components and The goal for this is to and there's some keywords there that I want you to highlight The goal the real the goal of the project is to seamlessly run Kubernetes workloads in their own confidential computing enclaves also known as trusted execution environments There are some very important keywords here seamlessly is the probably one of the most important and The idea is to with confidential containers is to say if you have a definition for pod for workload you take that definition you don't change it and With confidential containers, we are going to allow you to run that inside a confidential computing enclave inside a TE Okay, so seamless and see the the seamless aspect of the project is very important and With confidential computing in general and with confidential containers in particular We want to remove the CSP from the trusted computing base So basically with confidential containers, you're going to be able to run Kubernetes workloads without having to trust Your infrastructure provider typically a cloud service provider at all So you only have to trust yourself and what you give and provide to the workload and Last but not least We are aiming or and we are supporting multiple trusted execution environment implementations SCV for from AMD Intel TDX Intel SGX IBM SE so we want to be hardware agnostic. We're not bound to a specific implementation of confidential computing So those are the very high level goals of the project I also want to highlight the fact that this is an opportunist community obviously with the contribution participation from already a bunch of key stakeholders cloud providers like Alibaba and IBM Silicon vendors so the people who are actually implementing on in hardware the trusted execution environment the confidential computing hardware like AMD IBM Intel ARM is also starting to participate. Those are participating to this project and Open source vendors like Red Hat also are contributing So this is a very high level view of the project To be more illustrative. This is what it looks like a very high level from a Kubernetes standpoint So we run on top of a Kubernetes node with confidential computing hardware enabled and In these diagrams, there's a few very important point the first one is that you have a pod and your pod has a dedicated confidential virtual machine and Why a confidential virtual machine because in practice confidential computing is implemented as a hardware virtualization extension so when you talk about confidential computing when you're talking about SCV or TDX or SE you end up actually running something inside a virtual machine And what we're doing with confidential containers is taking that something which is your Kubernetes pod and Run it inside a confidential computing virtual machine So each and every one of your pod with confidential containers will run inside its own TDX SCV SGX enclave That's the very important point. There's another one, which is the enclave agent That's a controlling agent that's running inside a virtual machine that kind of sets the Confidential computing environment and this is a very high level a diagram. We're going to more details as we answer more questions So how does that relate to confidential computing? How does confidential containers use the confidential computing hardware and the implementation and features? Again With confidential containers each Kubernetes pod gets its own enclave gets installed confidential computing enclave What does that mean? First of all, that means that your Kubernetes pod is going to have its memory encrypted through hardware But not only memory. It's so it will also have its CPU states and hardware state in general encrypted So the host the infrastructure owner the UCSP won't be able to see your Kubernetes pod data and Kubernetes pod memory and won't be able to see your CPU registers won't be able to see your CPU state It's it's it's that so we're using the the memory encryption and the hardware encryption features of confidence computing But we also using with confidence confidential containers another very important feature of confidential computing, which is at the station running with memory enable is one thing with memory encryption enable is one thing but With memory encryption enable you enable you basically are saying that Your host is not able to see what your community spot is doing But you are no guarantee about what's your what your Kubernetes part is actually running the software stack that is running So it can be it can be controlled by the by the by the the CSP by the infrastructure owner And we want to prevent that from happening We want to control what's actually Creating this data that you're going to be encrypting with hardware memory encryption and to do that we use at the station the attestation features of confidential computing Basically with confidential containers you have an enclave stack So there's a stack that's running inside a virtual machine that's going to run your community spots That is trusted and verified through Confidential computing attestation and typically this is basically remote attestation So you have an enclave stack that's running inside your virtual machine That's going to be dedicated to your pod and this enclave stack is trusted It's something that you can verify from top to bottom that it's running exactly what you need and exactly what you want as a guest owner again to summarize this a Confidential containers Kubernetes part will basically use the hardware memory encryption of confidential computing for protecting your data while it's in use But it also will use attestation measurement and verification to make sure that what is producing this data is What you own and what you defined and you can verify this so if your CSP or infrastructure owner decide to tamper with what you're going to run inside is this pod You are going to be able to see that and not generate any of the data So you can verify that the data is is what is provided by the right software stack and then you can encrypt this data An illustration of this. This is the high-level view if we're going to a little more details First of all, we have this enclave stack that I that I mentioned which covers Everything that's your that your virtual machine your confidential virtual machine is gonna run from firmware to any 30 So that includes the firmware the guest kernel and the guest in 30 Plus an agent the enclave engine that's typically part of the of the unit already with confidential computing we are Encrypting all this so all the memory all the CPU state that this virtual machine is going to be using leverages confidential computing encryption Capabilities to encrypt all this and we also measuring the enclave stack So all the components that are part of the enclave stack the firmware the kernel the unit are the the current command line All the parameters that you want to add to your measurement you can measure that and Measuring means that you can verify it So there's a relying party that's outside of your node that can be used that can be controlled by you as a guest owner To actually verify that this enclave stack is exactly what you're expecting So before starting the pod before running anything on your on your confidential computing VM You're gonna be able to verify that what's gonna be running this is Kubernetes pod is exactly what you expect as not be temporary by by The the infrastructure owner Okay, so you can verify that what this workload is gonna be generating as far as data Is gonna be generated by the software components that you defined and that you want Okay, next question. How does that protect Kubernetes workload in a relevant and significant way? So as I said with coffee that containers we are basically taking the host The infrastructure owner the CSP whatever you want to name it whatever you want to call it We're taking this out of the trust boundary We take me we taking this out of the trust boundary by again using hardware encryption With hardware encryption you add the guarantee the hardware guarantee that your host is not gonna be able to see or tamper with Your memory and your hardware states It's not going to be able to mess with your CPU state with the CPU states that that that your Confidential virtual machine is running it's not going to be able to mess or even look at the memory that your Kubernetes workload is generating But that's not enough if you only have this you have no guarantee You don't know what's generating this data So you need to make sure that what's generating this data is what you define what you built and what you want and to do that We're using memory measurement and attestation. So the guests Runs again the enclave stack the enclave stack is completely measured verified by you as a guest owner through remote attestation Once you have this enclave stack booted verified Attested and you know, this is the right software components that are good that are running then you can start running your Kubernetes Workload you can start running but your Kubernetes workload by actually pulling Encrypted container images inside the guest So this is a major difference compared to the regular Kubernetes workload workflow that we all know about where you have the hosts pulling the container images and then the Kubernetes workload Actually running what the host pulled container the poles stores on your hosts And then your committees Kubernetes workload use that as an overlay FS and runs your communities workload with confidential containers The first thing you do is verify that your enclave stacks the stuff that it's running inside a confidential VM that your community spot is running on Is verified once you have done this Your enclave stack through the enclave agent is pulling the container image and keeping the container image inside the enclave so the container image what's what really matters to you what your workload definition is Kept inside this virtual machine image it kept inside this this virtual machine memory and it's protected So basically the host no longer owns the container image and no longer sees and no longer is able to temper with or modify this container image To go into more details again. This is the previous diagram What we do once we verified once we know that the enclave stack is trusted The relying party is gonna inject secrets into the confidential virtual machine typically the secret is An encryption key. It's an encryption key that you're gonna use to pull a container image that is encrypted and decrypted Inside your virtual machine. So you pulled an encrypted image from any registry public private whatever It's encrypted you keep it you keep it inside the virtual machine and you decrypted with what's been provisioned to you By your relying party after verifying that your enclave stack the component that your virtual machine is running is verified and you trust it So none of this is visible by the host none of this visible by the CSP or by the infrastructure owner So once you provision this as I said, you're gonna pull the container image You're gonna decrypt it and you're gonna start the container image from the virtual machine Nothing gets out of this Enclave of this confidential virtual machine that is fully encrypted and fully protected by the confidential computing hardware So this point having drawn you in technically Samuel's explained a lot of them gone into the detail. I'm gonna wind back a bit Kiddering perhaps more for those who didn't answer. Yes to any of the questions. I asked at the start So to start with if this technology is out there, we have trusted execution environments Why can't we just use them as runtime? Why do we have to worry about anything else? Well, it comes down to who we trust and it comes down to thinking of personas and actors Who is it? We trust and who is it? We decide not to trust So as we put this into a more cloud native perspective What we're what's happening here is that it's the person who's driving the workload driving the pod descriptor the helm chart The whatever that's defining the workload. They're the ones that are inside the circle of trust or the trust boundary Is it where whereas the administration of the cluster whether that be a managed cluster? Whether that be somebody with their own company or whether that be the actual infrastructure upon which we're running That's outside our trust So that boundary that we want to put left leaves the workload inside But the orchestration and management of it on the outside is such a way that however It's managed can't change the workload that we trust Can't get access to the data the algorithms or whatever else that we've got in our workload And that's the kind of separation that we're aiming for here So what does that mean? Well, it means that if we take a trusted execution environment today It's essentially a secure environment in which you can put your workload whether it be containers or whatever And if all you're doing is considering a pure VM approach a virtual server approach Then you define through your workload the API as you define what crosses that boundary So you're in complete control of what gets into that trusted execution environment Just because you control the workload of the application But as soon as you start to put that into an orchestrated world into a cloud native word There are other things that need to cross that boundary to do with the orchestration actions to do with starting the containers or monitoring What's happened and if you go to an extreme there are capabilities that would cross that boundary that for example would allow you to execute an arbitrary shell command inside a container for example Which doesn't really fit with what we're after here in terms of confidential computing Now that would be a complete Nightmare if you're trying to suggest that your workload is secure if somebody could do something that extreme as it were So I mean what we're after here is you could use a trusted execution environment to wrap different things and we could use it to wrap a node for example, but Essentially the management of the cluster lives inside that so that's why we settled on That boundary around the pod as a way of kind of neatly suggesting that Those who are defining the workload can have more control of from inside the pod Leaving the management of the cluster to happen as as normal and I guess this is the next key point where we're Putting some control Into the pod so Samuel mentioned about the pooling of containers Essentially, what we're driving here is that I want to define a workload that could be deployed in a cluster And I want to know that it's deployed as I expect it to be deployed It's deployed into a trusted execution environment It's using the images that I want that the orchestration actions that can be carried on out on it are in my control and so in that respect I Need to be able to define that such that the pod has control over that Because I can define the workload that's what happens in the pod So this is why the pod needs to take more of a control of pulling those containers in and the way I can be sure that it's deployed As I expect is where we link into the attestation Because I can measure the environment that confidential VM in which my pod is running ahead of time I Can control the containers of traveler if I encrypt them or sign them then that's something that can be provided as a result of attestation So that when it starts when my workload starts or whatever combination of pod start I know the only way they could have started is if they've received the secrets that I've provided and as a result of Attesting that the basic confidential VM that I'm expecting to happen is there and outside that I don't care what the Workload, sorry what the administrator is done. Yes, they are in control of denial of service and things like that But that's what their least privilege is that's what they're supposed to be doing What I don't want them to be doing is changing my workload starting it outside a trusted execution environment getting access to The data that I want to protect or the algorithms that I want to protect And the last thing that brings us all together is that we don't want to as some you'll say we don't want to pursue a Completely different paradigm for doing this. That's not what this is about We want to fit with the experience and expectations that are out there The last thing we want to do is to change the whole way you would manage a cluster and how that's thought about We don't want to change the whole way that a workload is is defined so there's an element of Sometimes new ideas come with some friction that changes what's there before we are aiming to reduce that friction So essentially nothing changes for the majority of personas and actors and what they do nothing changes You still manage a cluster in the same way. You still build containers in the same way. You still deploy them in the same way But actually everything changes for those who care about it because at that point you can start to say right Actually built my container now. I'm going to encrypt it and look after that key and that key is going to be provided as a result of attestation So there is an element that does change that you're concerned about But you're concerned about it because you're concerned about confidential computing and everything else can remain the same Which basically means yes If you have a workload definition a pot spec with confidential containers, you're going to keep the same thing But there inside your workload inside your virtual machine You're free to define and run it as you wish and define those boundaries But from a workflow perspective. Yeah, nothing changes pretty much So next question now that we talked about this, how can you use and deploy confidential containers? Quickly because I think Yeah, we're on time As part of confidential containers we have many components, but one interesting one Which is it's kind of a the integration point of the whole project is the community's operator So we have an operator for confidential containers that allows you to Deploy confidential containers as easily as possible Inside your inside your convenient Kubernetes deployment Basically that will allow you to deploy the the right runtime to build the rights enclave stack that that you want to configure it properly And deploy that and run it with the with the with the specific runtime that you define to through the operator That runtime I haven't mentioned it but for most of it is We're using the the cala containers runtime Which is a natural fit for running? Kubernetes work Kubernetes parts inside virtual machines. So we're taking this runtime We adding some features to it We're adding some configuration or all this or most of it is upstream and we are deploying this through the operator And the operator allows you to define your configuration Define your app API boundaries your enclave stack definition and build it and deploy that inside your communities environment And again to keep with our personas here having drawn you in technically I'm gonna take a step back again So if we're good, it's all right. Yeah, so you went backwards. That's confusing me There we go, so why do we care so We're not here to redefine what the use cases are for confidential computing there They're out there. They're well trodden path if you care about your data if you care about algorithms for example in AI Then you're interested in protecting them. Essentially if you're in Industries that have concerns about regulatory requirements or compliance or things like that or worst-case scenario If you your very existence as a company Depends upon protecting the data that you're privileged enough to be using Then confidential computing is a concern to you But what we're trying to do here is set it in the context of what does it mean? For cloud native what does it mean to how we would administrate our clusters and this comes down to? technical assurance Versus operational assurance and what I mean by that we can configure a cluster and we can configure Webhook sidecars, etc. That would check what's happening But fundamentally who's in control of that access control who's control of those webhooks Is it the person that we want to be in control of that because what we're talking about here is trying to have that control or at least Certain portions of that control Reside with the person defining the workload not with the person administrating the cluster So what I mean by technical assurance here is it doesn't matter how the cluster is configured It's not possible to do certain things Because in defining the workload you have made those things not possible So essentially if the cluster or environment in which you're running your workload Goes against the things that you've said in the workload itself are required Well, then the other station will fail you won't get the secrets the workload won't run and so that's what we mean by technical assurance versus operational assurance and that's that separation of personas where we're Putting more of that control into the hands of where we define the workload Because after all why should the person administrate in the cluster have privileges that allow them to change containers that are running that allow them Going back to the obvious example allow them to arbitrarily execute a shell command in a container So, yeah use cases as Jim said I mean, it's it's not really different from a regular confidential computing use cases Basically, you're taking the infrastructure owner out of the draw of the trust boundary that gives you a lot of potential use cases one one interesting one is that you basically can Run your sensitive workloads anywhere where confidential computing is Enabled so if you're in a cloud where confidential computing is enabled you take confidential containers and you can run it there You have the guarantee that your infrastructure owner, whatever whoever it is is not going to mess with it There's a lot of different use cases there that are interesting One of them also is basically about moving the workloads from your private cloud to public cloud public cloud when when Confidential computing is enabled They become as safe as your private color. Maybe even safer even internally even in your private cloud Maybe the infrastructure owner is someone that you don't want to have a look at your workload even internally With confidential computing, confidential computing, confidential container, sorry, you can run that Safely at least so basically moving from private to public if your public cloud is giving you access to confidential computing That becomes possible. Whatever your workload is And I think an element of that the next slide is kind of the what next scenario is at worst. So we're Interested in and we're not claiming that hey this solves security that would be madness to suggest that I think what we're talking about here is a progression that it's more secure than what was there before but it's a journey that we're on with confidential containers and What we really want to do is to start to connect with other groups here to understand high Existing projects and existing things in a cloud native world could either take advantage of Confidential containers or would work with Confidential containers to progress that story to make it more secure than before to start that journey to confidential computing So what's next? Well We're working towards our first official release for confidential confidential containers One interesting thing is that if you go back to this slide for example Currently what we're really working on is the all the left-hand side of this diagram Basically the runtime what's running inside inside a confidential VM? But what we realize is that there's a we didn't realize but we we kind of Acknowledge that there's a very strong Bound between that part and the right-hand side Which is basically all the attestation parts the key brokering service the attestation service and when trying to integrate Our project with this we realize that there's a myriad of different projects trying to do this So we are expanding the Confidential containers scope to not only the node side, but also to the attestation side, so we are working on defining a Protocol that is simple enough for everyone to comply with for attestation service and key brokering service in the Confidential containers context So we're expanding on this One thing that we are also working on is adding secure storage. How do you inject data volume inside a confidential VM? And we're defining this we have a good architectural design and a initial implementation for this and When we're gonna put all this together. This is where you're gonna have our first release Hopefully pretty soon in a few months. Hopefully so that's pretty much what's next for confidential containers Yes, I think we have five minutes for your questions. Although we'll be hanging around afterwards. So are there any questions? I would start first with online question if you don't mind questions So the first question which came through platform. How confidential computing? impact security monitoring tools like Falco Okay, so This is a this is a very generic. I think this is a very common question Basically, how do you observe a confidential computing virtual machine? And I think we've been thinking about this. There's no there's no good answer to this There's no magic, you know magic answer for this, but basically one way. We are exploring is is About saying if you want to observe something into the competition computing VM, you're gonna have to run sidecars But those sidecars needs to be Verified by you as a guest owner. So it's okay to run sidecars alongside your Kubernetes Confidential containers pod as long as you as a guest owner can say yes, this is the sidecar I'm expecting and I'm allowing this sidecar to observe what's happening inside my confidential computing VM So that's that's one of the things that we're going to do is we're going to have to run sidecars That's that's one way to explore it sidecar and basically allowing the the infrastructure owner to Inject anything in the pod is kind of Antinomic to the whole confidential containers concept. So there will be restrictions there will be things that you cannot do with confidential containers in that context and You know making sure that what you inject inside a pod inside a confidential container pod is Something that you can that can be verified by you as a guest owner is our current answer. Yeah I mean it I'll take the backwards step again Some elements of observability have to be in the control of the person defining the workload fundamentally And that's basically whether that being control of verifying which containers are added for that purpose Or whether it being control of some other means of controlling what can be logged or monitored within that context Some element of that control has to reside with the workload provider There were there are some questions from oh Oh Hello, my name is Nick Vidal. We actually have an open source project called an arcs and we we Combine web assembly with confidential computing and we are part. We're part of the confidential computing consortium And I wanted to ask are you of course you are an open source project? I saw your your github page, but are you also Part of the CNCF or the confidential computing consortium. Is there a company or a group of companies behind your project? How does that work? Thanks, so I guess the answer is here We are a CNCF sandbox project and If you look at the bottom of these slides, those are the companies that are currently contributing to the project so we're we're fully aware of an arcs existence and I think this is what James was highlighting we probably need to talk and understand how Those kind of project can live together and if there's you know, maybe there's some way to converge Maybe there's no way to converge and two different project different provide different values So we need to understand this and and discuss, but yeah, we are CNCF project We're not I mean we we participate to the the confidential computing consortium, but we're not an official part of the CCC Another question. I Was interested in finding out. Well, how do you if I'm running this framework on a public cloud? You said you have an operator. So how do you attach the operator? How do I know if I'm deploying the operator? While deploying the provider doesn't mess in with it and then everything is kind of I Think the answer is that you you can't you can't really do this So if the if the CSP really wants to mess with this it can but then you won't be able to Attest and verify the workflow that you run The only thing that you care is that you can when you start a workload when you ask for your Confidiation computing confidential containers ng and x workload to be running you want to make sure that this is the right container image So you can you can change the operator you can change the runtime that you that you run It doesn't really matter at the end of the day What you really want is to this runtime to be able to start your confidential container and Then you can verify what you're running. That's that's all yeah I mean it comes back to to the whole attestation flow you have to accept that that Rule that's Administrative cluster can change what's there But at the end of the day if that workload starts outside of that confidential VM that's measured it won't receive The secrets required for your workload to actually start and those could be things like simple keys to decrypt containers It might be certificates for TLS it could be anything like that that we're exploring But some element of that configuration is required for that Workload to start and it won't only start if it's starting in that attested confidential VM But the elements which are part of the operator logic they could be compromised I just depends what you have in there, right? They could be compromised Yes, but if you it doesn't even if they're compromised, but you're still running and if you if they can bear So let's say the operator changes the the guest kernel Okay, you change the guest kernel. This is going to be measured This is going to be verified and the attestation is going to fail So if the if the CSP changed the guest kernel that your confidential containers is going to run You're never going to be able to run your container image and maybe there's an element here to push on We would not imagine that the operator is also setting up the attestation Flow as it were that that measurement that's recorded in the attestation service needs to be done outside of the operator So the control of that is not within the control of the Kubernetes administrator Because if they could control that then you're quite right They could change anything they want and then you wouldn't know but measurement of that attestation is done by the hardware So if you change the kernel for example attestation will fail. You will never run your encrypted Privilege, sorry protected workloads in that case so Just one last very quick question and afterwards we can continue discussion in the corridor Yeah, thank you for the talk. I might or a one quick one Is I saw the attestation part was like outside of the workload Is there any way for the workload to assess that it's running in a confidential container? Like or does it make any sense? The workload that so you have two two parts on the workload The first one is the what we call the enclave stack, which is basically what boots the confidential virtual machine that actually talks to the hardware and gets the hardware to Generate the attestation report so the hardware has measured the firmware the kernel and the the component that actually even that Actually asked for the attestation report so Yeah, but let's say as an application developer, right? Let's say I want to just my container to double check you can it yeah Yeah, you can definitely do that if you if you know how to do this basically you need to talk to the To the kernel API to actually get the attestation report and make sure that you're running on the right hardware with the right components Yes, you can no more questions No more time for questions. Okay. No more time. All right, so we will be around Yeah, yeah, well I did the front of anyone has any last questions and please come up and talk to us. Thanks And on Friday, there's a CNCF tag Yeah, there's a CNCF tag runtime talk on Friday where we'll have five more minutes to talk about confidential changes. So