 Yeah, thank you, Pushka. Welcome to From Threats to Trust, Safeguarding Sensitive Data in Kubernetes. Hope this is the talk you all are here for. Otherwise, feel free to run. Quick introduction about me. I'm Moritz Eckert. My day job, I work at a startup from Germany called Edge for Systems. And there I have the pleasure to work with a wonderful team on open source, confidential computing software. We also happen to be the organizer of the biggest conference in that space called the Open Confidential Computing Conference. Yeah, personally, my background is in the software security, vulnerability discovery, binary exploitation kind of side of things. I spend a fair bit of my lifetime in captured-the-flag competitions with Team Shellfish. And yeah, if I'm not in front of a laptop screen, I like to ride my bike a lot and also enjoy a good cup of coffee. I can definitely recommend the IntelliGenser here in Chicago, so if you're into good coffee, you're welcome, definitely check that out. But enough about that boring stuff. Today I wanna go on a journey about trust, especially trust in our model of cloud computing and especially on the infrastructure side of things. So trust can run very deep. If you think about the trust model in our cloud and the infrastructure side, think of the iceberg meme here, right? You have your application on top, your code, your container, all the stuff you control. That's something you trust. You hopefully implement some rigorous testing, scanning, fuzzing, whatever you do to protect your application, protect the front door. But then of course, your application is not standalone in most cases. You have some dependencies, your code is stored somewhere in a repository. Some people, some machines have access to that repository and then likely there's some kind of pipeline that takes your code from the repository to a production deployment. And all of this, of course, comes with a certain amount of trust, right? Do you trust the code dependencies? Usually, I guess most of you will implement some form of supply chain security. You have some access control to your GitHub repository. So you have all of this controls to reduce the trust in these components. Then we can go one step further. Do you trust the admins, your developers, whoever has access to those dependencies, GitHub repository to your code? Do you actually trust the platform you're deploying in? Do you trust your cloud service provider? If you have the container runs on GKE, do you trust Google to maintain the security of that cluster? And if you don't only have code dependencies, you might use some kind of software service like a database. You potentially trust that database vendor, that provider of that service to keep your data secure. And that, of course, also includes the employees and whoever and their devs and their developers who have access to the system and trust them that they keep their credentials secure, that they potentially implement two-factor authentication and so forth. And, of course, we can go deeper. We can go deeper in the software stack, right? Do you trust the hypervisor? Do you trust your container runtime that it's not backdoored? How do you trust that the Kubernetes you're running on is actually the release Kubernetes and not some malicious, I don't know, modified backdoored version? How do you do that? Hopefully, or likely, you use the latest Kubernetes release signing with the six-door project and verify that signatures and make sure that you're actually using the correct binaries that were released officially. And we can go once to further. Do you trust that signatures? Who has access to the keys that sign those releases? Who has access to the machines where the signature process is happening, right? And if you run on the cloud, you might trust the hypervisor that is running there. It's not being maliciously used by the cloud service provider or anybody else. But do you trust the other cloud tenants that they're not exploiting the virtualization stack and attack you horizontally or vertically from there? And we can go deeper and deeper, right? Do you trust the data center employees that have physical access to the machines that they don't access the memory, even though you might consider that unlikely, right? And finally, you trust the actual hardware. You trust your CPU that it does decapitation correctly. Otherwise, yeah, you need to trust something. And you might say, well, that's a bit paranoid, right? Do I really need to consider trusting my hypervisor? Do I really need to think about the data center employee? Are these likely scenarios for me? Is that a threat model I need to consider? I think that's fairly important. Talking about the threat model and talking about where trust matters. I have a couple of cases here. One thing we've been working with was, or is investigative journalism. So folks that are dealing with information, sensitive information and information about the sources, the actual content that it has information about very powerful actors, like organized crime, human trafficking rings, potentially some government, so some state actors. Think of if you're a whistleblower or somebody that is highly vulnerable, who and what can you trust? So safeguarding the information, if you have some form of cloud deployment and be one of those vendors, what would be your deployment model and what kind of security measures you have to do to keep that data secure? And you can say, well, these are the paranoid. Naturally, these people need to be paranoid. Maybe something a bit more practical. And there, one of the cases, or a usual case would be more in the regulated industries, like healthcare, financial sector, everything that deals with information that is potentially sensitive should stay confidential. And one of the cases we have in the EU and specifically in Germany is the case of the electronic health records. So whenever you go to a doctor or hospital, you get some scans or some other information about your body, this can be stored. And nowadays, this can be stored electronically in some kind of cloud system, so it can be accessed from all over the place, and people can actually have one concise place for all of this information. And with health information, it's not like some credit card information that once lost, you can replace that credit card the next day you have a new number. Once this data is gone, it's gone forever. It's not like you will change your body composition. And health records are one example where we already see there's very, at least in Germany, there's very tight regulation. It's very tightly regulated on how you can implement such a system, who needs to be excluded from that data, and that goes down to the provider of the data that's being stored. And the provider of the system itself, where we see this regulation now is catching up in other sectors as well. So in the financial industry, we now are, it's now being worked out the Digital Operation Resilience Act that specifically mentions that data needs to be protected in all states, while transiting through a network and addressed, but also in use, or if that's not possible, if you can't encrypt it, it needs to be processed in some separate, isolated environment. And to be fair, these are fairly European cases where we see much, much more regulation in terms of these industries. But another case I brought today is a bit more recent, and maybe a bit more global, that we see is in the now upcoming AI and large language model hype. You've all probably seen the JetGPT case. When we look behind the hood, what happens there and how this is implemented, we usually see some of this kind of pattern where we have a large language model that is owned by someone, or might be open source, but currently is still largely proprietary and owned by someone who has VIP of this model of the weights. And then we have folks privately, but also in a corporate enterprise context that want to use that model. And at the same time, you might want the model to be trained on some more specific information, like your local knowledge hub, your documents, so that the model can provide you with a more targeted help for your specific cases, like generating a summary of your data, your reports, whatever. And they're not only about a regulated angle, but more from a privacy concerning angle. We see that the left hand side here, the request is very, very hesitant to use that, especially in an enterprise scenario. So the question is, how can we make sure that all the requests, the prompts that flow into that model, how can we keep that secure if it contains personal identified information or some kind of IP? How can the model owner ensure that whenever they deploy that, that their weights, their model is being contained, is being secured, and how can it securely access a knowledge hub without revealing the data inside to the model owner? So these are cases where we see that we need to be able to isolate the data that's being processed from the actual infrastructure, from the service provider, from whoever processes the data. So the question is, how can we achieve that? How can we safeguard that data from these potential threats? And one remark I want to make is that we can debate about that we can just go and run our entire stack in an on-prem in a private cloud scenario. And I think that's actually the case for a lot of these scenarios where people are forced to stay in their own private data center, in their own private cloud. And the question is, is that desirable? Is that something we want? Where everybody needs to maintain their own stack, the security of the stack, and or do we wanna profit more from the left-hand side? Do we want to actually being able to run those use cases in the public cloud and maybe have somebody else take care of the software stack, the security, for example, keep our hypervisor up-to-date? Is that something we want to offload to somebody that does that just specifically? And then we can profit from the scalability and all the rest in the public cloud. And I would highly advocate that. The left-hand side, in a general case, is probably what we want. And the right-hand side is only a workaround because we can't establish the trust in the public cloud. And that's what I would like to focus now is how can we get there? How can we establish this trust in a public cloud scenario? And as hinted by this Dora regulation, the missing piece is essentially that we can also protect our workloads, our data while it's being in use. So we already have the encryption addressed. It's pretty much a solved problem. We also, everybody uses TLS and some other forms of transport encryption. What's missing is that while the data is processed, while this data is being executed on the actual in the cloud, that there it's accessible and vulnerable to being accessed by some malicious actor. And there are different ways how we can protect data in use. There's multi-party computation, homomorphic encryption. What I would consider the most practical and usable one today is confidential computing. And that's the one I want to present. So confidential computing, the rough ideas that we have a hardware-based solution for isolating workloads in a environment such as the cloud by keeping it encrypted in use. So making creating a private isolated space in that stack and making this verifiable and accessible from the outside. That's the rough idea. In terms of our trust stack, the goal is that we can reduce this entire iceberg so that we trust the hardware that creates that isolated environment and our application. So we can eliminate the rest of the stack and don't get me wrong, this doesn't mean that confidential computing is a solved-it-all solution. We still want to use all of the other tooling and all of the other things like supply chain security and testing and the fuzzing and so forth protecting the front door of application. But in terms of trust and isolating against the infrastructure, the desire is that we don't need to trust all these intermediate steps, we only want to trust the hardware and the things we provide our application. And these isolated environments that we create with confidential computing are usually called trusted execution environments and you can define them by usually three properties. One of course is the encryption in use, the memory encryption and then there is some form of identity, some kind of way we can measure the identity of a workload. So that we can attest it remotely, we can verify remotely, that's exactly what we expect to be running in that environment. The latest generation of hardware usually does it by isolating VMs. So there's an extension to the processor that allows us to create an isolated VM. So the memory pages of that VM are isolated are only accessible from within that virtual machine contacts, not from the hypervisor, not from another VM or any other thing below in the stack. And that we can verify that entire context from the outside. And this is available in most of the server CPUs that you can get out there today, like Intel and AMD have solutions in their arm, has a specification that's yet to be seen in silicon, but that's already there. IBM has secure execution and there's also a risk five specification for that. They all are slightly different, but more or less apply that concept of isolating an entire VM. One note to that, what we've seen, I've mentioned the AI use case, and with AI we usually need to talk about GPUs. Now with the latest generation of Nvidia GPUs, they've released a concept where they have something in practice that allows to extend that concept also to the GPU. So think of an isolated VM and now we can establish a secure channel to the GPU and also have the GPU's memory being isolated for that context. So have the same properties as we have for the VM. And that means from within the VM we can securely use the GPU and process the data securely inside without losing the trust here. Okay, so we've seen the left hand side. We've seen that we can create a trust execution environment by isolating VMs, rating isolated VM in the cloud. So the question is now, okay, how does it help us with trust? How do I verify that whatever is inside that context is actually what I expected there and this is actually isolated from the cloud? How do we gain this trust? The first thing is we need to understand how the identity of such trust execution environment could look like. In the case of confidential VM, it can roughly look like this. We can have, it's usually being referred as a measured boot approach. That means we go from a well-defined state, the initial memory state of that VM, which is usually the firmware that's being loaded by the hypervisor. And we want to go all the way up in the boot chain so that in the end we have a booted VM that is verifiable from the outside. And the way it works is that every component in the boot chain is measured by the previous component. So the hardware loads the initial memory layout, loads the firmware, measures that firmware, creates a hash of the memory pages, stores that hash in a safe space, and transfers execution. And this continues. So the firmware loads the next component, potentially the bootloader, stores its hash, the bootloader then holds the kernel, the kernel loads an init-rd, and the init-rd might load a root file system and store the hash of that root file system, and then we transfer execution to system D or whatever our init is, and yeah, from there we can start, for example, a container on time in that VM. What we get then is a unique chain of measurements, specifically for that VM. So when we obtain that chain of hashes, we can verify that was exactly the image boot that we expected to be there in this VM. So given that we have this concept of a measured boot, what the CPU and the CPU features like an A and B SCV until TDX is what providers with is generating a report for that environment, and this report, of course, it contains some metadata, the firmware version of that CPU and other stuff, it can contain this chain of hashes, so that report is unique to that VM and we can identify the VM from there, and then it signs it with a key that's burned inside the CPU. So this key is only accessible by the CPU, so that we know that this was actually not generated by someone randomly, but this was actually generated by a CPU from, let's say, AMD that has access to that key. So the process of attesting that environment then is, we boot up this confidential VM in the cloud, we obtain this attestation statement, and then we need to verify it. So we need some ground truth about the measurements we expect. We need to be able to verify the signature, so there must be a certificate change with the hardware vendor that makes it possible to verify that signature. And once this is done, we can establish a secure connection to that environment, so think of this VM booted up. We use that remote attestation statement to make a key exchange, and then we can securely communicate with that environment and potentially send our data inside for processing. There's a proposal in RFC for remote attestation called RATS. If you're interested in the details and the inner workings, feel free to check that out. Yeah, what do we get now? We get the capability of creating isolated, secured environments inside a context like the public cloud. We can verify that from a mode. So essentially, we can create our own private space in the cloud, but not the cloud provider, and no other parts of the cloud stack has access to, and that's something fairly cool. Questions, okay, great. Now, how do I use it? Where can I use it? Where you can use it is pretty much most cloud providers have some form of confidential computing offering. It goes from basic bare metal machines with the CPUs that have the features to what's probably more interesting is infrastructure as a service, so you can create confidential VMs based on that, and then of course, higher level services as well. So the whereas is not a big deal. I would say everywhere in most clouds. The question is how? How do we use it? Since we're at the KubeCon, I think that's the most relevant question, and now we go to the juicy part of that presentation. How do we make use of this feature? Quick revision of the Kubernetes stack. Super simple here in this case. Yeah, you know, we package our application containers. These are usually grouped and managed in an entity called POT, and this POT is scheduled by a control plane by an orchestrator into some compute resources, some notes, and these are usually, when we talk about a cloud use case, usually VMs. So every node is a VM, and then we schedule our POTs inside. So how do we apply confidential VMs to that stack? I like to present three options here. You can probably find more fine-grained versions in between, but three main cases. One is we consider that as, consider our Kubernetes as one environment, one concise environment that want to isolate against the cloud provider, so we can deploy our stack inside while being isolated. That's one case, I would call this case the confidential cluster, and one implementation of that is called constellation, we'll see that in a second. The second case is maybe one step away is we don't consider the control plane, so we only take our worker nodes and package our worker nodes inside confidential VMs. You will get that from the hyperscalers, like a GKE with confidential worker nodes or an AKS. We'll also see that in a second in more detail. And then a slightly different concept is instead of protecting our cluster or protecting our nodes, we want to protect the workloads themselves. So protecting individual POTs instead of a cluster. The interesting question is how do we match this VM-based isolation to POTs? And also that we'll see. So the confidential cluster, as I said, the entire cluster should be isolated as a whole. What we need to do is run the control plane, and this could be multiple instances, of course, run that in confidential VMs, run the workers in confidential VMs. That's the rough idea. If everything runs inside a confidential VM, this whole thing can be seen as one shielded context. And then from within this context, I mean, it's just Kubernetes, so nothing really changes there. What are the challenging bits about that? First of all, of course, attestation. So how, roughly, how would attestation work in this context? So think of this. You want to create a cluster in the public cloud. You want to run all of the nodes in confidential VMs, and you want to attest this entire cluster so that every node inside this cluster has the correct identity and is running inside a confidential VM. We can, of course, as we've seen, do remote attestation with a single node. So we can connect to one node, test this node based on a ground truth like the expected measurements. And then, of course, we could repeat that process with all nodes, but this would be very tedious. Let's say you think more into the future of day two operations. You need to upgrade your cluster, roll out a new node image. You need to do this all over again or think of a scaling event for that cluster. So one alternative would be, instead of doing this sequentially, you do this more recursively and then verify the cluster transitively. So you verify the control plane or the first node of that control plane, then the control plane verifies itself so all of the other instances and just the control plane verifies all the worker nodes. Of course, therefore, the control plane needs also access to some verification, right? So it knows the ground truth and it knows it can verify this remote attestation statements, but given that, you can verify the entire cluster by basically doing this recursively. And then we can bootstrap this together. They can of course exchange credentials in that process and then create a cluster and all of the nodes are mutually authenticated. And with this concept, we need to also revise some soft problems like networking. If we don't trust, if you don't wanna trust the club provider, of course you need to make sure that all the networking is encrypted as well. So we keep the data secured in all states. And you can think of a lot of options here, right? You can implement this on the application layer. You could implement this via service mesh. Probably the most comfortable one would be to implement this on the CNI layer so that no matter how the application is being deployed, the connection between nodes is encrypted automatically. Yeah, there was an interesting talk on that from a colleague of mine at the Cilium con here, just focusing on how this can be done, for example, with Cilium. And then, yeah, storage, encryption address. Again, a soft problem, you would say. I can ask my club provider to encrypt the storage, the persistent volumes. The question would be if we do that on the server side, so we ask the club provider to encrypt the storage, we can say, hey, I have a key management system. So I keep control of my key, but yeah, if the club provider can encrypt the storage, they can also, of course, not encrypt the storage, so you trust that the storage is actually encrypted. So to keep the concept and to keep this isolated, we need to do this on the client side. Again, a lot of options here, application layer, but we can also decouple that, for example, into the CSI layer, encrypt the storage there, and the encryption needs to happen inside the confidential context. If we talk about maybe an external storage, like an S3 bucket, you can think of concepts like a proxy, but does the encryption inside this cluster for you. So yeah, we need to also revise those concepts in order to keep the state encrypted in all states, and make sure that the encryption happened in a context that we control. And yeah, as I said, there is an implementation. We do an open source project called Constellation that implements this concept of a confidential cluster. You can check that out on GitHub, play with that. There's also, for mentioning that, there's also a concept from the Reddit folks that presented a confidential cluster concept for OpenShift that follows similar patterns now. So that is the one option. I said there's maybe one step below, there's the confidential worker nodes. And to be clear, I don't think that's a good concept. The question is, of course, what's your threat model? But what that means is only your worker nodes are running inside confidential VMs. What you gain is the runtime encryption for whatever's running in that nodes. So consider the data center employee that I had on the iceberg. They couldn't go to the physical hardware, pull out that memory, and extract anything, because it's encrypted. But most of the other components in the stack, this could be a fun exercise for capture flag competition. I give you access to the control plane, you have full access to the control plane, all of the worker nodes are running inside confidential VMs. How do you gain access to those workers? Things would be, for most of you, probably a couple of minutes, since there is no isolation between control plane and the nodes themselves. What's more interesting though is the confidential containers project. That's also an open source project, and the idea here is that we isolate individual services, or individual pods, individual containers. Again, it's the infrastructure. So the main challenge here is then how do we get this confidential VM concept into isolating individual pods. And the idea that's being followed currently is based on the idea of cutter containers. If you're not familiar with that project, the idea for them is running every pod inside its own VM to isolate the hardware against that pod, so that you can run potentially malicious containers that have not such an easy way of escaping their environment and attacking the host. We turn that around, so we protect the pod from the infrastructure from the hardware. So that means instead of running every pod in its own VM, we run every pod in its own confidential VM. We can do this either nested or by a remote hypervisor concept where we say just create a confidential VM somewhere else in the cloud. We need to take care of the connection to that context. But nevertheless, the more demanding challenges and the thing that's currently being worked on is now that we, again, we don't trust the control plane here. We only trust the concept of the pod. How do we take care of orchestration and attestation? So consider you don't trust the API server. You don't trust the ACT. How do you provide a deployment file? Like, let's say you have a helm file, you deploy that. It goes through the API server. It could do whatever. So we need to restrict the access of the control plane to those pods to ensure the confidentiality whatever ever runs inside there. And then we also need, of course, make sure all the IO problems we also solve the confidential cluster concept are solved. So how can I consume storage? How do I do a networking? How do one container can trust another container and could securely communicate with each other and make sure that the other containers are also running inside a confidential context? So these are challenges we need to solve and for different environments, different solutions are being discussed. Unfortunately, I don't have the time to go through all the details. But if you're interested in that, there was a great talk from Magnus from Microsoft on the open source summit in September that basically focused just on confidential containers. So yeah, check that out if that's interesting. Yeah, again, confidential containers is a GitHub open source GitHub project. You can find it. Just to mention it, Intel also released a project called ACAN or ACON that is specifically for Intel TDX and follows similar pattern as confidential containers isolating individual containers based on Intel TDX. Okay, summary. So first of all, we've seen why this is interesting. Where is the threat model of protecting against infrastructure-based threat, isolating ourselves from the service provider? Where's that relevant? We've seen how we can solve the problem of providing that isolation in the public cloud, different concepts for different use cases, either isolating an entire cluster or isolating individual workloads. And we've seen that we can pretty much go ahead and implement that today on most cloud providers. Some references for further, for yeah, exploring that further. There was interestingly, there was a Kubernetes block article, the Kubernetes security block, I think it is, written by a bunch of people. There's the Confidential Computing Consortium that's part of the Linux Foundation, oops. That has some material on the topic. And of course, you have the open source projects. I think that's the most practical start and getting really hands on trying that stuff in practice. And if you're more into, oh, in our conference, the OC3 is coming up next year in March, where we have a bunch of topics on just confidential computing. If you're more into books, there's Mike Brazil's book on trust in computer systems. These are just for reference. But I'd like to conclude with a statement from Akrozinovich, the Azure CTO at OC3 last year, where he was setting, we're heading towards a fully confidential cloud. Just think a very promising vision and the kind of vision that changes the model we think about cloud computing today, where you can actually exclude the data from the service provider. And that's a very promising future. And there's a lot of problems, a lot of challenges to be solved. And yeah, if you find that interesting, as you've seen a lot of open source stuff going on, get involved. Hope that inspired somebody here. Thank you. To so many of questions. Yeah. Hi. You talked about the boot flow where everything, the first thing attests the second thing and all the way up to the OS disk. Secure boot solves all the problems until VM or machine boots. But what, how do you, what is that tool or component that does the attestation of the root disk, you know? As in traditional tools don't have it, so what are you doing? You mean, let's say you go up to the unit, or how does the attestation of the root disk work? Yeah, yeah, right now it only stops, as in Secure boot only stops before system disk starts. Right. What's that handover look like? So there are different ways people do that. To, for constellation, for example, the way it works is that the root disk is mounted as read only and integrity protected by DMVarity. So the DMVarity root hash is included in the, in it or D itself that's being measured. So you essentially hash that component and contains the root hash, which is then used for DMVarity to ensure the integrity of the root disk. Does that solve the answer? More questions? So for revocation procedures, how do you deal with that? So you have a, in the middle of the run, production run, the root is compromised or sign certificate is compromised. How do you do the checkup that during the... Okay, so the question is, for when we saw the attestation model, the attestation segment is signed by the CPU assuming that we can't trust that signature anymore or the firmware or whatever of the CPU and there's an update of that. How do we handle that? So with a constellation cluster, you would need to recreate another cluster and update, of course need to update the ground truth with that new certificate. And in place upgrade would not be so sufficient due to, yeah, you can't trust anything in that cluster anymore, yeah. So essentially we need to have another process that monitors the trustworthy of the signing certificate or the signing root. Say again, please. So we need to maintain another monitoring process, right? That would say, oh, is this signature still trusted? Within the cluster, there's no built-in mechanism to do that, right? No, the cluster itself does not have a reconciliation loop that automatically updates itself about the ground truth. So let's say, for usual cases where we just say like we want to deploy a new image because we updated the container runtime, we can push the ground truth into that cluster and tell it to only accept new nodes based on that new container runtime. Of course, the trust model here means if you consider that cluster being compromised already, you lost, right? But considering that we want to just deploy a security update, then we can do that. But given that scenario you described where we don't trust the CPU firmware itself anymore, well, then we need to update the security firmware and then the firmware of the processor and then go from there, yeah. I have one more question. So in the model where you are bringing confidentiality only for workloads or pods, how do you deal with logs and metrics? How do you divide it between, okay, this is application specific and this is cloud specific? Yeah, the short answer is this is not a solve problem. Yeah, logs and how you deal with observability and metrics in this case is there's no existing solution that does that perfectly for you. Of course, as you say, you can deploy the usual observability stack and then try to restrict the access or restrict what's being locked and try to infer what's application, what might be sensitive information, what not, but generally observability and confidentiality are, yeah. You need to level out and see what for that your threat model works or not. But are you seeing any patterns that are being employed right now or are users comfortable with giving away, let's say, specific metrics and then only keeping logs with themselves or some kind of mixed model? Yeah, so for constellation, that's where the most insights, since we have some production deployments there, is that usually the main goal is isolating against the cloud service provider. So if you have a metrics endpoint in what you call a safe space, then you're usually fine, right? And you log to that. Or you consider logs that are low-level enough that they don't contain sensitive information about the application itself. For confidential containers, this goes very much into the details now, but one of the concepts that's being discussed is, I think, I'm not sure what it's called, the data hub or something. So something that sits in below and takes care of any IO-related stuff or like storage and stuff, you can think of implementing some kind of logging probably also there. There was a research paper from IBM about how you can make a service mesh like Istio compatible with that concept, restricting the access from the Istio control plane to the data plane in the service mesh or the envoy sidecar service mesh.