 Hello. So my name is Christophe de Dinchin. Try to say that three times fast. I hope you learn since this morning. And so I'm working for Red Hat. Nobody knows what I do there because I don't know myself. Vaguely working on Continental Computing. And today I'm going to try to show how to go from a Continental Computing host all the way to the workload. So what we are going to cover today, first a quick overview of Continental Computing that's a repeat for folks who wear their faithful and wear their this morning. Then we are going to show how to access a guest memory to steal secrets. We are going to set up a confidential host, which is extremely simple as you'll see. Then we'll try to set up a confidential VM. Hopefully everything happens as we expect. Then we'll discuss some limits of memory production. We'll talk very quickly about confidential VMs for instance on Azure and other places. But Vitaly explained everything about that this morning. So there's a better talk. And things like confidential clusters as well with a quick example of SLS Consolation. And confidential containers I'll probably have to skip for time reasons. And there is a workshop on Sunday where we'll show you how to do that in practice. So you're welcome to join the workshop for containers. So the problem statement is how to protect data in use. That is data in memory. And we use so the problem is that confidentiality is really the essence of trust. And it's broken right now in the cloud because your infrastructure essentially sees your data. So the software that you have runs on the cloud, also known as someone else's computer. And the hardware resources there are owned by that host. You don't really buy for this memory or whatever. You just print it. You run various workloads on it so that's typically some things like containers. It can be virtual machines. It can be other things. And we have many mechanisms in Linux. Elected memory, fruit, all these kind of things are designed to protect the host from the workload. The other way around, nobody really cared about. In other words, someone who has root access on the host can fully look at the files for your containers, look inside the memory, and so on. And so that means that if you have competitors on the same hardware, they tend to not be very at ease to put really confidential stuff there because nothing tells them that their competitor did not pay the CIS admin more than they did. And so in the end, maybe their data is being shared. So we have solutions for data on disk or data in transit, networking, et cetera, so that the host essentially cannot really know what you have when you use an encrypted volume, for instance, normally you're safe. For memory, it's a different problem. And there's tons of secrets that run there. And these secrets, well, there may be a transient that may be moving fast, but you can still extract pretty much anything from it if you know how to look. And so what confidential computing first added was a way to encrypt memory. And the encryption is not necessarily super strong, which is why I'm not using a super strong encryption there either. But it's very fast. It happens on every write to memory. And so it really offers some serious production anyway. And then can I take that question at the end? That's an excellent question. The short answer to that is that the encryption is really normally not visible to the host at all, and dependent on the page, dependent on location, and other aspects that make it, in practice, hard to break. What could work well would be replay attacks on a given page, for instance. That would work better, because that encryption mechanism would be the same for that page. But we can take that later. That's a very good question. And when I say weak, it's not super weak either, right? It's still relatively serious. So there is also integrity production for the CPU state, essentially to make sure that the hypervisor cannot make your workload do whatever they want by changing the address of execution or some registers in the CPU state. And one thing that I covered extensively this morning was proving that what you run is what you want through mechanisms known as attestation, and that give you some cryptographic counties about what you're running. OK, so let's go to some real meat. So this morning was sort of a joke presentation. Today we are going to look at a real system. And I would have liked to do that live, but I explain in a moment why I did not do it live. So I'm going to comment what I recorded. So we have a FETA-R38 system. It has a relatively recent Linux, 6.2.14. And it's running with 16 CPUs. So that's an Epic 16 core. So that's a relatively serious machine with a CV and a CV S&P available on it. That's what I'm going to use for this demo today. So in terms of what the machine looks like when you remote connect. So that really depends on the machine. But you have typically web access like this. And you can look, for instance, at BIOS settings when it starts. That's the good old fashioned way of doing it. So I'm going to skip a little forward in the setup. You can access the BIOS the old ways with hitting the F12 or F2 or whatever it depends on the machine. What matters there is that we are going to look at the processor settings and check that we have secure nested paging enabled. So you can see that at the bottom of there. So that's essentially the thing that re-matters in your configuration there. And of course, the way you do that depends on the machine. So the rest is not specifically interesting. You just have to be relatively up to date with firmware because these technologies are relatively new. And so older firmware may have BIOS issues. OK, so nothing really, really special otherwise. We can boot the machine and look at the virtual console. And you get just your regular text-only mode Linux. This also supports graphical mode. So the more modern we are doing that is to go through the web interface like this. So you have, as always, the place depends on where you are. But you have system settings. And you can see again that you have your secure nested paging somewhere in the middle there. That's called 2-Fast. It's a bit above. So you see it's a secure memory encryption, secure nested paging at the bottom of the page. OK, so I think that's about for this page. So we can check that we have some host-level SCV support simply by looking at the message log. And that's essentially all we can do without additional software. OK, so in order to be able to do some kind of demo, I'll have to use a little bit of extra software. And I'm trying to give a sort of shout out to KCLI, which is a relatively convenient tool to do various things. So you enable a special copper for that. And then you install what you need. And I forgot one of the packages is there, which is Libert's human client. So nothing really complicated, but you need essentially a hypervisor, a manager for that. So you're not very surprised so far, I guess. So now we can check with self-color. We can check a little more about what our machine has inside. So it has support for a CV, a CVES, and a CVSMP. And now you all know what this means. We also need to record these numbers that we have there, the number of physical bits, reduction, the CVIT location, and stuff like that. We'll need that later. So let's just remember for now, 51.5. We can also check that we have a kernel that has enabled the AMD support. And we can check that our Libert also has support for this kind of features. So the DOM capabilities will tell us, if there is an SUV section in there, then your Libert also has support for that. So this is essentially a pre-flight check to be sure that your machine has what you need. And again, we have these reduced face bits and the 51. And of course, you can now notice that the number of reduced face bits is not the same between the two. And you're starting to scratch your head and say, OK, something in there does not cook. Well, which one will I use? We'll see. OK, it doesn't. So KCLI, I assume most of you don't really know what KCLI is. It's a sort of a quick user interface to create VMs, create clusters. And it has vice-backends that can use Libert. So here, I'm going to use Libert. And so I create a default pool for that. I essentially reset that machine initially before doing all that, except I forgot to reset the network. So let me just delete the network, reset, recreate, so that you can see how you get started setting up a special network for your machines. And we can skip a little. And now I have no VM. And these are the images I played with. One interesting thing with KCLI is you can list all the available images. And you see there's a number. And that's, for me, the best tool, for instance, to install things like R-Cos or REL, compared to the others. It saves you a few steps in terms of being able to deploy this kind of operating system. So this is essentially the basics for what you can do with it. So we are going to, in the first run, create a regular VM and steal data from it. So to run a VM, you use the KCLI create VM. The dash i is I want the Fedora 38 image. I want four CPUs. And I want four gigs of memory. And I'm going to pass a root password, which is one of the secrets I'm going to look for. So please remember that secret. We'll use it later. So now I connect to my console. And I check that the machine boots. So far, it looks good. I'm happy with it. And you see that there is some cloud-init magic happening behind my back. I log in. I type strumpf, which is the French for strumpf. And KCLI already sets up for me SSH and stuff like that. So that means instead of going through the serial console, I can use a more serious tool and use SSH, because now we really care about secrecy and stuff like that. So I'm going to SSH into my test VM. And you see that I'm user Fedora there. So KCLI has this idea that, depending on the image, it can actually create a user for you and do not necessarily log in as root. But you can also log as root. And we are going to do that to check that there is no ACV inside. So no ACV support in this VM, the way I created it. Just a regular thing. And let's try to now inject a typical C program in there. So that's high-quality C code. QE told me you'd better run that in a VM, please. So they told me there is a bug or two in there. I don't know. Looks good to me when I compile and compile. There's barely a little one or two. Let me copy that to my VM. So I have the scp command as well. That's fine. And I connect to it. And then I'm going to run. And of course, it's going to run perfectly well, like any C program. So it's not exactly what I expected. That's not the message I put in there, but close enough. Ship it. So now I have this program. And I'm going to do something slightly more complicated. So this one, I have to look up the documentation every time I want to use it. And I'm going to dump the memory of the guest. So that's why there is a pause there, because I'm thinking, OK, where is my documentation? Where do I copy and paste this stuff? And also because I discovered at this time that reinstalling everything caused me to lose my completions. That's also because. So I'm going to use the QMU monitor command and execute a dump guest memory. And the arguments are, it's very user-friendly. It has all the quotes and backslash and curly braces you want. And I'm going to use a protocol, which is file colon something to dump to a file, which why not just file colon, but I don't know. But it's essentially I'm dumping the memory of my guest into this place. You can also do something like gcore. If you do that, you're going to have some other stuff than the guest memory. That's why it's better to do it that way. And you see this disconnected from QMU system doesn't look really good. In reality, that's probably why my machine ended up being a bit screwed, but we'll see later. So now I can drape for my secret. That was in my C program. And you see, I have a match. So let me check with Emacs what I see inside my guest memory. Emacs is being completely designed to open 4G files. So it takes a few seconds to open. It does want me. 4G is big. Do you really want to do that? But then it displays the stuff the way it should be displayed with tons of zeros control at. And now I look for secret. And I find my, hey, really secret stuff right in there. So my C program does not emit that message, but at least I see it in the memory. So I know I did load the right program. Now what if I search for my strong password? Well, of course, I find this as well. And I find it in this nice little snippet of script. So hey, that's perfectly unsafe. I'm happy because that's sort of what I wanted to demonstrate, right? So success for some definition of success, of course. So let's try to fix that. And you'll see that it's really, really super easy. Don't worry. So let's first try to make our VM compatible with what you need for continuous computing. So unfortunately, KCLI still creates all style VMs. So I have to change the hardware to say I want a more recent Q35 stuff. So let me do that. I replace PC440FX with Commu. And then it complains because, oh, now I need PCI. OK, so let me fix that too. So I need to fix my devices. The good thing about Libvert is that it checks tons of stuff for you. And then it complains about the addresses being wrong. So I just remove everything. It's going to recompute a set of addresses that match. Usually works. And the next thing I need to do is replace my SATA disk with SCSI disks. And normally I should be more or less in a good shape after that. So what I'm doing here is only trying to make my machine compatible with the kind of hardware I need for virtual machine. So OK, now I forgot something. And I mentioned the SCSI stuff. And I did not do it, so let me fix that. And that's because the devices you are going to pass in your VM need to be protected somewhere. So you need to have IOMMU active everywhere. And to have IOMMU active, you essentially need to convert to Verteios SCSI and stuff like that. So that's what I'm doing here, converting to Verteios SCSI. And everything should be pitchy. OK, so let's start my VM and check that it's still alive. Come on, don't do typos. My AI is typing. That's why it's sometimes making typos. OK, so the VM looks good. And I can still connect. And of course, I still don't have any kind of ACB. I did not activate that yet. But at least the VM has the right kind of hardware. So I need to do tons of VM edits like this. And it's well documented in LibVert, on the LibVert.org page. So you see that, for instance, that's why I took the Q35 idea and things like that. But it also tells me I need to change the boot loader. And I need to do some adjustments to the memory. So we are going to do that. Let me skim a bit quickly in that. So you need to boot with OVMF, so UEFI. And you need to do some adjustments to the memory. You need to do some adjustments to the network. This is all relatively well documented. Let's skip relatively quickly through this, because it's not necessarily the most interesting part of the talk. But it tells you what I'm spending my days on, right? Who doesn't love that? OK, so I follow these instructions. And I keep turning and turning. And I try to start it again. And that's why I got my first LibVert hang. That's why I knew that something was up. Forget that. Restart, just restart some demons. And I can start my VM again. And now it boots in UEFI. So that step looks good. I'm still not completely there. Of course, I still have not enabled ACV. So you're starting to think, maybe this is taking a bit longer. Maybe this is not the best way. So before I can actually activate ACV, I need to activate it on the host and or set. So to do that, first let me give you a bit of terminology, because AMD is very light on acronyms. So ARK stands for AMD root key. ASK stands not for ask, but for AMD ACV key. CE key is a chip endorsement key. OCA is the owner's certificate authority. P-E-K is a platform endorsement key. PDAs is a platform Diffie-Hellman key. You need to know all of this. And there will be a question at the end. I ask questions at the end. Oh, and there are two more, like the tick and the tech. And you can read. OK, so let me do my host setup. Let me first reset everything. And let's verify the host. And you see that I have this nice chain. So this morning, I taught you about the root of trust and maintaining the chain of trust. You got a good example there with this sign, this, and certifies that, et cetera. And if I run it twice, I get the same results for PDH, PK, and OCA, which you remember what it means. If I do a reset, though, I'm going to get different numbers this time. So you see that the PDH has changed. And transparently, SCVCTL is actually talking to somewhere at AMD, which I learned the hard way when their server was no longer responding to the requests. So notice also that the AMD root key, fortunately, did not change when I did a reprovisioning of the system. So that's what you do when you reprovision the system. OK, so that looks good. Now I'm going to do the guest setup. So the guest setup is primarily adding a long security step to certify the virtual machine as confidential. And you'll see it's really easy. So it's all well explained on the document here. That's the XML you need to put. So I need to find a place to put it in my code. And that's why I'm going to put that stuff. And that's over. This is where my CBITPOS and stuff matter. I need to put them there. And I need to select whether I want to put one or five. Probably both work. There were some discussions on what exactly was supposed to be there. So I've got my long security section. But it's not sufficient, because I need to have actual keys in there. So in order to do that, I need to create a security session. So first, I will export my PDH, which is the, OK, everyone, platformed if you have none. And then I'm going to just check that it works with this setup. And then I'm going to generate the files that describe a given session for a VM. So now that's related to that specific VM. Normally, you would use the host name there, because you can transport this data to some other place. But I'm going to keep it simple. Just use host.pdh and do everything local. Normally, you can do that remotely. And the policy is helpfully explained in chapter three of some obscure documents somewhere on AMD website. It's also on a different page on the Libvert documentation. And you essentially have to orbit mass for stuff. That's modern style, 1980s style of user interface. So now I've created my session. And what this did is create a few files that all begin with that VM. So there is this God mode thing. There is a tag, a tick, and a tag, and a session, and all the stuff. So that looks good. I can probably make some progress from there. And put that stuff inside my file. So I'm going to insert where there is dh side. So dh should remind you, of course, that it's pdh. It's the same thing. So you insert the copy of the file there. And then you insert the copy of the session file in the session section. I think that makes sense. I mean, that's simple. That's a good user interface. Use emacs, use vi, whatever. I mean, just text. There is some remarkably low entropy on this sequence of As that I find really surprisingly low. I don't know why exactly. That's something I'd like to look into, why there are so many As. OK, so now that's edited. And I'm going to do it the scv old style way, scv scv es. I'm not going to try remote attestation, because that's complicated. So we'll keep the simple stuff here. And so what we do, essentially, now is that we are going to enter a brave, new, secure world where we are completely productive. So to launch a secure VM, the first step is to start the VM, but you start it in post mode. Because reasons. The reason is you have to check before you actually launch it. That's called pre attestation. Remember, I explained that this morning. So you have to check that it passes the test before you let it go. Then you do this dumb launch second for test VM that gives you a measurement of what you are trying to launch. And then you need to validate it with this helpful tool where you just have to copy paste the output of the previous comment. But of course, with different option names, like everything that begins with scv it does not begin with scv on the command line. Other than that, it's really simple. You do that, and you try to not typo too much. Oh, you have to measure the firmware, so you have to tell where the firmware is. And with that stuff, it looks pretty inoxy to me so far, right? Now I'm going to get my grade four. And I get this nice answer. Looks good to me. So now I'm allowed to resume my VM. I can say, hey, resume, console, and let's go. And what happens? So booting again into EFI. So what should happen there? I should see a CV. And I should be able to run my application without its content being visible. That's what I expect. So I do that. I connect. I type stromf, which is no longer so much of a secret. And I have a CV, success. I'm really happy. Memory encryption features active. Yay. Let me SSH into it from another window and run my little test forum. So let me copy the latest version where I upgraded nothing. I don't know why I need to copy it. I think because in the meantime, I recreated a VM from scratch. So I run it. It works just as well as before. And now I'm going to do my memory dump like before. But now I have it in the history, so it's easier. And I take the encrypted version. What do you expect? OK. Same here on the same thing. But let me now grab. Ta-da-da-da-da. That is a pretty. Huh. Let's look with the max. Something is wrong there. Hmm. I have to wait to see why it's bogus. It's slightly faster. And the reason for that is that we see the world secret as part of some fight system stuff, which is already, I'm already scratching my head. But I don't actually have my secret stuff with 3F. It's not there. So something is working. I have product it somewhere, something. But if I look for Stromf, oh, OK. Who can tell me what I did wrong? I scratch my head. I thought maybe it's something on the console that is going wrong. So I'd say if you can't answer, if nobody in this room can answer, it means I'm not the only one being stupid. So you're all as stupid as I am because I had to scratch my head for a while to figure out where did I go wrong, what did I do incorrectly. No, it's much simpler than that. So I double check. SCB is there. No, we have a problem. If you have not seen the WhatsApp talk, you search Google talk. WhatsApp talk right now, you watch this. OK, so it's time to tell the boss. Dear boss, my demo for next week doesn't work, which is why I completely changed compared to what I was supposed to talk about. Dear Ryan, my problem doesn't do CPCs of my secrets in there. And I'm not sure why. So that was my secret. The first experiment was with TicTac.toto, SprintShot, SCB is active. And so for my boss, isn't that the punchline? I'm just supposed to point out from that. OK, so the reason for that is that only memory is encrypted. What I did wrong, if I grab the stuff from my program, it's not there. That stuff is not visible. That part we succeeded with. However, when you do encrypted memory like this, you cannot send that to this directly. You cannot write to a network directly. So all the IO buffers that are being used by the kernel have to have the famous CBIT clear, saying it's not encrypted, which means that you'd better encrypt your disk. And that's the thing I forgot about KCLI, is that by default, it uses this good old card in it. And there is no encryption of the disk setup by default. So we need tools, and we need to build tools to keep that complexity and reduce the risk of misconfiguration. So first of all, how did I get there? You can blame Wainer over there. I'm pointing fingers there, because I'm also doing a workshop in two days with Wainer. And his workshop was about, I know it's Q&A, but it's the last session of the day. So I have a little extra time. So I have too many talks going. And he did a remarkable series of script for this lab. And he was using KCLI all over the place. And I said, ah, that makes it really shorter compared to what I was planning to present. So let me use KCLI. And I switched to KCLI and did not realize it wasn't encrypted until the very last step. So step one, prepare a list of commands for a live demo. Step two, prepare another workshop with Wainer. Step three, notice that this workshop uses KCLI. Step four, think, ah, great idea. I used that too. Step five, record a movie with all the planned steps, but where you replace carefully all the steps where you were creating encrypted VMs with a fun installation with KCLI. Totally forget that KCLI does not encrypt disks. Success. So that's how you fail spectacularly in public. Now, there are a bit of ways, and I'm going to show them that should be for another talk. You can set up VMs in the cloud. That was at 10.30 this morning with Italy. You can use confidential containers. That's the talk we have tomorrow until this. And you can deploy complete clusters, which I'm going to show on the next slide, with a tool like Constellation, for instance, which is careful to set up the VMs the right way for you. It's just that it would not fit in the 30 minutes that are still alerted to me after the end of the talk. And so essentially, you wait, and you wait, and you wait, and after something like 40, 50 minutes, you get a cluster, which is really set up the right way. So this is the recommended way to go if you want to really spend your money. If you have too much cash on hand and you're ready to pay for all your VMs and all your control plane and everything to be encrypted, go with it. That's very easy. It's then behaves like your regular cluster. So then you can connect to a node and check that it's encrypted properly, that the disk is encrypted, et cetera. And of course, we are working on this for Fedora and CoreOS. So what you see here is the creation of a Fedora CoreOS disk with ignition, with a secret injected, and that gets encrypted on the fly as the ignition step happens. And I have to thank Timothée Ravier for helping me doing that this morning. This is relatively new to the talk. Now, it's time for questions if there is any. There is no time for questions, but... So the question is about what is the cost of memory encryption mostly, right? And so in terms of power usage, it's hard to tell because it's not exactly the same generation of processors. So if you do processor with more transistors, but they're smaller, then maybe it's not consuming that much. Yeah, I don't think that using encryption adds much in terms of consumption of memory. Sorry, you have energy. And it does not add latency. We did some measurements. And in terms of latency compared to non-encrypted, it's the same speed. However, on the same CPU, however, what it adds cost is that you cannot do any kind of deduplication. You cannot deduplicate your disks because they are encrypted. You cannot deduplicate your container images because now you want to download them from inside your virtual machines. So you're downloading the same thing over and over again. And you cannot do any kind of KSM or anything with encrypted memory anyway. Don't ask me. So the question was, when do we get cheap hardware? So to be precise, yeah, to be precise, I think it does consume a little bit of power, but when you see that you have chips that consume 100W or 130W of TDP, I'm sure there is half a watt in there that is encryption, but I don't think it's more than that. The reason I think it's cheap, the reason I think it's cheap is that my phone does it on a battery for a whole day. So it keeps encrypting stuff all the time when it's communicating, and they accelerated in hardware or the encryption with the network and so on, precisely because the hardware could do it relatively cheaply. But to be fair, you've got the wrong Dinshan. You've got the wrong Dinshan. My brother Flo is the hardware guy in the family. I have a brother who does FPGAs and stuff like that. He could tell you everything about how much it costs to encrypt stuff. I can't. Sorry. And back to your question, when does that become mainstream? I'd say as usual it will probably take a decade. So the feedback we got so far is that there are a few markets that are already interested today because they have to deal with regulations constraining the kind of, their responsibility in terms of data leaks, and that's banks, medical, stuff like that. But also that because this becomes available, of course the manufacturers are not being all they can to make this mandatory. And so now it becomes a lot that you have to encrypt your data that way. And so maybe there will, in some cases you won't have a choice. Time will tell. We'll see how this works. No, so when I'm saying it's weak is that it's typically done in real time. It's still modern algorithms, and the algorithms they use are dependent on a chip. And frankly off the top of my head, I don't recall which one they use, but I think they have AES 128 somewhere, some label for the first generation chips, and now they are doing more complicated stuff. So it's not the stuff that you can break with your computer without paying a lot of energy. And by the way, I suspect that one reason that triggered these efforts is, I don't know if you recall like almost 10 years ago there was this big push about memory stores, and persistent memory, non volatile, RAM, and this kind of things. And everybody at the time thought that we'd have really cheap, non-volatile memory one decade later, so that's now. And so the problem became for the chip vendors, I need to make sure that if someone plugs one of these non-volatile chips, they don't get all the data that was in there. So that was probably a primary motivation to get started on this encryption stuff, initially. And in that space, so it had to be relatively robust anyway. So I apologize for the use of the word weak, I meant it's not as strong as you could do if you were really pushing the limits on your crypto, but it's pretty good. I'm out of time, I knew, I knew that.