 So I'm waiting for my machine to come up. It died on me just a little bit ago. So just as a quick introduction, the protected execution facility is a hardware architectural change that we're making to the power architecture. And because it's a hardware change, it also requires some supporting firmware. Most of my talk is going to be about the supporting firmware, because we're doing this for power and open power and for power Linux systems. And so that's what I'll be talking to you. But I'll also give you a while I'm doing it, I'll talk to you about what the hardware changes are. And I'll take questions at the end on any component hardware or software. Let's see. What else can I tell you without any foil? Unfortunately, I need these pictures. Let's see what's going on. I haven't plugged my dongle in, because there's nothing to see yet. And as soon as there is, I'll plug it in. So it looks like it'll be about another minute or so. So on my acknowledgment foils, I tell you that some of the concepts that we developed that are represented in this presentation were developed in research that was sponsored by the US government, the Canadian government, and the United States Air Force. And so on the acknowledgment foil, there's a long paragraph that we're required to put on any publication that results from this work. And there's a pointer that lists the contract number and all the official words that each government wants us to say. And at the end of the presentation, there's a pointer to the final report from that project, along with pointers to other things. I'm also supposed to tell you that these are my ideas, not IBMs, and that anything that we talk about is subject to being accepted and upstreamed, although a number of the patches that are required for this to work properly have already been pushed. And I have pointers to those in the presentation so that you can look at them. And there are more that are coming. You can ask me when it's going to be available, and I won't answer. What? Can't answer that question. That's about the only question I'm not supposed to answer. And let's see. What else? Yes, the team that developed this spans IBM research. Actually, the work was done, the initial research was done in IBM research, and then the team spans IBM Cognitive Systems, which is sometimes known as our power division. And it uses the LTC. Claudio was here earlier today and talked about the LTC and gave you all these locations, about all those locations and a couple of others we're using in this work. And then let's see. I don't have a map. So this has been going on for a while. Let's see if we can get this to come up. Yeah, I'm just waiting for it to wake up so I can put it into screen cell mode. It will be a while. Let's see, what else can I tell you? So that covers the first couple of introductory foils. We've had a lot of presentations about various challenges, especially this morning, about challenges in bugs and operating systems and hypervisors and all the other sorts of things that you're doing. So the basic idea here is that, well, let me wait till I get this thing to go, if you see it OK. So I've given you my name. I don't think this is quite awake yet. And this was my acknowledgement foil. So we'll skip that. I gave you that verbally. This is basically what I'm going to do is walk down. I'm going to give you an introduction. Then I'm going to give you an overview of the architecture. Then I'm going to give you lower level details. And then I'm going to summarize. And each time I'm giving more details about what we're talking about. We talked about this. I forgot to tell you that all of the software and firmware that's associated with this facility is or will be open source. There's some new stuff that's not quite out yet. We were talking about this this morning. Several presentations have been addressed at what do we do for cloud infrastructures? What do we want to do? We're working on regression tests. We're working on verifying. We saw yesterday there were lots of bugs that didn't get introduced. And they get fixed slowly. So there can, unfortunately, be bugs in what people call zero-day compromises. So what we're trying to do with the protected execution facility is we're trying to introduce this concept of what we call a secure virtual machine, abbreviated SVM. We're trying to protect the SVM against attacks, protect confidentiality and integrity of the SVM. We're integrated with trusted computing tooling. We rely on secure and trusted boot to get the machine up and running. We enable secrets to be embedded inside an SVM at creation time. We enable existing virtual machines to be converted into SVMs with new tooling. We have a smaller trusted computing base than our traditional servers or open power systems might have to reduce the attack surface. Exploiting the open source ecosystem we're going to be extending it by open sourcing all of the firmware associated with this. There's no limit in the amount of memory we can protect, and existing application code can run in an SVM without change. So what we see here on the right is this picture of a couple of VMs. And this thing with the white brick wall around it is the SVM and the red arrows are illustrating the fact that you can't attack that either from the kernel below or from another virtual machine running if either one is compromised that just won't get into the SVM because of this protected execution ultrvisor, which is the firmware component of the protected execution facility. Consequently, we protect against road administrators and compromised hypervisors. And we protect while in transit executing or stored on disk. And I realized that I didn't explain the in transit thing. Basically, the secure VM is encrypted. And so you can ship it around. And it's encrypted with integrity protection and integrity information. So you can ship it around over a wire or leave it on a disk as much as you want. Because when it goes to get executed, the protected execution facility will verify that it hasn't been tampered with before it allows it to execute. If it detects a tamper, it will not execute. So there's basically two things you want to do. You want to create one of these things and you want to run one. So you start with a regular normal virtual machine. You develop whatever applications you want to run in it. We're assuming that you will do this in what you consider to be a secure facility because we cannot protect you against what you put inside of that virtual machine. And then we give you some new tooling. And basically, one of the features of the architecture is now every machine has a public and private key. And so you have to collect the private keys of where you want this thing to run. And you give that to the tooling. And the tooling can make sure that your file system is encrypted. And if it's not, it will encrypt it. And then it does some other calculations. And it builds the integrity information so that we can verify the virtual machine. And it outputs SVM, which looks like a normal virtual machine on Linux KVM. So when you're running it, it starts like any normal virtual machine. It starts in regular memory. And during the brute process, it executes a new sys call called an inner secure mode system call. That goes to this protected execution ultravisor. And it checks the integrity of the SVM. Well, actually, the first thing it does is it moves the virtual machine to secure memory. And secure memory can only be referenced by the ultravisor in something running in secure mode. And then it checks the integrity. And then if the integrity check passes, it passes execution and continues. And during the execution of that virtual machine, all of the interrupts that occur during that virtual machine go to the ultravisor, not directly to the hypervisor or to the VM. So there are two basic kinds of interrupts. They're the hypercalls that the virtual machine would do. Those go directly to the ultravisor. The ultravisor puts dummy state in all the registers for the virtual machine, except for those required for the hypercall that has been executed, reflects it to the hypervisor, and returns the results to the virtual machine. We'll talk about this a little bit more. For asynchronous interrupts, what it does is it saves all the state of the virtual machine and then reflects the interrupt to the hypervisor. But no state that's not explicitly being passed to the hypervisor by the software in the virtual machine is ever seen by the hypervisor. So let's go to our higher level overview. So here's the base principles. Protecting integrity and confidentiality of code and data. Minimize the trusted computing base. The trusted computing base here is the processor hardware changes and our processor, which is huge. But we have to trust it. The TPM and the firmware that's associated with this. The firmware includes host boot, opal, and the ultravisor, but not ski boot. We get booted before ski boot. We introduce secure memory, which is only accessible to secure virtual machines and the ultravisor. We introduce a new power processing mode. Previously in power systems, we had hypervisor mode, supervisor mode, and problem state, for lack of a better names. Now we have ultravisor mode, which is higher privilege than hypervisor mode. So we have four modes instead of three. We have hardware and firmware that's used to manage this, and we enable secure virtual machines. But normal virtual machines can run on the same hardware. So all the normal ones run. So here's a pictorial overview. At the bottom, we have an arbitrary open power CPU chip with the protected execution facility. On the left, we have some form of non-volta memory. We're relying on the secure and trusted boot that Claudio talked about. The private key for the machine is in the TPM, and never leaves a TPM, is never exposed to software in the machine. This works because in the TPM 2.0, there's an interface that you may be aware of where you can say, oh, generate me a public private key here, and give me the public key. And then you set a policy on when that private key can be used. The memory can be divided into secure memory and normal memory, and you can divide it any way you want, except that you can't change that division after the system has been booted. So then you have the protected execution alt advisor that gets booted, and then the hypervisor, which in our case is Linux KVM. And then above that, you have the environment where the virtual machines run. You have the secure virtual machines running in secure memory. You have the normal virtual machines running in normal memory. They're all coming off disk. And this just illustrates the tooling that's necessary to convert a normal virtual machine into a secure virtual machine. So I already told you we were referring to some changes. And I told you about the firmware. We sort of went over this. You run secure virtual machines and normal virtual machines in the same hardware. To create an SVM, you're going to need the new tooling that we're going to open source as soon as we've finished all the little things we're doing with it. And the SVM executes insecure memory, which is under the control of the alt advisor. And the hypervisor and the normal VMs cannot reference secure memory. Cannot reference secure memory means that the hardware will not let them. It's not the alt advisor's decision. It's the hardware's decision. It won't let anything other than something that's run in secure mode reference it. So none of your devices can reference it, et cetera, et cetera, et cetera. So a little bit lower. The private key remains in the TPM. The alt advisor uses the TPM to get access to the symmetric key. In a minute, I'm going to give you a more detailed layout of the secure virtual machine. And you'll see how all of that works. The alt advisor uses a secure channel to talk to the TPM. So it says, here's something that's encrypted. Give me back the decrypted blob. The TPM does that and passes it back securely to the alt advisor. The hardware separates the memory into secure memory and normal memory. And then when the ESS call is received, if the calling SVM has not been modified, the alt advisor will transition it to secure mode. As I said, what it does is when it gets that call, it moves it all into secure memory. And then it does the integrity verification. If the integrity verification passes, then it lets it go into execution in secure memory. We're higher privileged than the hypervisor. Linux KVM has to be per virtualized to work with this facility. Most of the changes are in the architecture dependent section of the hypervisor. If the hypervisor needs to update the partition scope page table, we'll have to ask the alt advisor for assistance. If the hypervisor needs to return to an SVM, it will have to ask the alt advisor to complete the return. And we're also going to modify the heterogeneous memory management so that we're using that to help manage the secure memory rather than inventing something new. At the VM level, normal and secure virtual machines run on the same hardware. The alt advisor is not involved in the execution of normal virtual machines at all. They both get services from the hypervisor. The alt advisor, I haven't said this in this presentation. Whereas the hypervisor is millions of lines of code, the alt advisor is on the order of tens of thousands of lines of code. It's a little bit larger than anything that's been formally verified today, but it's not that much larger. And we are working towards certification. That question came up on one of the other talks. So I just thought I'd mention that, although it's not written on the false. We create them with new tooling. The secure VM starts as a normal virtual machine, and we use this new sys call to make the transition. Now there are some limitations. And we will not support, in the initial release, suspend, resume, and migration of a secure virtual machine. All of these things we can do, we're just not doing it in the first release, because we're trying to get it out on a certain schedule. We're also not going to support over committed SVM memory in the first release. We can do that. We can allow the hypervisor to page the SVM memory because the architecture makes sure that the hypervisor can only see the memory of the SVM encrypted with integrity. And we do not allow, right now, dedicated devices going to a secure virtual machine. So you can't get to it from a GPU card, from a PCI card, or from anything else other than a few things. We do not currently support transaction memory in SVMs. Transaction memories are the best side channel that anybody ever introduced into an architecture. We're not supporting them right now. Applications that use transaction memory will crash as soon as they do. So let's get to the lowest level details that I'm going to be talking about in this talk. So I'm going to go back for one for all, and then back to the other one. This is the overall layout of a secure virtual machine. Sloth is the bootloader, and on the right of that, you have a disk image. You can see that the kernel is a Z image. We don't care if it's an MBR or GPT format disk. This is a virtual machine that's booting. It's not booting native on the hardware. Go into Grub. Grub goes to the slash boot partition, and there can be one or more Z images there. Each Z image has this thing we call an ESM blob, which is on the previous file. I'll talk about it in a second. And then the remainder of the disk is the root file system, which is encrypted. Okay, going back one. So what this ESM blob is, there's a symmetric key which was used by the tooling to encrypt the verification information, which is in this blue box. It's encrypted with the public key. The symmetric key itself is encrypted with the public key of each machine the SVM is authorized to run on. In this particular illustration, this ESM is associated with an SVM that can run on machine A, B, or machine C. The alt-reviser asks the TPM to use the private key of the machine, which is stored in the TPM, never comes out, to decrypt the symmetric key. So the alt-reviser passes, for example, if it's on this green machine, machine B, it passes the green blob into the TPM, says give me the results. It comes back, assuming only if the correct firmware was loaded, it comes back, and then the alt-reviser has this symmetric key associated with this particular SVM, and then it can decrypt this blue blob, and it can get the integrity information for the kernel, the integrity information for the NITRAM FS, the integrity information for our tasks, a little bit of information about the symmetric key blobs, and the dmcrypt passphrase, so the actual root file system can be decrypted and the virtual machine can finish booting. Now, you might be asking yourself some questions about this. Are you gonna make every machine have a different public-private key? Well, no. We made it possible for you to make every machine to have a different public-private key, but it's also possible for you to have multiple machines with the same public-private key. That's up to the owner of the machine, whether they wanna use a unique one on every machine, or if they're building a cloud infrastructure, that's not necessarily likely. You can have the same key on multi-machines. You can use the protocols established in the TPM to save keys and a bob that you can restore if your TPM breaks so that you don't lose the activity. We don't pass the private key of any machine out of the TPM. So what that means is if you have some form of a crash that dumps all of the memory of the machine, including the memory, the secure memory, although our crash tools do not dump certain areas of secure memory, but if there were some sort of a bug, when nobody ever has bugs in their software, and all of the secure memory got dumped out, the most that you could expose would be the symmetric key of a machine that was about to boot, because after we get the machine running, we destroy the symmetric key and we don't need it. You only need it while you're in this process of checking integrity and stuff like that. And after you've set up and set up stuff. So that ESM blob that I just described is this little yellow box next to the Z-images, and that's what we have there. Okay. So boot changes. We had to change prominent because prominent made modifications to the kernel and other things that would cause the secure virtual machine to not pass integrity checks, but it was a straightforward change. If you wanna look at it, it's right there. We're using the Z-image format for booting these things. So we had to change wrapper, which is the name of the script or tool that writes, that converts a VM into a Z-image. We had to change it so that we could extend the ELF format so that you could throw an ESM blob in there and it would be available for the ultravisor. That change has already been pushed. You can take a look at it if you want. We may have to change grub to make config, we're still looking at alternatives, so that change has not yet been pushed up. So this ultravisor, what do we do with it? So if you're familiar with power, so it's higher privilege than a hypervisor and there are certain things that you can do in power processors that are security sensitive. These registers we call SCOM registers, they control the configuration of the machine. So if you're a hypervisor and you wanna read or write a SCOM, which is a legitimate thing for a hypervisor to do, you have to ask the ultravisor. There's also this thing called stop and restore state, which is used for suspending processors and stuff like that. Even if you're suspending a core and it doesn't have anything to do with a secure virtual machine because of the way that stop and the stop states work in power processors, you gotta talk to the ultravisor, et cetera. They page in and page out so that we can do the secure paging and then you get the idea. There's just a few interfaces into the ultravisor. Some of these can be used by a secure virtual machine and the rest of them are intended for the hypervisor. We allow secure virtual machines to share, they have to share unprotected memory with the hypervisor, otherwise they could never get out over the network. So that's what's going on with sharing a page. We allow the guy who's writing the secure virtual machine to say, hey, I wanna share this buffer with the hypervisor unprotected. We're assuming that if he's using it as a communication buffer, he will put TLS or some other form of correct communications on the data that goes through there so we don't have to worry about it. We can't do everything but we can facilitate what you wanna do. There may be a few changes to this list as we move forward and some of those may change. KVM, KVM being the hypervisor, we need it to give us a couple of extra H calls. We may have a few more additions. There'll be a talk on the changes to KVM at the KVM form in Edinburgh in October. And as I mentioned earlier, we're changing the heterogeneous memory manager so that we can use it to manage the secure memory rather than inventing something new. Kernel changes. These are kernel changes in the secure virtual machine. As I commented earlier, nothing can reference secure memory except the secure virtual machine in the ultrvisor, which means vert IO won't work unless it does bounce buffering. If we allow shared non-protected memory and so these vert IO changes that have already been pushed up for comment are similar to the changes that AMD introduced for their approach to secure virtual machines but somewhat different. And basically what we're doing is we're changing the low level infrastructure of vert IO so that all of the vert IO drivers don't have to change. So if we put the change in a place that's transparent to the driver so we're just changing that, you can take a look at it. VPA in power is a communications area between the virtual machine and the hypervisor that both the virtual machine and the hypervisor have to write. So that has to use the shared memory ESM call and that change is being developed and we expect it to be pushed in August. So this is my brief introduction to the hardware changes and it's brief. This is aimed at you if you happen to know power architecture. If you don't know power architecture this is not gonna make, some of this isn't gonna make a lot of sense and I apologize for that. But point I made earlier, the amount of secure memory you want to have in the machine is configurable and it's configurable per memory controller. So one memory controller can have more or less than another, that's up to you. We've taken one of the reserved bits out of the MSR machine status register named at the S bit and it indicates whether or not you're in secure mode. What this means is that there are now three bits that determine the privilege of an executing entity on the machine. There's the MSR sub S bit, the HV is the hypervisor bit and the PR is the problem bit. So PR meaning zero, you're not in problem state, PR meaning one, you are in problem state, problem state being the lowest privilege state. So you can sort of see a little illustration that tells you what the states look like. And there's one state that's currently renewed. We added a new register called SMF CTRL which tells the system whether SMF is enabled or not and a couple of other things about it. We had to add some new registers, UR, MOR, USSRR0, USSRR1, USPRG0, USPRG1. All of these registers mirror registers that exist for the hypervisor and power of architecture. The UR, MRR is a relocation register that the system uses to find the altivizer and the other registers are altivators and privileged so we can handle interrupts and other things like that properly. We have a new instruction, URFID so that the altivizer can start up a secure virtual machine. And so when you're in your MSR, so this is one, you're either running the altivizer or a secure VM and privilege, which is a state or problem state, which is the application. And as I commented early, all hypervisor interrupts go to the altivizer. There is one hypervisor interrupt that the altivizer does not reflect to the hypervisor. In our old systems, there was a hypercall to get a random number. That's not necessary anymore because we have a random number instruction in the hardware, but if you have a system that's using that call, that call is not reflected to the hypervisor because we wouldn't want a unfortunately compromised hypervisor to manipulate random values being used by the secure virtual machines. That's handled directly by the altivizer. And as I said, asynchronous interrupts go to the altivizer. So now my summary. So we're protecting you from the hypervisor, other software, if there happens to be a rogue system admin. The security domain is the VM when it's at rest and transit or executing. Applications, no changes to run in the SVM. The guest OS, we're creating some new Kconfig options so you can say I want this to be a secure virtual machine. The hypervisor, we have to pair virtualized. We're making those changes. All that will be open source, including a tooling for this. There are some changes for both OPAL and host boot associated with this and they will also be pushed out in an appropriate time. The secure memory gives you integrity and confidentiality protection. You can embed a secret in the secure VM because it's encrypted after it's created at all times. The memory size is limited to available memory based on how much you're going to allocate. Available memory minus the memory you allocate. You have to leave normal memory for the hypervisor so you can't use 100% of available memory because the ultravisor is not a hypervisor. And everything associated with this is already or will be open sourced. So this is the page I mentioned earlier on. IBM's been building cryptographic co-processors and secure hardware for about 20 years or so. The first being something called the 4758, which was a cryptographic co-processor which was done in the late 90s. And it was also tamper proof or tamper resistant whichever you prefer. If it detected your tamper, all of the secrets disappeared. And the link there is to the latest version that was announced either earlier this year or last year. We also produced something called the SecureBoo, secure processor technology. Several million of those have been out and around. We created a research project called SecureBoo Plus Plus. We also have announced on Z-Series, secure service containers, secure execution technology on IBM Linux 1, which is related to this work. And then finally this last link is to the final report for our DHS sponsored project. Sponsored by the government of Canada and the US government. Are there any questions? There's one right here. So I have a couple. First is, let's say you've got three keys under which give you a set of hosts on which you can execute, right? Is there any way for a VM to know which of those keys it's executing under? We didn't think of that. We could make a way, but there isn't one in the interface right now. Is that an important thing? I'm not sure yet. I mean, it's easy for us to provide an ultravisor interface that it can find out because we know the ultravisor knows. We just don't pass that information up to the VM at this point. So a second and very different question. If the ultravisor firmware were compromised? Yes. You would have the machine into the world. Is there any way of letting an orchestrator, I, whoever's gonna be sending the machine, the SVM image, know of that as part of an attestation? Okay, so in our secure boot architecture, which Claudio talked about, we're doing secure and trusted boot. So if you boot to the ultravisor, it hasn't been compromised in a sense unless it was compromised when it was created. Okay, if it is compromised, it won't boot. It'll boot and the ultravisor can boot, but we're using trusted boot to give the ultravisor access to the private key. So if the ultravisor's running and it's not the right ultravisor, the TPM can never get the private key, so nothing gets out. Okay, so why have yet another layer? Why did you not just extend the supervisor to do the hypervisor functions? Because the attack surface would be significantly larger. See, we found instances where we didn't wanna write another hypervisor or make it very complex and we found that when we looked at the VM to OS interface, as others have done, we found that it was fairly easy to secure it and save the state of the running VM and pass only the information needed for the call. So that's what we decided to do. I think if we tried to extend the hypervisor or something we would be including, we would have to try to create almost another level with just firmware and we decided to introduce a new hardware state because then there's a whole lot of stuff the firmware doesn't have to do because the hardware does it automatically much faster. Any other questions? Yes. Question about scalability. So you have a number of private, I guess public private key pairs stored. One per machine. One per machine. Yes. How many machines can you support? I guess that's limited by the TPM memory? No, okay. So each machine has a TPM on it, one or more TPMs. And so this scales to as many machines as you want it to scale because the ultravisor is only running on one machine. It runs on all cores on one machine. Yeah, you can, and the same public private key you can put on as many machines as you want. That's in the TPM infrastructure. So you can do what I think it's called replicating or duplicating a key. So if you have a machine and you're building up an infrastructure and you want this data center, let's say you're doing a data center, sort of obsolete, but you're doing a data center, and you want all the machines in the data center to have the same public private key, you can do that in our architecture. If you're building a cloud infrastructure and you want all the machines in this cloud, you have one instance in Dallas and another instance in New York and another instance in France. So you have different instances all the way around the world. And you want the particular cloud instances to have their own, each to have its own public private key, but you want them to be different because you're using that to obey GDPR and make sure certain computations can't run out, can't be used outside of their national boundaries. You can do that. So the ESM blog, yeah. Yeah. No, no. You make it as large as you want. Well, you put a million keys in there, it's going to take a little longer to boot. Yeah, but that's it. And do you have storage on your media for the size of, you know, et cetera? It's not going to be no scaling issues that we're aware of. Any other questions? Yeah, this is related to what he was just asking about. So with regards to the key management for the TPM, what kind of infrastructure or services are available to actually have visibility across your fleet for all of that key management? So we do not provide any way for you to see the private keys because if you were able to extract a private key and somebody stole it, they would now have entry to every secure virtual machine that was authorized to run on your infrastructure. So you can't do that. We provide no means for that. We don't tell you how to manage your public keys. We leave that up to you. You can use any key management infrastructure you want. What we do do is we enable you. We put a public-private key in the machine when we manufacture. When you get the machine, you can change it. When you get the machine, you can replicate a key from another machine into it, or you can make a new key. Or if, for whatever reason, you decide you really need it, if you sell your machine, you bought the machine, you keep it for five years, you sell it to somebody else, they can put in their own. They can change it again. So we allow you to change it. We're going to supply tooling so that you can update and change the keys. All of this stuff is in the TPM 2.0 spec, and we're going to supply software that makes it easier for you to use it so that you can update the keys. And then once you have the public key, then we're assuming that you would manage it as you would manage any other public key in your infrastructure. Kind of as a follow-up to that, you said you could import a key from another computer? You can, in the TPM 2.0, I can tell it to generate a public-private key peer. And then there's another command. I don't remember the exact name, where I can export that public-private key peer encrypted and store it someplace, and then I can restore that into another TPM or back into the same TPM. And I can use that facility to replicate the same key across multiple machines. This is TPM 2.0. We didn't invent this. We're just using it. We did this called duplicate. Duplicate, yeah, that's what it's called. Okay, go ahead, he's got one. Are there any occasions when you might need to be able to revoke a set of public-private keys? And if so, how would you do that? It is possible that you might want to revoke a key. The one thing I didn't point out is that with this scheme, once I authorize a VM to run on a piece of hardware, it's authorized permanently. And so if for any reason I wanted to not enable that VM to run, I would have to revoke the public key. The weakness of that is obviously that if I stop, if I disable that public key, I disable every VM that's used that public key, not just a particular VM. So then you get into other questions that we're also looking at and thinking about. So I mean, that's an important question. We're sort of assuming, we're not that concerned about it although it's an important question because we're assuming that you have some degree of access control management over those who are authorized to run things on your system. And so your first step would be to withdraw their credentials. And then so this other step would be more drastic if for some reason that wasn't sufficient. Anything else? Okay, thank you.