 Hi, all welcome And thank you for the introduction. It's great to be back at a Linux security summit and I wanted to take the opportunity today to Kind of give you guys an update on what's been going on with AMD's secure encrypted virtualization or SEV feature both what we have been doing within the last year and Kind of some of the things that we're looking at in the near future as far as open source support goes So for anyone who's not familiar with this feature, this is a technology that we introduced about a year ago now into our server line of x86 chips and the SEV feature is designed to offer protection for virtual machines from the hypervisor and so this is kind of a change in the traditional trust model of virtualization and and There's a number of potential scenarios where this is interesting kind of the most obvious one is a public cloud type scenario and Where we have this situation where From the guest perspective, so kind of the cloud customer, you know, they they want to to use a cloud But they see the potential risk of the hypervisor and the access that it has Obviously the hypervisors code that they are not able to control directly But the customers have various obligations including sometimes legal obligations to protect the data that they're working with And so that's concerning and on the the hypervisor say the cloud vendor side, you know It's a similar situation. They want to be able to offer more security to customers and really In general, they tend to see the visibility as more of a liability That is the the visibility they have in a customer data as a liability that that They are concerned if someone comes asking for that You know, that's not really what they want to be providing And so the the net result here is that, you know, we believe there's this opportunity to Try to offer some security enhancements in this space So the AMD SEV feature is designed Primarily to protect against two different classes of attacks on the one hand. We have physical access attacks, which means someone is kind of physically on the motherboard and they're probing the DDR bus or They're doing a cold boot attack or something like that. And so Because we have Memory encryption is part of this feature then we can offer strong protection against that We also are offering protection against certain types of admin access attacks. And so this is like Someone with root access trying to do a memory scrape of a guest Trying to inject code Into that guest to get it to execute that or just as kind of an extra layer of resiliency Against a potential hypervisor bug or side channel attack or something like that that can potentially allow One guest to escape its sandbox and try to see what else is going on and So just kind of at a high level the way that this feature works. There's a few components This is rooted in something called the AMD secure processor, which is kind of a security subsystem That's embedded in our SOC. It's got an arm core on it. It's got some private resources and this implements the key management and For the VMs it provides an API for doing VM lifecycle management and So this API has things to help with starting these encrypted guests generating attestation reports Assisting with migration things like that. This is publicly documented. We have a spec on our website that details this and from the Kernel perspective the secure processor just looks like a PCI device. So it has a standard MMI interface and This API is intended to be called from the hypervisor as it's doing these various lifecycle tasks So the hypervisor is responsible for still managing the resources in the system calling the various functions that it needs to on the secure processor and Then within the guest we have support for the SUV feature within the OS kernel And this is primarily because there is a new x86 page table bit that's defined as part of this architecture and That bit allows the guest kernel to indicate pages of memory that are to be marked private meaning that they're encrypted with the guest specific key and shared versus shared pages which are accessible to the hypervisor and so the way this Set up today in Linux for instance is that the vast majority of pages are marked as private Except for a few pages that are necessary for either para virtualization features or things like DMA So the nice thing is that once we have that support within the guest kernel There's no application changes that are necessary so all the applications within the VM just run as normal and they just default to using encrypted memory and And Kind of an example support environment for this today We have supporters. I'll get to it in KVM QMU And then we require using OVMF as the BIOS within the guest VM to help to bootstrap that and And I say this is this is now commercially available So I've got a Brief demo video of how this works and hopefully this will this will play But essentially the setup here is that we've got a server. It's running two different virtual machines one with this feature turned on and one with it turned off and We've kind of made it pretty with a little website here, but essentially these are Just commands that someone with with root access on the box is able to execute and so If we play this Yeah, so we have just looking for a QMU process and then pulling up the memory map for it and Then just dumping the memory with a DD command and So the first page of memory is is mostly zeros so that's that's of course as expected if you do the same Same Same thing but with a VM that has been run with this encrypted virtualization feature So we have that on the right side here We do the same thing and the data is Essentially random and that's because the hardware is aware that we're running in the hypervisor mode versus a guest mode It only allows access to the key when it's running in get in this particular guest and so when you try to view that memory from outside then then you just see the cipher text and So then what this is going to show is Just a text editor. So this is like a user editing file in that VM. It's just sitting in memory it's not actually written to disk yet and the the admin user on the on the Native box that's running these is going to basically do a grep command on the memory and Of course on the left side. It's very easy to find the text. That's it's in this file and When they do the same thing on the right side then They find nothing and so, you know, this is about as exciting as a security demo get so that's that's what we got And this is this is a two gig Of ram guest so as you can see it's pretty easy to do it. It's pretty fast Okay, so what's been going on in the last year as far as this feature goes so The main thing is that we've been starting to upstream support for this feature as well as We have the SMU which is secure memory encryption feature, which is a simpler form. That's just Memory encryption support for a native system And so we we have support for that starting in in 414 and then we we kind of had two sets of patches for the scv support so we had the Guest patches that made the kernel aware of this new page table bit so that it could be run inside one of these VMs that went into 4.15 and Then the KVM patches that interfaced with the secure processor and did all the Appropriate management for these VMs went into 4.16 And so depending on when some of the recent commercial distributions picked up their kernels There are certain ones that are capable of being run out of the box today as either an scv guest or an scv hypervisor We do have a website that I'll mention a bit later that has kind of getting started instructions for each of these distributions So if you're interested in kind of playing with this and want to know the Options to give the QMU and all that stuff. We do have that documented The other thing that we've just gotten up fairly recently is just some enhancements to the vert.io GPU support we had basic vert.io stuff working initially, but there's some rearchitecture of the GPU driver that was needed to Work with how DMA works in the scv world. And so so that was recently completed and And so that's kind of where Things are right now What we're gonna be working on here in the very near future One thing is migration support for these guests. This is something that is supported in our secure processor API We actually do have prototype patches available today on our public github page if anyone's interested in trying those out So we're gonna be working on getting those upstream. I think right now they need some optimizations So we're gonna do that. We're also looking to open source something we call the scv tool very shortly and and this will also go up on github and this is really just kind of reference code for parsing attestation reports validating certificates things like that so All the formats are documented, but hopefully just some reference code will be helpful for anyone trying to use this We're also gonna be doing some support for kind of save and restore of these virtual machines and Then the other big thing I'm gonna talk about more here in a sec is Support for a feature we call scv es Which is our scv with encrypted state support and so this is kind of the next major feature in the the scv roadmap So the es feature I talked in more detail about this last year at the North American Security Summit And this is also a feature that's available in hardware today. It was it was shipped as part of the Initial launch we did last year and The idea with this feature is to not only protect the memory space of virtual machines But also to protect the register state across world switches so There's two main things with this feature The first thing is that there's now hardware support for kind of this really large atomic world switch operation so We've expanded what we call the VM CB which is the structure that holds all the register state for the gas to Contain basically everything that's needed. So gpr state floating point state and so on and there's a single Atomic switch that happens between the hypervisor mode and the gas mode and Then and that structure is now encrypted so the hypervisor cannot actually see what the gas contents are across these world switches The second piece of this is that for the cases where the hypervisor does need to be involved in emulation for Mio or or certain instructions We have a new exception that we call a VC or a VMM communication exception and The idea is that when the guest does something that needs support like a CPU ID The hardware throws this exception and the exception handler then does this para virtualization thing with the hypervisor to To process that and I'll talk about that in a second This communication is done using a structure called the guest hypervisor communication block Or a GHCB and this is a new thing that that we're adding so With the ES architecture we have divided the types of VM exits into two categories So we now have this list of we call automatic exits Which are the only things that actually cause you to exit a virtual machine and go back to the hypervisor And so that's things like asynchronous interrupts certain instructions like halt and shut down certain types of page faults and Then everything else which is the vast majority are these non-automatic exits Which are the things that cause exceptions within the guests. So I said you got CP ID You got your MSRs. You got your IO ports, whatever the hypervisor is choosing to intercept instead of actually Intercepting back to the hypervisor. It now produces this exception in the guest and So the way that this works is you know the guest starts by doing something Say like a CPU ID and now instead of doing the exit. We'll get this exception So in response to the exception, there's this new handler inside the guest VM. That's going to Read this error code. It's going to see oh, I try to do a CPU ID I need to Write some information to this GHCB structure to tell the hypervisor I want to do a CPU ID and so it's going to do that and it's going to do this new instruction We've added called VMG exit, which is basically a hyper call In response to that. We're actually going to completely exit the guest We're going to save the state and crypt it we're going to go to the hypervisor The hypervisor is going to read the GHCB. It's going to see we want to do a CPU ID But instead of actually writing the results directly into the registers for the guest it's going to write it into this GHCB data structure and It'll then resume the guest where we go back to the handler And so the exception handler will see these new values it can decide if it likes them or not if it thinks they're safe and It can then copy them into its own register state and it can go back to the guest so the basic idea is that all of the Kind of intercept behavior things they all get funneled through this centralized exception handler So the rest of the code whether it's in the kernel or in user space doesn't have to be aware of the security feature so one of the One of the piece of feedback we got very early on with this was that it would be really nice if there is a standard protocol for doing this para virtualization stuff between guests and hypervisors regardless of vendor and And so we decided to take on the challenge of trying to standardize this and And I mentioned this last year as well, and I honestly hoped that we would be a little bit farther along than than we are today It's not anything to do with this feature It more has to do with the fact that many of our security and kernel developers We're kind of busy earlier this year. You guys might know why so But the good news is that is that we are coming back to this so just last week I believe we posted publicly a copy of we call the ghcb software specification It's not that long. It's less than 20 pages. I think and This is kind of our first draft of our attempt to standardize this and so this is open for feedback there should be a mailing list thread on this pretty soon and And and we've been shopping this around both within the open-source community as well as with the proprietary vendors and The the specification here it defines a few things of course defines what the the memory of this structure looks like It defines which specific exceptions guests are expected to handle and That's because there's a lot of potential intercept bits in our architecture, but most of them don't need to be set and so we're defining exactly what the minimum subset is that's required for support and With each of those we're defining what values the guest is expected to provide for the emulation and what the hypervisor expected to provide back And then there's also a few things in here about how we do bootstrapping And how we we bring up multi vcpu environments NMI handlings and other kind of weird oddity and so what we're really hoping to do here is to provide this unified interface so that Your guest OS can boot and doesn't have to be aware of exactly what hypervisor it's running under things like that so if you look at the document what you'll see in part are a number of Entries like this that basically say Okay on this particular type of exception. How does the protocol work? So for instance if the guest does a read MSR instruction it will generate this exception and then the guest is expected to set a few fields in the GHCP structure to say You know, I want to do a read MSR and here's the RCX values so indicating which MSR I want to read and in return the hypervisor supplies Rax and rdx similar for CPU ID. There's different things Really one way to think of this is it's basically a sparsely populated Register state structure. So just instead of everything being supplied Only the values that are actually needed are being supplied to the hypervisor so As I mentioned feedback on this is welcome. We started receiving some feedback already That I did want to share You know, there's been a few things that I think have gone pretty well. I've actually been pretty impressed with how well Kind of people have worked together as far as both the proprietary and the open source community So so I think that's really great In the current specification we have the layout of the GHCB matching that of the legacy MCB so basically the state structures look the same And so that's really helpful from a compatibility standpoint because they can use the same memory offsets everything It also turns out that there's quite a lot of similarity across hypervisors in terms of which Behaviors need to be intercepted and how those intercepts need to work and that kind of makes sense because it sort of falls out of the x86 is a quite a bit and In this type of a virtual machine environment the hypervisor really has to be very hands-off It has to let the guests run as much as it can until it does something that it really has to step in for and so Because of that we see we don't see much variation here and And also the way that we negotiate this protocol So we do have this version so it can evolve over time and there's a way that as the guest boots It can find out what version the hypervisors supports and respond accordingly There were a few interesting challenges that we did discover during this This initial review and I say we're just getting started on this one is so there's this VMM call instruction which is traditionally used for hypercalls and Of course it kind of makes sense that different hypervisors would use VMM call for different things What we also discovered is that their calling conventions are completely different and there's no way to make them mutually compatible So we basically had to throw our hands up on this one and we basically said that you know Look, you're gonna have to customize this particular behavior depending on your hypervisor We also had some some challenges around things like AP startups So this is how we boot the other vCPUs if you have multiple cores within your guest Traditionally this is done using a inter-process interrupt or an IPI which tells the processor is where to go and start executing but We don't want the hypervisor to be able to supply an arbitrary instruction pointer for them to do that So we kind of had to rework this protocol a bit where the hypervisor can wake up a core and then the core has to go and get its actual Jump address from its encrypted memory So this is kind of a new protocol You know we have a prototype. I think it works. This would be be something to review and Then we have a few other things that are really not supported in the current spec So hardware debugging is is not really supported many you are probably aware of a couple years ago There's a CVE about how Hypervisors have to intercept debug traps because otherwise the guests can spin forever There's kind of a processor deadlock situation that can happen and they can do a denial service And so because of that we're not really able to make use of these in one of these isolated guest environments We also don't have support for SMM today and I hope that stays that way, but we'll see So the the final thing I just wanted to mention is kind of an update is We have started looking at ways to leverage SEV support with containers and so we started to get involved a little bit with the Cota container project and This is a project that Works on running containers using hardware virtualization And it uses kind of a stripped down version of the Linux kernel I think based on clear Linux and we have been looking at ways to enable the SEV functionality within that project and It turns out that's actually not too difficult because the kernel support that we're already upstreaming kind of flows into that with some modifications to the runtime and You know the way this works is very similar to a normal container. It's just that there's a new option for specifying an SEV runtime and When you do that then you get a Container that's that's using this feature and if you were to look at the memory from outside of the container It would be the same as that video that I showed earlier that it would all look to be encrypted because it's in a VM So this is not upstream yet but we do have We do have that those patches on our public github page So if that's something you're interested in playing with I would encourage you to check that out So, you know overall we're we're making progress around SEV Where we have a few areas that we're working on and the encrypted state feature is kind of the big one that I'm hoping we're gonna make quite a bit of progress on within the next year and In part of that is trying to standardize this interface between guests and hypervisors and so I hope that works out We do have a new Developer dot AMD dot com page. So that's got links to all the specifications related to the SEV feature as well as We have a public github page that has kind of getting started instructions for if you want to start running some of these encrypted VMs You right now a lot of it's still script based. We don't have any nice GUIs yet, but I'm sure you guys can figure it out and And and as I mentioned, there's a tree on there as well with some of the kata stuff if you want to play with that So that's what's been going on any questions I haven't worked with AMD yet, but I worked with trust zone a ram trust zone. So you mentioned before All registers state so the question I trying to understand the difference Could you please define the all register state and second question is Why do we need to encrypt those? When we do is when do and do a switch because usually One shouldn't have access to in different Registrate state as I understand and the third question is if they encrypted where This encrypted state is stored or kept. Yes. Thanks. So if I can remember all three of those So the the register state so it first of it specified in our public x86 doc So if you want the full list you can look through that but essentially It's all the all the VM state that used to exist. So kind of the basic control register or segment state It's all the general purpose registers. It's all the floating point registers And it's a certain set of MSR is related to like system call behavior and things like that The register state is encrypted so that the hypervisor or somebody with root privileges not able to see it and gain information from that For instance, if you are there can often be privileged information in registers Like if you're using the x86 AS instructions The key for your AS operation is actually in one of the XMM registers in the floating point unit And so, you know, that's something that we would ideally not to just have their sitting in memory If you take a and say interrupt while you're in the middle of an AES operation The the data itself it's stored in in normal memory There's no special location for it. It's just that it is encrypted with the guest specific key So it looks basically like any other page of guest memory if the hypervisor tried to read it So it just sees the ciphertext Thanks, sure Thanks for talk first. Is it the actual contents of guest memory access accessible through JTAG so on our parts When when parts are configured for production the JTAG interface in general is blocked off. So There is no JTAG Debug support at all on a production AMD part So so no Okay, and second does it something does it this feature some has something to protect against side-channel text or Make it more difficult Yeah, we don't have any specific side-channel protection You know the the one thing that we're able to offer is just because the the memory space is encrypted with kind of a per guest key You know if you have say a side-channel that lets you kind of read memory with hypervisor permissions You can't use that to see the plain text of another guest right because the hypervisor normally can't see the plain text of other guests But there is nothing specific for like side-channel attacks within a guest or anything like that and What about CPU caches? So in terms of the security or the side channels or So the way that this works with the caches is that each cache line is tagged with the address space ID that the cache line belongs to so The data in the cache is stored in the clear But it's tagged with the owner of that data So only that VM can access it if the hypervisor tries to read that same address We'll treat that as a cache miss because the tags don't match so There is no way to Kind of leak data directly through the cache But as far as kind of general cache side channels, I said we don't have any specific features there Thank you So other than the GMC be registered state most of this seems to be about Confidentiality rather than an integrity of the system. Is there Anything that prevents the hypervisor from say replaying the same Like snapshotting a page running the guest for a while and then replaying the old encrypted page Yeah, so in general in this generation the answer is no That the memory is it's just Encrypted it's not integrity checked. We do some integrity checking on the state page specifically Because we see that as being kind of an easier target for that. So there is some protection there But in general, you know rollback attacks are not not blocked today Okay, so is it fair to say that in the present generation? This isn't a protection against an actively malicious. That's correct. Yeah There's questions up over there Well, I had two questions in the previous was the first question. So I won't duplicate it Is there any relationship to any TPM? Keys and the keys inside the VM. I know that you guys have a TPM in SCP So is there any relationship at all between the keys and the association between a key that might belong to that application to Yeah, there isn't today, you know the the functionality that the secure processor provides certainly has some similarities to a TPM in terms of the way that when you set up one of these things you basically do a measurement of the initial biosimage and That gets turned into basically attestation report for the guest but it's not a TPM specific interface and today we don't have a way for the guest VM itself to actually call any services on Secure processor. It's only there for the the hypervisor to use So if the guest wants a TPM like functionality, it basically would have to provide it itself and software But your scp. I thought implemented a TPM as well. So inside the same space Yeah, so in our client generation of parts we do support we call firmware TPM So we basically emulated TPM 2.0 We don't support that in our server line of parts, which is the the line that supports this feature that that's mainly because We do is kind of a cost-saving measure and for server the cost-saving isn't as important And you know the firmware TPM is not certified like a discrete TPM is Unfortunately, I think I have to stop for discussion, but the good thing is now the coffee and tea break is there So I think we can maybe continue the discussion there. So let's thank David for the talk