 Hello, everybody. Thank you for making the time to come to this on a Friday afternoon. My talk today is about hypervisor security. It's labeled the elephant in the room because, at the moment, people seem to be closing their eyes, covering their ears, and hoping that the particular virtualized room they're in doesn't catch fire and sink their entire business. I'm going to talk a little bit about breakouts, about the potential for them, and whether or not the threat is theoretical or real. We're going to address some of the data leakage and virtual machine compromise issues that people will face in the cloud. My name's Robert Clark. I've been the lead security architect at HP Cloud for a year or two now, and been involved in OpenStack for about two years. I've been involved in developing security solutions for a lot of the challenges we face in public cloud and now with private cloud as well. And I was one of the founding members of the vulnerability management team at OpenStack, so I've really been around since the start of the point where OpenStack considered security to be a real priority and issues that needed to be dealt with. Since then, I started the OpenStack security group, along with Brian Payne from Nebula. We found this gave us a good set of perspectives from a core private cloud and a core public cloud to be able to address security concerns. Because we don't have a security group talk this summit, I'm just going to very quickly tell you a little bit about the security group, because if you're in this room, you're interested to some extent in security in OpenStack. We were established coming up on two years ago. We do a number of things. We issue OpenStack security notes that sit alongside the OpenStack security advisories, and they serve to provide extra security information for people deploying clouds. So they may involve third party bits of software, or they may involve common configuration mistakes with OpenStack, and they serve to provide more security in deployment. We consult with the OpenStack VMT. So when vulnerabilities come in, the vulnerability management team is very small. And if, for example, it was a hypervisor security issue, they may well reach out to one of us in the ISSG to run through the issues with them and decide whether or not it's a big problem, whether it can be deployed, whether it needs to be fixed immediately, or it can wait for a release cycle, stuff like that. Among security initiatives, we have the OpenStack Security Guide, which I'll speak to you very quickly, because that is important for this talk. And as of about this morning, we have 97, 98-ish members in the OpenStack Security Group. So one of the main things that we can say we achieved in the last six months was the publishing of the OpenStack Security Guide. It's available on the docs page on openstack.org. It is available in HTML, PDF, and you can buy it in tree form. It was built in the space of a week in a documentation sprint involving major participation from all these organizations I've got up on my slide. And it's very valuable. This talk will speak to a number of points in the Security Guide. I would encourage anybody who's serious about actually deploying OpenStack and letting real people run amuck on it to spend a day or two reading through this guide, because there is some really, really good information there. However, it was written in a week. If you find problems with it, please feel free to submit a patch or just reach out to me directly and I'll make the changes for you. And these are the people that were involved. Right, so I'm going to speak to you a little bit about virtualization. This is a conference talk rather than a design talk. So it's going to be a mixture of higher level and lower level information. Compute platforms serve to essentially aggregate compute loads such that you don't have to have as many servers sat around. And there are various other reasons you might do it, obviously. HP Cloud has been selling and making clouds in some description for many, many years. HP has been selling virtualized platforms for a long time. Virtualization technologies really fall into three categories. And the lines between these are very, very blurred nowadays. It used to be that you had hosted hypervisors, paravirtualized, and full virtualization. Nowadays, depending on how you interpret those different things, Zen could be full hardware virtualization. KVM can have some paravirtualized elements depending on the way you use it. So these are very broad categories. And one of the things I ran into when trying to create this talk, and as I say, it's kind of pitched at a number of levels, is trying to work out a nice way of modeling a virtualization stack in one model. Because if you look at, say, Zen or KVM, they are very different in their approach. But there are a few fundamental similarities. We have a device emulation layer and a hypervisor. And the hypervisor in some situations, like KVM, will actually be a part of an OS. So KVM sits within the kernel. And you have a full Linux OS. QMU provides the device emulation. Those of you that are more familiar with the data paths that happen in typical virtualization stacks will be aware that you can draw a lot more lines between different components on this diagram. But we're trying to keep it at a reasonable level. Zen operates slightly differently. So Zen is closer to what would be a type one hypervisor, a bare metal hypervisor. It still makes use of QMU to provide a lot of its address spacing for machines. So that's all your addressable devices, apart from CPU, MMU, and a couple of other bits and pieces. So this is what our generalize stack is going to look like for the rest of this talk. And I'm going to discuss different ways that different parts of this stack can be attacked. The results of those attacks, whether or not those attacks are realistic in today's world, whether they might be in the future, and some of the mitigating controls you can put in place to protect yourself from these various attacks. So for this, I'm going to introduce an actor. Our actor is called Mal, short for Mallory. That will make reference to some people. Mal is a malicious virtual machine. This could be a VM that's been created for malicious purposes. So we run a public cloud, and a very real concern for us is that somebody will create a virtual machine to either directly attack us or attack one of our customers. In a private cloud, this could be exactly the same for a customer of the organization or put an employee in the organization who, for whatever reason, is deciding to carry out malicious actions. Equally, this could have been a perfectly benign customer virtual machine that has been compromised by some outside attacker, which we see in public clouds fairly regularly. So what does a compute host look like? So a compute host, for the purposes of this, is just a container with a few virtual machines within it. So in this example, Alice has two virtual machines. She's co-tenanted with Bob. Bob also has two virtual machines. This compute host has a hypervisor, being Zen and or KVM, and device emulation provided through QMU. And on the left here, if you can see that, we have a kind of combination of the base layer that would either be the Linux kernel, Linux OS, or DOM0. This is the best way I could find to explain these attacks in a way that would be fair to both KVM and Zen. So we have a compute instance. Mallory has gained a point of presence on this compute instance by either compromising another instance that was running on the host, or by simply asking to have a VM created. There are a few attack options open to Mallory, straight away. Mal can go after another virtual machine on the network just using straight up network attacks. Mallory can sit there, create a virtual machine, deploy Metasploit, and go straight after another machine. And if security groups are in place or your chosen networking rules aren't as effective as you would like them to be, that may happen. In most clouds, this is a valid attack vector, even if you have to route your traffic out to the border and then back. Virtual machines are almost always requiring some sort of network connectivity, and therefore you need to take appropriate steps to protect them. We're going to look at this a little bit more in a moment. Mallory going directly after the KVM or Zen layer and how it might interact with QMU. So we're first going to look at how KVM can go after QMU. So there are a number of vulnerabilities that have been recorded over the years that would allow a virtual machine to attack the QMU layer. QMU is very large. It supports a large number of different devices, all of which have different interfaces. They all have wide address spaces that are accessible. Although QMU has grown over the last five years to have many, many contributors and to become a very large and mature project, it is still sat on a foundation that was mainly written by one guy, Fabrice, whose surname escapes me. And he really did a lot of this stuff for the first five or six years of QMU's life. And there's still some legacy code there. And while he's pretty much borderline genius, it's a lot of code for one person to write, and there will be more bugs in there. So if an attacker were able to compromise QMU and gain a point of presence within QMU, they could potentially attack the Linux OS. So in most situations, QMU is going to be inside some sort of process running on a Linux machine or on a Zen Type 1 hypervisor. In either case, QMU is running in Zen. It'd be attacking DOM zero. But in either case, you end up attacking a Linux OS. Typically, there's a possibility for access to other resources once you compromise the Linux OS. So at this point, you'd still only be a user of Type, whatever Type you gave QMU. So you can access other parts of the Linux OS. You can potentially access different OpenStack services because OpenStack is very trusting of what's running on compute hosts. There are ways to contain what can happen after a QMU compromise, which we'll look to you later. And of course, the obvious thing that people are going to be looking for there is a privilege escalation. If someone's able to gain root along this path, then they basically own the compute host. Once you own the compute host, you own all the virtual machines within it. Another method of attack that we actually see a little bit more focus on, if you look at the vulnerabilities over the last few years, they do seem to be focusing right now much more on the core hypervisor technologies. Someone's able to compromise KVM or Zen. They may be able to read the data or write to the data within other virtual machines, but might not necessarily be able to break out of those hypervised processes to really do much else on the system. However, KVM is based inside of the Linux kernel, and therefore, there is an obvious escalation path there. With Zen, you're basically interacting directly with the hardware at that point, and you face a fundamental hostile processor problem. So nothing running on that machine can trust what's going on there anymore because Zen has direct access to the hardware. And the result of that is everything's compromised, and we all have to go home. So what does this mean for a cloud? So I've been focusing on a single compute host, but clouds aren't single compute hosts. There are many thousands of compute hosts potentially. And they are made up with various supporting services. The supporting services in OpenStat right now trust each other quite a lot. I'll come back to that in a second. But what does it mean to be a cloud of this size? It means that somebody could, again, create a virtual machine on one of your compute hosts. And if they have a vulnerability that they can leverage, they may be able to, through one of the mechanisms we described earlier, gain root on that box and essentially own everything that's on it. However, if you consider a cloud where you have a lot of virtual machines that have the same vulnerability or a public cloud where you essentially pay to pop up VMs wherever you like, you can end up with lots of parts of your cloud compromised at the same time. And that allows you to go after various different services. You may, for example, if Alice and Bob are on a machine that someone wants access to, they can continue to pop shells on cloud hosts until they find the machine they're interested in. Alternatively, they can just compromise one more quietly. And then this was the case as of a few months ago in OpenStack, you could then, if you had compromised a compute host, you could just say to your block store, hey, I want the block store for Alice or Bob or Pepsi or Coke or whoever that company that I'm paid to get lots of information about is. And that seems like a bad thing. So we'll come back to how to stop that happening. One thing that we get asked about a lot in customer engagements is virtual machines, slide channels, affecting entropy, all that sort of stuff. So side channels tend to focus on cryptographic mechanisms. This is because it's a very strong area of academic research, and it's a nice place to be able to show things in a really provable academic way. Virtual machines deployed for any sort of production use, typically at least some of them have some sort of secure transport on them. Web servers, voice over IP, VPN, chat applications, pretty much anything nowadays that's interacting across the web, at least, will be using some secure protocols at some point. So I'm going to talk you through a little example of a documented side channel attack in Zen. That was in a simplified version of Zen with certain things turned off to make this easier. It's important to recognize that, because you can't necessarily go out and own AWS with this, because they run Zen. They've done a lot of smart things there. But if you were to deploy it very naively or deploy it in a way that the academics have, then you can do some interesting stuff. So if you can consider this example of Alish, who is interacting with Bob over a TLS or SSL tunnel. Consider a man on the side, or a man in the middle, who can be privy to this data, but not necessarily interpret it or understand it. Once SSL's in flight, you're invariably talking about either RC4 or above, so more than likely AS128 encryption traveling between these two places. So unless you are able to decrypt the initial key exchange, you've got very little chance of actually breaking the traffic and understanding what's going on. On the compute host here, I've highlighted the CPU and the level one cache, because those are the specific things that are targeted in this attack. And when you look at side channel attacks, whether or not they're poisoning entropy or attempting to recover bits of keys from other machines, they invariably tend to enter out to CPU caches. So Mallory's back and has gained a point of presence on a compute host that they care about. And using the techniques that were described in the paper that is referenced here later, they are able to interact with the level one cache very, very rapidly. They continuously allocate large segment sizes. They continuously reading and dumping and reading and dumping. And by doing this, they're able to collect fragments of the key exchange that happened between Alish and Bob. And once you've got the key exchange, you can derive the key, and then you can replay the entire SSL session and read all the data. So it's fairly trivial, then, for Mallory to pass that information off. You can pass this, shows it very simply. A slightly nicer method of using this attack would be to have Mallory have several point of presence, or perhaps even a legitimate account on your cloud, spin up 20 or 30 VMs to do the number crunching for you from the one that was performing the side channel attack. So as I said, isn't this all a bit theoretical? I have been in meetings with virtualization vendors where I've been told that hypervisor attacks and hypervisor breakouts are purely theoretical and academic, like the side channel vulnerability we spoke to a minute ago. So in 2008, Cloudburst was presented at Black Hat. ImmunitySec demonstrated a breakout of VMware's hosted product. VMware at the time said that they accepted there was a vulnerability in the hosted product, but it didn't affect the SX or ESXi. ImmunitySec said otherwise. The reason this came about was because it was found in 3D drivers that were compiled into VMware that weren't used for anything anywhere. There's a lesson in there for later. OK, so the Zen Onage Trilogy, Joanna from Invisible Things Labs and the rest of her team released three different vulnerabilities in Zen. They involve three different types of attack. The first one went after direct memory access mechanisms within Zen. The next one was a system management mode attack, which essentially allows you to interface with machines before they've really come up. System management mode is sort of a transient mode between your base OS and full. And blue pill shimming, which is quite interesting. Blue pill would allow you to use the virtualization extensions on the host system to move Zen into its own virtualized space to essentially shim yourself underneath between the type 1 hypervisor and the CPU layer, which is useful. Virtunoid, Nelson L. Harsh, KVM, full breakout, attack to dangling pointers in the KVM code, essentially pointers in computing for those of you who want to know. Pointers point to different bits of memory. When they're deallocated, they should be destroyed correctly, but sometimes that doesn't happen. And then if you're very, very clever and have lots of time, you can find out ways to point these to things you do care about and either read from them or write to them or do other nasty things. sysret64 was a Zen vulnerability. It affected the power virtualization context switching mechanisms within Zen. So although sysret64 is a command that you'll find in a number of different types of systems, it only affected the context switching within Zen. And again, allow for a potential full breakout. And to come back to what I was saying earlier, full breakout means everything on that system is owned. VMDK has left the building 2012. The SSI file handling logic error is a classification for this vulnerability. However, if I were to tell you that it meant you could load the block store of any customer on your cloud, you might consider it to be just as bad as a full breakout. Essentially, yeah, you can attach the storage, the database, all the information that a customer has running on their cloud, apart from their VM run state and its ephemeral memory. KVM had three CVEs come out on the same day. That was a fun time to be me. They range from denial of service and memory leakage to a full potential breakout. I've listed the CVEs here. They're reasonably well written. The IOAPIC vulnerability was an out of bounds read, which had the potential to mean that somebody could read the running state of another virtual machine. An obvious example of an exploitation for that would be attacking the key exchange of a machine that was doing cryptography. If you consider that a number of people run VPN gateways in the cloud, then all of a sudden you get access to a lot of different people's data. The time vulnerability was a use after free, which again comes down to memory management and abuse of pointers. And the setMSR is a function that is called low level in a lot of systems. And there was a buffer overflow in the code. It was just badly written and able to be exploited. So I've run through, I think, four or five different vulnerabilities over the last five years. We should convince you that people need to start taking this a little bit more seriously and be a bit more aware. And if that doesn't convince you, this really should. So this is the number of security vulnerabilities in virtualized products for the whole family, from everything you can think of that comes into virtualization. This is from the excellent IBM X-Force report from 2010, which is why the data runs up to 2010. And you can see that there is a massive increase in vulnerabilities reported. Now, this is more than likely due to many eyes. More people are recognizing the importance of virtualization products and going and looking at them. Doesn't necessarily mean that they're full of holes. But given how many were found in the last 10 years, I'm inclined to believe there is still more to find. This table here is, I'm not going to speak to everything in it. It breaks down the different types of vulnerabilities that have been found in virtualization tech. It's a very recent paper that's referenced here from some guys at Princeton. I suggest anyone seriously interested in virtualization technologies goes and has a read of this. Again, it's nicely presented and breaks down all these attacks type by type. You will see here that Zen and KVM are pitted against each other. At the moment, KVM seems to be winning in terms of having a lower number of vulnerabilities. While I think you can deploy both securely, I'm not convinced that this is necessarily indicative that one is much more secure than the other. What is obvious is that you can configure either of them to be incredibly insecure. So I think perhaps at this point, this is a good question to ask, it seems a little bit tricky. So there are a few things you can do. You can turn on relocation read only for all your binaries in your virtualization stack. This will basically obscure the global offset table for people who like getting down and dirty in binary. And what this means is exploitation is very, very hard. Stack canaries are a fairly well understood technology now, essentially inserting known values into the execution stack of binaries and seeing if things overwrite them. It's a great protection against buffer overflow. Never execute and DEP, both essentially the same technology, CPU level tech that is supposed to, again, stop binaries moving to the wrong places. Position independent executables are a prerequisite for address-based layout randomization. So if you have ASLR enabled, you're doing PIE anyway. These are very, very useful for making it much harder for a successful exploit to gain a point of presence. So you may still end up with denial of service type situations because people are able to make things crash. But gaining reliable exploitation becomes much harder because knowing which bits of your address space to jump to to gain functionality becomes much harder. And this line at the bottom is a compile line of how to do all this for QMU. If you want to compile QMU to support all this or have it all built in or if you're not sure if your vendor does, send them this and ask them if they do it this way. And this is taken from the OpenStack Security Guide. So not only does the guide have fairly high level information and pretty pictures and stuff, it also has some pretty low level information on just how to get it done. So reducing the tax surface is important. The Cloudburst vulnerability is a good example of this. Cloudburst actually combined a couple of vulnerabilities in different systems that didn't need to be in VMware for what it was trying to do at the time. If you look at a standard compile of KVM or a standard compile of QMU, you will see that you will have things like 3D graphics turned on. You will have multiple network devices that you've never heard of and never ever want to support. You'll have sound support. And more than likely, the entire Bluetooth stack supported in your virtualization stuff. We don't have a huge use for Bluetooth in our data centers, so we turn it off. Mandatory access controls are very useful. Anyone who's looking to deploy this stuff at any sort of scale really should be considering using either SE Linux or App Armor. They both have pluses and minuses. They both work in slightly different ways to try and achieve the same things. That is to say that it makes it much harder for processes that have been compromised to change their behavior to be able to access different parts of your system that they wouldn't ordinarily be given access to. It basically gives you very, very tight granularity on what processes are allowed to do. This is an example of Svert running on a Linux KVM hypervised system. In Linux KVM, VMs are running in separate processes, and each process is labeled by SE Linux. And SE Linux can then track what the process is doing and block it from doing anything it shouldn't be doing. So if a process running a virtual machine suddenly starts trying to make calls to other parts of your kernel or tries to reach out to different devices or any other sort of horrible behavior, SE Linux or Svert, in this case, is able to detect that and alert on it. And if you have good monitoring and alerting in place, you can just isolate the box, shut all the VMs down, kick off your incident response procedures, all that fun stuff. So to go over what I just mentioned, these are all points that are spoken to in the OpenStack Security Guide. These are all things that will stop the majority of hypervisor attacks and hypervisor breakouts being able to damage your cloud significantly. There's very little protection against really big zero days that come up in virtualization technology. However, you're not necessarily likely to be affected by these vulnerabilities when they are found. The reason for that is because a vulnerability like that is worth a lot of money and will be used in targeted attacks by people who really don't like you. It won't be used. It's not going to turn up in metasploit on day one, or it's very unlikely to. Is anyone who's going to disclose it that way will be looking to do responsible disclosure? Now, if everyone gets hacked tomorrow because of a massive zero day, it's really not my fault. But for the most part, if you apply these different technologies, if you go and read the OpenStack Security Guide, which goes into much, much more depth on every single one of them, then you are going to put yourself in the best possible position. And you will have done your due diligence to protect your cloud from these sorts of attacks. So these are the parts of the OpenStack Security Guide I've referenced. Again, I can't say strongly enough. Please go read it. Please contribute. Thank you for coming to my talk. I hope it wasn't too scary for you. And if anybody wants to do any more research, here's a bunch of links that are incredibly useful. And obviously, my slides will be online, so you don't all have to take photos. Thank you. Any questions? Sure. So if some kind of exceptions happen, blame it. Absolutely. OK. So the question was around the timing of the attacks, and whether or not you will have time to respond to attacks when they happen. Is that right? And that's not just for the side channels for the other type of attacks. OK. It's really very much context dependent. If you have a lot of these technologies turned on, then attacks that attempt to circumvent them, especially things like ASLR, tend to be very noisy. ASLR doesn't work in the kernel. For obvious reasons, you can't randomize the kernel space quite so much. So attacks directly attacking the hypervisors will generally be very quiet. A lot of the things that are going on here aren't, they're not traveling over network interfaces. They're the direct interfaces the kernels have, the direct interfaces that the compute instance kernels have with the systems that are providing their virtualization. So they're exposing, in one way or another, they're exposing PCI and that sort of stuff. So you're talking about very, very fast buses that can, albeit virtualized buses, that can be interrogated very, very quickly. I would expect there will be traces of most of these types of attacks. Where you draw on the line in terms of inspection very much depends on what your agreements are with your customer. Certain customers aren't going to like you, EIP tracing all of their compute instances, for example. Very, very paranoid ones, but, however, may pay you a lot of money to do that. Sure. So I'm not comfortable speaking exactly to what Red Hat have done, because I haven't spoken to their guys about it directly. We did have a couple of guys from Red Hat working with us on the OpenStack Security Guide. I know they have worked very hard on a lot of this stuff. They contributed heavily to things like SBIRT, so they do take the virtualization stack security very, very seriously, much like we do at HP Cloud. So we apply, wherever operationally possible, we apply all of these technologies. As with anything, there are going to be points where you have to make trade-offs between actually having a cloud that works and a cloud that is ultra secure. That said, I think if you combine all of them intelligently, you can make some really strong security assertions about your cloud. As to the direct question about whether or not Red Hat have compiled out parts of KVM, I don't know. So I think that the question was that I didn't speak directly to the, are you talking about the LiveVert drivers and stuff like that? Sure. I didn't speak to that because I'm not particularly expert in Zen. My day-to-day work is in the KVM world. We do a lot to isolate QMU from basically everything else that's on the system, because there we have it running in a quite a discrete process. I would say that that's not particularly well addressed in the OpenStack Security Guide either. And if you would like to have a conversation about contributing to it, that would be great. OK, thanks. Thank you very much.