 All right, let's welcome Alexander Bulyakov, who is going to present about Bending Space to First Virtual Device and Beyond. And over to you, Alexander. Hi, thank you. Hopefully, everybody can hear me all right. So let me share my screen. And let's go to my presentation. So my name is Alexander Bulyakov. I'm a PhD candidate at Boston University and an intern at Red Hat Research. And I'm excited to talk to you about the work that I've been doing on fuzzing virtual devices and hypervisors and not just that. So if we look at hypervisors, they serve as the backbone of a lot of our cloud infrastructure. They're the software that allow us to run multiple VMs on the same machine. And when it comes to multi-tenant cloud environments or even just any cloud environment, one of the things that we expect from our virtual machines are that they are isolated from each other. So what I mean by that is that, say, something really bad happens and either you have a malicious tenant on your server or one of your VMs was running a buggy application that got compromised. Well, if that VM gets compromised, the entire hypervisor shouldn't be affected by that. Because if the compromise is able to escape out of that VM barrier and hop onto the hypervisor, then that means that the entire server is potentially compromised, along with all of the other guest VMs, which is a huge concern for anybody running a public cloud or a private cloud. And so let's look at how one of these compromises might happen. Well, what's the hypervisor's job? It's to run on top of some hardware and it's to partition the resources of that hardware among the different VMs. And the way that hypervisors do that is that they provide a model of that hardware to each of the VMs, a separate model. And those models are implemented in software. They're written in code and many oftentimes a model might be thousands or hundreds of thousands or even millions of lines of code in size. So if the guest OS, if these virtual device models are the interface that basically govern accesses from the guest OS to the hypervisor, then if the guest is compromised, it's logical that they'll attempt to attack these virtual device models. And so one of the critical interfaces that hypervisors are trying to ensure is secure and is protected is this virtual device interface. However, unfortunately, attacks against these virtual devices and vulnerabilities in these virtual devices aren't unheard of and they're not even rare. They're found every couple of weeks in every single hypervisor that I've seen. And a lot of the time they're quite serious because hypervisors much like operating system kernels are often written in memory on safe languages such as C or C++. So as a result, we hear about these bugs coming out every couple of weeks and also sometimes the bugs that actually make into the tech news like the venom vulnerability which was found in a virtual floppy device, I believe in 2015 which allowed a malicious VM to potentially take over the entire hypervisor. So if we want to protect against these kinds of things, well, what sort of proactive measures might we take? Well, if we look at software in general, one of the key things that people have found to really help to catch these types of bugs is testing. So what I mean by testing is you have some piece of code and you manually provide, so for example, in this case is a calculator and you manually provide some inputs such as two plus two and you make sure that the code returns four. And you can manually write a bunch of these test cases and make sure that the outputs make sense. And another thing you can make sure is that the code returns and doesn't even crash. However, the problem here is that you as the developer are basically the bottleneck because there's only so many test cases that you can think of and only so many test cases that you can write that will fit inside the code base. So this is where fuzz testing comes in or fuzzing. And the idea here is very similar, right? You're running your code, but instead of manually writing test cases, you let the fuzzer basically feed random inputs into your code. So you lose something like you, maybe you won't know if the calculator returns the correct number back because you're just feeding in a bunch of random stuff. And also a lot of your test cases won't make any sense to the code that you're testing. But if you do this millions or even billions of times, eventually you might run into an input that causes some sort of nasty issue, such as a floating point exception or a memory corruption issue. And fuzz testing over the past couple of years, there's been a lot of research that's gone into it and people have combined it with, for example, coverage information where each time the fuzzer runs a new input, it looks for any new lines that were triggered inside your code. And if an input hit a new line that it's never seen before, then the fuzzer saves that input for later and it says, hey, this input was interesting, I'm gonna save it for later so I can mutate it. And hopefully that will find even more interesting code. So this is called coverage guided fuzzing. And also you can build your code with sanitizers, for example, which will let you catch bugs that would normally be silent, such as heap related issues that would normally be silent and maybe you would have never noticed in the first place. So this is sort of a different sort of testing, but it has helped find thousands of bugs in a wide range of code. And there's a lot of research that's been going into fuzzing over the past couple of years. So combining what we just learned about hypervisors and how crucial that virtual device layer is with what we just found out about fuzzers, it might seem that virtual devices are a natural match for fuzzing and we should be fuzzing in all these virtual devices. So that was basically what my project, that's what my work is doing. It's fuzzing virtual devices and specifically I'm doing this for QMU, which is one of the most popular open source hypervisors and it's used by systems shared, Red Hat and by a lot of other systems as well. And one thing that makes QMU unique is that it has dozens of not hundreds of virtual devices that we might want to fuzz. And also in the past there was already work that showed that first of all, there's bugs in QMU virtual devices that were constantly being found. And also you could write device specific fuzzers for QMU, which would find bugs in a narrow set of devices, for example. Okay, so if we are going to undertake this journey of fuzzing virtual devices, we need to understand the technical challenges and really what our goals are here. So first of all, as we'll see in just a bit, the interface, the virtual device interface is quite complex. There's multiple different facets to the virtual device interface. And also it's not just complex, it's enormous. There's a innumerable amount of operations that you can perform over the entire virtual device interface. And also the code of QMU itself or really any hypervisor is usually quite large. And because when we're fuzzing, we're interested in achieving hundreds, if not thousands of executions per second, the performance of our fuzzers is important. We can't let QMU's large size slow us down too significantly where we're only running one or two inputs a second. And the other problem is that virtual devices are stateful. So they'll absorb state from your inputs. And that means that if we don't do any sort of cleanup in between inputs, we're gonna hit on indeterminism where different inputs will affect each other, which will really hurt fuzzing performance. And if we look at the higher level picture here, remember that QMU has dozens of devices that we want to fuzz. So we need a strategy that will work for a variety of devices, not just one or two devices. Ideally, we also want to fuzz continuously. We want to always be fuzzing and be on the lookout for new bugs that appear in the code. And also when we do find bugs, we really need to make the developer's life as easy as possible because debugging virtual device code is already hard as it is. So whatever we can do to make life easier for the developer who's gonna be looking at the bug report is definitely worthwhile. So I mentioned all these complexities about the input space on the previous slide. So let's just recap that. There's two main address spaces that CPUs, use to interface with virtual devices. And that's the, first of all, the port IO space where you have this set of 64K addresses that is directly mapped to, where some of these addresses are directly mapped to virtual devices where the CPU uses instructions such as out and in to communicate, directly communicate with virtual devices. And what happens is the CPU invokes this instruction and then that request goes to the device, the device does whatever it needs to and potentially responds with a value to the CPU. So this is a synchronous action where CPU does something, virtual device does something and then the CPU continues running. And then we have the same thing, but in memory. We have memory mapped IO, which acts very similar to port IO, where instead of using an in or out instruction, you use a reader write instruction, but essentially the behavior is, that has the same synchronous behavior. And one layer of complexity here is that actually these regions in port IO and memory mapped IO space can shift around. So one of the port IO regions is the PCI configuration space, for example, where if you write some values there, it might end up completely shifting around what your MMO and your port IO space looks like. So we don't know, before we start buzzing, we don't always know which, where these regions are gonna be in memory. And then finally, probably the toughest part of the equation that we encountered was the DMA or direct memory access paradigm that's used by virtual device, that's another part of the input space. And unlike port IO and memory mapped IO, the way this works is the device initiates it. So the device will reach into the guest's RAM and it will pull out or read or write to buffers inside that RAM on its own. And the way the CPU interacts with this is it basically provides the device with addresses of data in port IO and memory mapped IO, or it provides the device with addresses of data in RAM over port IO or memory mapped IO. And then the device will asynchronously reach in and pull out more data. And because this is an asynchronous action and the device, the CPU doesn't have direct control over this, this is quite a tricky interface to fuzz. And basically it combined, you have potentially gigabytes of input space that you're trying to fuzz here. And this is really intractable for any out of the box fuzzing method. And to emphasize this point, here's a real bug that was, or a real input that led to a bug that was found by our fuzzer. So the details here really aren't super specific on the right here, on the right hand side here, I have the way the developer actually interpreted this bug, but the takeaway here is that there were three modes of IO that were involved in creating this bug. So there was port IO up here, memory mapped IO down here, and then a DMA over here, all simultaneously required to trigger this bug. The values that were written are actually quite important because as I mentioned, the address of DMA regions is passed over port IO or memory mapped IO, which is exactly what happened here. And also there's quite a lot of instructions and their sequence is important. So for a purely random process, just spitting out random instructions to random addresses, this would be completely intractable for a fuzzer because there's just way too many, the input space is way too large if we want to find these types of bugs. So maybe we can take inspiration from an analogous system to hypervisors which I think is a kernel, the Linux kernel is probably a good analog where when you're interacting with virtual devices, you're using port IO and memory mapped IO accesses to directly invoke some actions, much the same way that user space processes in Linux, you use system calls to tell the system to do something. So for example, if you want to print something out to the screen, you use the right system call and then you basically switch into the kernel and then the kernel does something. The virtual devices work very similarly. And DMA accesses that I mentioned are this asynchronous way of accessing data in RAM. There's an analog in the kernel as well where the kernel can actually reach into the processes, memory space and also read or write data from there as well. For example, your hello world string will be read asynchronously by the kernel. And the size of the input space is comparable because both the kernel and virtual devices have this idea of RAM or memory. So if you look at the solutions that have been used in the past to fuzz kernels, usually they involve these like really extensive specifications to fuzz individual interfaces within the kernel. So for example, if we're fuzzing the kernel virtual machine or KVM interface in the kernel, the syscolor fuzzer, which is one of the most successful kernel fuzzers that's in use today. Syscolor has a like 1000 line specification that's designed specifically to fuzz KVM. So and it took years to develop this kind of specification. And basically, what we found was that by leveraging the nature of hypervisors, we would be able to avoid a lot of this complexity when fuzzing virtual devices. So first of all, if we look at Portaio and MemoryMap.io, remember that they're synchronous. So the hypervisor needs a way to trap accesses to certain memory regions or Portaio regions when the CPU accesses them. So in order to do that, the hypervisor basically needs to keep a mapping of memory regions to virtual devices in somewhere in its state. And so instead of basically fighting against the hypervisor and trying to randomly locate where these memory regions are, we just hook into this table and we have a, we have the list of memory regions that we should be providing random inputs to for Portaio and MemoryMap.io. However, as I mentioned, Portaio and MemoryMap.io aren't really the most important and most interesting part of the equation here. That would really be DMA. So when we're dealing with DMA, we've already written to Portaio or MemoryMap.io and now the device is trying to asynchronously fetch some data from the guest's RAM. And how does it connect these two things together? How does it convert the guests, the physical address provided by the guests to a virtual address in its own process? Well, it uses a DMA API. So it calls into some functions that hypervisor provides to basically help extract data out of the hypervisor, out of the VM. And again, in this case, we leverage this API and we hook into it. And what that allows us to do is basically fill the DMA region with randomized data just before the device actually reads from it. So by hooking into this API, we basically reduced or in reshaped the input space so that we don't need to be doing any guessing of guest memory addresses, which are exactly what made the input space so enormous. So we implemented our prototype with these hooks on top of LibFuzzer, which is a fuzzing engine that ships with LLVM. And one of the benefits of that is that all of the instrumentations and all of the modifications that we made are done at compile time. So there's no overhead, there's very little overhead at runtime for collecting, for example, coverage information. And LibFuzzer is very lightweight. It lives as a separate library inside the overall QMU process. And basically the QMU process that we use for fuzzing is very similar to a normal QMU process, except it has this reshaping and hooking functionality that I just described on the previous slides. However, LibFuzzer doesn't reset any state for you, which is a problem that I mentioned early on. So we actually added a custom method for resetting state as a fork server. So basically each time we run a new input inside a QMU process, we fork it so that it's actually running in a separate process. And then once the input is done running, we can just destroy that process and start off with a clean slate for the next input. So we, using this fuzzer, we've found and reported well over 100 bugs at this point. We've upstreamed this fuzzer, it's upstream in QMU and it's actually running on OSS fuzz, which is a continuous fuzzing infrastructure for open source projects. And so the QMU is being continuously fuzzed as we speak right now. And we fuzzed a wide range of devices, including network devices, disk devices, as well as like low level devices, such as a keyboard and mice to find and our fuzzer finds bugs in them equally well. And because of the way we reshape the space, we're actually able to find bugs that are actually usually tricky to identify, such as double fetches, where we perform two DMA accesses to the same region. And as a result of our work, we actually published a paper that will appear at Usenix Security next year, which is really exciting. And if you're interested in this work, keep an eye out for our paper to be released in the coming months. So the other part that I mentioned was the fact that we really need to make life as easy as possible for the developer, because the developer is not always going to know how the fuzzer works. The developer's not always going to know how the fuzzer works and the developer already has a backlog of bugs that they're probably looking at. So as much information as possible that we can provide to the developer, we should. So let's look at what we get when we get a report from OSSFuzz, which is that continuous fuzzing infrastructure that I just mentioned. We get a stack trace of the bug and then we get a raw reproducer or a raw dump of the input that triggered the bug. So this input is usually unreadable to a developer. It doesn't mean anything. And reproducing it requires building the fuzzer. And also, the stack trace, it's useful, but usually you won't be able to fix a bug just by looking at a stack trace. You need to actually reproduce it and hop in with a debugger to really dig in to see what's going on. So we have a pipeline that before we create a report and report it to the developer that we use. So first we replay the binary input and we record the CPU instructions that would be invoked by this input. And as a result, we have something that already looks a lot more readable than this binary blob. But remember that our fuzzing process is sort of strange. It has these hooks that really change the way DMA works, for example. So the next thing that we do is we need to reorder, we need to unbend the DMA space and we do that by basically reordering some of these DMA commands, which we describe really in much more detail in our paper. And in the process, we can also minimize that we can get rid of commands that aren't needed to trigger the bug. And as a result, what we get is basically a one-liner command that you can copy from an email and reproduce the bug in your existing build of QMU. So you don't need to know anything about how the fuzzer works. Just copy this, you hit the bug, and you're ready to debug it and fix it. But we didn't stop there. We actually also made a program that converts these single line reproducers into regression test cases. So remember how I mentioned testing is also a very important part of preventing these types of bugs. Well, once you do fix this bug, you can just copy this code and commit it as a regression test. So all future releases of QMU will be regression tested against this bug to make sure that it doesn't come back. So all of these things come together in order to create a report that we create on the QMU issue tracker. And hopefully this makes life as easy as possible for the developer, and they really don't need to know very much or anything at all about how the fuzzer works in order to be able to work through these issues. So the main points that I think were important from this project are the fact that really you shouldn't, if you have a really complex target like the kernel or something like a hypervisor, don't shy away from modifying that target in order to make it easier to fuzz. And definitely, once you do have a fuzzer, try to upstream it because then it becomes, it's more than just a weekend project that you use to find like 10 bugs or something, it's going to benefit that project for years to come and you'll potentially be able to catch critical vulnerabilities before they appear in the code. And also invest the time to make sure that you make life as easy as possible for the developers that will be on the receiving end of your bug reports. And what have I been working on now? Well, I'm sort of applying these takeaways from fuzzing hypervisors to fuzzing the kernel because I think that there's a lot of parallels, as I mentioned, in the middle of the presentation. So the kernel is definitely a much different beast. It's much larger and you have some challenges like you can't just use the forks system call in order to create a copy of the kernel. And you also have, for example, file descriptors, which are something that you don't really have in when you're dealing with virtual devices. So far, this appears to be a promising direction and we're able to create fairly interesting behaviors in the kernel, so stay tuned for that. And I want to give a big thanks to everybody who maintains and reviews and mentioned my progress on this project and my co-authors on the paper. So, and with that, open for any questions, I hope you enjoy the presentation. Thank you very much, Alexander, for such a nice presentation. And yeah, I would definitely love to check out your papers afterwards. So let's see if we have any questions from the audience. I'll also link the breakout room and then the chat so that people can gather over there and talk to the speaker. Yeah, so I think there is one, is this the one? No, okay, I think this is one. So we have one question over here. Eddie asked about like, are you going to continue working on KMU as well as the kernel, or is someone else taking over KMU fuzzing? So that's a great question. Right now I'm the maintainer for fuzzing in KMU. So, but really my work has switched over to fuzzing the kernel mostly. So I'm mostly just doing odd fixes and also improving the state of bug reports to the KMU project. But we have, there have has been more interest, there's students from other universities and other developers and other companies who have reached out and there's actually, I believe like six or seven people on the reviewer's list now who see all the changes that we're making to the fuzzer. So there's definitely more things to improve there and I'm excited to see what happens there. But right now my primary work is the kernel. Yeah, yeah, that would be great. Finding students to work on KMU, it took me a while to get used to the code side. It's not, I know if it's something that you can get used to in a couple of weeks but yeah, if there are students who are interested in the sort of thing, I'd love to talk to you. Any more questions? We don't see any questions. So if you have any more questions, you can just go to the breakout room and chat with the speaker. So thank you very much, Alexander, once again. Okay, thank you. Thank you.