 Welcome everyone to my talk or like our webinar on fuzzing the Linux kernel. My name is Andrei and so I've been working with the Linux kernel fuzzing and other stuff for about five years now. I both worked a little bit on making bug finding tools like fuzzers and memory detectors and currently I'm mostly working on memory tagging which is an exploit mitigation technique. And my experience with kernel fuzzing in particular I like to split it into like three parts. The first one was about fuzzing the network subsystem via syscalls. We found a few hundred bugs and I even wrote three local privilege escalation exploits for three of them. Then I worked for a while on external network fuzzing and this is about let's say you have a Linux machine and it's receiving external data over network and the kernel is supposed to parse network packets for example if you have a TP packet that the kernel receives it's supposed to figure out which port this packet is directed to and then based on that decide which application is supposed to receive this packet. And this is also a source of memory corruption so if you send a malware on external packets you can find something in the kernel. And finally I've worked on fuzzing USB. This is also similar you connect malicious USB devices to a kernel and the driver for a particular USB device is supposed to be handling packets messages from that device and if there's a memory corruption that's also source of errors. So for all of this work I use syscalls so I'm a little bit biased but in this talk I don't want to focus on syscalls too much. I want to give an overview of the Linux kernel fuzzing approaches in general and some of my tips and some references. I'm gonna give a lot of references to different things that people did when they approached Linux kernel fuzzing. So for today we're gonna start first with the fuzzing theory in general so what is fuzzing, how does it work, what is coverage-guided fuzzing and so on. And then we'll try to apply the theory to the Linux kernel. After that I'm gonna describe two of the like biggest other most famous fathers that happened over the last few years and this string can syscaller and then I'm gonna describe a particular approaches that you can use for fuzzing and at the end if we have time I'm gonna go deeper into coverage coverage collection for the Linux kernel with K-Call. All right let's get started with the first part. So this fuzzing, fuzzing is a technique to automatically find bugs in computer programs and that's really easy to use you just generate random inputs you feed it into the program you execute the program and you check if it crashed it's great you got a bug and if it didn't crash just keep on going and generating new inputs. So it's really straightforward you can as long as you can execute a program you can you can use fuzzing and try to find bugs with it. And what kind of programs can have further targets? Well anything the most the simplest examples are user space applications and libraries and those are quite straightforward but the fuzzer can also target kernels right? The fuzzer can also target firmware basically anything that accepts some kind of input you can target with fuzzing. The definition that I gave earlier fuzzing is feeding in random inputs until the program crashes. If you kind of split up this definition into five parts whenever you're trying to make a fuzzer you have to address each of those parts. The first one is about the program how do we execute the program? It's straightforward for user space application because you just run it but maybe it's more difficult for the kernel and then if you're fuzzing part to the firmware it's also can be quite difficult to execute them. Then how do we inject inputs? Again if our program expects inputs on the standard STD in it's easy but then when you're trying to inject random USB devices and trying to fuzz the kernel this can be more difficult. Then how do we generate inputs? We can just generate random data or maybe we can do something smarter. How do we detect crashes? If the application just crashes let's say user space application crashes with a segmentation fault that's easy to detect right? But then for example if you have an out-of-balance right of just a few bytes these does not necessarily lead to crash immediately or maybe it will not lead to crash at all in your particular case. So in this case we want to be able to somehow detect bugs better than just relying on the application crashing. And finally ideally we want to automate the process and so we don't have to run the inputs manual like run the program manually. So most of these questions are answer to most of these questions depend on what kind of target you have what kind of program you're fuzzing but there's one that has some common theory behind it and I'm going to start with that and this is one about generating inputs. So let's look at an example let's say we have an XML file parser. How do we generate inputs for it if we're going to fuzz it? The most simple idea that comes to mind is to just generate random data. Unfortunately generating random data doesn't always work. For example let's say our XML parser expects the inputs to start with an opening XML tag. In this case if we're going to be generating every byte of the input randomly there is a very low chance that we will be able to guess all of the four bytes have this particular values. So the file that will need a lot of guesses before it can get past the header check. And then inside our parser there may be multiple headers like that multiple places where we need to guess the particular values of bytes or stuff like that. So random inputs are not good and they're good sometimes but not always. So random inputs doesn't work good what do we do? Well the obvious thing is to generate better inputs. Now how do we do that? And there's been a lot of fuzzing work in the last years and people have been mainly using three approaches to generating better inputs. The first one is about generating structured inputs that's also called structure where fuzzing. The second one is about guided generation that's most notably coverage guided fuzzing and finally we can just collect a corpus of samples. So let's look maybe to closely each of them. First of all for structured inputs in case we have an XML parser what we want to do instead of just generating random data let's try generating correct or more less correct XML files. So in this case what we do we come up with a grammar that describes an XML file. So instead of just generating random bytes we'd be generating random tags and those tags can have random attributes. And thinking about the case that we had previously where the XML parser would expect the file to start with an opening XML tag we can actually build in into build this in into our grammar. I mean in this case this is the grammar that I took from one of the fuzzing tutorials and it doesn't have this feature but this is can be easily extended. The second thing is about guided generation. So I'm going to explain it on example how this work. So first of all we choose a random input. Typically when we use guided generation we are going to be maintaining a corpus of relevant inputs and I'm going to explain what relevant means a little bit later. So we choose a random input we can choose one one of the inputs from our corpus or we can just come generate a new one completely. We mutate it mutating means changing it in some way. For example we can click bits we can add data we can remove chunks and stuff like that. We execute it and we see did we get any new code coverage. So the idea here is that whenever our program tries to execute our input or like to process our input it's going to execute different instructions depending on the input rate and based on that we can understand if our new input any good. If it didn't produce any new code coverage within our program that's probably our input is not very good and we just throw it away. But if it produced new coverage compared to the input that we were like previously processed that means it is somehow more relevant and we add it back to the corpus. To this way doing this process over and over we kind of go in deeper into the code exploring new instructions explore exploring new paths within our program. To this way we can go deeper and deeper step by step. Previously I this is what's about coverage guided generation right but the guided generation does not have to be only relying coverage. There are many different types of signal that we can use code coverage is the easiest one and more relevant usually that's why people use it but you can also for example rely on memory state. So let's say each new input that we process we're going to be tracking a value of some global variable and whenever this global variable changes into some new value with our new input we remember that input. So you can still use that in certain cases. This highly depends on the buzzer on the target that we're fuzzing and yeah but still I wanted to point out that the coverage is not the only signal that we can use. And finally of course we can combine guided generation with the structured inputs approach and we can mutate inputs accordingly. So in case of XML we can instead of mutating bytes like flipping bits we are going to be inserting and removing tags. And the final thing that people do is collecting corpus. So in case of XML you just scrape the web and you collect a bunch of different XML files and then you start mutating them you can also flip bits or bytes and feed them into the program. Or you can combine with the previous two approaches so you can do first of all structured fuzzing although in this case remember that if your input typically when you're doing structured fuzzing your first are going to be generating inputs in some internal representation right and then kind of like converting them into actual XML files. But in case you have an XML file as your sample you need to first parse it and convert into that internal representation before you can mutate it. But generally this works just as well. And of course you can combine this with coverage guided fuzzing. And to understand fuzzing better I think the best the best exercise that you can do is you can write a simple puzzle from scratch. And here are links to articles by people basically they just wrote a hundred lines of Python code and they wrote a simple puzzle for a target that they chose. Some of those further coverage guided you can do all kinds of stuff. I mean just just try to figure out how it works by itself by yourself and this is a great way to learn about fuzzing. All right this was like some generic theory about fuzzing I don't know if you have any questions on that. All right let's now let's try to apply all that theory to the Linux kernel. So as I mentioned before whenever you're trying to make a fuzzer you got to answer these five questions. And in case of the kernel instead of running some abstract program we're going to be running the kernel. And what I'm going to do is I'm going to describe my thoughts on each of these questions. So let's start with these two and these ones are about injecting and generating inputs. So when it comes to the kernel and actually when it comes to any kind of target like the very important thing that you need to understand is what kind of inputs does my target have? So what kind of inputs does the kernel have? Kernel is obviously program and it processes something. Right so the first kind of input that the kernel have is syscalls. So whenever you have a user space application running it executes syscalls and the kernel is supposed to be servicing those syscalls. So if you're fuzzing the kernel we can target syscalls. But then if you think more about it we also have some external inputs. So something that already mentioned is if the kernel receives a network packet it's supposed to parse it. If you have a new USB device connected it's going to send some data to the kernel. And we have like a lot of different other external buses. And in some certain cases firmware can also interact with the kernel and send some external inputs to it. And this is also a target for fuzzing. So how do we inject inputs? In case of syscalls this is really straightforward. You just execute the binaries that perform the syscalls. That's it. But in case of external inputs that's maybe not trivial. So I mean if the inputs are generated by hardware you could take a piece of hardware and actually use it to generate inputs. But that scales fully and does it always work. The other two approaches that you can use is actually either injecting inputs from user space or using hypervisor or emulator. So an example if we're doing network fuzzing we can actually... There is a thing that's called tuned up interface. And basically if you set up the jeff tuned device properly whenever you're writing data into it it's going to be going through the same parsing path in the kernel as if the packet was received externally. There is something similar for USB that's called the gadget subsystem together with the dummy driver. You can turn it into something similar. But most of these approaches require some kernel support. So the kernel is supposed to provide a way for the user space to insert like to inject this external inputs. And another approach that you can use is actually injecting those inputs from hypervisor. An example would be for USB. You can use QMU and you can use USB radio which is a protocol for actually transferring USB data from the host to the guest. And this way you can fuzz the kernel. There is a fuzzer that's called PUSBF that does exactly that. Alright, now let's talk about generating inputs. As I mentioned, the simplest approach to generating inputs is just generating random data. A little bit smarter approach would be generating structured data. But we have a problem. The kernel does not accept data as inputs. It expects syscalls. You could limit your fuzzing service to a single argument of a single syscall. But that would be like we would like to find an approach that works in general for all kinds of inputs that the kernel has. Okay, so most syscalls are used as an API. So this is something, an API is a term that I personally use. I don't know what the official name for this in the fuzzing community. But the idea is that whenever you use a sequence of syscalls, it's going to be a sequence of calls. Each of the calls can have multiple arguments and those arguments are structured, which means an argument can be an integer or it can be a structure or a pointer to a structure or something like that. And then some of those calls might return, return values. And some of those return values might be used in subsequent calls. So here's a small example. This is a, we're opening some device file with an open syscall. It returns a file descriptor. Then we're passing that file descriptor into a yoctl as a first argument, and passing a structure as a second argument. And then we close it. And in case of passing the kernel, this sequence of calls is our input. So you can think of it, you could think of like a single input being a single syscall, but that would be wrong because you have this return values that can be passed between syscalls. So typically, when you think about passing the kernel by syscalls, you would think of a single input as a sequence of syscalls. And this is what I call API aware fuzzing. And yeah, inputs are API call sequences. And whenever you fuzzing, right, you should be generating your inputs according to some structure. In this case, our structure is a call sequence. So whenever you're generating an input, you should be generating a call sequence, and you should be mutating accordingly for coverage guided fuzzing or stuff like that. So for example, inserting new calls, removing calls, changing arguments of existing calls. And I'd say this is like the biggest thing about Linux kernel fuzzing. And this is what syscaller realized. And that's one of the reasons that syscaller was successful. But then we can look at some of the let's let's take a break here. Is there any questions about this part? Because I'm going to be talking about more difficult stuff on the next slide. I don't see any questions in the Q&A or the chat on day. Okay, then let's keep going. This is this is the input structure for the simplest syscalls. But then we have other syscalls that might work differently. For example, they may not be as straightforward, or they don't accept just simple structures as arguments. So the first example that they have here are the two syscalls clone and syscaction. There might be others. And the idea here, let's look at syscaction syscalls. Let's say you have a API sequence of calls, a sequence of syscalls, and one of those is syscaction. Syscaction can be used to set up a signal handler for one of the signals. And essentially, whenever that single signal would be triggered, the kernel would transfer the execution to the handler for the signal. So you have a single API sequence that actually calls sysaction. But then whenever the kernel would transfer the execution to the signal handler, you can actually launch another API sequence. And a similar thing happens with clone, because you can be executing one set of syscalls before the clone then do the clone. And you can be executing one set of syscalls in the program that was cloned in the parent. But then you can be executing a different set of syscalls in the child. And this is, and it's kind of similar to the, what most syscalls do, but it's still a little different, because if you're fuzzing these syscalls, you have to be accounting for that. You have to remember that you can actually divert the execution in this case. And you need some way to describe those kind of inputs. And there are also two, or maybe more examples that are also kind of like different. And the two examples I have here is BPFA KVM. And the difference with those is that instead of just accepting structures, like static structures, they actually accept code that consists of instructions. And the problem is that whenever you generate the code as a sequence of valid instructions, even those instructions are valid, the code might still not be like valid in general. So for example, if you try generating just random sequences of BPF instructions, the BPF verifier is going to reject them in most of the cases. Same thing can happen with KVM, you're generating random instructions, even those instructions are correct. One of your instructions can cause a null pointer to reference or like to reference some random address, and that's going to crash almost immediately. So just by describing correctly, those instructions separately, there is no way to go very deep when you're fuzzing this kind of inputs. So this is something that's called I call script that we're fuzzing. And I think this is something that needs to be approached differently. I saw an approach that's called Fazili, and this is a fuzzer that is used to target JavaScript calls. I mean JavaScript parsers, JavaScript, what they call virtual machines, because they have the similar, similar issue. You can just generate random JavaScript code that needs to be correct in some way to actually trigger anything interesting. All right, Andre, we have one question in the chat. Does fuzzing examine the correctness of outputs, or just examine whether the cardinal will crash? Yeah, it's by the correctness of outputs. You mean you pass, as far as I understand, the question is about, so let's say we can be passing random inputs into the cardinal and waiting until it crashes. But then we can be passing random inputs into the cardinal and checking whether its behavior is correct. So fuzzing can be used for both. I don't know of any examples of people. Actually, I think CIScaller, what Dmitry tried to do with CIScaller is, so there's something that's called Juvisor. And Juvisor is an interpreter of Linux programs in a way. So what you could do, and it's supposed to work exactly as the Linux cardinal, but instead it's written and go and runs in the user space. So what you could do, you could try to run a sequence of CIScalls against the cardinal and then try the same sequence of CIScalls against the Juvisor and compare the results. And if Juvisor, so Juvisor claims to be identical to the cardinal, I mean, within some limited sense. And what you could do is you could actually check that the behavior, like the output of the cardinal and the output of Juvisor is the same. So this is something that you can do. And this is something that's been applied quite a few times to different... In fuzzing, when you have two implementation of the same algorithm, what you can do, you can just provide different, like, same inputs to both algorithm and compare the outputs. So this is something that the fuzzing can be used for. All right? All right, let's... Now let's look at the external inputs from the point of view of input structure. So if you look at network packets, they might seem like random data, but actually it's more like API. An example of that, actually, let's look at TCP connection process. Let's say our kernel or our machine, Linux machine, has a socket, a TCP socket in the listening state. And externally, we're trying to connect to that socket. So the way this works, you send a sync packet, you receive an ACP packet, and you send the CNAC packet. The problem here is that when you receive the ACP packet, it's going to contain a sequence number. And to actually establish the connection, you need to parse out that sequence number and use it in the CNAC packet. So this is like an API. You're expecting... You're kind of like parsing an response. You're getting a return value and you're reusing this value in a subsequent call. The same thing happens with HTTP cookies. Although network packets might be a little more difficult because you can be expecting... So you're not only getting responses to your calls. You can be getting responses at any point of time. So this is something that's similar, but actually more complicated. And then USB. USB is also quite weird. So USB is actually fully host driven. So if you're trying to emulate a USB device, you're not making API calls or calls of any kind. You are responding to them. And you don't actually know which kind of call is going to arrive next. So your fuzzer has to account for that. And this is... This is something that's very unusual about kernel inputs. So the overall idea here is whenever you're trying to write a fuzzer, you need to understand what kind of inputs do you have. And you need to generate and mutate those inputs accordingly. No questions. All right, let's keep going. So now let's talk about code coverage. So to do code coverage-gated fuzzing, we need some way to collect code coverage. And there are three approaches that people usually use. The first one is using compiler instrumentation. And that's what cakeoff is for. I'm going to have, if we have the time, a whole big section about cakeoff addent. But the idea is that you're relying on a compiler to insert certain types of callback into a code. And then from those callbacks, you collect the trace of execution. And that's how it works. Then you can use an emulator. So let's say you're running a kernel in the emulator. And emulator, what it does, it has a loop that executes instructions one by one. So what you can do, you can just hack into this loop and dump out extraction pointers. And this way you can get an execution trace. And people have been doing it for QMU, they've been doing for Unicorn, there are probably other implementations for other kinds of emulators. And finally, there is an approach of using hardware tracing features like Intel Pointer Trace. There's something similar for AMD as well. And the way it works, basically there is a way to tell you a CPU, like I want to trace this particular, like I want to trace the CPU. And the CPU is going to start dumping out trace into some particular address in physical memory. And then you can inspect that afterwards and rely on that for code coverage. Okay. So I answered the two questions about inputs. Let's move on to the three other questions. And these ones, I'm going to just briefly cover them. So first, how do we run the kernel? There are two typical approaches that you can use. The first one is using physical device. The second one is using a virtual machine or an emulator or something like that. The good thing about using physical devices is that you execute the kernel in its native environment, which means you will have all the proper hardware and you can actually pass device drivers. When you're passing in the emulator, you are kind of only limited to what emulator can, the emulator can emulate. But generally, if you're passing a very generic core subsystem of the Linux kernel, it doesn't really matter which which approach you use because both should work just fine. Then the bad thing about physical devices is they're hard to manage. So they're hard to restart, they're hard to get kernel logs from, they're hard to debug and other stuff like that. And also, if you're running your fuzzer as root on your device, there is a chance it can get bricked. So in case of the emulators, this is very easy to deal with because if you're, for example, your kernel hangs, you can just kill the virtual machine, you can just restart at any time. Most of the emulators have some gdb stuff, so you connect a debugger and most of them provide a way to collect kernel logs. When it comes to scalability, I think, I mean, it's much easier to spawn more VMs than to buy more devices in case of using physical devices. But that's, I mean, the only problem with buying more devices you might encounter is that in case you're trying to fast some devices, it's really hard to acquire. So you will, like, let's say you just have a single device and you can't actually get any more. So that's, that's something that can cause a problem in case you are trying to scale it up. All right, now about how do we... Andrei, one question. I think the question is answered, but I just want to give you a chance to see if you have more to add to this. If the fuzzer runs on the same host as the kernel that is fast, then disrupting the kernel may prevent test results from being collected. Do we generally run the test from a second host? Yeah, it's a great question. This is actually true. So I'd say this can happen, although in practice when you're running a fuzzer on the same device that is being fast, typically you have, you still have enough time to somehow pass, or like, I mean, it depends on how the kernel crashes. If it's a really hard crash that kills everything, of course, you will not be able to, like, even realize that something happened, except that device crashed. In case of the kernel, usually the crashes are soft. I mean, it means that the kernel still can be working in a certain way, so you can still have some communication channel to the outside world. But this is certainly a thing that can be hard to deal with. But in case you're running it in an emulator, so you can actually, it depends on your emulator, but some emulators allow you to actually save the full trace of what's been happening inside, and this is something that you can rely on. So even if the kernel dies and it has your fuzzer that tries inside the emulator as a user space application, it dies together with the kernel, you can still use, like, the crash down from the emulator to observe what's been happening. So I'd say there's no good answer to this, but this is something that definitely can happen, and you should be able, I mean, you should be thinking about that when you're making a kernel fuzzer. Thank you. All right, let's move on to detecting bugs. How do we detect kernel bugs? Dmitry Vyukov gave a session about this last week. This was also a mentorship session, so I definitely recommend watching that and checking out the slides. And the TLGR is just use the dynamic bug detectors that we have for the kernel. We have a lot of them. Most notably, we have Kazan for detecting memory corruptions, but we also have four detectors for infolakes, for data races, and stuff like that. The only thing that I would add to this list is you can actually I think someone accidentally muted. Yes, I think that is the case. If you have a question, please ask, yeah. Yeah, anyway, so the thing that I would add to this list is you can actually write your own bug detectors. You don't have to write something really as complicated as Kazan. You can just add some asserts into the code that you're trying to fuzz. For example, you know that in some particular place, a particular subsystem is vulnerable to a particular like logical fold or something like that. So instead of trying to generate a test case that crashes the kernel, like after that logical fold has happened, you can actually add an assert and try to find a test case that triggers that assert. This is something that can be useful in certain places. And when it comes to automation, of course, there is a lot of stuff you can do. You should ideally, if you're writing a further by your own, what should you targeting for is, for example, you should be able to just leave the further running for the night. You go to sleep, then you look up in the morning, you check out the fuzzing results, and you should be automating up to that point at least. So you should write some code to monitor kernel code for crashes. You should restart crashing VMs or physical devices in case you're using physical devices. Some more advanced stuff like deduplicating crashes. This is the case where we have multiple crashes. They're a little bit different, but still this is the same root codes. So you can actually like bucket them, put them into the same bucket so you don't have to inspect all of them. Then you can generate reproducers. And even at some point, you may reach the point where you want to report bugs and track fixes automatically. Actually, if you reach that point, you probably should just reuse syscolor or base your fuzzer on top of syscolor, because syscolor already is doing that. So you don't have to do it on your own all over again. Right. Any questions about that part? Don't see any questions at the moment. Okay. So let's briefly look at two fuzzers. The first one is called trinity. The second one is called syscolor. And this has two fuzzers that's like the two stepping stones in terms of Linux kernel fuzzing development. So trinity appeared many, appeared many, many years ago, and this fuzzer essentially is doing what? It's running an infinite loop, and it's trying to call different syscalls in this loop. So this is a fuzzer that does not generate inputs in the same that I mentioned before. It does not try, doesn't try to generate a sequence of calls as an input. They're just trying to generate infinite sequence of syscalls. But this fuzzer is still API aware in some sense because it knows about different kinds of syscalls, and it knows that some syscalls are supposed to be accepting these particular structures, and it tries to generate those structures. But there is no coverage guidance or any guidance in any other sense. So syscolor was a huge improvement. It was started on like five years ago, and syscolor is something like Trinity, but better. So first of all, syscolor introduces a notion of a test case. So instead of just generating an infinite stream of syscalls, it generates a finite sequences and executes them. And also includes isolation for running those sequences. This isolation is not perfect because syscolor still reuses the same kernel to run multiple inputs, but it's as good as you can get. So it tries to use different kinds of sandboxes, namespaces and stuff like that. Then it is coverage guided. I mean, the moment you have a notion of a test case, that means you can actually mutate those test cases and you can use the coverage guided approach and it relies on kickoff for that. And instead of hard coding all of those descriptions of different syscalls and structures into its implementation, it provides a language to describe them. This language is called syslang. And finally, it introduces a lot of automation. So eternity is usually binary. You just run that binary and everything else you have to do on your own. So you have to monitor the kernel output on your own. If you have Mcrash, you have to restart it on your own. Syscolor has all that built in. And also it even goes much further and it introduces something that's called sysbot and sysbot is a it's a few dozen of syscolor instances that run in the cloud, in the cloud. They're targeting mostly the mainline kernel but some of the other kernels as well. These syscol instances they are automatically reporting all of the bugs that syscolor finds. And essentially syscolor is something that goes deeper due to coverage guidance. It finds more bugs and it's easier to extend. But Trinity is still even though it finds less bugs I think it's still useful in certain cases because it's really easy to deploy as a binary. Just build it, you drop in the binary you run it and that's it. Syscolor can also be run in standalone mode but it's a little bit more difficult to set up. All right. So what I'm going to talk about next so okay this is what's all the theory about kernel fuzzing. Now if you want want to be doing kernel fuzzing your question is like what do I do now? Like how do I approach this? What can I do? And I want to describe these four approaches briefly and these are the approaches that I've seen people doing and all of them have been successful in a certain way. So the first approach is to build kernel code as a user space application. The second one is to reuse a user space fuzzer and third one using syscolor and the first one is just writing fuzzer from scratch. Actually see a question in the Q&A box. Yes, go ahead. Yeah, okay. The question is I've seen KVM specific syslong descriptions but I was wondering if Q&A is used to launch kernel then does nested support and then does nested support has to be enabled on host in order for fuzzer to use KVM specific syscalls. Yes, as far as I know the way KVM fuzzing works so you do run fuzzer in Q&A you do enable nested support and inside of Q&A syscolor tries to create another like a nested virtual machine but instead of using something like Q&A or KVM it has an implementation of its own KVM machine that designs specifically for fuzzing. I hope that answers the questions. I'm going to actually link if you want to see more about how this works I'm going to in one of the next slides I will have a link to the implementation of this particular part so you can check it out. All right, let's get back to our approaches. So the first approach is building code in user space and this works great for the code that's actually easy to separate from the rest of the kernel I mean the great thing about it that you don't need to bother about emulators you don't need to bother about kernel like virtual machines you can just reuse all of those nice tools that we have for user space fuzzing and here's a couple of couple of approaches that people have been doing the first one is for fuzzing BPF so that what they did is actually moved the BPF subsystem kind of like out of the kernel so they would run the verifier in user space and I think some other parts of BPF as well. And the second approach where they've been fuzzing the ASN1 this is like a fuzzer for crypto keys they actually also moved it to user space and fuzzed it there. So this works in case your subsystem like the part is even to target is easy to separate but for most of the kernel inputs that is not the case. All right, the second approach is to reuse a user space fuzzer and in this case you just take a user space fuzzer you can take AFL or libfuzzer this is as a fuzzer that people usually use for fuzzing user space application. AFL was the first fuzzer to actually introduce coverage guided approach but the problem with that was that it would actually read all of the inputs from a disk which is quite slow. Then there was libfuzzer that also as far as I remember AFL the first version used instead of using compiler instrumentation they would do some dynamic binary or like they would rewrite the binary dynamically to extract the coverage which was quite painful. And then libfuzzer came and introduced coverage guided fuzzing with compiler instrumentation and instead of also instead of reading inputs from the disk it would read them from memory. And there are also many user space fuzzer many other user space fuzzers that you can use. So instead of if you take a user space fuzzer so first of all instead of calling a function of a user space library like the thing that you normally do when you're fuzzing user space apps you interact with the kernel. For example you call a particular syscall or a sequence of syscalls. And the other thing that you will need to do you will need to plug kernel coverage into the fuzzer. So all of the user space fuzzers they rely on a certain type of coverage that is if so the kernel this is completely different. And this works fine for fuzzing inputs that accept blocks of data like file system images certain netlinks netlink stuff and other stuff like that. But usually as we discussed kernel inputs aren't blocks in case you're fuzzing syscalls it's a sequence of syscalls so you would want to somehow like be mutating sequences of syscalls. So you need to write some kind of generator a mutator for those inputs. And one of the approaches that they saw it's actually about fuzzing the XNU kernel which is an iOS kernel but you can apply the same idea for Linux kernel. So what they did there is they would turn structure to where you're fuzzing into API where but by basically representing a sequence of API calls as an array of structures. And then you write the custom generator mutator and that just works. All right. If this is a good time there is a question in the chat or stateless are they imply the chain of dependent API calls? If they are stateless is that possible? Is it possible? Every input, right? So you have to approach this. You can either approach this as a stateful approach in case you don't restart the kernel and you're going to be getting something like Trinity. But in this case if you're using coverage-guided fuzzing it might break because some of the inputs that you previously used might lead some state in the kernel. Or what you can do is you can try to sandbox those inputs like so they don't effect that they don't produce any side effects between them. And this is something that CISColor does and it's really this is not a very this is not an easy thing to do. I actually recommend to check out some of the CISColor talks to figure out how it's being done in case of CISColor. But yeah this is definitely a problem. Another thing that I saw people doing I don't think I have any links here but instead of rebooting the kernel after every input what you can do is called snapshot fuzzing. So you save the snapshot of whatever state you have in the kernel and before running your first input then you run the input then you reverse the snapshot. Of course to do this efficiently what you would need you would need some kind of emulator that allows you to actually take snapshot quick and restore the previous state. Right I hope I hope I answered this question in some way. All right let's move on to CISColor. So there's been a lot of CISColor talks over the last years I don't want to go over like each of them in details. I don't want to mention the same stuff in details and the thing that I want to say is CISColor is really good to fast API based interfaces because this is what this color was initially designed for. And usually if you have something that looks like an API like a CISColor sequences with structured arguments and there's just simple stuff like passing return values between them you should use CISColor it's a great tool. Some tips about using CISColor the first one is don't just fast mainline with the default config because I mean we already have CISBot that fuses the mainline kernel with the like a bunch of config options enabled. And if you're going to be fuzzing it locally most likely you will not find anything new. So to actually find some new bugs what you do with CISColor the first thing you can do is to add new descriptions. The second thing you can do is to tighten attack surface which means instead of trying to target all 300 of CISColor that the kernel has just choose three of them. And it usually works great if those three are related. So in case of network fuzzing for example what I've been doing for a long time is I would enable like the TCP sockets I would enable descents CISColor the receive CISColor and set socoped and I would be fuzzing that like four CISColor just those four CISColor and something else that you can do is to fuzz distribution kernels this way you can actually detect bugs that have been fixed in the mainline but have not been fixed in the district kernels. Then something that I want to mention is that CISColor is extensible. You don't have to just use CISColor as this and first what you can do is you can build your fuzzer on top of CISColor and regarding to that KVM question that was before so here are the links of the KVM father that's built on top of CISColor. So there is actually a separate package called iFuzz within CISColor to generate x86 instructions valid x86 instructions. It still suffers from some of the issues that I mentioned like you still have to generate valid sequences of instructions to go very deep but it tries to address them in other ways. Just check out check out these files that I provided. And what I also did is I also built a USB fuzzer based on top of CISColor. If you recall USB inputs are not API based but what I did here is I actually managed to well basically with USB fuzzing what I did is I introduced a notion of an API call and that API call would receive a request from the host and try to understand what request this is and based on the requested with generate response. So this is something that can be done and you don't have to just run CISColor as is you can build on top of it. And finally you don't even have to build on top of CISColor you can just reuse some parts of it. For example the crash parsing code or the VM management code. So CISColor actually has a code to manage different devices different types of devices that CISColor can run on and it's all behind the same interface. So there is the same interface for managing QMU instances there is the same interface for managing Android phones and stuff like that. Andrei we have a few questions if you want to take them. The first one is have you considered generating fuzzing scripts as correct sequences of system calls using the model-based testing approach when we have some formal models for describing the behavior that is allowed. No I have not considered that I've never heard about model-based approach but so yes I mean generally I have not I can say that I've considered any particular approaches to resolve this kind of question but I wanted to point out that generating instruction sequences is something that wait okay sorry I like I was confused because it says generating fuzzing scripts as correct sequences. Okay anyway the answer to this question is no I've never heard of model-based testing but maybe this is a viable approach that you can use. Yeah I was confused I thought this question is about like instructions in terms of KVM, BPF and stuff like that. Okay that sounds good thank you. The second one is so if fuzz produces if fuzz not quite sure what the question is if fuzz this is if fuzz is the yeah this is color part that generates x86 instructions yes it produces guest code correct. So actually if you you were talking about how are we doing on time by the way yeah it's fine yeah okay so in case of iFuzz there is a actually what's called a pseudosys call it's like an API call that tries to set up a guest like a guest VM into some interesting state a guest CPU into some interesting state and then there is the iFuzz part that tries to execute random instructions inside this guest. This is how it works briefly. Okay one more question have you tried to use a symbolic execution to assist fuzzer? Have you tried simqmu qmu to fuzz kernel? Yeah I've never tried that personally I saw people doing that I'm going to have some links at the end of the presentation where you can check out some papers that people are trying to do that my so I haven't tried that but my overlook's impression is that symbolic execution works in certain limited cases but it hardly works for such hard targets as the kernel because I mean symbolic execution is you're trying to generate the next input based on so you're kind of trying to figure out what kind of input should you use to actually go deeper into the code and you're like trying to solve a set of equation in case that your target is really big there is a lot of equation equation that you need to solve and it kind of is getting slow One more question Andre does syscaller specifically try to generate test cases which trigger code paths who are not covered yet? No, not as far as I know So this is something that's related to symbolic execution as well but yeah, it's not trying to do that Okay, second part of that question is does it explicitly try to trigger the code lines which their color is red in sysbot or does it just try to cover new coverage? Yeah, it just tries to mutate the inputs that you have and if they cover something new that's when it registers them but it does not specifically try to apply mutations to try to cover new lines of code now Thank you, that's all we have right now Okay All right, the final thing that is the final approach that I wanted to mention is writing a fuzzer from scratch and this might work really well for targeting some weird like for writing a targeted fuzzer for some interface that is not API based and instead of going on a lot of into a lot of details about that I've linked here three articles and I definitely recommend reading all of them so the first one is about writing the world's worst android fuzzer and what they're doing there is just trying to go through different kinds of device files that an android phone has and try to write and read data from them and that actually works for crashing android phones The second one is also really interesting it's about ebpf fuzzing and they're using kind of like a hybrid approach there because they manage to they run the ebpf verifier in user space and they're fuzzing into user space and trying to generate a sequence of ebpf instructions that passes the verifier and then they pass the same sequences of instructions into the kernel and try to figure out if these sequence of instructions can trigger a buck in the kernel and they're also using a very interesting approach to actually detect logical bugs within bpf programs in the kernel so definitely worth checking out and the final article is both fuzzing the x86 entry code this is yet another type of inputs that the kernel has instead of just trying to target the syscalls what they're doing is they're trying to set up all the registers in like weird states and then trying to call different kinds of instructions that are supposed to call a system call as an example in 80 or syscall instructions or other stuff like that so definitely recommend checking all of these articles out right any questions no not at this time okay now I want to share some generic fuzzing tips and the first one is read the code this idea here is the ultimate target that you're trying to fuzz is the code so instead of relying on some documentation or some specs like in case of usb I read a lot of usb specs but then I ended up writing those are not always relevant and what you should do you should try to read the code you should understand what kind of inputs it expects how it behaves and based on a fuzzer like write a fuzzer based on that and also what really helps is that when you're fuzzing some particular subsystem you should identify a part of code that you're trying to target and if you're doing coverage guidance fuzzing the part of code that you're trying to cover this really helps the second thing that I wanted to mention let's say you wrote a fuzzer so how do you know if this fuzzer is any good these are some simple tips the first one is check code coverage so you don't only use code coverage for guidance you can also use it for inspecting what your fuzzer does and if you follow my previous tip and you actually identified the layer of code that you're trying to target the part of code that you're trying to target just make sure that your fuzzer covers it that then what you can do is you can actually inject bugs into the code that you're trying to fuzz and check the fuzzer finds them or you can revert fixes for certain boxes CVEs and also make sure that your fuzzer finds them so this is in case your fuzzer does not find any real bugs just make sure that can find at least injected bugs or the bugs that were in this code previously and another thing that I wanted to mention there is something that's called like if you have a fuzzer you can work on two things first you can work on improving how fast it is which means you can try to achieve more execution per second or you can improve how smart it is so you can improve the way you generate inputs you can improve maybe use some more relevant guidance signal or maybe some other stuff and my general tip is to focus on smart first before focusing on fast it would be really interesting to see if this I mean this is my experience and it would be interesting to see if there is like some formal if there is a way to formally verify this I actually saw this paper I don't sorry I've linked this paper I mean when I share the slides you can check out the paper it's about something similar but still a little bit different so it would be interesting to see whether focusing on smartness is actually a better approach focusing on making buzzers fast all right any questions Andre what is the first question is a simpler one is what is the paper and called or others okay let me let me open it I'm gonna copy the name into the chat I mean I'll share the slides later so that's good so there is a very there is a question that's rather long do you want me to read that out or will you be able to check the chat I can read it out okay I see okay for fuzzing complex ioctal handlers they have really yeah sorry let me read this first okay this is the question as far as I understand it's about whether we can actually use this symbolic execution to guide our fuzzing and can we build that into cscolar the answer is yes we can the question is if you can actually manage to get it working and working fast enough because if you're if you're trying to generate your inputs and you're trying to solve equations for one hour then obviously it's not worth it I mean unless the result of your solution to this equation is the input that actually triggers the book immediately the this is something that you can do but nobody actually implemented that so that I mean yeah there is no no implementations that I know of that work good okay so that is one more question in the question Q&A so does cscolar provide valid ebpf programs when fuzzing no cscolar cscolar generates valid ebpf instructions but then the ebpf program that cscolar generates is just a an array of those instructions so it might work okay in certain cases because the instructions that are generated they're good I mean there can I mean there are arithmetic instructions or instruction that tries to access bpf maps but cscolar does not try to generate a consistent input in terms of a single program like a sequence of instructions it just tries to generate those instructions separately and I think this is something that can be improved within cscolar and you can achieve some interesting results with that thank you okay all right let's talk about the final part which is about collecting coverage with kcoff and so what's kcoff kcoff is a tool that specifically designed for fuzzing and for collecting code coverage from the Linux kernel it's available upstream you need to rebuild the kernel you need to enable config kcoff and the reason you need to rebuild the kernel is because kcoff is based on compiler instrumentation and it's able to collect coverage from both user threads and background threads and also software interrupts and the second part requires kernel code annotations but I'll describe that in a second so first how does it work the way it works is it enables a certain gcc and clang flags to insert excuse me insert compiler insert callbacks into each basic block so consider if you have an if close it's going to insert a callback before the if close inside the if close and after the if close and then all of those callbacks callbacks are used to actually collect the coverage trace so the way you use it is there is a debug phase debug fs interface you open cskernel debug kcoff you m up the buffer this is a buffer where the coverage will be collected and you call the kcoff enable ioctl now after you do that whichever cskos you execute coverage that happens in their handlers will be collected into this m up buffer so this is done like each sanitizer kcoff trace pc call we'll see that kcoff is enabled and it will know that it's supposed to collect to put coverage into this m up buffer so kcoff collects coverage from the current user thread by default and this is done on purpose the reason is that let's say you try to execute an input a sequence of cskos and to do coverage guided fuzzing properly you would only only want to collect coverage that is relevant to this input and since the kernel has a lot of different stuff happening in backgrounds there are interrupts there are different kinds of threads the schedule can executing you don't want to collect coverage from every single thing that's happening in the kernel so initially kcoff has been designed so it only collects coverage from cskl handlers that happen synchronously there's a problem though with this approach I mean it works let me put like that it works great for what this color has been doing and it's I mean yeah anyway it works right but there is still a problem with this approach and the problem is that if you have an input that you're trying to execute some of coverage that's still relevant to this input might be executed in some background code so let's an example would be let's say you have a device handler and when you open this device handler or like when you create an instance of this device you open a device file a background thread is getting created and then the data that you're writing to the device is actually handled by the background thread instead of being handled synchronously in the cskl so in this case you will not be seeing this coverage when pausing so we've tried to address this with a kickoff change and the solution that we came up with is that we thought about like can we actually somehow automatically decide which coverage is relevant and which is not and in certain cases it's it's it's not possible since that we went with the approach of actually annotating the parts of the code in the kernel to collect this background coverage and the way this works is let's say you have some background thread and there is a section of this in the threat implementation that you want to collect coverage from so what you would do you would put two calls the first one is called kickoff remote start the second one is called kickoff remote stop and since you need to somehow understand like which which strategy you're collecting coverage from there is a unique ID that assigned to each of those sections now this is what's the initial idea now the the problem is that you have a lot of different background threads and they might be using different IDs but how do you pass this unique ID from user space or maybe how do you know to which threat a particular unique ID belongs so let's say you have a particular background thread and you only want to collect coverage from that how do you actually connect to the threat and do that so for this part this caller actually makes or a kickoff actually makes distinction between two different types of kernel threads I call them global and local I don't think that's the official kernel name I don't know the kernel name has say an official name for that but that's that's what I call them it's a global background threads as the threads that are spawned from unique code during boot typically and typically there is a fixed number for example you are booting the kernel and you have a single network device and the network driver for this device will spawn to threats one for handling the input data on this device one for handling the output data for this device but then you have that something that's called that they call local background threads and these are the threads that spawned from syscall handlers and let's say you're opening some device file and inside the syscall handler for this device file a new thread that handles this particular instance of this device file is created and we kickoff makes a distinction between those so for global threads there is no easy way to pass these IDs from threads to your spacer from your space to the threads so what we do is we use predefined IDs let's look at an example this is a hub event thread and this is a global thread that is spawned for each USB bus when the kernel boots so this thread handles all the USB data that is going through a particular USB bus and the way we do so we have the kickoff remote start and the kickoff remote stop annotations in there but what kind of ID do we pass so we pass an ID that's fixed here actually use kickoff remote handle and the kickoff remote handle just think of it as a simple function that actually mixes its two arguments so something that we wanted to address here is to make this extensible so the ID is generated based on subsystem ID and in case of USB based on the bus number and the way you would collect coverage from this particular global thread is you would issue a kickoff remote enable instead of just kickoff enable and you pass this particular ID into the handles array of the argument of that IOCTL so you can connect to multiple threads and you need to actually know which IDs the threads use and the definition for this kickoff remote handle of kickoff subsystem USB is actually put into a UAPI header so it's available to both the kernel and the year space so I don't know was this clear are there any questions about this part so far right let's keep going then so this is for global threads but then for local threads it's actually simpler because local threads can be created from today are created from syscall handlers that means we can actually pass unique IDs from user space and the way this works you also use the kickoff remote enable syscall and you use a different argument of the structure that's passed you use a different field of the structure that's passed to this IOCTL and the field is called common handle so what this IOCTL does it actually saves this common handle inside the current pointers current points to the current user task there is a when kickoff is enabled there is another field inside the current task struct and it's called kickoff handle where the common handle is getting saved and then in the syscall handler that actually creates that local thread what you do is you take that kickoff handle from the current and you pass it into a structure that passed to this thread so normally whenever a kernel thread is created some kind of data is going to be passed through it and in this case we are looking at example of vhost and whenever you open a def vhost device this function is going to be called and it's going to create a thread that handles this device so what we do here we take the kickoff handle and we pass it through def and then inside the implementation of the thread what we do we take the handle that we passed and we call the kickoff remote start and kickoff remote stop this way we can actually pass a common handle from a single kickoff remote enable call and to collect coverage from all the local threads that have been previously annotated right something that you need to keep in mind when you're using this stuff is when you're fuzzing from multiple processes in single VM there is an issue with global threads so the issue is that if you want to be fuzzing from multiple processes you want to know which global if there's only a single global thread that handles inputs from all those processes there is no way to know which coverage is related to each input so what you need for using kickoff with global threads is you actually need a separate global thread for each fuzzing process and the only the only actual user right now in the kernel of this feature is USB and with USB we create a new USB bus a dedicated USB bus for each fuzzing process and this way each fuzzing process has its own each USB bus has its own background thread and each of those background threads has its own ID so the fuzzing process can connect to the background thread and only correct coverage from it for local threads though you don't have any problem you can create multiple threads within the single syscall handlers or within the single program you can call multiple syscalls that create threads so it's all going to work we have a question Andre is there a user space utility to parse the output of syscarnal debug kickoff I mean similar to S-Trace on some program there are some tools so if you go to syscaller repository and you check out the tools director I think there's something that's called cake I don't remember what it's called but there is a tool to actually visualize the kickoff done so you pass the whatever you collected from kickoff you pass the kernel image and it's going to actually show you the it's the same thing that you see on syscaller dashboard so you're just going to show you the which lines are covered and which lines are not so just check the syscaller tools directory there is another question which is also long in the chat would you like to take and read that I think it might be hard to follow if I started reading okay this one is still about symbol execution say in an ioc tool handler we have a case statement like cmd equals that beef would it be worth extracting statements like this in advance before fuzzing the syscaller so you don't waste iterations trying to drive that beef before getting inside the case statement or will syscaller with kickoff be able to figure out this easily yeah it's a good question so there is a single instance like okay not a single but there is a very important instance of this behavior that I saw when I was fuzzing USB and when you're fuzzing USB when you connect a new USB device it passes a certain ID like a device ID vendor ID to the kernel and based on that ID the kernel is trying to figure out which driver to load and the code that tries to figure out which driver to load it's basically a loop consisting of three lines it goes through a list of device drivers so whenever a new USB device driver registers itself in the kernel it puts a link to itself into this list so there is a very short loop that tries to go through all the kinds of device drivers that there are and tries to tries to find out the IDs so each driver has a set of IDs that for the devices that it handles and the loop is trying to match the the IDs of the device that you have with the IDs that are inside of the drivers right and the problem here is that if you rely only on code coverage there is no way you're going to be able to connect or like to go into different kinds of devices because the code is always covered there's always at least a single instance of this loop that's being executed and what we did here is we actually used exactly the approach that you mentioned so when C-Scholar tries to generate USB IDs IDs of USB devices beforehand we run a like a program that tries that extracts all of the IDs for all of the drivers that are enabled in the kernel and then whenever we generate a program we actually use one of those IDs so this is something that you can use in certain cases this is one of the only ways to deal with stuff there is another question in the Q&A box since the last talk last week I have been trying to fuzz a rooted Android device but C-Scholar attempts to ADB push CIS fuzz during the boot splash screen and get a permission denied error do you know of any resources for setting up a C-Scholar Android environment and if not anyone with experience using CIS-Scholar for Android fuzzing I have seen the set of docs on the GitHub looks like they're looking for resources for more help yeah I mean the you can check out the CIS-Scholar documentation has two pages the first one is called CIS-Scholar talks the second one is called CIS-Scholar research articles and they're both linked from the CIS-Scholar main page they might have some details but I don't have an answer to this question so what you have to do here if you see that the fuzzer you're using CIS-Scholar and you see that something is going wrong for example in this case your CIS-Scholar starts copying the fuzzing binary before the Android phone boots that means there's something wrong and the only way to fix that probably is just to go read CIS-Scholar source code and try to fix it in there so if no one maybe your phone your Android Android version that you're using works a little bit differently than the one that's been used when this code was developed so probably there's not going to be a straight up answer to this question so the only thing to deal with this is to actually read the code understand what's wrong and try to fix it yourself right okay let's yeah let's switch to the final part a few final notes the first one is developing fuzzers is engineering so what I mean by this is that if you're writing a fuzzer if you want to write a good fuzzer you got to be good at writing code so the reason that CIS-Scholar has been such a success is because it has a lot of engineering experience and a lot of engineering thought put into it and yeah the better like you can't okay anyway that's that's what's the point so anyway you got to write be able to write code and the second note that I wanted to give is I think you've reached the point where whenever you have a good fuzzer it finds a lot of box so if you look at CIS-Scholar dashboard there's going to be a few thousands of box and not all of them are dangerous so fuzzing became kind of like the new static analysis in the sense that static analysis produces a lot of box but most of them are false positives right and unfortunately not all the box gets fixed so there's no way for you let's say you're trying as a security researcher you're trying to find box that matters if you have a few thousand box it's going to be a very hard work to actually go through each of them and try to figure out what each of them does something that might be one of the interesting approaches that I would like to see in the future is trying to figure out which box actually matters if you have a thousand box maybe there is some approach to find out which of them are exploitable but this is not something that I've seen I've seen some papers that are talking about automatically exploiting box and this is a step in the direction that I would want to see people going just a side note and then I have a list here of different kinds of materials you can check out first of all there is the first two links is just a collection of links to articles and I mean the first the first block of links the collection to links article research papers about kernel buzzing in particular so there is a lot of them you can check out you can check them out then a few people to follow on Twitter first of all you should follow Dmitry Yukov who is the author of C-Scholar then there is Gamozo Labs who is Brandon Palk and he I'd say if you're interested in doing more advanced buzzing stuff this is the person that you would want to follow he's actually doing streams on Twitch where he takes some target and tries to fuzz it and a few months ago he's been fuzzing an Android phone which was a really interesting thing to watch but definitely you can check out what he does and then whoever else like whoever else's work or whoever else's article I mentioned in the talk you can also find out what they do for all of them and then finally there is a telegram channel or where I post the links that are related to links kernel sometimes so you can also find some materials there all right do we have any more questions a couple of questions we have how does C-Scholar deal with kernel code that triggers timers kernel code that trigger timers I mean can you give an example I'm not sure what exactly this means so maybe you mean like how does it deal when you have a maybe you have a C-Scholar handler that's actually instead of doing stuff on its own it creates a timer and does it in a timer so in this case if you want to be doing so C-Scholar doesn't really have any specific handlers for these cases I'd say the thing that you can do if you want to be collecting relevant coverage is to actually use the remote cake-off coverage stuff that I mentioned so whenever you have a timer it's typically executed in a software interrupt context and whatever I mentioned about background threats you can use the same annotations to annotate interrupts and this way you can actually connect certain types of interrupts to certain types of inputs with C-Scholar but this is something that you have to do manually so there is a follow-up does it collect coverage for the code that runs in the timer I think that's... Yeah, that's what I meant so by default no because it's software interrupt context and cake-off ignores that but you can use the annotations and that is the only thing there are some limitations so if you actually open the oh anyway you should try it there are some limitations to when it works but this is fixable there are like I've only implemented it as far as the cases that I saw and in some certain cases you may need to also work on cake-off a little bit so you have to improve cake-off but generally it should work yeah One more question systems may be particularly vulnerable to boot is there a way from for example boot arcs to start fuzzing early maybe from a special system the unit I've never thought about that yeah I mean maybe there is I sorry this question is something that I don't really know what to answer okay that's an interesting question because sometimes early bugs you know or sometimes we do not know how to debug yes that's a great question but okay that's any other question that makes me think yeah sorry there's not something related but they just made me think that maybe it makes sense to try to make a fuzzer that tries to fuzz boot arguments of the kernel maybe someone already did that and this is something completely different and it's more about finding stability bugs than security bugs but still that would be interesting great any other questions we're coming up on almost we have five minutes left left at this point okay if there are no questions thank you for joining us today Andre thank you for the talk it's very informative and awesome I want to leave you with some resources to continue your self study and education and we have a and we really hope that this talk this webinar today and other webinars in the series are will be helpful for you to continue a journey of learning to be more effective and productive participation in the open source projects we will leave you with a few additional resources for your continued learning the Linux kernel Linux Foundation mentorship program is designed to help new developers with the necessary skills and resources to experiment learn and contribute effectively to open source communities outreach remote internship program supports diversity in open source and free software in addition Linux foundation foundation training offers a wide range of free courses webinars tutorials and publications to help you explore the open source technology landscape the next foundation events also provides educational content across a range of skill levels and topics as well as the chance to meet others in the community to collaborate exchange ideas expand job opportunities and more you can find all events at events.linuxfoundation.org please continue your journey of learning and thank you Kristen do you have anything to close with no that's it thank you so much Andrei and Shuwa and hope everyone has a great day thank you thank you for inviting me thank you Andrei thank you Andrei bye bye bye