 Welcome to the second slot of the CSU or BSD card. With us is actually Gohavsky, who's been working on fuzzing, fuzz systems in that BSD using the American Fuzzing Loop. So, give him a nice welcome. Good morning, everyone. My name is Maciej and as was spoken, I started working on NetBSD fuzzing with American Fuzzy Loop and Kiko. So, I have this pleasure to be your presenter before the lunch break. So, it's not ideal, but hopefully you'll be able to make it. Let's start with the outline. So, today I will be talking about the fuzzing, but the fuzzing is part of larger initiative inside the NetBSD, which is NetBSD quality assurance. So, first of all, we'll touch on this topic and also I have a couple more slides, which was done in this area. And then I will touch a little bit what the fuzzing is. Probably everyone is already familiar with the fuzzing is, but just as a reference and some background, I'm kind of obligated to do this. And then we'll talk about the work that I was doing for porting the American Fuzzy Loop to fuzz NetBSD kernel and file system subsystem in general because I'm mostly interested in file systems and some work for coverage and Kiko that I was doing with some members of the community. Then I will show some basic fuzzing setup and how you can easily do the fuzzing of the file system. So, I will try to convince you that fuzzing in this way is really simple and everyone who is writing some kernel code can also do this by its own, so you don't have to do some very complicated setup with multiple virtual machines, some servers. You can just do this locally, just like a very low entry bar. And I will end up with some conclusions. And as I said earlier, I have also a couple of slides from my colleague Kamil about the state of NetBSD QE. So, I plan to do those slides after and then I will be able to take a questions. So, as I said, we have this initiative inside the community to improve the quality of NetBSD. So, we have many different ideas and a couple of people working as a community to improving the quality. And this started a couple of years ago when we got some security researcher finding multiple bugs in NetBSD. Some of them were in some kind of stale code or not frequently used. But as I remember correctly, there was something like 80 NetBSD bugs which were fixed very quickly but people started thinking, okay, so how we can make sure that our quality as a small community will be able to scale and how we can improve the quality. So, we have sanitizers and NetBSD is currently very well sanitized across the BSDs. And so recently we got MK sanitizers which is a really nice tool to get our user space for fuzzing because when you are doing the fuzzing, you usually also would like to run with sanitizers, couple kernel sanitizers, leak sanitizers, address space sanitizers and different behavior sanitizers. We have a security team, so if you find any bugs or have some concerns, you can talk with them directly. So my part of the work is related to fuzzing with AFL, American Fuzzy Loop, but also there are a couple more fuzzers. This summer we had a couple of students working on SysColor finding a lot of interesting things inside the SysColor. We also get people from, I think from Google to run also the SysColor because SysColor is a Google project on NetBSD and even yesterday we got a couple of reports, I think three or four crashes. And AFL three-fours, another Google summer of code project, also very interesting. And we also have ramp kernels, which are very interesting topic. I won't be talking too much about ramp kernels today, but also encourage you to research a little bit this area. So there are other things like stack protector, static code analysis and hardware bug mitigation that also people looking after. I'm not very familiar actually in this part, but also if you have any other idea how we can improve the quality, feel free to join us. The simple way is just go to NetBSD QE on the free note. There's a couple of people over there. So I also started this spring working on the AFL. Some kind of like a not very old, not very old member of this community. By the way I found this really nice and also I think the entry bar is very low and you can start doing some interesting things quickly. So starting with the fuzzing, I won't be talking too much about this, but just a simpler reference. It's how you can write the simpler fuzzer. You can just get some random data. Let's assume that you have some binary that you like to fuzz. This binary takes our argument up half to the file. This file can be for example 1k in size, so we can just get 1000 bytes from random and put them as a file and then run the fuzzing binary with this file. So this actually is the simplest way that you can do. From the other side as you can imagine the space of all possible inputs is exponentially large and most of those inputs won't really fit to your program because usually the program expects some particular format of the data. So you may have a program that needs some image, for example JPEG or PNG. You may have something that requires some document. So if you are fuzzing in this way and just getting the random data you most likely will be iterating over and over again on some header checks, on some entry code of the program and you won't be executing the paths. So how we can do this fuzzing process much smarter? So we can start thinking first of all, if you thought about this problem, you can try to improve somehow the way how we generate the input. So we can for example have some idea what the input format is. So if we know that we are fuzzing the, for example, images we can take some input as an example like the image file and then try to change and play with bits from it. And this is kind of the way how mutuation fuzzing works. So mutuation fuzzers essentially take some seed and then have some idea about the, let's call it grammar or structure of the file if you will and then play with this structure, flip the bits from one side put some things in other section and run the program. By design most of them do not take the feedback from your program and from the other side this feedback thing is something that is used commonly in evolutionary fuzzing. So evolutionary fuzzer is the fuzzer that can iterate many times and observe the behavior of the program. So the simplest way to think about this is you can have some feedback coming from your program which for example can be execution time. So if you know that this thousand of bytes of all zero execute for example for one second and then if you flip one byte somewhere and then your program will execute this input for 10 seconds that means maybe this one bit really does matter and move your input to some other execution path. So maybe this is something that you should consider during your fuzzing and over the time we create another generations of those inputs and also we take the feedback after every generation. So this is also the way how American fuzzy loop works. Someone may argue American fuzzer also is doing some mutational of the state and that is also correct because today's software mostly is very complicated so it's not just one or another one but you combine those approaches. So as I said, AFL is evolutionary fuzzer which takes coverage as a feedback so we have a feedback loop based on the coverage of the code. So AFL uses the map or array of the pair source and destination branches so the way how it was originally implemented when you compile your program you compile with specific AFL compiler this compiler does some custom instrumentation so it put additional instructions on every branch in your program so then while you are executing you have some chart memory and inside this chart memory every time when you hit the branch you have couple of instructions which see the source branch and destination branch so you get this pair and you put this pair to the array because of that the fuzzer get a hint so it's already this path in the execution graph so this path already was covered or maybe this path wasn't covered and I can take this as a hint ask my feedback to create another generation of the input from the other side in NetBSD 8 we had a cake of coverage device which provides us PC trace, program counter trace and comparison trace so the way how PC trace works you run your process and then inside the kernel we also have instrumentation which are done during the compilation so you have to compile your NetBSD kernel with option in order to use cake of you cannot get just normal image and run coverage on it you need to recompile it so it's very simple you just go to the config and you enable cake of and it should work so then you have every time when you execute the kernel code for specific process you can collect the coverage data inside the buffer inside the kernel space and then the buffer also is mapped or can be mapped because you don't have to do it but this is usually your case if you are doing the fuzzing so this buffer then is mapped to the user space by mmap and you can see the coverage data so in order to make American Fuzzy Loop works on NetBSD with cake of I needed to fix a little bit this tracing so as I said we had the at least long list of addresses in the kernel space and we needed to convert it to get the different to essentially create the 64k buffer for the fuzzing so this conversion is very simple you just go through the whole list and you see the current PC counter and the previous one and then you do the XOR on them of course you also have to make sure that you are in line in the size of your array and that's mostly it but we had some discussion how to do this properly so the first idea was okay so let's leave coverage device addresses and do the conversion on the user space site which end up that is not a very good idea because NetBSD kernel is a little bit more verbose that we expected so your trace can be like multiple thousand of entries and doing the work first of all inside the kernel collecting and then doing conversion in the user space doesn't really work well because you are going through the same list twice and inside the fuzzing the performance is really important so I decided to do this inside the kernel as to do this inside the kernel as a kind of like an addition to the cake of other things that American fuzzy loop require was just simple things like changing the shark memory to the end map for the coverage device opening the files so not much here inside the user space to change but as I said because I put AFL inside the kernel space we were talking a little bit how to do this properly and there was some work for example done in Linux where people decided to provide additional trace type which would be AFL in order to handle this correctly which is not ideal because AFL is not really the tracing format it's more like the standard for the program so in order to handle this we modify the cake of device so there would be some changes coming soon where you can have your own modules for coverage so you can register your module for doing the coverage and this registration requires a structure so this structure essentially requires you to provide open free operation you will have hooks to coverages right now we have PC and CMP coverages but in future there will be more and after you register your plugin when you do anything from user space to coverage device you will be hitting those functions but from the other side you will still use the same code as a cake of device so we won't have duplication of the code and also we will have one common code for dealing with the coverage which from my perspective is a really good thing so as I said generic cake of module which move the responsibility of doing anything processing anything with coverage data move it to the plugin we still handle the raw coverage inside the cake of and the data is easily accessible using the callbacks we can support potentially multiple different fuzzing for every fuzzer you just need to write your simple module which is usually just take a look for example an AFL module or like a sample module that also was written and then you can see which operations essentially will be different from your you implement them and we should be ready to go also we can do stuff like filtering the coverage data which is also important so the simple setup of the fuzzer I describe a little bit the shark memory how it works and also cake of so we have a coverage device and the plugin so the whole setup will require a couple of more components so we need also a wrapper for AFL so the wrapper itself is something that is specific to your type of fuzzing for instance in my interest there are file systems so I need some way to get my code executed inside the kernel inside the file system stack somehow so I need to get my wrapper and write inside this wrapper some mode operation I need also to prepare the block device but I won't have block device straight away so I need to get a file pretend that this file is a block device and then run amount on the block device and during the call I will collect the whole coverage data from the syscalls, from VFS, from in core file system then translate it to AFL and we'll have everything inside the memory the fuzzer itself will analyze this data and will do some modification and create different files different generations of the inputs the next thing that I found really interesting in this project was I had this setup and then I started thinking I would like to run some kind of hello world for fuzzing so I create some simple hello world for fuzzer which is something that is in many different fuzzing tutorials that you can find online so you just write, you just hard code some magical pattern and then you get input from your fuzzer and you are trying to break the magic and you see how fast it takes for example to break something like four, five, six characters magical password hard code is password so I run this and it was running and running and after a week my fuzzer was still not able to break the magic password and they started doing okay so I know we usually think about the fuzzer as some kind of like a magic process that just take a program as a black box and execute on it but if you essentially would like to see what the data are visible for your fuzzer what you can do for example as I did the debug the process I get the list of the trace for my mount process I translate it to the particular functions and then it was not able to find anything related to my code which is kind of surprising so then I get all of those traces and count unique entries for those traces what I found was essentially we have first couple of hundred entries was essentially the kernel code not related to our fuzzing and especially we had a lot of a lot of built on memory operations the page faults so there was about 20,000 entries and from them 1500 for example was related to the page faults so my initial thought was hang on the moment because the way how AFL works it detects the part of the source and destination branch so in theory we should be able to see that okay so this is something that we see but we already saw this execution branch because it is in the map but it turns out that you can have multiple different combinations of those source destinations so essentially you are not fuzzing the file system you are fuzzing just the virtual memory subsystem so how we can fix that so first of all I was thinking because the instrumentation are compiled time operation I can just go to every of those functions every couple thousand of them and change and remove them from the trace so I can add no instrument function keyword from GCC and let's try this and as I turn out because at that time we still had GCC7 so GCC7 for compiling the kernel and GCC7 does not it does have this keyword but essentially this is not working for 7 so in order to make it work I started compiling the kernel of GCC8 that was a long story, a lot of warnings I don't know if anyone compiled with GCC8 the kernel not many people but I realized maybe it would be easier to get those information in runtime so I create a black listing of those addresses so how it works you get your addresses for the function that you would like to exclude from the fuzzing put them to the list and IFL have Iocto operation where you can put addresses to the list this list in the kernel space is sorted so finding the entry is not the big overhead it's still in some operation which probably is better to do this statically but from the other side I wanted also to keep going with my research and then you have the filter which every time when you have a trace and you end up in the virtual memory you just do not include this to your to your coverage data which reduce the space a lot so coming back to the coverage benchmark so that's the coverage benchmark so we have six characters that our fuzzer needs to guess so it's called panic lottery so before a small reminder those are really approximate numbers you shouldn't quote them anywhere it's just the way how I was playing with the fuzzer but at the very beginning I run my process to just guess this input for two weeks and it was not able to it was not able to win the panic lottery after applying the filtering I was able to fix that in less than 24 hours and also I used the same seed input so it makes sure also that because sometimes the fuzzing can be tricky it depends what are your input data what are your starting conditions so I make sure that this is the fuzzer see the same thing so I run this on the virtual machine I get the snapshot and run the second one with the second one with the filtering enable if you do something similar in user space this is something that every how-to-do AFL fuzzing manual in internet is doing as a first kind of first slide you will be able to break this in approximately couple of hours so next thing I spoke about the coverage so in order to so next step will be to provide the wrapper so for my case I need to I wanted to file for example FFS file system so we need first of all to make sure that we set up our process for fuzzing and we will be executing the same and we will be executing the correct part of the kernel code so in this case we are focusing about the mount but mount came also with we need a device that can be mounted so you need to do the preparation you need to do the cleaning so the good way is to start with bash and then move to the C or you don't have to move always to the C you can do C++, Rust and anything that provides you good performance but the reminder here is performance is the key for the fuzzing so do not use the bash script as a mount as do not use the bash script as part of your fuzzing loop because that will slow down the process a lot and the fuzzing of the kernel is already slow because of the way how the communication with the kernel works so make sure that you get everything from your mount code and try to be as close to raw syscals because the performance is very crucial okay so let's then do the local setup so we have already our wrapper we'll create the file in this case that would be ATK of size vndconfig to create it as a blog device then create the new file system on it someone may ask why you don't just leave your fuzzer to guess the structure of the file system but you already give it as a hint that this is the structure of the file system you can also get just a zero and then run your fuzzer with just a zero but that will take some time for your fuzzer to guess and also I was working on this blacklisting so I was thinking it's probably a good idea it's probably a good idea to have your fuzzer works on already pre-part set of the data and then you can just run your fuzzer you can then just run your fuzzer with minus k switch this is also something new and you need to provide the mount script be careful with this because essentially you're just fuzzing the binary that you're running on so do not cut the branch that you're sitting off and every time I talk about this people ask me how many bugs did you find how many kernel crosshairs are you able to report so let's do something very simple to also show people that we are not only finding the kernel crosshairs so here I have my director with the fuzzer and let's run this script the same thing run the fuzzing 90 operations per second and something went wrong so this actually is not something that is wrong in our script but this is actually the bug inside the kernel so what is happening we are doing the mount I have a little bit of a small time but we are doing the mount on slash mnt1 and we can see so we have mnt1 inside the kernel but if we will try to do any operation now after mounting on corrupted value there is no file stat mnt1 it also doesn't work so during my fuzzing of the ffs file system I also find some kind of like a logical bug so this for example is not a simple crash that you may expect from the fuzzing of the kernel but this is also something that you may find there are some logical bugs which also will be nice to handle okay so I will probably skip the conclusions because 10 minutes some resources so we have netbsd blog it's a really good place to find your to find some information what we are doing inside the kernel for the QV there are also two posts that I provided about the fuzzing with AFL I would like then to move to another slide like that I got from Camille and okay so this came from Camille unfortunately was not able to join us today but to give also some other things that are done for QV as I said earlier we have MK Scientizer which is really important thing for quality in the kernel which essentially is your starting point if you are trying to do the fuzzing of the user space because if you will try to compile everything with different sanitizers it's almost impossible I was trying to compile FSDB and it's really hard so MK Scientizer is really important and current is covering about 95% of things that we have in user space there are still a lot of issues over there so you are more than welcome to help with that so MK Scientizer essentially provides you a way to use other sanitizers and compile everything with the same settings so you can have other sanitizers undefined behavior sanitizers shred sanitizers memory sanitizers, leak sanitizers additionally we also have LeapFuzzer Stuck Harder in Xray which is also coming in the future so I already said MK Scientizer already works and pass about 95% of upstream tests 100 issues still to be fixed so then we can also take a look about the fuzzing the kernel code inside the user space so this is the RAM kernel fuzzing here people also use the Hunk Hunk Fuzz together with RAM kernel so this is the way how you build MK Scientizer nothing special here you install the Hunk Fuzz Hunk Fuzz is a little bit more complicated project than something that it just shows you in the terminal so we need to spend a little bit more time to configure it properly you need to do the mount for different subsystems for the RAMP change the root then you also need for a RAMP kernel you also need kind of the wrapper because the RAMP kernel is also a user space library so as I use the wrapper for using for just fuzzing the kernel is something similar for the RAMP starting the Hunk Fuzz fuzzer with some prededicated corpus and then observe the crashes so 60 minutes after starting the fuzzing we got the kernel crash also for the mount so this is also fuzzing the mount but not inside your kernel but actually inside your RAMP which is running in your user space so you need to go through a lot of logs for example I found this a little bit tricky but it's easy to doable it can be improved a little bit but this was reported a couple of weeks ago so you have essentially out of bounds for your image and after we found your bug you fix the bug, you repeat and we found another bug and this is how we improved the quality so kernel signatures I already covered them so undefined behavior run on all ports kernel Kali is kind of our special sanitizers there are still a couple of things that are not merged yet for example I saw Maxim running threat sanitizers and already be able to catch first bug using it so the project that I show on one of my first slides SysColor so SysColor was run by tonight as a Google Summer of Code you can also read on our NetBSD blog about it so currently also was improved by Vladimir from Google and as I said he started finding some crashes and this is running 24 by 7 and I'm fuzzing over kernel another interesting project three-fourth AFL done by Anko as part of Google Summer Code so he was also able to find a lot of interesting bugs also using sanitizers most of them were already fixed but also we can read about them on our blog another interesting thing done by Maxim there was some USB device so currently Maxim came with some software device that is able to just inject the packets to the USB stack so when you have some idea to connect the fuzzer to this device and then start fuzzing the USB stack and what else improvements to virtualization so very neat action for everyone validate your kernel code with sanitizers so right now you have a lot of good tools in NetBSD we have a lot of progress with sanitizers so if you are doing the fuzzing do the fuzzing plus sanitizing there are still a lot of things that you need to improve in the future but slowly but surely we are progressing yeah so I already covered all of those so that's all that I have for today so do you have any time for the questions? who has the question stop being shy thank you guys