 allowing many tasks to be delegated to user space. Bruce Wayne has prepared a fantastic guided tour on what we can do with and to that user land. Please give a warm, fuzzy welcome to our next speaker, Bruce Wayne. Hello, I'm Bruce. Probably already you know me, but I will introduce myself. So I'm no longer a superhero. Now I'm working in IT because working as a superhero is pretty much, you know, it's a hard task and it's easier to work in IT. So by day, I am now a pen tester. By night, I'm trying to fix some open source projects, including the NetBSD. So how many NetBSD users do we have here? Any of you? Few, yeah. So for the guys on the stream, it's a full tent of NetBSD users. I want to also tell that it's my third camp, as you may suspect. It's super cool. I met here so many friendly people and I'd like to give an applause for organizers and volunteers and all people that let it happen. Thank you. So the NetBSD, so it's just like any other operating system, but the better one. The BSD, so you might think that B stands for Berkeley, but in fact it's Batman. It's a NetBSD's multi-architectural and it can run on top of many hardware, including the Atari or Amiga, one of the name servers of the NetBSD running on top of Amiga. And it's not just a kernel, it's a kernel and the user land. So we are rather a cathedral, not the bazaar. And I'm going to talk about RAMP kernels, which is implementation of the any kernel idea, because my opinion is pretty underrated NetBSD feature. And I will try to convince you that it's super cool. I will show you a few demos, even a crash, the great NetBSD for you by using RAMP kernel. So, sorry. So, have you seen Ilja von Spründerl talk about the BSD last year? It was at Defcon and then at CCC in December, I think. He was talking if the old BSDs are created equally, basically it was an audit of the free BSD kernel, of the NetBSD kernel and the open BSD. And in fact, he found like 60 bucks in NetBSD. For sure, we weren't happy about that. But one developer fixed all those 60 bucks overnight, so it's a pretty, pretty good result. But we are not happy about it, so we decided to do some quality improvements. And there are many projects ongoing to help us nile more bucks in NetBSD. Most of them, or some of them, rather, are connected with fuzzing. We have a few projects sponsored by Google Summer of Code, which includes AFL34s and fancy stuff like that, or CISColor. But today I'm going to show you how we can fuzz NetBSD kernel in the user space. Besides that, we have also things that led us to nile bucks in the kernel space by using address sanitizer on top of kernel or undefined behaviors, sanitizer, et cetera, et cetera. And in fact, we are now, well, the most sanitized BSD in the world. So many interesting projects are ongoing. But the question for you, what is the logo of the NetBSD? Orange flag rate. So let me also tell you some trivia fact about NetBSD. Do you know who is proof? Any ideas? So in fact, it is Julian Assange. He was the NetBSD developer like 20 years ago. And still his code is in the NetBSD. So if you are using SL Attach, I think it's, yes, SL Attach program, then you have the Julian's code. So the NetBSD logo looked like this a few years ago, but now it's like this. But we're going to pretend like the logo looks like this because it would be easier for me to let you know about the Ramp kernel. So let's pretend that this is the NetBSD kernel. We can call this demon like maybe Robin. So the Ramp stands for the runnable user meta programs. And I don't really know. I have no idea what it means. So I decided to dig into documentation and let you try to understand it. So I created this painting to let you know how it works. So basically we want to decompose the NetBSD kernel into pieces, which can be run on top of anything. And in our case, anything means that we are going to run it on top of Linux so we can fix the Linux world with our implementation. But in fact, the any kernel means that you can run these parts of the NetBSD wherever you want to. From the architecture point of view, it looks like this. So there is a little library called Ramp, which gives the primitives that led us to run the FS stuff or VFS or INET or whatever we want to. And there is some glue between Ramp and something. It doesn't matter right now what it is. But for example, if we want to run the NetBSD kernel on top of the user space, then we should implement the Ramp user to give the primitives from the user space so we can run Ramp on top of it. For now you can think about the whole thing that this is just a very light virtual machine. It's not necessarily true but less or more is like this. But in fact, I also told you that it can be run on top of anything and really it can be run in kernel space, user space, outer space, it really doesn't matter. So why this is cool? So you can run the TCP stack or file system or any other part of the kernel in the user space. I will show you why this is cool. Later I have a demo. And in fact you can also debug things in the user space which is pretty convenient. Also for a developer it's very convenient to run it. I can speed up my development process because I don't need to reboot my station, etc. Have you ever tried to debug kernel? Any of you? Is it easy task or rather not? You have to set up, yeah, it's very complicated stuff. Sometimes you need serial port and connect to machines. It's not that easy. I'm going to give you a demo that in NetBSD it's easy. And in fact you can also use the user LAN tools on top of the RAMP kernel so you can use GDB or Balgreen or whatever you want to. So the first demo is to show you the GDB debugger. Oops, let me scale it. So the resolution has changed. And I don't know if you see it. It's okay? Okay, so have you ever tried to use GDB? GDB has built in the debugger into the NetBSD kernel. You can get into by some magic key sequence. And now we are running in kernel space the debugger. But it's pretty simple, the debugger. I mean that many features are not here and using that it's pretty inconvenient. So I will run the RAMP kernel on top of Linux and show you how you can debug the NetBSD kernel in user space. So I executed the RAMP kernel here in the GDB and it is listening for the connection on port number 10,000 because the RAMP CTRL is just a tiny script. Less or more it's the same as a SSH connection to RAMP on server. And let's pretend that I want to debug the ICMP protocol. So let's set a breakpoint for a function ECMP input. Do you know what ECMP input does? Any ideas? No ideas. Yeah, less or more so. Let's set a breakpoint, continue a program and what do you think, how I can trigger that breakpoint? Sorry, you have to scream because I can't hear you. Any ideas? Pink, yeah. So we're going to send pink to the RAMP machine. So let's try to do it. Oh, but first of all I'm in the GDB right here so let's continue kernel here. Yes, so RAMP, pink. I'm not going to show you the details and talk about the details here because I want to just show you the idea so you can explore things by yourself. So let's send just one packet there. And as you can see the breakpoint was fired so we can in a very convenient way debug it just like any other user LAN program and we exploit this fact, sorry, we exploit this fact to write tests for the NetBSD also. I will show you how it looks like. So for example in this test case in user space we are checking if our network stack is working properly and we can set up in the user space program the whole network or the whole network configuration so it's a really powerful tool. And another demo is to... Oh, sorry, yeah. Another demo is to show you how to run the TCP-IP stack of the NetBSD on top of the Linux so you can show to the Internet only the NetBSD implementation which is running in user space and then we are going to run on another box HTTPD which is going to use that TCP-IP stack on top of Linux. Do you get it? Pretty much. So you can ask me why I want to do it. I have no idea but it's cool so let's try to set it up but first of all I need to delete my breakpoint so I don't break the demo so let's delete all breakpoints. Yeah. Where is my cursor? So first of all I need to configure TCP-IP stack on top of the Linux and the driver which lets me to do that is called VIRT which just creates a tune device on top of Linux so I can simply configure both RampKernel and Linux so let's do it. Yeah, so we're going to use Ramp if config VIRT0 create so we created the VIRT0 device and we have to assign an IP here so let's assign IP like this what's your favorite? Netmask pick any. 24. Whoops, sorry. I hope it won't break my demo but let's see so it should be configured. Let's see. Yes, it is so we should also set up the Linux part and there is a crazy program right now on Linux if config is no longer an option so I hope that you will help me because I'm not the Linux guy there is IP... let's see if the tune zero is here we have to set it up so IP, ADDR, ADD, ADD 002, 24, DEF, tune zero yeah, and I have to set the interface up and as far as I remember it's IP set... sorry, link, set tune zero up what was wrong with ify config? I have no idea but for some reason now coolkits use the IP so we have a connection I guess let's try to ping 001, yeah we have so now we created this part and now we have to run HDTPD which will proxy Cisco sockets Cisco from box one to box two so HDTPD can use the TCP IP stack of the RAMP server so let's try to do it so this is another box on the left side I have a box two on the right side I have a box one as you can see it's the netBSD and netBSD is shipped with the HDTPD that HDTPD is called Bozo HDTPD and it's a pretty simple demon IP exec HDTPD is B minus F minus E so I'm going to listen on the part 999, 9999 and which directory you want to serve to the internet through the netBSD stack which is running on top of Linux etc, yeah we well in it's pretty standard I believe but we should also do another thing if we execute the HDTPD like this it's going to use the native IP, TCP IP stack so we have to preload something which is called userlib ramp hijack so we simply hijack the socket stuff and proxy to the other TCP IP stack and I wanted to serve the ETC ah there is ETC so let's see if it works links HDTP 001 was the TCP IP stack which we configured port number 9999 which file from the ETC you want network file password file, password WD shadow won't work, ah it's going to work but there is no shadow in the BSD world what is the file called in the BSD world any ideas? master.yes, master.swd yeah so it works so as you can see it's pretty cool but yeah why we did that I have no idea but I will show you another how we can apply the ramp kernel another way so let's get back to the presentation well the cool thing is that if for example I am fighting with Joker so if he wants to exploit my TCP stack if it's running in user space I have another layer of security so maybe that's why we did that yeah and about fuzzing I guess that all of you already heard about it so I'm going to just present a story of the invention of fuzzing so Professor Barton Miller in 88 was working from the remote on his computer and it was dark and stormy night and the thunder storm caused that his commands were mutated so and he noticed that that mutations are making the programs to crash so he thought that maybe it's a good idea to use it and see if other programs behave like that so what would you do if you were Barton Miller he made an assignment for his students to test it and they were able to crash like 50% of the Unix tools of I mean the existing Unix tools and 30 years later the fuzzing went mainstream and there are many flavors of fuzzing one of them is dump fuzzing and if you ever tried fuzzing actually the dump fuzzing works and if you work for academia then probably you are trying to use some feedback driven fuzzing which uses the SMT stuff or not theory and things like that and I have a comic strip for you about that just back to me yeah probably you know that comic strip I don't know who is the author but it is so true dump fuzzing usually works pretty well so don't be scared that you are doing simple things because I will show you that it works so to test NetBSD we created something called foosrump which is just a fork of builtrump which lets you to cross compile the ramp for any POSIX compatible system and in our case any means that it works on top of Ubuntu and don't try any other platform because probably it's not going to work but if you want to port it then we will be more than happy we also realign the baseline from NetBSD 7 to NetBSD 9 which is going to be released when it's ready but actually it was branched so we used the current version foosrump and we have also an AFL support and I'm going to show you how it works and what problems we encountered so in case of allocators many subsystems in the kernel used a pattern where they were allocating a big chunk of memory and the problem is that if you want to use address sanitizer then address sanitizer is not aware what is happening inside of this big chunk so instead of allocating one big chunk we wrote a patches to allocate just the small ones so we can detect if something is happening between those chunks and we rewrote stuff like Kmem, pool, etc and also the problem is that ramp kernel if you compile the application which uses the ramp kernel in order to avoid the clashes between the name functions ramp renames every function of the kernel with the prefix rampNS and the problem is that address sanitizer is not able to see that rampNS memset is in fact memset so we created just a simple library to expose the memset instead of rampNS memset and using it is as easy as doing simple all the reloads so address sanitizer can be happy again and what to look for? the kernel is a little bit different than the user LAN program and I mean that if we have a leak in the kernel or something like that and it can be triggered from the user then probably by repeating one leak you can stop the whole kernel so this is pretty dangerous and in fact if you use the ramp you can also use the address sanitizer feature which is called leak sanitizer to detect those leaks so I'm going to show you the damp use case of ramp kernel so well I started this project like two or three years ago and it was my first approach to fast the kernel I created a very simple fuzzer which looks like this I'm going to show you the configuration file of it it is here so I just provided the prototypes of the syscalls that we have in the netpsd and it's pretty well easy because I just paste the prototype from the codes and that's it and I had thanks to Minerva Leap I had a fuzzer in five minutes and you can execute it like this let's try with ten iterations so it simply calls the random functions with the random arguments and in fact we were able to find a lot of bugs using it which is quite embarrassing because we are rather not supposed to have bugs like that in the kernel but if we can use address sanitizer then we nailed bugs which are not necessarily caused the panic of the kernel or things like that it was just harmless bugs but in fact they were bugs I have maybe here I have here the example report from the fuzzing session let's take this one so we found bugs like two years ago and it was fixed and then I had a two years break and I wanted to check if I can use AFL on top of the RAM kernel well it wasn't trivial task to port but we managed to compile RAM using AFL clunk and things like that and I'm going to show you how easy it is to to fuzz the NetBSD with AFL so if you want to do it you have to fuzz RAM clients, yes for now I will show you the FFS example which is just the file system so we have here simply we are mounting the file which is provided in the arguments of this client so we just wrote the 15 lines of code and we are able to run it in the AFL and find the bugs and I will show you that we in fact found some bugs but let me lock here well I'll show you the crash we found recently in X2 file system yeah, I know, thank you so in order to mount the crash EMG you have to do it this way so it's kind of a loop in the Linux world and then we are going to mount it X2, fuzz, fuzz, zero, MNT and what do you think, what is going to happen? crash, yeah so it crashed and it was found thanks to 15 lines of code in AFL it's division by zero bug in the X2FS but in fact if you want to exploit that you have to be a root so it's pretty harmless and that's why I show it publicly right now and we also decided that the FS staff is well I would say that we expected to find many bugs in file systems because if you mount unknown image then you are not responsible so we thought that fuzzing network could be more interesting and our first approach was to use just the raw sockets in drum kernel but we had a problem that we didn't know when the packet is handled by the kernel because there is, for example, if you send the IP IP packet to the netBSD it's handled by soft interrupt and we don't know when it's going to be fired up so we decided to, well it's another cool feature that from the point of view of the application you can call any kernel function which is cool but well we are abusing here our API a little bit so I'll show you how it looks like let's get back to the terminals we have here a program called net input net input is a program that reads a packet from the standard input and simply pushes it to the network stuff we call here function fuzzrumpIPinput and we implemented that function in our kernel I will show you the implementation SRC sys netinetIPinput fuzzrump so yeah, here we have implementation of the function that feeds the network stuff so it just puts the data into the Mbuff and pushes it to the IP input function we also had to pretend like we are the software input but it's harmless and what do you think by using this approach how many bugs we have found so far and you should ask me Batman but what about the checksums and stuff like that if you mutate packets then you're going to have invalid packets because of the checksums so I showed you that we are feeding the loopback device and the checksum are turned off there so we don't have to care about it and how many bugs we have found what do you think? 20, any other ideas? so we have found nothing nothing yet at least which is both pretty cool as a developer I'm really proud of it as a backhunter I'm pretty sad about it but it's not that our effort is meaningless because now we have a corpus of the packets that covers pretty much of our network stuff so we can test every change by running this corpus if anything is broken and in fact it wasn't a big surprise because the network stack is pretty well tested in the wild if you ever connected any bugs to the internet you know that a lot of very special packets are there everyone is trying to scan you etc etc so it's not a big surprise but at least we are not notes because we run the LBSD kernel in the user space we can compile it with the AFL and finally we want to cover more drivers especially the VIFI stack and the Bluetooth stack if you want to help us you would be more than happy to cooperate with you we can also try to integrate this project with the OSS fast by Google so we can have reports from them that we have a broken kernel or something like that so I know also that other operating systems like Linux have the same things as Ramp I don't know if it's working or not maybe you tell me for example in Linux there is a live OS have you ever tried to use it? not really but I think that it's not as advanced as Ramp stuff so maybe you can work on it and I also want to say big thank you especially to Michal who helped me a lot do you have any questions? great so if you have questions we have a microphone angel standing over there we have a microphone angel standing over there and we have perhaps some questions from the internet for Batman no questions for Batman the internet is very disappointing this evening but here we have a question thanks for a nice talk as I understand you started your project before Ciscoller started to support BSD the Ciscoller was implemented last year I think as a part of the Google Summer of Code I started my project in 2017 but my free time I'm a superhero so my free time is... well I don't have much free time but anyway it is really great thank you very much and if we compare your approach with I would say native fuzzing inside of virtual machine do you have any... I have better things currently there was a blog post on the NetBSD blog about fuzzing file systems using the IFL and cakeoff which is coverage from kernel there is a special device in the Dev3 so you can run Ciscoller and see which functions were triggered and the guy who wrote this blog I think he was able to run like 40 executions per second and using this approach we can do few hundred tests per second like in case of the network stuff we are able to inject like 10,000 packets per second and the problem right now is that it's my hobby project mostly I think that... well you can use the cakeoff stuff on top of the RAM kernel so getting the coverage is pretty easy you can run the IFL and see which branches are triggered or not and then you can put the test cases for that branches inside your corpus and run the IFL again but you know it takes time it's not necessarily exciting task so if you want to help us with it then you are more... well I'll be very happy about it and small additional question about Net so do you have some descriptions of protocols network protocols to have more efficient fuzzing or just leave it to IFL? well I built my corpus I took the test from the HongFas implemented pretty much same thing for Linux but it's not working in the user space but they have a big corpus of packets and I simply took it and tried how it works with the NetPSD last one and then we have to ask the internet if there's any internet questions so this is your last one as I understand you fuzz under root or super user yes well it doesn't matter if you run as root or not you can run as a normal user but the thing is that if you want to use super features like the VIRT interface and things like that you need a super user power and maybe I showed you the demos as a super user because I'm a super hero so that's it great any last thoughts from the internet? no last thoughts on the internet then a last thought from us here for Bruce Wayne let's hear it