 Let's have a big hand for Ilya Van Sprundel. Awesome, okay. Yeah, so as previously mentioned, my title is RLBSD Created Equally, a survey of BSD kernel vulnerabilities. Start off with who am I? My name is Ilya Van Sprundel. I work for a company called IOActive. I did spend the last five years in America, but I recently moved back to Belgium a couple of weeks ago. I'm a computer steering consultant. I do pen test, code review. I break stuff on a profit. I've been going to the CCC Congress since 19-C3, which has been a while, and I've spoken a couple of times before. I think this will be my seven or eight time. And let's see how this goes. Right, so my organization is sort of split into three, four-ish pieces. There's an intro, which will be the next slide, and then there's a little bit of data that I'll show, and then there's lack of data, so I want to test a couple of things, and once I've tested that, and I've got some results, I'll present my results and my conclusions. Right, so what does this talk about, BSD kernel vulnerabilities? It's a comparison between the different current BSD flavors. As for what I expect from the audience, if you, basically, I expect some basic Unix kernel-ish knowledge, not too deep, but some would be nice. People that I think might appreciate this talk are, you know, if you're a low-level security enthusiast, if you're a USD or BSD geek, and then if you're a Linux guy, you might like this too. And then generally, people that are curious about OS internal security, I think might enjoy this presentation. Before I move on, I have to mention, you know, some people that I call this standing on the shoulder giants because you don't do this by yourself. You build upon work by others, so there's a number of people that have done interesting BSD kernel security research over the last, I don't know, decade and a half, maybe two decades. Some of them might even be at the Congress. So, Silvio Noir, a bunch of others that did some really interesting work. So, really, this presentation started with a post to a mailing list years and years ago by Tio Dorat, who, if you don't know who he is, is the main guy behind OpenBSD. And I remember reading this 12 years ago, and it's been sort of stuck in my head since then. And it's basically a quote from Tio where he goes, you know, if the Linux people actually cared about quality, you know, as the OpenBSD people do, they would not have had as many local root security holes as they had last year, which is that year, which is 2005, and Tio goes, how many is it? Like 20 or so? And so, I think that was, you know, when he makes that sort of, oh, 20 or so, I was like, really, you know, is that true? What's the data behind that? Can we look at some of this? And so, I basically started looking at the last, almost 20 years of Linux kernel vulnerabilities and what the numbers are. And so, I went to CVDDels, and they basically keep a record of his stuff. And it goes back to 1999. And if you look at, you know, the first three or four years, it's, you know, in the teens, and then, starting 2004, 2005, it sort of jumps up, goes up into the hundreds. And then, this actually, this table is from July of this year. So, the 2017 numbers aren't correct. And so, in July 2017, it was 346 Linux kernel vulnerabilities, but as of today, because I checked earlier, it was 434 Linux kernel vulnerabilities. And so, as you see, these numbers are kind of sort of growing. Now, CVDDels doesn't have the same numbers for, like, BSD kernel issues. They have, like, general BSD numbers, but that's all encompassing the OS. And I wanted the specific BSD kernel numbers. And so, what I did is I sort of scraped the free BSD, open BSD, net BSD numbers, and sort of put them into a nice little table. And if you look at the tables here for the BSDs, you'll see that the first couple of years, they're kind of on par with Linux, but then the Linux ones kind of explode. But the BSD ones all sort of more or less stay the same, either single digits or low teens. It doesn't really go over 17, and that was only open BSD a year ago. So, if you look at these numbers, TIO's observation was pretty, he had a pretty astute observation. In fact, 20 was a low estimate. It turns out it was significantly more. And so, you see that numbers for BSD and Linux over the years are not the same, we're not on the same level. And so, what I asked is, are they on equal footing? Obviously, there's a really large community behind the Linux kernel. If you look at the BSDs, it's a small group of developers, a small group of users that are using these things and compiling themselves and so forth. And so, I wonder if that's the reason, or if there's a reason why the numbers aren't the same, or if it really is, as TIO said, if it's code quality. And obviously, one of the things I consider as well, maybe it's the mini eyeballs thing. And I know there's, mini eyeballs thing is, you know, very convoluted thing, but there's some truth to that. And so, I was wondering if there's some truth to it in this case. Well, and the only way to really do that is to test it, to go and see what the code is really like. And sure enough, there's a guy that did this 15 years ago, Silvio, and I did a blackout presentation about this in 2003, 14 years ago. And basically, he did an audit of the BSD and the Linux kernel and sort of drew some clues from that. And one of the things that he said was, he said, well, there really isn't much difference between the Linux kernel and the BSD kernels in terms of security auditing. They're equally-ish, broken. It's easy enough to find bugs in both of them and the same kind of bugs. Now, that was 15 years ago. Now, half things changed since then. The other thing he said was that he only spent a couple days in the BSD and then three months on Linux. And so, if you had spent longer on the BSDs, would you've gotten more bugs or would you sort of hit a limit, right? And then the third one is sort of the, if you look at his presentation, and this is 15 years ago, he mostly looked for, or mostly found trivillator draw flows and info leaks and nothing else. So, the only thing I had is, what if I expand the number of bugs I look for? What if you also look for, you know, race conditions and things like that? And so, the data's interesting, but it's kind of a little bit outdated and a bit too limited. At the time, I think it was perfect. For today's purposes, it's not quite good enough. And so, the only way to really figure out if those numbers line up or don't line up, what the reason behind it is, is not to dig in myself. And so, I spent April, May, June and July of this year looking at BSD kernel code. I don't know exactly how much time I spend on each one of them, but I would say they're more or less equal. And so, basically what I did is I asked myself, if I was going to look for bugs, where would the bugs be? Where do I think they would make mistakes? And so, I made a list of sort of attack surfaces that I wanted to go look at. And none of these are, I think, overly surprising. Obviously, the common stuff is like, yeah, obviously, I'm going to take a look at some SysCalls, and I'll take a look at the TCP-IP stack, and then, yeah, I'll take a look at some drivers, and I'll look at the compact code, and the trap handlers, and the file systems, and, you know, other networking stuff. And I actually spent three-ish months looking at that. And so, what I wanted to do is I wanted to jump over some of the things that in each of those areas that I found, and sort of discuss that briefly. And I have a couple of demos in here as well. And then once I've run through that, I'll sort of run you through my results and my conclusions. So, yeah, so let's dive into SysCalls. Obviously, this is prime attack surface. I don't think it should be any surprise to anybody that, you know, if you're looking for security bugs, the first thing you want to do is look at SysCalls, because that's the thing where user talks to kernel. And the first thing that you can see to observe is that among the three big BSDs, there's a difference in the amount of system calls that are implemented. Right? Free BSD has well over 500. Net BSD has almost 500. And open BSD has, you know, about 300 and change. So, right then and there, there's a clear difference between attack surface between three of them, right? But even sort of, regardless of attack surface, my assumption was given that these are so obvious that they're not well tested, they're less likely to contain security bugs. And so, you know, I go along and I start looking and I start looking at system calls and I start playing around with them. And sure enough, this is an OB system call and it's the census log. And you get to give a number of bytes and basically that gets passed on eventually to malloc. And the thing about ancient BSD code, no BSD has this if you pass an unbound length to malloc you get an instant kernel panic. And sure enough, that's what happens if you do syslog and give it a really long value, the thing blows up. And I actually have a demo of this. I don't know how this is going to work on the screens. Let's see if I can get this to work. Yeah, hold on. Yeah, okay, crap. It's not mirroring on my screen so it's kind of annoying to see this, hold on. Okay. This is my little syslog example. And sure enough, boom, that caused the kernel panic. Okay. So yeah, that's the syslog and that's the kernel panic. There you go. And so what's happening is basically, use the syslog, syscall and you pass an unbound value and cause a panic in malloc. This is not BSD 6.1. I haven't looked at the later ones. I know it's been fixed since, but it's been there so it'll be 6.0. This is a fairly recent bug. So, and here's a second example. This is a system call on FreebieZ called KLDstat which basically gives you statistics about loaded kernel drivers. And this thing basically has this stat buff from the stack fills it out and then sends it back to userland. But it doesn't fill out everything in so you get initialized bytes in there. This has been, I found it in FreebieZ 11 but it's been in there for just over a decade. It doesn't fully initialize a structure and basically it sends it back to userland. And then, hold on, let me see if I can show you a demo of this too. This is KLDleak and then and then basically all these bytes are initialized from memory that's basically flashing over the screen. So that's basically bug number 2 where you get to basically leak a whole bunch of undersized kernel memory. So basically, my previous assumption was wrong, right? When I said that I assumed that because there are system calls and they're so well tested that it's less likely to see bugs in them, well that turns out to be entirely wrong. You can find bugs in system calls in the BSDs within fairly quickly, fairly easily bugs that aren't too esoteric or too weird, fairly straightforward bugs especially newly added system calls but even the older stuff as you can see, a 10-year-old system call in FreeBSD still had a trivial infallig. So those things do occur. So now that we know there are bugs in system calls and they're fairly defined let's move on to the TCP stack. This one, again, my assumption is it's very well tested and less likely to have bugs because as we all know, the BSDTP stack has been around for ages since the 80s at least and it's incredibly well tested and highly unlikely to find bugs in there and for those who don't know what TCP stack is well, it has things like IPv4 and IPv6 in there and UDP, TCP, ICMP, IP stack, Ethernet and things like that. It's been around forever and so my assumption is unlikely to have bugs in there. So I started looking around and I found FreeBSD stack which basically parses PPOE packets and then it's a little bit convoluted, I guess. But basically the idea is that you take it back and you start parsing it and then if it contains the exact right tag and the right error message basically it basically does this thing where in the BSDs the way this BSD stack works they have this thing called Mbuffs and it's basically, it's buffer management around data that you send in and out and so there's a whole bunch of APIs around it and the idea is that if you don't use the APIs correctly, things go really wrong and so in this case you see this thing called M pull down and basically if M pull down fails it will free the Mbuff you passed it and in this case you'll see if Mbuff fails there's error code where basically they go to done they basically free the Mbuff except if the thing failed the Mbuff's already freed so this is, even though the code's a little convoluted to read, it's a relatively easy bug to trigger it affects all BSDs 6.1 but it has been there since 2004 so it's been around for over a decade it turns out when I was looking at this and I found the bug, I checked, I'll check NetBSD and it turns out that NetBSD guys have the same bug but they fixed it but they never released any kind of advisory for it and so clearly they never told you it'll be easy guys about it and so it'll be easy guys to only fix it when I emailed them about it and so this bug was I mean if you know what you're looking for this bug was relatively easy to find and so again my previous assumption was wrong bugs in the gchip stack do occur with some frequency in particular newer code but also it turns out Mbuffs are hard very complicated and it's error prone the APIs aren't overly consistent about half of them sort of when they fail they free the Mbuff the other half when they fail they don't free the Mbuff things like that where it turns out if you have to use Mbuffs it's not quite that easy you have to really know what you're doing and there's a number of cases that I found where that doesn't happen and I'll sort of circle back to Mbuff misuse later on when I talk about the wifi stack anyway, this wifi stack turns out has bugs so moving on I looked at some of the drivers obviously because it turns out that there's a whole lot of drivers among the BSEs for all sorts of things really and given that Unix has the thing where everything is a file the way you find your drivers is you go to dev and then in dev it will have some kind of name and then you can do file, you open them and then you can do file operations on that thing so open, read, write, close and so forth but obviously iOctl is the one that you want to look at because that's generally where most of the interesting attack servers will be so this is this is the dev crypto on NetBSD which has this thing where they basically have this very, very trivial intro flow and then here you get memory corruption and it's not this is just one snippet but if you look at the actual driver it's like iOctl command bug iOctl command same bug 7 or 8 times the same trivial intro flow shows up and then sure enough you trigger it so the one I showed was this iOctl but there's 7 or 8 others and then again let me see if I can demo this yeah I think it's still running it'll leak your memory till the end of time it won't actually crash the memory is guaranteed to be there and it's just initialized so this is NetBSD and then crypto overflow and then boom sure the moment I hit this you get memory corruption yeah so basically this is a class control flow leading to memory corruption this is definitely exploitable and then this is a similar driver one this is FreeBSD this is a driver called KSIM which basically gives you access it shows you kernel symbols for FreeBSD it's an optional driver you can load and they have this interesting thing where you know they do they implement opens in column and they do open they basically in the the driver files driver filescrupters specific data they basically take a copy of a pointer that's very specific to your process and then when you do nmap they find that pointer and they basically use it there is an interesting issue here with expired pointers where if I open the device and then I pass that filescrupter along to a different process and I kill the original process and then that other process does an nmap it's basically using that original pointer to the original process but the original process is dead so you have this wild pointer that's being used this is essentially an expired pointer and this is very problematic it turns out because you basically you're using a wild pointer that can point anywhere to read and write to and I wanted to I can't really download this in live because my export kind of sucks and it takes 3 or 4 days to actually hit the bug I could probably optimize it I just haven't and so you'll have to do with a screenshot but this is essentially what happens if you hit the bug and then sure enough if you look at this here uptime 9 days 1 hour so it took 9 days to actually hit the bug but I did hit it alright so that's it for the driver stuff and there are many many many more driver examples but I figured 2 is enough so compact code is a thing in the BSDs where basically they allow the OS to sort of emulate a different OS so if you have a binary that you can file for Linux and you run it on a free BSD and you turn on a free BSD Linux compatible emulation layer and you make sure that you have the right Shirt libraries and stuff installed then your Linux binary will run perfectly or more or less perfectly on your free BSD or net BSD and they have these OS behaviors for different operating systems and so especially net BSD has quite a few of them free BSD has less of them and then open BSD pretty much killed most of them but it's essentially an air freighter different OSes or older versions of the OS so you have 64 bit and you're supporting 32 bit versions of the OS all that stuff goes through the compact layer and basically that stuff basically has to emulate a whole bunch of system calls and I found a really nice quote by T.O. about compact layers where he says well you know the people that rely on the compact layers don't care enough to maintain it and the people who work on main systems don't care about compact layers because they don't use them and the colleges aren't aligned in the same direction and basically says compact layers rot very quickly and then it's absolutely true and so an example of this is this is SVR4 compact code on net BSD SVR4 is ancient nobody has used that thing probably in 20 years but it's there and they support it and this is the thing called streams which is it's kind of like sockets but not and streams have been dead for a long time but net BSD still supports it in SVR4 compact and they have this thing where basically this function it gets called with this BND pointer and basically the content of which is defined by user land and what they do is they basically grab a value out of it uses an offset and then basically sort of do reference thing and read data out of it so this is basically arbitrary you know read anything from memory out of bound read that obviously can cause a kernel crash or potentially in full league but the really interesting thing about this code is if you scroll all the way up and you look at the comments it says yes this is gross so they know their code sucks this code has been around since 1996 so it's been there for 20 years I mean there may be people in the room that are younger than this bug but yeah this is really crappy code and this is what TMN when it said combat code rots really quickly they wrote it 21 years ago they never look back it's there probably hasn't been used since 1996 too and then so when I told them about it and they fixed it the log message basically said well we fixed the multitude of holes in SVR4 stream code and then they said we should have never enabled this by default and then it's a minefield so what I had gotten to do is when they fixed this thing they also changed their kernel config and they turned this shit off by default which you know right so the next thing I looked at are trap handlers and the thing about trap handlers is that trap handlers basically do this thing where they handle any kind of sort of exception or fault which can be division by zero or system call or peak point or in valve memory access or a very very long list of things and this is essentially mostly this is sort of hardware sort of this in saying hey this kind of fault happened then you have to deal with it some can be triggered by user land some can only be triggered by hardware some can only be triggered by kernel the code to handle this and deal with this is usually incredibly incredibly nasty it's terrifying to just read this code I can't imagine writing it and it's very architecture specific like the Intel ones are different from the R which is very different from MIPS and so on and even amongst the Intel earlier to later there were many changes in between and so I didn't really feel like I like my sanity and so I was like okay how about I just fuzz it and then it comes out to okay well how do you fuzz exceptions and I'm like well I don't really know how to fuzz specific exceptions but what if I just execute random instructions surely those things will hit some kind of exception and when I say random instructions I mean like super random like I don't even know what the instructions are basically what I do is I read from that few random and then in that page I make a read while executable fork off a process basically set a function for it to it let that process do its thing it will die usually after section 2 or 3 the thing is dead but you just keep doing that in a loop in a loop in a loop and it generates all sorts of weird traps that the kernel now has to handle and sure enough if you do this on on free BSD you hit a couple of bugs there's a a Zen builder even though I wasn't using Zen there's some send signal bug there's all sorts of weird things that happen so this is one of them this is not a one of them and I could but I don't the thing is because it's so random you never know when it hits but essentially this is all the code it takes I mean I have more advanced versions of this but this is really all it takes to cause it so if you write something like this and you run it on the BSDs and you know it will hit kernel bugs anyway that's it for trap handlers and I'm happy with that because I don't want to talk more about trap handlers so yeah the next thing I was looking at are file systems and so the texture of file systems obviously the easy part of the attack surface is sort of the oh well you mount the USB stick or whatever and that's the part of the file system and that's true I believe in my view that's attack surface but it turns out in recent years there's significantly new attack surface to file systems and this comes in the form of Fuse I don't know if you guys know what Fuse is it's essentially userline file systems and so what that means is all of a sudden all these Unix VFS layers that have been around for many many years that in previous years only took data from trusted drivers in kernel where data structures were you know more or less trusted like the data in there you figured was more or less accurate because it was given to you by kernel all of a sudden all that data is handed to the VFS layer by a userline process and the assumption that the data in there is trusted is no longer valid and so if you start looking at the Fuse implementations for the BSDs the first thing I thought was like oh well it's Fuse it's BSD sure it's one implementation and they all do the same thing well turns out that isn't true turns out all three BSDs wrote their entirely different Fuse implementations there is no code share at 3.0.3 of them they're all entirely different my understanding is my view from looking at the codes that the net BSD one is the most complete in terms of features the free BSD one seems to be the one where the arguments are the most controlled and constrained in terms of the amount of validation being done and then the open BSD one is basically has the most minimal amount of implementation or features implemented compared to the other two if you look at Fuse it actually supports Ioctl but all three implementations of these do not implement Ioctl but they implement pretty much any other FASM operation read write you know read get attributes and attributes that type of stuff and so for example this is the open BSD get CWD system call it gets called by system call and basically they'll do this thing where they go to the VFS layer and say get attribute and then they fill out this VATR structure and they assume everything in the VATR structure is more or less sane because it came from a kernel driver but given that you now use Fuse all data in VA is no longer trusted it should be considered tainted and you should validate it and of course they don't and so they take this massive length value from it and it passes the malloc and as I mentioned before the old museum malloc when it sees a very large value goes kernel panic so this right here was a bug and then if you look a little bit further in get CWD they'll do another call to the VFS layer this flop read tier and basically they get this Ioctl structure back and previously when you did that kind of read tier again it was files some data but it was handed back from an in kernel class and driver so the structure was more or less trusted and in this case it's really just a buffer and in a length and so the content because it now comes from a use line process is no longer trusted and so if you look at what it starts parsing the name length of an entry in a directory it sort of just assumes that it's more or less valid and then it uses that for a mem move and that can cause an out of boundary so basically the problem with Fuse and I would imagine you see this in Linux 2 but I haven't looked yet is that if you do if you want something like Fuse you have to modify your VFS layer because your VFS layer has all these assumptions where it assumes the data it gets from your file system is valid and it's no longer true so that was a sample of what it was but it had to be as similar bugs this particular bug the bug was there since 2006 but I think Fuse is not quite at all and so this has only recently been a real issue and then obviously if you look at the actual file system handler would you give it a file a blob of data that you mount so this is the X2 parser for freeBSD and this is very brief look at this I did not do an exhaustive search but I was like you know grep for this thing called b-read and I'm like oh well you're looking for some kind of string and if the string isn't there you just panic well that's pretty bad basically I can just give you a list of X2 partitions and this thing will cause a panic so obviously the file system parsers in the BSDs are not what they should be I suspect if you throw a simple file system fuzzer against these things it will blow up in all sorts of bizarre and weird ways I literally just spent 5 minutes on this and I was like this is very very broken because clearly they're assuming file systems again this bug's been there for I don't know 7 or 8 years or so but I suspect when you start hammering on the file systems you'll find more bugs right so yeah networking beyond this VIP stack and I sort of mentioned BT Wi-Fi and ERDA but I really only have Wi-Fi in my slides the other two I kind of threw out because I didn't get to it but when you look at the Wi-Fi attack surface there are sort of really two things you have to look at right one is the stack itself and then one are the actual Wi-Fi drivers right and ideally you want to have this thing where you have only one stack and then all the drivers sort of call into it which is what the BSDs did and Linux has this too but in early days of Wi-Fi every driver came in so full stack but luckily we're sort of past that and there's one stack for all the drivers so yeah the stack it looks like the DCIP stack I mean it's obviously parses different protocols but it's all in buffs it's in and out it's stuff that gets passed back and forth the main input function is IEEE 802.11 underscore input and once you start reading that you sort of find the entire 802.11 parsing stack and that function is called from all the Wi-Fi drivers and so sure enough we start reading that eventually you find this function called IEEE 802.11 underscore EPUL underscore key underscore input and this basically deals with EPUL keys and it turns out if you look at the structure for this thing it actually has two different lengths one is length and one is payload length and this this payload function does validate length and payload length but then it has to do this thing called an Mbuff pull up that is to say it has to actually pull up enough continuous buffers to make sure there's that much length and it does this for the payload length or the actual the other length field and so turns out that the actual length field is used to actually read beyond the thing to read the buffer and so because the pull up isn't done for the length but for the payload length instead if your length is bigger than your payload length then you can end up having out of bound reads and this can be triggered by remote Wi-Fi frames so again this comes back to Mbuff mishandling as I said previously in the TCPIP stack part mbuffs are hard this was in OpenBSD 6.1 when I was on it again and it turns out the bug has been around for about nine years the drivers the Wi-Fi drivers this is the interesting part the Wi-Fi drivers are either so Wi-Fi drivers in BSDs are either PCI or USB PCI is when you put the card and then it basically has DMA access but USB is different, USB is packet based and so it comes out in a question where it's like do you trust your Wi-Fi radio what if somebody compromises the Wi-Fi radio and then from there on tries to own your OS and so for the PCI when I was like okay well if your PCI it's kind of game over because they can do DMA and that's not longer true when you use things like IOMMU but let's just assume that if it's PCI and you handle it wrong okay fine but when it comes down to the USB drivers for 8-211 due to protocol is essentially packet based and so your USB host really should be able to parse the packets correctly and not blow up if you get bizarre USB packets from your Wi-Fi radio turns out they're currently not really doing that and this leads to very very trivial heap smashes and I have I wanted to have one example but I really gave you 5 examples of like the same thing so this is one driver where they go like okay there's a length field that can be up to a page long and then okay we'll go do we'll create an Mbuff cluster which can be 2000 bytes long and then we'll just do a mem copy with that length into that cluster without ever validating if that length is bigger than a cluster and obviously that means you get trivial memory corruption and then this is a different Wi-Fi driver and they say okay let's grab this length field out of the USB packet and then we'll basically use that and copy into an Mbuff that can be up to 2000 bytes and again very similar heap corruption bug and then this is another driver and it's a very similar thing with a mem copy with the length and then this is another driver very similar thing where they do they take a link and they do an Mbuff heap corruption bug and there's more of these but those are the ones that were very trivial so this is basically wide open attack surface across all BSDs across various different I mean there's an atmel, there's a real attack, there's a whole bunch of these Wi-Fi drivers this code is very trusting of the USB packet it just assumes everything is unright I think when these drives were recommended nobody thought about the attack surface on this one and then so that's more or less it for the attack surface I had there's a few more things I looked at and I kind of want to talk about like I didn't really know how much detail about it but so two things that I showed up that I sort of had that seemed like there were many bugs in there so one is there appeared to be a certain amount of easy detectable null DRFs in the BSDs I spent some time looking at it in free BSD I did a very quick rep on that BSD but I suspect there's a similar amount on that BSD so the way you call malloc on the BSDs basically is you pass it a flag and the flag basically says no weight or so I think the default behavior is that malloc always waits and when it succeeds when it returns it always succeeds it never fails but there's a way you can pass the flag which is no weight or can fail and what that means is if it can't fulfill the request within a certain amount of time it returns and it fails the request and so what you can do is you basically grab for these flags and you see anytime where there's a malloc done and return values in check then you get a null DRF and the reason these things show up is because by default the pattern is malloc cannot check return value and you would think okay well that's an easy pattern if you write in kernel code you would never make that kind of mistake except because the general way malloc is called is where it can never fail and so there's almost never a malloc return value check except you have to do this when you're doing no weight or can't fail and it turns out there is turns out there are quite a few cases where when you do malloc no weight or can fail people don't check return value and so I think these are pretty greppable and static analysis wise and I don't know why people haven't done this yet but there's quite a few I think I briefed so when I did grep on that because I think I had like 15 or 20 I can't remember how long the list was on the free BSD another thing that was interesting to see is that the BSDs have you know this direct rendering stuff in kernel right direct rendering direct rendering infrastructure for those who don't know these are basically the graphics drivers that are in kernel and so when X11 runs it sort of talks to these graphics drivers in kernel and that whole thing around it is called DRM or DRI and so this sort of because this sort of came from the open desktop people and it was initially developed separate and then a couple of years ago it was moved into the Linux kernel and then what to be as easy as they kind of forked it but it's essentially more or less the same code base and so the BSD guys kind of have to have it otherwise X doesn't really do much anymore these days and so it's interesting when you see the struggles they have with this and if you look at the main OpenBSD developers responsible for maintaining DRM DRI in OpenBSD he said well he goes all this loose code that we are importing it's not going to be reviewed by any OpenBSD curl developers because they refuse to read any code that is not conformed to the BSD KNF standard and so you have all of these rigorous code review standard practices on OpenBSD except all this DRM DRI stuff they will not touch it and so all the banners you see there in Linux kernel all that stuff is pretty much in OpenBSD and that BSD and free BSD as well okay so that was sort of it for running through the sort of the types of bugs I was looking for and the things I found I mean I could have spent I could have done another 2-3 hours running through a bunch of bugs but you know it would get pretty boring so what was my result after I was done about 3-ish months I had about 115 bugs in total 30 bugs in free BSD about 25 bugs in OpenBSD and the line share was in that BSD so about 60 or so in that BSD it was a very wide spectrum of bugs pretty much everything under the sun that you can expect straight up heapsack smashes race conditions expired pointers, double freeze integer issues, underflows, overflow, assignments bugs, logic bugs I had a typo somewhere where the wrong variable was used but it turns out that one existed too and then you had these weird things where the wrong thing was happening on the wrong structure because there was a typo division by zero I had some logic bugs in there I mean pretty much anything you can imagine is in there it turns out kernel code is not written by gods and they too make mistakes and they make plenty of mistakes actually so I found basically bugs amongst all 3 BSDs amongst all of the attachments I mentioned within that entire spectrum but it was interesting once I sort of got a grasp for all 3 of them and what was where and how it was done it's interesting that I think I can make some observations about the co-equality and just looking at the bugs just the numbers of the bugs you can see the same thing and so there's sort of a I think OpenBSD when it comes to code quality in their kernel is the clear winner and it comes the quote that I had from the original where it says oh it's code quality and that's part of it it became obvious to me it's code quality but also a tax reproduction a combination of those two that seems to be a winning formula and OpenBSD there is enormous tax reproduction if you compare it with the other BSDs there are many things they do not have they don't have a lot of kernel modules they have relatively few devices they have virtually no combat code they removed their latest combat code a couple of years ago they removed their entire Bluetooth stack because they thought the code sucked and they just deleted it so no Bluetooth support on OpenBSD they have significantly less system calls they have more than 200 system calls less than free BSD and they got support for a whole bunch of old architectures and that in combination with code quality I think is a winning formula the code quality is interesting because it really does show if you look at the code the trivial low hanging fruit stuff is almost entirely gone OpenBSD integer overflows, signage bugs are virtually gone in OpenBSD the Wi-Fi driver stuff is something where I still found it because it was a tax service they never thought about but anything that they know is a tax service it's highly unlikely you find integer overflows there or signage bugs simply because they know about them they inform their developers about what that looks like and they have this every commit gets code reviewed by at least one or two people they know exactly what these patterns look like and it turns out when you have a process like that integer overflows simply don't occur in a tax service the other thing is that they had fewer in full leaks than the other BSDs in terms of the clear loser, yeah that was OpenBSD, I think you can see that in these bugs, I mean they have 60 the other two combined is less it wasn't at BSD tons of legacy code, tons of compact code it turns out when you have legacy and compact code and you haven't looked at it since, yeah, you're going to have bugs they have these ISO protocols in there which I don't know if anybody actually uses those these days it's really ancient code from the 80s which is written by IBM and then it was important at BSD and it's been there ever since and nobody knows what it does, but it's there and a whole bunch of this really really old code the other thing is that their code seems to be less consistent with security code quality compared to the other BSDs unlike OpenBSD, they have tons and tons and tons of issues and signage bugs they're quite, I mean just that in one article I showed that, you know, eight in a row flows right there so there is definitely that sort of code quality difference and I don't mean that as a disc I mean I understand that, I mean building, maintaining, improving OS is really really hard and if you think it's easy, you know, try it so I understand that it's very hard to do an OS, but there is really a difference between if you look at one and you look at the other, you can see there's a difference in code quality when you look at the attack surface and then FreeBSD is somewhere in between really it's hard to place it but it's not the code quality isn't as bad as NetBSD but it's not as good as OpenBSD so obviously when I found these bugs I talked to the teams and I sent them emails and I said, hey, here's a list of bugs I have you guys should probably go and fix this so I emailed the OpenBSD guys and you know, Tio gets back to me I know about a week or so later and the first thing he does he says, oh, I'm sorry I took me a week to get back to you, I was on vacation and he says, oh, these bugs look good this is definitely problematic we should go and fix these things and then he emails me back and says the next two or three days you should see fixes sort of coming in our CVS and sure enough in terms of bugs I reported less than a week and they fixed all of them right? yeah, I think that's good and then a couple weeks later the OBSD guys basically made individual patches and firefighters and said, okay, if you have an openBSD and you want to fix these bugs, here's the patches here's how to do it so I think that was great, I think that's perfect I think that's the way it's supposed to be sort of the, you know, within a week response, a few days later things are fixes they are fixed in place and then like a week or two after that all the patches are probably available there's advisories this is exactly the way a security process should work OBSD guys have done everything right as far as I'm concerned so FreeBSD, how did that go? well, it sort of started off similarly where I got a response within a couple of days, maybe a week and I got a bug back from these guys and said, okay, well we've seen all your bugs and we filed them in our internal bug database and this was July 14th and sure enough, this is from the email and I've blacked out what hasn't been fixed yet or what I'm not quite sure of there's three narrative that have been fixed, right so since that email not all that much has happened it's five months later two advisories have been released and so obviously those two bugs have been fixed and then there's the third one, this ARC-E EDCAP I saw CVS commit for that, so I know that was fixed all the others it's up in the air, I don't think they're fixed I quickly checked CVS right before I came here they look like they're still in limbo somewhere so you know, this is where we're at FreeBSD okay NetBSD, first of all when I emailed them these bugs a list of 60 bugs this shit got fixed overnight seriously seriously this is obviously unbelievably impressive right, I don't know how you can do that but clearly there were two developers that were like let's fix this shit, let's fix it right now so that was that was very impressive and as I mentioned it also turned off the entire SRR4 compact subsystem and if you look at the commit message they said oh no, the email they sent me said basically and this is mirrored in the commit message they said something that should have been done a long time ago so that is incredibly impressive that was late July 2017 five months ago let me tell you what's happened since absolutely nothing bugs are fixed in CVS there are no patches for current there are no advisories what this effectively means is if you're running NetBSD today all 60 bugs are there and you can go to their CVS and see exactly what the bug is so I wish they had followed up with some patches and advisories so that kind of it started off really really really well and then they kind of dropped the ball so coming back to my sort of the earlier where I began my presentation sort of the okay, are things in equal footing well, I think bugs are still very easy to find in the BSE kernels probably about as easy as in the Lewis Kernel even OpenBSD wasn't I mean, there were certain things I didn't see in OpenBSD but it wasn't like it was hard to find bugs in there there's very, very level of quality between the two you know, three of them it depends on the age and who wrote it and under what circumstances the most consistent set of quality I found by far was OpenBSD, again this comes back to their rigorous review process every CVS check-in gets code reviewed I think that's a process that simply just works with the exception of the DRI DRM stuff, the OpenBSD guys have the same shitty code everybody else has and apparently OpenBSD developers refuse to touch it another thing I have sort of that I think should happen because I found I had a couple of bugs where I was like okay this bug's not OpenBSD but it was fixed in that BSE or in the other thing around where I'm like oh this isn't that BSE but the OpenBSD guy fixed it 15 years ago, right so those things from time to time did happen with some frequency and so ideally I think the maintainers of the BSE should talk more amongst each other now I understand that's obviously easier said than done because in the last 50 or 20 years they have diverged and there's different philosophies and ideas on where the BSE is supposed to go and obviously you know, there's big egos involved as well and so getting these guys to talk isn't always very easy and it doesn't always make sense but by and large I think there's enough commonality still between all the BSEs that it wouldn't make sense that when it comes to things that are attack surface and fixes it would probably make sense if these guys talked more with each other the other thing is that obviously when it comes to thing you know, if you look at codebase alone that tells you something about the attack surface OpenBSD's kernel is about 2.8 million lines of code, NetBSD is about 7.3 million and NetBSD's this was 11.0 I don't know what the latest one is the FreeBSD was about 9 million so that alone tells you that OpenBSD can have less bugs because they have less code it's that easy so obviously this plays a part you can't have a bug in code that you don't have and the other thing is obviously sort of accidental versus planned where if I haven't if I haven't implemented something yet then you can't have it yet but the other thing is obviously there were choices to make to delete code on purpose and this is something you saw in OpenBSD where they chose to delete their Bluetooth stack they chose to delete their Linux Compat layer and obviously it's a double-edged sword right you lose functionality but you gain security and try to find that balance but obviously cutting code it cuts attack surface and generally means you have less bugs right so yeah more conclusions basically going back to what I usually think I had which was the mini eyeballs thing yeah I think that's a factor I think it really does matter right I think if you have more people looking at something more bugs are going to be found and so I think one of the big reasons why the numbers are off when you compare those tables initially I think a large part of it is more people are looking at a Linux kernel and so they're going to find more bugs and they probably can't explain everything and you can say what you want about the people who write Linux kernel but there are just orders of magnitude more people looking at that code they're just going to find more bugs and it shows in numbers and yeah that's pretty much it that was frightfully awesome okay I'm quite convinced people want to ask a bunch of questions we got about 9 minutes let's start with you over there one sentence one question mark thank you for your awesome work and nice presentation are you interested in exploiting so you had a crush crushes and how about making a proof of concept local privilege escalation or RCE remote code execution given that my plan was to report all the bugs I saw really no point in writing foolable exploits because it's wasted effort to me the only way if I'm going to write an exploit I don't want to write an exploit for a bug I know it's going to get killed so I didn't see some of these bugs would have taken me weeks or months to sit down and write code for I don't think it would have been very useful for me to write foolable exploits I know there's a shock and all factor to exploits and at times they can be useful given my understanding or my assumption that the people on the other end I was talking to people like DO that are very knowledgeable in this area I didn't feel there was any need to write an exploit and so I didn't does that answer your question showing exploits usually helps to convince people in self-protection technologies like we have jar security and self-protection mitigation is obviously not a good thing we should keep innovating and scoring new mitigations it's just that I don't think this would have contributed much to it there's a question from the internet please listen to the dark side yes and thank you the dark side how would you suggest to improve cooperating between the different bsd's that's a good question I don't know I know Tio said a few things about it's not easy to talk to other people and so I don't know how to get these guys in a room and get them to talk or something obviously there are some differences between the two there are certain things where there are differences they don't necessarily have to talk to each other but I don't really know of any specific way to try to massage them to get them to talk to each other all I can say is find the right maintainers for the right subsystems and if some guy fixes a bug it's something where they go this is a tax surface reach out to the guy on the other bsd's drop an email and be like hey guys maybe you should look at this too beyond that I have no good answer really okay thank you is there another question from the internet not yet not yet how about one last question we got five more minutes anybody at the microphone who I'm not really seeing anything out there okay over there on the left excuse me my question was about methodology did you use any automated tool or did you do everything by hand where is this coming from on the left side I didn't see you either sorry in the back yeah well I didn't 9% of this thing was straight up reading code I opened up an IDE and I was just reading it there were a couple of times when I used some grep to look for some patterns but beyond that no it was just me reading code that was pretty much it there were no other tools involved thanks I think we'll take one more sure yeah one more here didn't you want to that was a big chance could you describe your motivation of spending three months on that work yeah I can I know I think I once heard it described being you know more boring train dry I disagree with that I enjoy reading code I think it's fun I think there's a number of things you can learn from reading code and you get this sort of interesting kick from it if I find a bug sometimes you use the code and you're like oh they screwed yourself I noticed this and they didn't so there's a little bit of that but generally enjoy it and I know some people don't but for me it wasn't a debt march it wasn't a stretch I had no problem and when I say three or four months I mean on and off because obviously I've worked on things so it's like evenings and weekends but it wasn't hard it wasn't difficult I was just like okay let's open up my laptop and I'll spend the next three or four hours looking at the system call or something I didn't think it was hard I understand it's not for everyone I tend to enjoy code review thanks again for your work awesome okay thank you very much am I overseeing a question somewhere no now's your chance the internet yes there's one final question from the internet while the bsd's may not be cooperating is there at least a common mission statement or any high level thing they could at least agree on this is a non-security bsd's question I have no idea I think there are people here that are probably a much better place than I to answer these kind of bsd questions yeah I have no idea okay if that's it thank you let's have another final hand