 And this second one was what an older version of OpenSSL produced as a result. So you had a bug where this mathematical function in a very rare corner case produced a wrong result. And I found quite a couple of these kinds of bugs with this strategy. Yeah, this is the patch where this was introduced. At some assembly, I have no idea what it does. And now another tool that is very good to combine with fuzzing is Address Sanitizer. And this is really something I'm currently running around and trying to tell free software developers that they make use of this tool, because it's really, really simple and it can sometimes quite easily find bugs. So Address Sanitizer is a feature in the compiler. It's available both in GCC and C-Lang and it can be enabled with this flag minus F sanitize address. So if you only want to take one thing away from this talk, if you write software in C, then use this flag to test your software. And here's a little code example. So we're allocating a buffer and then just writing zeros into it and then we're freeing the buffer and then we're trying to read from the buffer. So it's a use after free bug, kind of standard use after free bug. You're freeing a buffer and then still trying to read from it. And this is, I mean, this is a very simple example, but this is usually the kind of bug you have in applications like browsers these days when you have security issues. Now, here I have this code and now if I compile it, okay, yeah. So what happens if I run this is essentially unpredictable. So in this case, I have zero and this will also usually happen because we're not doing anything between the freeing and the reading of it. But we cannot rely that the compiler will do anything predictable in this situation. Now, if we add address sanitizer to it and also at G, which just gives us some debugging information, then, and I have to do this. Then, okay, repeat. Then we get a very nice error message. It says, he views after free. It tells us in which line in the code it happened and it tells us, okay, this is in some memory region that was freed in this line and it was allocated in this line. So we get a lot of information that will help us to analyze this bug. And the thing is with use after free, most of the time these bugs don't crash your application. Also if you have out of bounds memory reads, they also usually just your application continues running and something strange may have happened. And so it's hard to find these bugs without such a tool. So address sanitizer is very helpful that it allows you to find these kinds of bugs. Yeah, this is again the error message. And it's maybe you know wall grind which does similar things, but wall grind is quite limited because it works on the already compiled executable and some classes of bugs cannot be found with that strategy. So it's a bit like address sanitizer is a more advanced version of wall grind and it's also much faster because it operates at compile time. So yeah, and what you ideally want to do is you're combining fuzzing with address sanitizer because then you will just find more bugs and find these subtle bugs that are otherwise very hard to find. You probably all remember this one, the Heartbleed Bug which was essentially an out of bounds read bug. You had some buffer and a length and if you gave a server a length that was longer than this buffer, it would just read that and send it back to the user. And I did a little experiment with that. I created a small tool that was doing a handshake with OpenSSL with itself and then added the possibility that you could just swap out one of the TLS packages with something you gave it on the command line. And by that, I was able to fast the handshake of OpenSSL and then using that with an older version that was vulnerable to Heartbleed after a couple of hours, it rediscovered the Heartbleed Bug. Now you can say, okay, it's easy to find a bug if you already know where it is. But I would argue that in this experiment I didn't use any information that was very specific to Heartbleed. It was just a handshake. It was just a normal fuzzing how I would have done it otherwise if, yeah. I took it six hours and another tool called Lipfuzzer which I will also talk about later a bit more. The developer of Lipfuzzer wrote me then an email and said he has replicated this experiment with Lipfuzzer and it only took five minutes. So, yeah. Then something else I tried was I asked, okay, if you have address sanitizer and it finds all these marks, can you compile a whole system with address sanitizer enabled? So usually you use it for testing for one application but if you compile a whole system, okay, that may find some more subtle bugs. And I created the Gen2 system where everything except some corp packages was compiled with address sanitizer. So, you have some problems if you want to compile G-Lip C with it because what essentially it is doing is it is intercepting the memory allocation functions of G-Lip C and if you use it with G-Lip C with it then you will get some weird dependency conflicts. And GCC also didn't work because it's compiling these base functions that it adds in executables. So, I had to exclude those and a few more core dependencies but the rest of the system everything was compiled with address sanitizer. It is slow and it takes a lot of memory but it is possible to run. Yeah, and what was to be expected is that just by doing this, I found a lot of bugs in a lot of packages and I have a list here. I made it a bit longer this morning the stuff I found recently. So, you see a lot of stuff that like Bash or Python or Syslog and also things where you say, okay, maybe this is security critical like the Pigeon OTR plugin there was a use after free just by using it. So, no fuzzing involved or anything just starting it and if you opened a certain dialogue it would crash with the use after free bug was fixed in the last version. Yeah. So, yeah, I think this was quite worth it. It led to a lot of bug fixes and yeah. There were some challenges with it. So, the one thing with address sanitizer is you can use an executable with address sanitizer with a normal library that's not using address sanitizer but if you do it the other way around it does not work. So, if you have a library like let's say you compile your system open SSL with address sanitizer then all the binaries that are not compiled with address sanitizer will stop working. So, if you're recompiling your system with address sanitizer you have to check in which order you do it that nothing breaks and you also cannot easily just exclude one package. If you want to exclude something you also have to exclude all the dependencies. Then there was some build system issue. One was with lib tool that when lib tool is compiling a library then it filters out all compiler flags that it doesn't know about and it didn't know about address sanitizer. So, that broke. It's now fixed in the git code of lib tool but they have not made a new release yet. And then, and if you have some custom memory allocation in an application that usually also doesn't work. Yeah. Then, warning in past talks I had about this. I said, okay maybe this is something you may want to use if you want to have a very secure system but this is probably not a good idea because address sanitizer was designed as a testing tool and not as a production tool and this introduces some problems and someone on the OSS security list showed a way where how you could use address sanitizer to become root if you have a suede executable compiled with address sanitizer. So, this introduces security issues. So, at least in the way it's currently working it is not usable practically as a production protection mechanism. So, it's a good testing tool but it's not something you want to run in the real world. Yeah, but yeah, it works. I have a server running with it. I can serve web pages and yeah, found a lot of bugs with it. I still have a large number of log files of bugs from that that I have not reported yet. I haven't gone through yet. I probably somehow have to crowdsource that because there are so many of them. Yeah. But if you are involved in any free software projects please just compile it with address sanitizer once and run the test suite and see if something pops up with that. Okay, some other tools that are worth mentioning. Libfuzzer, I already mentioned earlier is it works similar than American FuzzyLob but it works on the function level. So, it's not calling an application. It's calling a function. So, you're writing a wrapper function that takes a buffer and a length and then you feed that into the whatever function you want to test. The advantage of that is, first of all, it's targeting different things. Like it's a function in other software and it's much faster than AFL because AFL has to like fog a new process for every tested input and this just is running in one process so it's much faster. But it's a bit tricky to use because you have to be very careful that you're not, for example, that you're not leaking any memory inside your test function because then just the memory of the process will grow till it's too much because it's executing this function millions of times. And so, it's not as straightforward to use but it's also a very useful tool especially if you integrate that in the development process of a software. I think then it's a very powerful tool and it's part of LLVM. So, yeah. Then there's KZen which is like address sanitizer in the Linux kernel and this is just an option you can make menu config under debugging you can just enable that. And yeah, when I tried this the first time then I immediately got some warnings that my graphics driver was accessing some invalid memory. So yeah, it's probably not used as often as it should be. And there's now also a tool called syscaller which is adapting a strategy similar what libfuzzer and AFL are using with this coverage-based fuzzing to the kernel with syscall fuzzing. And I'm also aware that there are people trying to fuzz other kernel inputs with similar strategies. Yeah. So then there are other sanitizer features in the compiler. So one is Ubisun which detects undefined behavior. So maybe you know that in C there are some things like if you have signed integer overflow or if you have a shift left by a negative value. These things, the standard says that they are undefined. So you cannot rely that they do anything predictable. And Ubisun finds these things. A bit of a problem with that is that there are so many of these bugs and usually they don't do anything bad because most of the time the compiler will just do what you expect it to do, like an integer overflow. Just okay, if you have max plus one it gives admin that usually happens but you cannot rely on it. So yeah. And there's also a threat sanitizer and memory sanitizer but these are a bit more tricky to use. So the really nice thing with address sanitizer is that it's just you're enabled and it usually just works. And there's some work done to adapt AFL to networking because the standard AFL is just testing file inputs. So you have executable and give it some file and yeah. There's a patch for it to let it fast TCP inputs. And there's also another attempt where you're using a preload library that will take something from standard in and feed it into a TCP port. But these things are quite brittle and it's not a straightforward to use. So you have to play around with it a bit. And there's certainly room for improvement here. So. Yeah. And there's a port of American fuzzy log to Android. I have not tested this myself but yeah, if you know about Android security it's probably a good thing it is needed. They should fast test their stuff. And what's interesting is like I mentioned earlier that some of the stage fright bugs were found with AFL and what happened there is that the guy who found them ported a stage fright library to standard Linux so he could fuzz it. So yeah. Yeah. And at the end I want to ask you like who is involved in some kind of C development or C++ development? Please hands up. That's yeah, maybe a third. And are you already using fuzzing or are you using address sanitizer or and who is not using both fuzzing and address sanitizer from the people? Okay, so now you know what you have to do next. It will improve your software quality, definitely. Yeah. Thanks for listening. Yeah. And I'm, I think we have time for questions. Ah. You give pictures on this? Right, so my first question would be like what is your attitude to static code analysis maybe model checking? Static code analysis. Okay, it's actually I have not used it a whole lot and my impression was that it will usually not find the most interesting bugs but I don't have a lot of experience with it so maybe I'm wrong on that. No more questions? How did you find the Pigeon OTR plugin? Did you do you really click on the UI? Yes, I just I had the gentle system built with address sanitizer and then I started the Pigeon and started the chat session and so the bug happened when you tried to authenticate the other contact. And another interesting thing about that one is it was already reported six months earlier in their bug tracker but it was ignored. This also shows up that your fuzzing project or your initiative mainly targets file input systems or the attack surface is file input but there's a whole lot going on from user input or on embedded systems where you can't do this file stuff thingy. Yeah, so I mean the Pigeon bug was there was no fuzzing involved. It was just I mean I talked about these two tools AFL and address sanitizer and this was just address sanitizer. So this is just use this to compile your software and test it with it. But there was also but the developers from Libre office they did something where they emulated the clicks on their GUI by based on an input and use that for fuzzing which is also quite interesting approach. Yeah, I think the file input should be extended to all user input not only mouse but recognize text fields and fill them in because almost no user interface today uses only file input. Yeah, but it's just files is the easiest thing to do. Yes, it's the low hanging fruit for starters. What kind of information do you need to provide to AFL about the structure of files? Nothing, you give it an example file and the rest it does by itself. Okay. Okay, if there are no more questions then. Yeah, thanks for listening. Thank you.