 One, two, three, does it work? Hi, everyone. Thanks for coming. I'm Max from Google Chrome Security Team, and I'm mostly working on fuzzing of Chrome on Browser with components and other open source projects for those who maybe think like what I might forget in Open Media Room. I had an internship five years ago in Intel Corporation working on media SDK. I've been trying to implement H.265 and code events, so I remember some stuff like macro blocks, variable lens coders, IP frames, so I feel pretty comfortable in Open Media in the room. And agenda of my talk, I'm going to briefly explain what is the fuzzing, how you can do it, how we do it at Google, why media processing projects seems to be a good target for fuzzing, and in the end, I'll give an overview of voice as fast project. It's like a fuzzing as a service for open source. Okay, let's start fuzzing or fast testing. It is also called so. It's basically a pretty simple routine. You just generate some data. You use this data as an input to your target. You monitor your target for unexpected actions like crushes, some inconsistency, some security bugs, and you just repeat this process until you find some bugs, some unexpected stuff. Why fuzzing is good and why people use it. Together with memory tools also called sanitizers, address sanitizer, memory sanitizer, undefined behaviors, sanitizer, and so on. It helps to find a lot of bugs, almost every type of memory corruption bugs, any other crushes like new drafts, some code exceptions, data races, like any stuff you may have in C or C++ written projects. All these troubles can be not, I mean, not all troubles in the project, but all types of these problems can be found. And it also is useful for finding logical bugs, but it requires some smarter approach to your fuzzing target. I will show some examples later. And one more good point, it also finds not only crushes or breakages, it also finds hanks or out-of-memory bugs when you just, let's say you have one kilobyte video file and you try to decode it and it takes 80, I don't know, 80 or 100 gigabytes of memory. It's definitely a bug and it also can be easily found with fuzzing. Speaking about fuzzing approaches, we can split it into three types. First one is generation-based when you write some generator to generate data for your father, because basically you can read your random and feed it to your target, but it doesn't make sense in most of the cases. So you write something, let's say, HTML page generator. You just took all names of HTML tags and you generate some text based on these elements. Then you try to open page generated by your generator with a browser and sometimes you will find some unexpected stuff in browser. Another approach is mutation-based when you do not generate data, you already have some data like you have valid video files, valid HTML documents, depends on your target. And you have some mutation engine which reads existing targets and mutates them, duplicates some parts, interlaces them, everything which can be created or programmed. And another approach is stuff which we call like a modern approach, it's a guided mutation-based fuzzing, mutation-based side part is already explained, guided means that fuzzing engine which generates inputs and mutates inputs, it tracks coverage of targets for every input generated. So let's say fuzzing engine generates some video file, it feeds it to the coder and it sees which parts of code are covered, then it mutates, this input generates another one, feeds it to the coder and see if new coverage has been reached. If new coverage has been reached, fuzzing engine saves this input as let's say interesting one. If nothing new has been reached, it just ignores it. Due to that you accumulate files over time and your coverage grows over time. So if you fuzzing for let's say, I don't know, months or week or several months, you can get a pretty high percent of coverage of your target. How does it look usually? Let's start from the bottom, these just three lines. This is what fast target essentially is. You just need to write a one function which has arguments data and size of data. That's it. And you use this data as you want. Usually you just feed this data into some target API which you want to test. In this example, target API is some artificial example which has a bug. We are short on time, so I just disclose it. When size is three, you have buffer over read, right? You have only three elements in data array, but you try to check element within the X3 which is like four sediment. And this example works in just three iterations. It finds this bug, but of course this is an artificial example. Another nice real world example is our LibxML read memory father. You can find it on GitHub, but it just has one call inside this fast target. It just calls XML read memory function, which basically just parse XML document from memory. And with that one liner, we found about 30 bugs in LibxML for over last year. There are, there were, here, buffer overflows, user after free, all this stuff. That example of fast target is actually Libfather defined interface and Libfather is an engine for guided in process buzzing. It is not the only engine in the world, but it is the one from which we started our, our project. It actually a library which provides main, so you just write the function which you saw on previous slide. Then you compile your target with specific flags, which is basically some sanitizer tool, address sanitizer, memory sanitizer, depends on which build you are going to test. And you need to specify coverage type, which will be used to track coverage when testing different inputs. And that's it when you write fast target, compile it, and run it over some corpus. Basically, you can start with empty corpus. Corpus is a set of inputs, set of test cases, initial test cases. If you have some, you better to put it into this directory because it will give some understanding of, of the target to a buzzing engine. But you can also start with an empty directory. It works. Why media projects are written in CC++ are good targets for buzzing, basically because it has a lot of stuff working with raw memory. And if it's C++, it can be some logic with objects. And if you write it, you definitely know which, how many troubles you may have in this. And another point, because media processing is actually used everywhere, you know, this is better than me because, like, any device with a battery and Wi-Fi connection definitely has several codecs and libraries for the current media and not only devices in this room, but everywhere. And due to that, media projects are definitely a target for attackers and some most-known cases from previous two years, three years. You've probably seen FFMPEG and Thousand Fixes article from guys from Google security team. I hope you heard about stage fright. It was a vulnerability in media library for Android which allowed to hack Android phone just by sending an MMS message to, to the phone, another example from one and a half years ago is viral video. It was an FFMPEG vulnerability in parsing of playlists. So it allowed to read local files from, from the remote server on Facebook, on YouTube, just on every video hosting service in the world. Yeah, this is all links, so when you, if you're interested, you can then open the presentation and read more. Another example from last year is ImageTradic. It was a backend image magic library for image processing. It also affected Facebook and you may be recently heard that one guy got a $40,000 reward for reporting that Facebook uses vulnerable version of image magic and it allows to execute code on, on remote server. And another example, the most recent is in Chris Evans block. It was a bug in G streamer which worked on Linux disk tops. Basically, you download a file and open the directory, download directory in File Explorer. And usually you see a preview of, of the image and this functionality of preview was, has been abused by Chris Evans and he wrote an exploit to turn it to remote code execution. And yeah, just a quick note, we sometimes at, at Google and then open source project here from some maintainers or programmers that their projects have been fast. But actually it, it doesn't work this way because you know, how fuzzing works is these techniques are changing the project itself changes. And to get high coverage and to cover most of your project, it actually takes time. So the best, the best approach is to make it in a continuous way. You just take new build every day or twice a day and fuzzing existing corpus as is built. And a good example just from the last week, a bug in OpenSSL. We took, it took about 550 CPU days to find it. It was very interesting bug because OpenSSL has two implementations to multiply big numbers. And it was a logical bug which the fast target actually used data, data in its size as a two big numbers. And then it's called big numbers multiply two different implementations. Slow one and optimized one. And then it, it was comparing the results of this multiplication operations. And if result isn't the same, it was a problem. So this is what has been found. It was carry propagation bug in optimized implementation, which used almost everywhere because when you do it, you want to use better, faster version. Yeah, now about SSFuzz. As I said, I'm from Google Chrome team and we are doing fuzzing for previous maybe six years already. And we have a large infrastructure about 15,000 of CPU cores to run for others to de-duplicate crushes to check fixes and so on. So like when you try to do it at scale, you find, you will meet a lot of problem because some targets are crushing, are crushing every minute and you got thousands of similar bugs per day. And you need to understand which bugs are unique, which are not. And few months ago, we just copied, cloned this infrastructure and made it free and accessible for large open source projects on which, let's say, the modern internet relies on. Yeah, so this is all free. We have already more than 50 projects with about 200 fast targets. A lot of bugs found and most of them are already fixed. And yeah, we encourage every open source project to try fuzzing, to just try write a simple fast target to test your code. And if you can and would like to join to this project, this would be awesome because like all you need just to write this fuzzer for a target, all other issues are handled by existing infrastructure. And essentially, it's fuzzing as a service and it is free. Yeah, here I have an example of bug found and file it by our, we call it cluster fuzz, it's the whole infrastructure. It's cluster fuzz. And it is a real example from FFM pack. Yeah, it has a type of bug, stack buffer or flow. It has a minimize test case to reproduce all the info like stack trace configuration, which memory tool has been used. And of course, for security bug, they are accessible only by people who specify your security more maybe you and other developers not, it's not open until you fix it. Yeah, and also it's not only file as bug automatically, it also verify fixes like this is the same example. The maintainer, maybe he's here, Michael, I don't know. He wrote that patch applied and like, thanks. Then one guy from Google said that cool and if everything is fine, cluster fuzz will market as fixes and close the bug. And then even the same day cluster fuzz has detected this issue as fixed and close the bug. So you just like, you get back, you fixed and you shouldn't do anything else. Yeah, we also have some stuff to help you. I'm sorry you don't see it, but it is not very important to see. It's a statistics dashboard. It has some numbers, some persons for every father, such as number of those cases executed in given time range coverage by functions and by code edges. It has links to coverage report. I will show an example on the next slide. Speed of generating of inputs links to corpus, which grows over time. And there are a few more columns like performance analysis. I will also show an example. So you get analytics for your fathers also. An example of coverage report for another FF impact father. You don't see probably these lines, but you also should not. These are different source code files and percent of code covered in every file. If you click on file name, you will see more detailed view like which functions have been covered and even which lines are covered and which lines are not covered by existing fuzzing corpus. Another feature is automated performance analyzer. It checks your father logs every day. And if your father has some issues, basically set of issues is not big. It's known issues every time your father can be crashing every time like you run it and it crashes in five seconds. So it doesn't work technically. And you need to fix it as soon as possible because you just waste CPU cycles. And there are other issues which are similar for some fathers. So this performance analysis let you know them without checking logs for hundreds of fuzzing runs which are happening every time on the infrastructure. Here are links for looking at other examples of fast targets in Chromium in existing OSSFAS projects. We actually have only FF impact in OSSFAS. So if you are working on other decoding projects or I don't know, VLC player, I see a lot of guys with heads. You are super welcome to take a look at it, to ping me and we definitely should work something out on that. And I've seen that GitHub already has a lot of fast targets for, I don't know for which projects, like, but there are thousands of them and it's also a great place to take a look. Yup, that's all. Thank you very much for your attention and I would be glad to answer any questions. Oh, yep, sorry, a minute of promotion. I'm giving a lip-fazer workshop in April in June and if you manage to be there, please come. Okay, so we have one first question there. Yeah. Another one here. We actually pushed all the tools to the mailing lists and stuff, but it was never merged. And we think your tools are not very good and your corpus is very good, not very good, but I don't see how GitHub, we can improve that because you use the test case, you use the regression testing as your corpus, which is okay, but you don't test the full range of features in the decoder, you only test essentially the default features and not more obscure features like slice spreading and interlacing. So, and I haven't quite understood how on GitHub you can add more test cases and add more features, more test things in the programs. We already have this, but we'd like to do it. And I don't know, could you explain how to do it? Yeah, so the question is how developers, maintainers of project can affect what we do at OSS fuzz and can add some complications to fuzzer or inputs and so on. Well, basically our idea is to move responsibility for fuzz targets and for corpus to developers. Actually, we are trying to do not own fuzz targets, do not own this source code of fuzzers. For FFMPEG, we initially had one fuzzer in Google internal stuff. And this fuzzer supports about 60 different configurations. So we just pushed that fuzzer to FFMPEG repository. And then FFMPEG guys helped us to, let's say integrated in our existing pipeline. So if you have something to improve, the best way is to do it to push it inside your target repository, FFMPEG or another one. We should find those guys who rejected your pull request and ask them why this happens. You're also welcome to create a pull request on OSS fuzz. And... I don't understand. Yeah, yeah. But where are these files stored? How do I upload files to GitHub in the corpus? It's not at all clear, let's put it. We store corpus into place, in three places. First of all, you can upload it just to OSS fuzz repository and add one line to build a such script there. Another way to store it in target repository is to remove your project. But again, in build a such script in OSS fuzz, you need to copy these files to output directory. And third place is a cloud storage where we run fuzzers on different machines and all of them are syncing files to common place. And project maintainers have access to this common place and they can download existing corpus. They can drop new files in there. So if you are not maintaining of the project, you don't have access there. But if you are, maintainer or developer who is responsible for bugs, you have access there. And it is written somewhere on OSS fuzz page, but yeah, I agree that it's probably hard to find. Thanks. It's the fuzz that's just generating the random input and triggers on runtime error, but not using this random input to trigger at the unit test that is based on the project. So the question, if I got it, like fuzz engine, fuzz engine generates inputs and feeds them to the target to get on time errors, but doesn't feed it to existing unit tests, right? Yeah, but essentially it's somewhat similar to unit testing, but it is better because in unit testing, you have some input, test input, some functionality and some expected output. That's it. In fuzzing, you only have this functionality and you do not have a specific output. You have millions of them, they are generated and you also do not have any output. You can have if you want to do some correctness check and if, as an example, this open SSL multiplication. But technically, basically, fuzz target is a unit test, but continuous unit tests without fixed input and output values. And we are trying, in Chromium, we are trying to convert existing unit tests into fuzz targets to have them as an enhancement. Like, we would like to have unit tests because people write them and they are really important, but copy their functionality, which they are testing into fuzz targets and do it in continuous way is great because it finds a lot of stuff. Thank you very much for the talk. It's very interesting. I'm part of the research team in Imperial College, London. We've been working on fuzzing graphics drivers and actually we can crash the machine through a scroll web-GL page, increasing on Chrome. It's really interesting. So about the question is more like, do you test, so this is more about all media processing. Do you also test graphic drivers or just drivers and more general? And I would be interested to know, you mentioned 550,000 cores. Is this what you offer to other people to do fuzzing or do you have even more cores dedicated to fuzzing inside Google? Yeah. Let me start from the end. The question is about number of CPU cores, which we are using 15,000 of cores. It's internal Google, internal Chromium infrastructure for Chromium browser. For RSS fuzz we have, let me do not lie to you, maybe 4,500 CPU cores at the moment, but we add more when we are getting more projects. And the first question was about fuzzing, not only media projects, but drivers, graphic processing stuff. Yes, we have some of them. We have, you can go to Chromium code search, like Chromium code search and look for GPU fuzzer. You will find there an example or maybe two of existing GPU fuzzer. Yep, thank you.