 Minna-san, konnichiwa. Wadashiwa, you, this. Hello, good afternoon. I'm Xiong-si. And this is the first time I'm representing my company, speaking internationally. So if you cannot tell, I'm very nervous right now. But let's begin. So my name is Xiong-si, and I work at Tsukse. Under the hardware enablement Taipei, Taiwan team. And I mainly work on VPF, but that's not the topic of the talk today. Today I'm going to talk about a brief walkthrough of how to work with OSS-Fuzz reported issues. And also a bit about how to submit patch to the Linux kernel mailing list, LKML for short. But my main focus today will really be the OSS-Fuzz reported issue. So let's dig into it. The journey begin here. And at Tsukse, we have a dedicated security team. They will monitor and track security issue for us. It was a normal hot and sunny Monday, because it was always hot and sunny at where I live. And that's why I'm wearing this. And anyway, so I got a report from the Tsukse security team saying there's an issue with the VPF. I'm the package maintainer. So therefore I have to look into it. And they for me the CVE number, and so I can look into it. The first thing I check out is the CVE description that's here. And it says that the VPF version 0.6 has some buffer overflow and the VPF object open function. That is quite specific, but not enough for me to fix the issue. So I keep on looking. There's this reference section in the CVE report. So I follow that link, and this brings me to somewhere else. So I could see that there is some information that I already know, but also some more new information that I don't yet understand. And also a few links to this website called OSS-Fuzz. So that brings a question. What is OSS-Fuzz? Looking into the documentation, it says that it's a free service that runs Fuzzer for open source project. And it's a service that was started by Google back in 2016. And later it's done in collaboration with the Open Security Foundation, which is all open SSF for short. Now, all that is good, but the problem is that I do not know what Fuzzer is at that point. So I had to look into it. So what are Fuzzer? Well, I find that instead of saying what they are, it's easier to describe them by asking like, what problem are they starting to solve? And the problem that they're trying to solve is around bugs. If you think about it, to fix bugs, there's actually two parts of the equation. First, the bug has to be discovered, and only then can the bug be fixed. So this raises the question. Can the bug be discovered before the user, before the developer, or even the researcher sees it? And basically, can the bug be discovered with minimum human involvement? And as you have guessed, the answer is yes. And Fuzzer is one way of doing so. And briefly, the Fuzzer works like this. So Fuzzer is a piece of software that feeds random input into a target. The target in my case is libbpf. And when it feeds this random piece of input, it also checks for unexpected behaviors. For example, program crashing will be an unexpected behaviors. And when such things happen, it's a bug. And we have found one. So that's basically how it works. And knowing how Fuzzer works and knowing what a Fuzzer is, we can look back at OSS Fuzz. So it's a free service that runs Fuzzer. In other words, it's a free service that tries to find bugs in open source software. And actually, it focuses more on the security issue of open source software. So back to this OSS Fuzz report. Well, I know now that this is a bug report, of course. But secondly, I also know that this is actually a bug report that is filed by software. So unlike your usual bug report filed by user, where you can interact with them and ask for more input and ask more questions, we don't get that benefit in this specific issue. But that said, OSS Fuzz actually provide us with a lot of helpful tools to develop this issue. But to get into the details, we had to get our hands dirty. And the first thing is that I want to see the issue for myself. With that, I can kind of get into more details. And OSS Fuzz helps me by providing a reproducer test case. So what is reproducer? Back to our introduction of Fuzz. Fuzz feeds random input into a target and waits for a bug to happen. Well, this random piece of input is the reproducer for such bug, which means that if I run this reproducer provided by OSS Fuzz, I should see the same issue again. The first thing is that I first need to set this to run the reproducer, I have to first set up the environment. And it's actually quite trivial. So the first thing to do is clone OSS Fuzz. And the next thing is to run OSS Fuzz provided helper to build the image. Here I have to provide a project name, which is, of course, the BVF. And this will try to get some container image, which could take a while, depending on your network speed. It may take 10 to 30 minutes. And this build a container image for building LibBPF. So it only creates this container image without LibBPF yet. It's just the build environment. So the next thing I have to do is clone LibBPF. And with that done, I can now finally build the BPF itself with a fuzzer. But for that, I have to give it three arguments. First is, of course, the project name, the BPF, and then the source to the BPF. And lastly, the sanitizer. So what is sanitizer? Again, referring back to our fuzzer introduction, we have some check for unexpected behaviors. And the sanitizer actually are the checks. So since there's quite a bit of unexpected behavior, different type of them, we also have different type of sanitizers. For example, for uninitialized memory use, we have the memory sanitizers to detect them. And for the shift out of bound, there is the undefined sanitizer. But for a flow, we had used the address sanitizer. And there's also one for data race, which is the thread sanitizer. So you don't really need to know all this because all this fuss report actually tells you that. In our case, we're using the address sanitizers. So that's what I'll use. And with that, it builds the BPF alongside the fuzzer. Now we have completed the environment setup, basically just four steps. And I can move on to running the reproducers. For that, I just download the link provided in the report, rename it, just to make it easier. And then again, using this OSS fuzz helper script, I gave it the reproduce command. And the BPF is the project name again. Reproducer pass will be the one that I just downloaded. And there's also a fuzz target name. So fuzz target is basically what interface or what function the fuzzer is testing. And again, this can be seen in the report. We'll be using the BPF object fuzzer in this case. And actually BP, the BPF only have one fuzz target. So it's always this one. And we put that into the argument. And now I can start to see something. So there's this address sanitizer report saying there's a heat buffer overflow. So if you think about it, it's just like six steps going from knowing nothing to reproducing the issue locally on your machine. So I'll say that's a very nice thing to have. And with the issue reproduced, the next thing is to try to diagnose what's wrong. And the sanitizer and OSS fuzz actually helps you a lot with it. The sanitizer report contains a lot of information. The first thing it contains is exactly at which function and which line of source code does the issue happen. So we can tell, we can see that it's here. But this heat buffer overflow issue actually requires more digging. You need to know the root cause by looking at where it is allocated. And for that, you look at the second part of the address sanitizer report. And I can see that it's occurred in the same function just a bit higher up. So I can tell that while the heat buffer flow happens here, the memory was actually allocated here using this some section count. And I can tell that the section count comes from elsewhere, from another field. So this is a point that it's hard to get too specific into the issue itself. So for the next part, I'll kind of just skim through and mainly for time constraint and not digging too much into detail. But in short, with just print f magic, I realized this number is zero. And looking into the spec, there's a part of the f spec saying that such a number can be zero. And in such case, the real value is stored elsewhere. So I have to look somewhere else. And just so happened that I could use some helper functions for it. So I could use this elf get section header number and to retrieve the real number that I'm looking for. So I apply the change. And next thing I want to do is to see if the reproducer still could trigger the issue. So I had to rebuild the fosters again, rebuild the BPF because it's compiled and I change the code. So compiled it. And the next thing is to try to reproduce again. So this time I see that no address sanitizer report coming out, which means I have fixed the issue. So all is great. So next thing is just to commit. And that concludes the first part of my talk. The first part about how to use OSS files, I hope it's clear enough that it's quite easy to use and you can surely, well, not surely, but most of the time you should be able to find the OSS report and reproduce the issue locally on your machine. And now move on to the second part, submitting a fix. So if you just ask as a random project, like where should I submit the changes, you can say GitHub and you'll be right like 90% of the time, but not for the BPF. So although the BPF has a rep thought, a repel on GitHub, the changes should be submitted to the BPF mailing list because the BPF is integrated into the Linux kernel source code. And then for that, we have to post our patch to the BPF mailing list. So it's not really LKML, like the topic kind of, the subject of this talk is kind of off, but please forgive me. It's, I think the process is close enough. So what I'm going to say next should apply to LKML as well. So posting patch to the BPF mailing list. So if you have not worked with mailing this before or you just heard of it for the first time, it's essentially a discussion form kind of like Stack Overflow and like Reddit, except all the exchanges are done through emails. And there's quite some specific requirement for, for sending emails, especially sending patches there. The first and foremost, the most important part I would say is the email subject because that's what everybody sees at the first site. And the email subject actually contains two parts. The first part in closing the square bracket is a prefix. Inside the prefix, you have to specify that I'm sending a patch by, by saying patch. And secondly, you have to specify what tree this patch should go into. So tree is essentially just what rep hold that your patch should go into. For that, you can look into the BPF mailing list developer guide and it tells you that there's actually two BPF kernel tree. The first is BPF. The second is BPF next. Now, since I'm sending a fix, I see that BPF tree itself is for fix. That's where I'll send. So I use BPF for the tree. Up next is the version. So what version of the patches patch set are you sending? So this is the first time I'm sending this patch. So version one. And for version one, you actually can just skip it. So that's fine. Now we move on to the last part of the prefix, the sequence number. This is also, this is mainly for a situation where you're sending multiple patches at once. And in my case, in this simplified case, I'm only sending one patch. So one over one, I can also skip that. This leaves me with the prefix of patch BPF. And up next is the component part. So for component, I find the easiest way to find it out is just to look at the Git log and put in your file and see what others says the component it is. So in this case, it's pretty clear it's just the BPF. And that's what I use. So last, and probably the most important part is the actual subject that you'll be using. And I find that it's really hard to come up with a good subject, like just right away. So what I do is I actually skip this part and postpone it until later. So I'll move on by generating a patch. Well, actually the email file that I'll send for my patch. This can be done with the Git format patch. And recall that we already have a prefix that will be used in our emails. So I gave it to this command with the subject prefix. And I also specify which commit it should output. Since I'm using the last commit, I just use hit minus one. And this will give me an email file that I could further edit. Fire up the editor. And this is likely what you'll see. So what I need to do next is to fill it up with some commit message. And I find that usually when sending patch to BPF mailing list or OKML, I find it's pretty nice to have some template. And the template that I'm using is basically abstract, followed by content, followed by summary. So abstract is just very short sentence, saying what this does and why I did this. The content will go into a bit more detail on the reasoning behind this change and followed by the summary that enclose everything. Since it is a fix, I usually get into more specific details. So I'll say what the symptom of the issue is and followed by the root cause. And so for my patch, in this case, this is what I'll say. For the abstract, I look at what my changes. So I replace a field with a helper call. So that's what I'll say. I say this commit replace this field with this helper call. And as to the why, well, because I want to fix a heap of flow issue. So that's it. Now I can move on to the symptom. For the symptom and for OSS Fuzz, I actually have a very good summary. And that is the sanitizer report. Paste it in and the maintainer usually will have a pretty good idea of what's going on. And for the root cause, I use the EF spec content, a bit summarize and reduce, but that makes a pretty good group cause discovery. And for summary, I just go over the reason again, followed by the changes I made. And now I can go back to fill the title. So the subject is, firstly, it has to have the components I added, but I also need the main part. And I find that it's usually pretty easy to extract something for the first part, the abstract of my comment message and just stick it there. So in this case, I decided to say, this just replace some field with the helper function call. Equally, I think I could also say that this commit fixed a heap buffer flow issue, but that felt a bit too general. So in this case, I thought it's better to be more specific and see my changes. Now we're almost done. You have to still add some more things. So first is the certificate of origin. So any patch you send to LKML, the kernel source code must come with a certificate of origin. Secondly, it would be nice to say that what this patch fix. So the main use for this is like this from maintainers like me or others can discover that there's some issue with the code base and releasing and apply this fix to some older release product. And it really helps with the whole process. And thirdly, if you have some link to some external reference like this case, it'll be nice to add it. And I had to do a bit of trick to make it work. But if I had the time, I'll talk a bit more about this. So up next, before sending email, you also have to configure your get config to probably send the email. And my configuration, my setup is basically MSNTP plus mute OAuth to script. And with that setup, I can actually send all the patches through Outlook. So if my setup could work with Microsoft, it probably could work with any setup out there. So it should be worth trying. And with that, I can send out the email with the send email command as the name implied. So you have to send it somewhere, which is the BPF mail in this. And it's also nice to have a dryer on to test your script, test this, whether it works before actually sending out. So once you make sure that everything works nicely, you can actually send out a message. And to be very complete, this would be the command line I would actually use. So the one on top actually just sent to the BPF mailing list. But ideally, you want to send also to the maintainers, not just the list. So just to make sure they get it and they see it. So for completeness, this will be the one I'm actually using. But I think if it's like a very simple fix and just one shot, I think nobody will show that you by sending just to the BPF mailing list. And when after your mail and send, I start a code review. In my case, the BPF maintainer says that there's something that bothers him with the code I have. And he suggests some fix that I should do. So it does the change. I make the use of helper function closer to the actual allocation as suggested and commit a man format patch. This time it's the second time I'm sending my patch. So I had to add a version to and also slightly added the patch. So when you're sending another version of your patch, you need to specify what changes are made. And usually this is done in something called the cover letter. So when you're sending multiple patch, you'll send the first kind of like a summary of what your whole patch set does. But since I'm only sending a single patch, in this case, I'll put my changes just before the stats of the diff. And this way I have something that the emails reader would see, but also would not make into the final get repo. And so I say that the changes I made, I actually go against a different tree. And here's the changes I've made and then send it. So that's pretty much the, I guess, the end to end process of going from finding an OSSFAS issue into having your patch accepted at the Dennis kernel source. And if you're interested, I think there's a lot of opportunity for contributing mainly against OSSFAS, but of course also against Dennis kernel. But this slide will be mainly about OSSFAS. So OSSFAS actually tried to build and fuzz a lot of different target. And right now there's actually quite a bit of target there failed to build, which means they're not getting fuzz. So if you want to help, you can look into that. And there's also a tracker for all the OSSFAS reported issue. That's also something you can work in. And there's some more resources, mainly on fuzzer and on synthesizers. And that would mark the end of my, I guess, the main talk. So yeah, arigatou gozaimasu. Thank you. So any feedback or questions? This is just some feedback. I work on the team that supports OSSFAS. If you, there is money that Google will give you if you help improve OSSFAS. So check out the bark hunters page of Google and you can learn more there. Yeah. Yeah. Thank you for that. I forgot to mention. Yeah. I think there's some, as mentioned, there's some bounty. Let's see if I can open. So you really get paid, I think, quite a good sum of money if you try to integrate it into the existing open source project, the first one, right? So is there some, like, a threshold or some requirements for the open source project to be considered part of the award program? It will get checked for the criticality of the project. So, like, just picking some random project no one uses won't get a bounty, but you can get up to US$5,000 for really critical projects. All right. Thanks. I'll also add, like, this also includes, if you scroll down, improving coverage as well, which is, yeah. Yeah. So one thing I think I learned while looking at fuzzer is that besides issues that crashes or memory sanitizing issues, fuzzer could also be used against to find some logical errors, I believe. So I think one thing that math-related library could be fuzz is that if you have two implementation of the same library and you could try to give them the same input and if they give you a different result, that means something is wrong and this is a logical issue that someone should look into. All right. So I guess that sums up my talk. So if you don't remember anything, just remember that you can get paid by integrating OSS fuzz into important open source projects. And thank you, everyone.