 Okay, good morning everyone. Thank you very much for coming to our talk on fussing the CNCF's landscape. My name is Adam and I work with David at Adalogix. And over the course of the last almost two years we have been working on securing and we have been working on improving the security posture of CNCF projects by way of fussing. The work has been funded by the CNCF and in this talk we will present some of the experiences and the results that have come out of this and we will look forward and consider where we are in the greater picture of this work. A quick run through of what we will go through in this talk, a quick fussing intro for those that are unfamiliar with fussing. Next we will go through the OSSFOS project and why it is so important for the open source ecosystem. Then we will touch upon and go into the state of fussing CNCF projects. And in that regard we will go into how to fuss a CNCF project and how we approach that challenge and some of the results that have come out of this. And lastly we will consider future work in terms of, in the perspective of where the work that has been done now and where we will be going next. And also just to note here, we'll present a lot of work that we have been doing but we will also touch upon projects and data where we are just tiny pieces of the whole puzzle because there are other people fussing various CNCF projects and they deserve a huge amount of credit in the data that we show as well. Okay so I'm David and I'm gonna give a quick intro to fussing and the main reason for this is amongst other reasons to kind of argue why it's useful to people who might be interested in fussing but might not believe it's really working. And how many of you are familiar with fussing? How many have heard about it before? Okay so around, I wanna say 40% or something? How many have never heard about fussing before? Okay so I think maybe this talk is jumping a little bit too ahead then. So fussing from a high point of view is like automating test case generation. So it's very similar to writing unit tests or integration tests but it's a way to kind of like generalize the input that a test case uses. So on the left here you see traditional ways of testing let's say my API and if you were to test that with various different inputs you would just have a set of lines in your code that says that calls my API with the different inputs. And the way you would generalize this in fussing is you would have this essentially a loop that calls into my API but instead of giving a specific input you would just ask the fuzzer to give you this input. Like you will ask the fuzzer to give you some input and that will then kind of like the point is then that the fuzzer can generate all sorts of inputs. And this is the input that the fuzzer will generate is often considered to be just random input but in fact it's kind of input that has been constructed by way of a very complex system underneath that is the fussing engine which approximates in a sense a very useful input for the fuzzer if one can argue. But how this will look in actual code is right there. So here we have the simplest case of a fuss driver in the Go language where you just have a function called fuss and this will then be called by the fuzzing engine which is kind of like underlying in your program with the fuzzer input. And then you just call my API. So this is the exact way that you would write a fuzzer for my API in Go lang. So as I said before there's this myth that fussing is just random input it's not gonna work to my application that uses highly complex data types. And this is kind of like a response we often hear when we approach CNCF projects especially CNCF projects that are written in memory safe languages. So in memory unsafe languages CNC plus plus fussing has been around for quite a number of years now. I think fussing was first started around 30 years ago but it has really had this evolution or resurgence in a sense the last 15 years or so where this coverage guided fussing came to be. So we often get this response when we say we would like to fuss your project we often get a very sort of critical view from us it's not gonna work because it's just random testing and it's just a false notion in a sense. There are hundreds of academic papers in the last decade exploring how to improve fussing and it did start 30 years ago with basically just cadding from random and using that as input to your program but that's not how it is anymore. It's highly structured tools that try to pick the right inputs to your code such that you will execute essentially as much of the underlying code as possible. And when I say modern day fussers here I really refer to coverage guided fussers. So the underlying principles in coverage guided fussing is that you have a given corpus which is just a set of test cases and these are the test cases that you work from when you want to generate new input to an API. And the fusser will start with a given corpus it could be just a random element and then it will start to exit and then you will have to compile your code so your target project that you want to analyze you have to compile that in a certain way such that essentially the compiler can insert extra code into your program that the fusser will use to essentially trace execution in the target and it's then going to use this ability to track execution in the program to identify whether a given test case is good or bad and the way it's going to do this is it's going to pick seeds from the corpus it will execute the program with that seed and then it will track what was the kind of like let's just call it execution path of that program. Then it will save the execution path into kind of like a set of we have explored this already. Then the next time it will so like just run again pick a new seed from the corpus mutate that seed so that means just modify it randomly in a sense and then it will execute the program again and again trace that we see a new trace in the program it's not really traces but I think it's more intuitive to think of it that way and if there is new coverage say a new trace with this new input then it will save that seed into the corpus but if there isn't new coverage they will just kind of like forget about that seed and the main point is that you build up this corpus iteratively where each seed in the corpus essentially executes the target program in a unique way and the benefit of this is that you will reduce the complexity for like exploring the code and I'm going to try to argue that through a small example here. To say for example you have a program that checks that has four conditions that checks a byte in each like in a given data buffer and if you were to guess four bytes correctly say A, B, C, D just four characters you would have an input space of two to 32 because a byte is eight bits and four bytes is thus 32 bits which is two to 32 but because we have this notion of corpus and coverage guided we have to guess each byte essentially one at the time and when we are to guess one byte there's two to the eighth chances of getting it correctly. So that's one, you have a chance of one out of 256 to guess the first byte correctly. So let's say we guess the first byte correctly then we save that seed in our corpus and we just call our current seed so like a data buffer with A and then we have to guess the next one and that's again to guess the next byte we again have a chance of one of two to 56 of guessing it correctly and that kind of keeps going until we have guessed A, B, C, D correctly and again the reason we do that is because we save whenever we see that we guessed it correctly we save that and we move on to guess the next byte. So the point is that we have to guess a maximum 256 times four times which is 1024 which is much less than two to 32 and that's kind of the ideas behind coverage guided fussing and why it reduces the complexity of exploring a program from just random guessing and this thing is called coverage guided fussing and that's kind of one of the breakthroughs in fussing that essentially made it much more applicable and what almost everybody uses today. Okay, so which box can you uncover with fussing? Fundamentally, fussers are just test case generators they explore the code. They don't actually detect the box. The thing that detects the box is just how the program executes and we then have this concept of sanitizers in memory safe languages and these sanitizers are essentially kind of like buck oracles that you compile into your code which can check for things such as buffer overflows, valid invalid dereferences and use of the freeze and all these kind of things which are common things in memory unsafe languages. In memory safe languages which is what most of the CNCF projects have written in in particular Go, you essentially don't have sanitizers so the things that you find at the moment are stuff such as uncored exceptions, out of bounds, nil pointer dereferences, timeouts, in some sense deadlocks and out of memory issues and these are kind of the things you can expect when you fuss a GoLang project. You can also do various things such as behavioral testing but I don't think I will go into that now. But point being, you can expect the fussers will not detect an issue in itself. It's actually when you will just explore the program the various ways it executes and then potentially you will find that if you execute it this way it will throw a nil pointer dereference. Okay, so fussers are pretty complex to manage. As I said, we need to have this corpus and the fusser will build up the corpus over time as it explores the code and that also means you shouldn't run the fusser from an empty corpus whenever you run it. You kind of have to incrementally build it up over like each day over the years and so on and there's also a lot of other issues that comes with managing fussers such as how do you keep track of all the bugs, how do you report things and how do you share results between everybody and this is what we have OSS fuss for. So OSS fuss is a service run by Google which is OSS I think it's open source software fussing. And it's essentially a Git repository where you upload a dog file, you upload a build script and then you to a given folder in this GitHub repository and then Google will just start running all the fussers that the given build script will build. So this is where the vast majority of open source projects are being fussed by way of now and I think that OSS fuss has around 600 open source projects so essentially all the very security critical ones and all the ones that we will be talking about this case in this talk is also on OSS fuss. And yeah, we have a link here to how to integrate into OSS fuss. Please check it out. OSS fuss might be a little bit large because it's just in a sense a big framework that lies underneath it and some maintainers are actually not that interested in getting into OSS fuss because they feel it's a little bit too heavy. The same developers of OSS fuss has developed this concept called claustrophos light which essentially runs and manages all of your fusses in your CI and you don't have to so like do the whole integration into OSS fuss and claustrophos light is also straightforward to integrate into your project. So if you are an open source maintainer or similar, check it out. Okay, so I think I've already talked about this. Let's now talk about the CNCF landscape being fussed. So yeah, so here we see an overview of the CNCF projects that are currently integrated into OSS fuss and quick note, Istio will also fall on this list once it joins. At the logics we have written a lot of fusses for Istio which also has resulted in a high severity finding. We have a blog post about that on our blog. It's a pretty interesting case if you are a go developer. So I encourage you to check that out. There might be a CNCF project, at least I can think of at least one that do have fusses and do a pretty good job at writing fusses and maintaining them and making sure that they are not broken. But not all CNCF projects that do maintain a fussing suite are integrated into OSS fuss. And because of that, it's a little difficult to say that these projects are running their fusses correctly for the reasons that David mentioned because running the fusses especially once you get start having 10, 20, 30 fusses in your system, in your project, it becomes a very complex task to run these fusses. So if you can think of a project here that do a good job at maintaining a fussing suite but it's not on this list, that is the reason. And if you are such a project, please reach out to us and we'll be happy to integrate your project into OSS fuss for you. So let's talk about how to fuss a CNCF project. And this is the approach that we take. Yeah, that is our practical approach to pretty much all CNCF projects that we approach that have done no prior work with fussing. David mentioned we have worked on some projects that have worked on fussing for years and that is another story. So this is more in terms of taking a project from zero to having a pretty solid fussing setup. So we start with an initial integration and to us that means we start by writing a few initial fusses and merge them into the CNCF fussing repository where we maintain all fusses that we write for the CNCF projects. We are open for contributions there and please check it out if you wish to. Next we integrate the project into OSS fuss so the fusses run continuously and in the OSS fuss integration we instruct OSS fuss to pull the latest master branch or release branch and pull the fusses from the CNCF fussing repository so that we can work on the fusses through the CNCF fussing repository and OSS fuss will catch those updates continuously. The next step is we write a lot of fusses to get coverage up and yeah, I mean at this point we have made the initial integration and the next step is really to increase coverage of the whole project, starting with complex parts of the code base. Finally, we work on delivering this over to the project maintainers and owners and if we, the goal on this final step is to make the owners take charge of the process, the project owners take charge of the fussing from here. We want, it is our goal that it is the projects, it is the projects that they take complete ownership and they can do whatever they want, delete fusses, modify fusses if they see it fit and in this step here, step three, we work with the projects on doing that. So there are some pros and cons of having someone like OSS data logics integrate a CNCF project, integrate fussing into a CNCF project and one of the pros is that we have a lot of experience with fussing, I think at data logics we have contributed fusses to more than 200 open source security projects, sorry, open source projects so and we have gone down rabbit holes, we have seen how to quickly get from zero to one and effectively but of course we don't have the months and years of experience that the project maintainers do and naturally on the third one as well, maintainers don't have that much time to first learn about fussing and how to best integrate it into their project. They have plenty to do already. I believe we had a project where they had had a GitHub issue on the fact that they should integrate fussing into their project for two years, this issue had been open and at data logics we were able to pretty quickly get fussing started and the engagement resulted in a high severity CVE. So yeah, yeah, I think that's it for this one. Yeah, so in terms of the last step in the process of integrating, of working with the CNCF projects, a few things that we do when we wish to hand over the fussing suite to the projects themselves is that we work from the CNCF fussing repo initially and we encourage the maintainers to move the fussers upstream to integrate into the test suite, CI, nightly builds, et cetera, and which will make maintenance easier and should help the projects avoid downtime in the fussers. There will often be a list of bugs found in this engagement and OSSBOS has capabilities to reproduce these very easily and this will be something that we'll help projects with. We encourage projects to write fussers specific for that target project and simply further develop the whole fussing setup, how the projects see fit. A quick note on our GoFuzz header project, if you are a project that is implemented in Go, we have a very neat little library here that makes it quite easy to fuss structured data types. You simply initiate a fuss consumer where after which you can quickly create random or pseudo random versions of structs, maps, slices and much more. So a quick note on that, if you wish to use that. Okay, so hello, hello. All right, so you can fuss a lot of different projects written in a lot of different languages and essentially OSSBOS supports projects written in Golang, CNC++, Rust, Python and Java. The bugs you look for in a project kind of depends on the language that you are fussing and threat models of the projects that you are fussing may not always be present. So you can go multiple ways when you are to say fuss a project. You can either kind of like extrapolate from the testing suite that they have. You can try to identify the whole and the result of that is that you will find most likely a lot of bugs but may not be security relevant bugs because you might not attack the threat model in a sense. And CNCF projects are often written in memory safe languages which means that if you are looking to fuss CNCF projects most likely you will turn into go or say mainly go to be honest. But at the same time even if you are fussing a project written in Golang you can always look for the dependencies of a given project. So for example, Flux, it's a GitOps tool that is based on that it's written in go and it's like a bunch of custom communities, operators, controllers, sorry. And that depends, Flux depends on the other go project called git to go and git to go itself depends on a library called libgit which has 60,000 lines of C or C++ code I can't remember, which essentially means even if you are just interested in fussing memory unsafe projects you can still go for a lot of say CNCF projects that may not be written in memory unsafe languages because they will have dependencies to memory unsafe languages. Okay, so what are the results here? And the results we are going to present in this section is essentially the data given to us by OSSFUS on this URL. So they make all of the results publicly available by on box.chromium.org and you can use the following query to essentially get the results that we're going to show and I want to just give a note here that this is an over approximation from the perspective of the results are the results but it will be the fuzzers will like not necessarily only find legit bugs the fuzzers themselves can have issues which will be reported as a bug in this manner. The engines can sometimes have an issue which will also report a bug. So take it with a little bit of grain of salt and don't expect that all these numbers constitute exploitable bugs. So these are the results from OSSFUS reported and potentially what we can see here is that there are two projects Envoy and Flunbit written in C plus plus and C and they have approximately well above 200 bugs each reported and notice again that this does include security bugs and also non-security bugs as set by OSSFUS. So if you look at the query you can see I have this slight type here where we both include security and box security types. So it's not just security types and this include all forms of well all forms of memory corruption issues that you can imagine. And notice here that Envoy has been fossing for I think they first integrated into to OSSFUS since 2016 and I believe that the team has done like a tremendous job in getting fossing all over their project. So in the sense that there's a lot of bugs is a good sign from the perspective of they have analyzed a lot of their code base. We don't have all of the goal line projects here and you can see that the numbers are quite a bit lower but this is also a reference to they are not as mature in their fossing approach as the two other projects that I mentioned here but these are the numbers that you can expect and in a sense it also goes to show there will be work involved here because imagine sitting at the receiving end of Envoy and you have to analyze 860 bucks to verify how to fix them or et cetera, et cetera. A lot of them will also be issues from the OSSFUS side that will be fixed by itself in a sense but there will be a lot of work for the developers to deal with this stuff. So if we were to plot the issues that are reported here we can plot them based on how many issues are closed on this monorail so the database that I just shown or and also how many are open and in that sense we can see how often bucks get introduced or get found by the fussers and how often they are fixed. So here we have a project called Argo and the red graph that you see up here is the amount of bucks closed on the database. That means whenever a bug is fixed it will turn from open to closed. So closed is just gonna go upwards depending on how many bucks gets found and fixed whereas the blue line shows how many issues are open. So ideally it will go a little bit up and then it will hopefully go down depending on how fast the bucks are fixed. And here we have an example of say when Fussing was introduced quickly a lot of bucks was found but they were also very fast to fix it and we can also see that it's kind of a look-a-rhythmic curve, the red one which means that the fussers will find a lot of bucks in the beginning and then it will start to fade out as they kind of have explored all of their potential or get to explore all of their potential. We have another project here, Kubernetes and notice that the x-axis shows the date and the y-axis shows the amount of issues closed or opened and here you can see so like a similar graph look-a-rhythmic in a sense and there are a few bucks still open in the Kubernetes buck tracker. We then have an example of LinkedIn 2 proxy where you can see that again the red graph will have a look-a-rhythmic sign but also a look-a-rhythmic curve but you can also see that they are very fast at fixing the bucks. Whenever a box occur, the next day is fixed. So it's kind of like a great, they do it really well, the LinkedIn team. By the same time you can also see bucks kind of stopped being found. Let's say six months ago and you can ask why. Most likely that's because they don't contribute new fuses to their code base which is also an interesting thought. So if you were to, for example, if you knew how to fast-rust projects, most likely you could go and help them out. Fluentbit, another example and again we see it, it's like the blue bar keeps being low whereas the bucks keep increasing and that's also a sign of whenever, whenever a bug is found and fixed, that one gets put on the red bar but the blue one kind of stays constant and that means every time a bug is fixed a new one is found. So when you first find a bug, it will kind of like run into that bug all the time. So it will kind of be detected all the time. When you then fix it, it can finally progress further in the coverage and that's why the red will continue to grow whereas the blue line stays stable. And here is the Envoy project, yeah. Tudorback, give that to Adam. Yeah, so let's talk about where we'll go from here. Let's zoom out a little bit in terms of where we are in the greater scheme of things here as David mentioned over the last 10, 15 years, a lot of work has been done in making fusing generally available for open source projects, for the open source community that has included writing really great fusing engines, sanitize us and bringing this technology to all sorts of languages as David already mentioned. So we have quite mature fusing engines now but of course we are seeing changes in this as well. For example with Go 118 where the fusing engine is getting integrated into the language itself. So next over the last few years we have worked on bringing these fusing capabilities to the CNCF projects and like you see that is like more or less where we are in that story. So let's talk about the next steps, what will happen over the next couple of years. Like we mentioned, we want to see maintainers involved. It will make a huge difference in terms of taking fusing from the initial step to really approaching it from a domain specific knowledge kind of approach. Next we will see more bug oracles and sanitize us for memory safe languages. There are a bunch of vulnerabilities that fuses don't detect, they do detect some but there are a lot that they don't detect and we want to find those automatically. Next, yeah, again mature fusing even more based on what has been done now and that involves involving the community and the maintainers. Could I just interrupt here? So with regards to this, can you go one back? With regards to this bug oracles sanitizes one important point to highlight here is that we have Java fusing available but it didn't detect the lock for J issue. Then the developers of this Java fusing engine came up with a bug oracle to detect command injections and they found lock for J in a matter of minutes with the new bug oracle. And you'll see a lot more of that where people will try to identify those type of high level issues that may not be just detectable by the program crashing, for example. Yeah, so I guess we have already gone through the maintainer involvement. That is one of the steps that we want to see over the next couple of years. But yeah, so David had an example of a real world scenario where a bug detector was improved to detect new vulnerabilities. And let's talk about like some possible scenarios where that will occur in the cloud native space that the CNC projects can benefit from. So when we talk about bug oracles or sanitizers we consider it from a perspective of it being integrated into fusing where the developers of FOSS harnesses and the tests itself don't have to tweak or change settings of the FOSSers but the same FOSS test can be used to detect all these different box and vulnerabilities. So for example now in Go as David mentioned we detect, we have at least a series of box that we can detect automatically like out of ranges, out of memories, timeouts, near the references, et cetera. But it's very likely that we'll see race condition sanitizer for example that will not only detect possible race conditions but might also try to slow down a program in order to win a race and produce complete reproducible test case. Next logging, we have a series of vulnerabilities associated with logging. For example, can users input arbitrary data into the logs and thus create fake log entries to prevent an audit trail. File handling, that seems, that is a big issue also in the CNCF landscape. I believe the talk after hours is about a vulnerability associated to file handling. So I encourage you to check that out but basically can we read arbitrary files, can we write arbitrary files on the system that we're not supposed to command injections. We added a logics, we found a high severity CV I believe in Flux in last winter and that I think believe they found another one just last week and yeah, it continues to be an issue in the memory safe languages and we might see a bug or a sanitizer that will find these bugs. So the main point here is that when we get these sanitizers into the fossing itself, into fossing itself where developers don't have to do anything extra to make use of these sanitizers, because we have coverage of all these CNCF projects, they will benefit from these new developments immediately and that will have a massive impact. So a single sanitizer will be piped out to all these CNCF projects without much or any further work. So the point here is that we already have hundreds of fuzzers now for CNCF projects, but there are a lot of attacks that are not necessarily caught which actually could get caught even by the cobwebs that exists in the current fuzzers and if you come up now, the next ability is if you come up with these bug oracles, all these fuzzers will be able to find say whatever bug your bug oracle may find. I think that's it. We'll leave this conclusion as such, but in short, we fuzz a lot of projects, found a lot of bugs, it works really well. It's not fully automated. As you can see that there's a lot of bugs to digest. It takes time to write these fuzzers. So you will often hear fuzzing is security automation, but it automates a part. You still have to do a lot of work yourself. I think that's it. Thank you.