 Welcome to this talk about building safer things. Thanks to GoFuzzing. My name is Jeremy. I'm an application security engineer at Grafana Labs, I joined six months ago. Used to be a backend developer long time ago, mostly coding for security products in Java and some.net. Go is not my main language, but I like the design decisions they've made from a security point of view. Started in Grafana Labs in March 2022, so that's where they're new. So we are in the beginning of the security journey, but because Grafana Lab is all about transparency, there are things we already wanted to share. And you will see in the presentation, there are many things they want to get a repose that our public and all the PR and all that. So you can follow up on that and see that it's real data. What we will start doing is, even though Go has good secure defaults, we'll see that it's still possible to suffer a pass traversal. And we talk about GoFuzzing, trying to, and I know it can help like detecting pass traversal. Then we will try to first mark complex code, see that writing predicates that are needed for Fuzzing can not be that easy. And then we'll follow up with next steps and like once you have GoFuzzing, what should you do next to make the more value out of it? Because Grafana Lab is all about transparency, we'll share first or first zero day that took place in December 2021. So I was not yet an employee, but I discovered a lot about the context thanks to public blog posts and then public repose when the issue was fixed. By that time, the security team was only one member, the size of Thomas Owen. And basically what happened, a security researcher responsibly disclosed a pass traversal in the code of Grafana on December the 2nd. It was rather bad, scored a high, because it allowed to basically download any files running as a Grafana process from the server. And out of excitement, this researcher tweeted that he found a pass traversal, not the specific place in the source code that was available, but just that he found one. And a few days later, we identified on December the 7th that, well, this vulnerability was actively exploited, making it a zero day. So other researchers, when they were about, hey, there's a pass traversal in this code, they could find it because they knew that if they look for something, they will find it. It was also fixed on December 7th. The question like you may be asking now is, yeah, but isn't it supposed to have a really good standard library with safe defaults, and then why did it happen? Well, in fact, the method 5pass.clean is pretty. Let's see why. If you just read the documentation, quickly, you just, like, somehow read the first sentence and say, yeah, it cleans, clean returns a short pass, equivalent to pass. So it looks like a really good candidate to filter any 5pass you could receive as input and to get the canonical form out of it, and then somehow you can conclude that you will be safe. But if you read in details, and I don't ask you to do it now, because it's kind of really confusing. There are several steps, and it's not easy to understand each of those and all that. And I'll try to make a recap. It's just like if you just use simple rule, 5pass.clean will remove any dot-dot sequence, which has a sequence that are dangerous from a password point of view, because that's the one that can allow you to jump to a top-level folder. So it will remove all of those dot-dot sequence in all the inner elements and also on the first element if it starts with a slash, which means that if the first element does not start with a slash, it won't filter it. It's more proof you just go to the go playground and you give an input that dot-dot slash data and then you see that it's not cleaned as expected. So the function is doing what it says in the doc, but the developer may be expecting something else. So now let's look at the vulnerability in detail. So it's easy. It's just in two lines. First thing we were doing, we were cleaning a web input. So we received a 5pass, and we were just executing 5pass.clean and thinking that we will get the canonical form out of it. And then we joined this 5pass to a plugin directory. And so somehow we're expecting that all the files would be contained within this plugin directory, which was not the case because of this, let's say, unexpected behavior. What's really interesting, if you see the source code, a few lines below, you will see that we have a comment that's about the security linter, Gosec, that we are running. And the developer says that, well, it's safe to ignore Gosec warning G304 since we already cleaned the requested 5pass. So somehow it could feel like it's a bit of pity because we've, like, somehow a tool that's put it something that could be dangerous and then we reached a conclusion that, well, it's not. And if you read the details of this Gosec warning, kind of, like, awkward thing is that there's a recommendation in it, and that says, yeah, well, you should just use 5pass.clean. So let's say any reasonable developer will get this warning, will look at the corresponding doc and say, well, now I'm fine, I'm using 5pass.clean. But it's not because we have this password or so. So how did we fix it? Well, somehow, try it in an easy way. We just add a slash in front of the 5pass we received so that now somehow we are making the clean method as we expected it should have been. And one unit test has been added to check those relative pass stuff and making sure that we don't get outside of the plugin directory. Yes, you could say that, yeah. One unit test may not be enough. Also sharing for transparency, side discussions we had on this topic and were three different proposals. One of them was normalizing all the URLs but in the business logic, you don't get dogged slash so somehow you do the filtering before. It happens in your logic. Another interesting comment was a way to silence this GoSec rule. But it's also clearly said that, yeah, we have the risk of getting blind to this issue because maybe it's not fixing it. It's just tricking the GoSec tool. And the last question, the last point would be really more interesting for us. The point about making a security helper library out of it so that other developers don't get tricked by that. And then, yeah, a question you can ask yourself is, okay, but how are we confident that only adding a slash and front of it will fix all the possible cross-travel issues we have? So that's why now we will be discussing about your fuzzing. So fuzzing in a few words, the goal is to extend unit tests by predicates. Predicates are somehow general truths and somehow we want to extend the unit tests by predicates that say that those kind of things should never happen. Like for example, for the plugin stuff, you say, hey, I never want that the resulting file pass is out of my plugin folder. And then what does fuzzing do? Well, in fact, it's an engine that will generate a really big amount of pseudo-random inputs and test them against those predicates. So now it's available natively in Go from version 1.18, which is really good because you don't have to depend on an external tool that is not well maintained, that is hard to use, why you need specific config files to have it running and all that. And what's really nice with the Go integration is by default, as soon as it has, if it finds something that breaks a predicate, it will create the corresponding test data and then as soon as you relaunch the test data will be including in it. The default implementation is a never-ending loop because you are generating random inputs. Well, in fact, there are kind of, there are deterministic permutations on inputs, but it never ends because you have so many possible inputs. Of course, you can configure it and say, yeah, you can stop after a given amount of time and with the assumption that if you don't find something quickly, it could be safe enough. It's multi-strated. You can choose a number of workers and say, of course, you want to use. So there is a rather nice tutorial, but we will just use the past traversal which we've been discussing about. So what has been done is in this GitHub repo, there is a really simple example of a clean path where the validation logic has been extracted into this really simple method where we just do slash plus or parameter and we clean it. And so what do you have to write to have the fuzzing test working? I won't focus on all the details just on the important points. First thing, you are declaring our test cases. So this is a list of inputs. There are four, three over here. And those somehow correspond to a unit test. And what's really nice with the Go implementation is if you run it as is, so without launching the fuzzing engine, those will be considered as regular unit tests. And then really in the specifics of the fuzz method, you are telling which parameter you want to fuzz. Then you throw your parameter in the helper library. And then you write the predicates. So for this example, there are two. The first one is, yeah, well, you expect that the clean output starts with a slash because we're really adding the slash. And if we don't have a slash anymore, it smells bad. It's been at somehow an escape has been found. And then the other predicate is making sure that we don't have any password or sole sequence. And the password or sole sequence is a slash dot dot slash, which means that somehow you are able to move up one directory. All you have to do is either you just launched this really small but what you have to do is make sure you have a correct version. Run the regular unit test because of course if there are a few inputs you've already provided break unit tests, it's not worth launching a B engine crunching millions of inputs. If there is already something that you know that break the predicates. And that if you really want to start a fuzzing loop, you just have to add a minus, minus fuzz and the name of the test you want to fuzz. So it looks like super easy. But writing the predicates is not straightforward. It's a bit more trial and error at the beginning. If you will write predicates that are not totally correct and fuzzing will find valuation, but in fact they are valid outputs. And then fuzzing will also identify counter cases where it's really hard to decide if it's a valid or invalid output. But it's worth fuzzing is really useful because it forces you to make requirements more explicit and because there will be less ambiguity you will have less vulnerabilities because you will know what would be the behavior of your software under let's say world conditions. So for example, and to be totally honest, I didn't get the correct predicates first. So I saw that I would want to check this past level. So looking at dot dot slash, because for me it was enough to say, hey, where was this character sequence? I'm able to move up a directory level. And then you just long fuzzing on it and it really quickly finds a valuation and it provides you the input and the output. And it's really hard to figure out if it's a legit input, like if you receive as a five-part something that's called zero dot dot slash zero. And is the corresponding output valid? Well, it starts with a slash, but does it make sense? Does it kind of pass through us all with that? And here you will have the question about like, should I be more strict on input validation and do not trust that? Or should I try to do something in my validator or to get rid of that? So that's why now we will be looking at more complex helpers. So I took this example and I somehow extracted the source code. We have plugins in Grafana and plugins are used to add features. And they are basically just a zip file that you unzip in a folder on the server. And what we don't want, of course, is that within this folder that there are sim links that point out outside of the plugin directory. So for that, we have a helper method that's called eSimlink Relative Tool. It's rather arbitrary. So yeah, well, Go is not my main language, but even though I had to read it three times, it was a bit hard to figure out what it was doing. So it's doing the clean. And it's calling the file path.trail to know if it's a relative pass or not. And then it has a return connection that's basically looking for ..slash, which is just like the sequence I was using before on that project I was using before that was not really okay. So when doing the code review, I was just, yeah, well, maybe fuzzing can help decide if this validation logic is doing its job or not. So for that, I had to write a first test. And then it become obvious that the predicate is not straightforward, so somehow add to extracted into a function called expected result. And then in the fuzzing logic, the only thing we are doing is easy output of the helper different from the expectation from the predicate, and if so, we found a violation. And then the predicate is implementation here. Well, when you need to express it, then you quickly realize that somehow you have to reimplement the logic of the validator and it's a pattern that's quite common in unit testing. Either you're just looking at an input A and you say you get B and input C and then you get D, or you try to be more generic and somehow you have to do implementation number two and compare implementation number two with your original one. So it's what I'm trying to do here with the naive implementation looking if it's like, if it looks like an absolute pass, trying not to use the standard library to be sure we are fully covered and then doing a naive job of pass and all that. And then what's really interesting there? And then as soon as you launch fuzzing, a violation is identified, something like some pass that would be dot dot slash dot dot, where the validator says, yeah, yeah, it's legit, but then my code is expecting that it's rejected. And then when I found that it was unclear to me what should be the behavior. So we discussed that in this PR and then the conclusion was that, yeah, we don't want to accept it. So fix has been made. Again, if you look at the fix, doesn't seem obvious that it's the, let's say, perfect solution for it. So then fuzzing is again useful because you fuzz the new implementation and then you get no violation in a reasonable amount of time. And then you can conclude that the fix should be good enough. Yeah, so writing predicates can be hard because as we see here, it was rather straightforward for the example one, where it's just a one-liner. So if the helper is small and you can easily express predicates with sentences like should not contain this character sequence, but if the helper gets bigger, you will need this complex validation logic to implement it. And of course, if you just copy the original implementation in the first test, it's useless because you are comparing output from the expected output. And if the expected output is always the output because it's a copy test of the implementation, you're doing just nothing. So you need to write your own implementation and for that, you need not to get biased by the existing one because otherwise, again, you would be just comparing similar things and maybe a corner case is missing and you need to think about it. But the more complex question I faced was, yeah, well, to which extent should I trust the standard libraries because we saw that 5.5.3 was tricky. And so for that, I wanted to share my lessons learned when I was trying to validate the file storage API of Grafana. So it's a new API that will allow to do file uploads on the server. So of course, pass-driver-sol is a big concern. And there's a rather complex validate pass function in this API. So add to be implemented. And it was really time-consuming. I did not identify any violation, which like somehow is good because it means like the current validate pass seems to be doing the job. Yet I'm not 100% confident that I did not miss some corner cases because it was hard to re-implement this complex logic. Okay, so next step. We still have some fuzzing for it and all that, so it can be useful. So what can we do to get the more value out of fuzzing? Well, first, yeah, as we said, you should make security helper as simple as possible. And then when it's done, you can easily include fuzzing in the CICD pipeline. So you make sure you will not have any regression. So let's say in a few months, some other developer decided to re-factor this example to logic and he breaks it. And then fuzzing could find a violation quickly and we can realize that the validator is not doing the job anymore. Of course, once you have those security helper that you can consider trusted because they are fuzzed, you should communicate about them so that more developer uses them. And then if you want to make sure that they are really used, you could use some static analysis tooling like SAMGRAP so that you can check if, like in your repositories, if those security helper are really used instead of the dangerous features. So to wrap up, so, yeah, you've been warned that fightpast.in can be tricky when you try to protect from past reversal. I hope that you are convinced that fuzzing can be useful in real life as it improves your automatic testing coverage and it's identifiable to you to identify counter cases that would not be obvious. And in the end, it should make you more confident with your security helper libraries. And we've also seen that, yeah, while go fuzzing is easy to use and really nicely integrated and it can be really efficient as long as you target simple functions. And now I would be really happy to take all the questions you have about that. Thank you.