 Alright, good afternoon everybody. I'm Jared DeMott gonna be talking to you about fuzzing and I am just happy A static and amazed that this house is packed Dan Kaminsky talking in the other room And I didn't I wasn't sure I was gonna get too many people in here today, so That's awesome. I'm gonna saw Atlas or Johnny long raise your hand yesterday. Do you guys see that? Who loved the video of Bruce Potter? Do you guys like that? Oh, man, that was to me. That was hilarious That was just off the hook. So let's jump into this. I got a lot of stuff. So I'm gonna move pretty fast The the range of people I think in here goes everywhere I've talked to people who don't know what the word fuzzing means and I've talked to people that have been doing this for five plus years So I'm gonna do a background current state talk about Context-free grammar fuzzers talk about generic fuzzers talk just real briefly about how web and file fuzzing kind of fits into this a Little bit talk about some formal metrics and kind of advancing the state and where some of my research is going with this So for those of you who do not know what the word fuzzing means It's basically an automated testing technique to find bugs and software. That's it That's the short version and there's a lot of debate Longer about what what else exactly it means and maybe fuzzing is only random and maybe it's you know Maybe it's structured and we'll talk a little bit about that. So Peter Olhart from the former Microsoft says a highly automated testing technique that covers numerous boundary cases using invalid data from files network protocols API calls and other targets as Application input to better ensure the absence of exploitable vulnerabilities and then the word fuzzing comes from 1990 with Barton Miller I a lot of people apparently say I can't even attest to this personally But for modem application tendency to fail due to random input caused by line noise on fuzzy telephone lines He did some research in 1990. So to get everybody on the same page what I like to do too is just give a really quick example of Fuzzing just up front show what it is and then we'll talk more in detail about it So say we have this fake little protocol and its user Jared. That's a client and the server says okay Good user. Give me your pass password my lane pass and the client in the server says all right cool and then client Here's a command. So that's that's that's the fake protocol Now, how would you fuzz that? Okay? Well first we'll talk about just the data and then we'll talk about monitoring the server But basically you could you could like see in loop three. I send user Jared and he says okay, and then I give some random data So that is not truly random fuzzing right because we have some protocol knowledge We send the first leg we get the second leg and then we send some crazy stuff in the third leg, right? So that could be maybe just seeing how it handles arbitrary binary data. That's test one say Or maybe these tests are randomized a lot of different ways to write fuzzers and we'll talk about that too In the first loop you see I send 50,000 ffs. Maybe you're looking for long strings or again how it handles non-ascii data Second loop we see 12% ends in the middle of the Jared for the user named Percent ends we all know what that is right. What am I looking for? Who knows? Format string bugs right and if you don't know what that is there's lots of good data out there So those are what are called attack heuristics sending a long string sending a format string Sending a null byte at the end instead of a carriage return Changing the delimiters right these are like in a text protocol that the normal sequence of events is like string delimiter string line Ending maybe changing delimiters or line endings or chaining commands There's a lot of different attack heuristics that should and could be built into a good fuzzer. So Just again, so we're all on the same page with the word fuzzing since I use it like a million times in this presentation Other words there's there's sort of like two industries that are now I hope colliding because I think software will be at its best when the QA world software testing Who does software testing quality assurance raise your hand anybody out there and no that's good I like that and then there's a security research is doing fuzzing to find bugs and software. Who's doing that? Raise your hand. Let's see. Yeah, there's more of those guys. All right me too So anyway, if the QA guys learn a little bit more about what the security guys are doing Things are gonna get better and if the security guys know a little bit about regression testing and some of the QA things And then their testing could get better too So the word that the QA guys tend to use is robustness testing or negative testing or all kinds of other words stochastic means random Monkey I've heard boundary stress some of these are a little bit different, but they can be used in the same context I'm just gonna use the word fuzzing today So how does fuzzing fit into software testing if I'm Microsoft and I want to fuzz to make my software better Which if you went to the Vista talk at black hat was pretty good. Actually they talk about that they do that, okay? Well software testing as we know is difficult. It's tedious. It's expensive. It's like half the budget But it's all we have it's really the only way in the primary way that you have to gain confidence in your software that your Software is any good pre-release post-release you have this whole group of third-party security reaches that'll you know be glad to find bugs And resell them and do this whole thing that we do as an industry now So testing again, we probably know if we've seen this before there's black box dynamic Which is like you know you have a real binary running on a real server and you're sending real stuff to it Really interacting with it and there's white box Which is more like code audits and scanning through source code and and both are good should do both There's some pros and cons to each the pro of black box is that if you send real data to a real server running on a real box You'll find a real bug and you can really exercise remotely if you're scanning source code and your your your lint or your whatever tells you Hey, here's a stir copy. That's kind of naughty. Then you're like all right. Well, let me go look into this and then You find out. Yeah, that is a problem when the config file gets loaded by the admin who has to be admin to load the config file Then that could be a problem, but you can't really exercise that bug remotely So there's a slight difference in terms of usability, but all bugs should be fixed from a QA standpoint, right? Bugs are bugs, but it's just in terms of attack surface, which will just explain Pretty briefly in a minute. So how does fuzzing fit into that quality assurance life cycle or how does fuzzing fit in? We know how fuzzing fits in from a research standpoint, right? We want to find bugs because we find a bug we do whatever we do want to do with it, right? But from a QA standpoint you have formal methods and software engineering You know you should have this formal process of developing software and part of that is your quality assurance group And part of that is your testing group and part of your testing group is fuzzing It's just one small piece of it. Okay, and fuzzing actually is kind of gray boxy I mentioned that and there's a lot of other testing that should take place unit integration blah, blah, blah, blah, blah so is fuzzing better than Reverse engineering is fuzzing better than source code auditing is Fuzzing better than gambling. I don't know. I think so, but I'm a little biased there So the answer is no, it's not okay. It's it's really not it just supplements all those things You should still reverse engineering. You still need source code audits. You still need testing. You still need pen test For a lot of reasons and one of those reasons is Probably fairly obvious in that fuzzing is not a catch-all solution. It doesn't work against Every kind of app if you have this special Java web thingy and you want to see can I go from login PHP to logged in PHP fuzzing doesn't really do a lot for you there, right? That's more of a pen test kind of thing sending random data To that might help you and actually we'll talk a little bit about how that could be interesting but traditionally especially if you're looking for memory corruption bugs which are still the majority of all the bugs out there Buffer or flows heap overflows stack overflows pointer overflows You know all these kinds of corruption memory corruption type errors in C or C++ apps That's where the goodies are and if you have a server that was written in C That runs on a box that doesn't have any heap and stack protection and all this kind of stuff Fuzzy because I guarantee you'll find an ode in it Well, not guarantee it, but there's a good chance. There's probably some bug in there somewhere Okay, so there's a lot of people that have been doing fuzzing for a lot of years actually and when Miller first did this thing in 16 years ago In 1990, I don't know if anybody really cared about his research He didn't get much academic respect because all he really did was basically take a random stream generator and Pump those bytes to like LS and other unix utilities and it was sort of like everyone in the academic community was like wow That's kind of lame, but it was amazing. He like crashed like half the apps on all the unix boxes at the time So things have gotten better since then you can't just send truly random stuff anymore won't help you a whole lot these days It could but it's been fuzzing has been effective for a long time and we just saw the month of browser bugs, right? We all know that so fuzzing is doing its thing still. It's alive and kicking. I'm making a living doing it. So yeah Okay, so who who does this fuzzing thing software companies ought to be doing this And I think they are now probably starting about a year ago They really jumped on this bandwagon said hey, we're getting killed the bad guys are their fuzzing and they're finding bugs And we got like a multi-billion-dollar budget and we're not finding them. What's up? So they're fuzzing Vulnerability analysts are fuzzing Soft fuzzing companies are creating fuzzers to sell. Okay. I don't know exactly how many fuzzer companies are out there There are some I'll talk a little bit about Cody now McCann and Finland just briefly they do some good academic work at a Lulu. So there's academic work. I'm doing that too and there's hackers Lord knows So fuzzers are built bottom line end of intro because they work, okay But the question is how do they work and Okay, so at the bottom of this slide You see there's this thing called a tax surface. So a program is big most programs are pretty big, right? And I talked about that mythical program that read a config file and there was an error in it and stuff like that That could be considered part of maybe like the local attack surface Maybe not depending on how that exactly worked There's definitely an attack surface from an external point of view, right? And that's the one that's most interesting for remote vulnerabilities There's this bit of code and there's the parsers that are in that code that read the traffic that comes remotely That's the attack surface. I'm talking about there could be a tax surface Jesse Burns gave a talk at black hat talking about the the inter-process communication attack surface There's an attack surface there between programs within a box and there's local attacks or so figure out what your attack surface is and Then the second most important part and I think it's kind of overlooked because we all know we've got this source data That needs to become fuzzy that we somehow deliver to an application. We know that right we got that down But the other part is how do we monitor the application to make sure? That we notice when something bad happens and that's actually a really tough problem. It's not just You know run something and hope it dumps core. That's what it used to be. That's not what it is anymore You can still do that actually and that still works in certain cases, but it's getting tougher One way is to attach a debugger. Okay, you take you start the process in a debugger That can be tricky too because some processes are really hard to start in a debugger at the end of this I'm gonna give a demo on a tool that I wrote and I'm gonna it's on dubcott dubcot's the application I'm gonna run GPF against GPS to fuzzer. I wrote general-purpose fuzzer so and show show a bug in that multiple bugs in that actually and It's really hard to start dubcott in GDB because like the parent process starts and it starts like an off process Which starts like three children process for each protocol and if any of them die The parent will just kick them off and restart them the whole thing So it's a little tricky to just quote-unquote start dubcott in GDB so you can monitor logs you can You know application logs system logs. I'm just gonna be doing two ways I'll actually show that it dumps core because I do a U limit dash C unlimited to get to dump core And also by tailing the bar log We'll see we'll see a message in the mail log itself But there's there's a lot of different ways and one final note about the debugger that was really interesting You guys maybe notice that there was a send mail bug this year Well, that was actually a timing issue a race condition and if you attach a debugger to it Okay to monitor and try and find this bug You really can't find it because it screws up the race condition right the debugger changes the timing of race conditions in some cases So you can't even unilaterally say that we'll just start it in a bugger and fuzz and find bugs right so okay enough said on on that Okay, so again one one more sort of intro slide so why why do they work again? Well a few reasons There's this general goal of qa guys to do functional testing in the past I think this is definitely changed now But in the past qa was like does our program work like it's supposed to and the hackers were like I don't know How this program works? I don't care how this program works. I want to find a buffer overflow, right? So there was a definite Difference in approach and goal and that made you know two different approaches two different results not terribly surprising, right? And that part of that is sort of what I call gap coverage, which is like even if that wasn't true Internally we bought this really expensive fuzzer on Microsoft We paid a million dollars for a fuzzer and it it rocks, you know It found like 99 of a hundred bugs and here I am maybe I'm this like Lamo hacker and I wrote this fuzzer and it sucks But it doesn't really matter because it found one bug And that bug was exploitable. Okay, so you see what I'm saying There's a notion of a gap that the tools are different and because of that you can still find bugs Even if your fuzzer is an uber cool So and the other the other deal is code coverage I just like to talk about this because it's interesting come from kind of an academic Excuse me standpoint, which is basically like in the past qa guys were given sort of like incentives or bonuses sort of like Our goal is to get 80% code coverage with your test to a sweet against our product If your tool achieves that and the faster you achieve that you can get a bonus or something like that I don't know exactly how works. I've actually never worked in qa. I'm sorry if I'm insulting the qa world, but And so they would do that they would write a tool that was based on just getting code coverage They wanted to cover as much code as possible But the problem again go back to format strings or anything else Well, they went through that the sprint f or whatever, but they didn't give it a percent n, right? So it didn't crash because they didn't give it the right data when they got code coverage So code coverage tells us something very very very useful, but it only tells us half the battle, which is this If you cover something you know that you cover But if you haven't covered it then you haven't tested it and you certainly haven't found bugs So if you haven't covered it, you know it tells you that right you tells you that you haven't even tried So you need to get code coverage code coverage is a good metric and it's an important metric to test against anything fuzzers Testing sweets, whatever, but it doesn't tell the complete story. So these these guys marketing my fuzzer gets 80% code coverage and yours only gets 50 You know, it doesn't really okay. Well, let's think about that. It doesn't necessarily tell the whole story. It does tell us something though Okay, so General fuzzing types of tools There's generation fuzzers, which is what the majority of fuzzers. I think out there are which is this I Make an FTP FTP product say I want to test it. I buy an FTP fuzzer It fuzzers FTP it finds three bugs. I fix them. I run it again. They're not there regression testing the whole thing that works Okay, the other type is Mutation or capture replay which is sort of like I use ethereal I capture any session whatever and I'll talk about this is my tool that I wrote GPF is kind of like this You can capture FTP Ike whatever and then you can replay that right and you can replay it with false injected in certain spots And if it's a cryptographic protocol, you can't just replay it directly, right? You need some user function that lets it know. Hey when you do this you do a hash and it's kind of complex Actually, Ike's not a real good one to use for for mutation Which is why I actually wrote an Ike fuzzer too separately because Ike is a is a tough nut to crack with a general purpose But it can be done and and general purposes are great against a lot of the protocols too and hopefully we'll see that there's fuzzing frameworks you know and there's old school straight-up random generators and Yeah, there's a lot of different kinds. So of the categories, how do they create this semi valid or some people call it semi invalid? It's the same word. It basically means like most of the stuff in the protocol is right Okay, and that's key that the rate of faults be relatively low because if all your stuff Is totally random you'll never make it past like leg one of the protocol, right? I showed that little example client said this server said this client said this server said it Well, if all you ever send is totally random junk in leg one the server's just gonna boot you right? You're gonna get reset and you'll never make it into the further legs So if you want to fuzz like the 20th leg of some arbitrary protocol 19 of those 20 steps need to be exactly right, okay? So you can see that the rate of faults need to be fairly low in general when you're fuzzing So to create these faults you could just randomly do it, you know, you could say pick a random leg Pick a random position on that leg pick some random data of a random size with some random bytes and stick it in and that actually works fairly well It's kind of cool You know, maybe you don't want to do that. Maybe you want to do be very cyclic very deterministic and exactly how this thing runs I want to fuzz every byte of every variable the variables are basically like what we saw like it was user space Password, you know line ending all those pieces are considered a variable string delimiter string line ending You could say for every string pick a spot in the middle and insert one to ten thousand bytes by one using every byte Okay, that's deterministic could run forever. All right, and that gets into this whole like infinite runtime deal If you talk about true randomness So in the library is is a good one test cases a good one library is kind of like I have this list I've ten thousand naughty strings that I like one's big one has format stuff one has whatever one has hacks You know and I just replace each variable on the line with a string from my library. That's good. That works really well Test cases are allowed the ones for sale. Do this. They're like sort of like my tool has ten thousand cases against FTP How many test cases does your tool have? You know so you can you can have like a number of already preset test cases you run it you run it again That same kind of deal. There's a lot of different ways to do this and a lot of them are really good so Funny stories about fuzzing. I was I was talking to Dan Kaminsky at lunch day two black hat and I Told him hey, I do fuzzing too. I'm giving a talk on it He does fuzzing too and and he's like dumb or smart and I'm like hmm. Well, I knew what he meant Okay, because I've been in this I've been fuzzing long enough to know what he was asking But it's kind of a strange question. A lot of people ask Ask ask that is it intelligent fuzzing is a dumb fuzzing and dumb doesn't necessarily have any negative connotation like your fuzzer is dumb And mine's intelligent It doesn't have the connotation that one is worse than the other what it really means is dumb means more randomness and Smart means less randomness and either extreme sort of like in politics, right? If you're way on the left or way on the right, you know You can get you can get lost but if you're sort of that line of holiness right down the middle that we're looking for and fuzzing is kind of the Same way if your fuzzer knows too much about the protocol that means it follows the RFC exactly and does everything exactly right I never expect more than 200 bytes here, so I never send more than 200 bytes Well, then you're not really fuzzing right you just have a client and if it's totally dumb totally random We already talked about run times infinite and you're not going to get anywhere fast So you want something in the middle faults relatively low. We've talked about that So the question here is which fuzzer of all those that we just talked about is better commercial open source Generation generic There's actually really no research done on this and it would actually be a really good research problem If somebody else is looking to pick up a PhD in fuzzing. I this would be a great problem I mean code coverage for instance could be a metric you could say well one fuzz one fuzzer gets better code coverage than the other But what like we talked about that doesn't give you a complete picture Okay, and you could say well number of bugs it finds That's actually a fairly good metric, but it's fairly hard to determine against software that hasn't been tested yet Because you really don't know if there's bugs or not right so you could you could take software You could maybe write software that only one person knew of the bugs that are in it and the people that are fuzzing don't know Where these bugs will be and you could maybe have kind of a shootout that would actually be pretty sweet. Um, nobody's done that Okay, so that's all the intro stuff and we're gonna go into context-free grammars I'm not a context-free grammar expert I admit that I've never written a fuzzer that uses context-free grammars, but the people at Ulu University secure programming group came up with protos I know you guys have probably heard about that if you've been fuzzing for a while and They did some really good research on that and I'm just kind of gonna I'm just gonna blow through this pretty quickly and not really Talk about how you know the BNF language and all that all that stuff works But basically here's an example you can properly and completely define a protocol Using context-free grammars like transfer as a reader or write a Read is a read request and reads a read request is a dollar one a file name a file name is a character A character is a one through seven F right you can you can and if you want to write a fuzzer that does context-free grammar fuzzing You could say basically do this right thing or do this bad thing So here's the here's the description. It's normal, but 30% of the time do this thing It's not normal. Okay, and you can just iteratively define a protocol With those kind of faults injected in that way, and that's actually a really a really good way to do it So protos talks a little bit about their their interface roundup There's a lot of data on this slide my slides are on applied sec comm my tools my slides my paper everything's there Go grab it. In fact, you want to grab my tool off applied sec comm not off the DEF CON CD because it's slightly updated so One of the things that I liked about the the third point of this limitation of vulnerability testing is what I talked about earlier that Not everything can be monitored It's really I think somewhat impossible to monitor the server and every snare if you attach it a bugger You could miss a race condition if you don't attach a bugger You're probably gonna miss some other bugs if you're just looking at logs Maybe it doesn't generate logs all the time in certain scenarios if you're looking at CPU utilization memory utilization You can catch some kind of runaway integer, you know There's a lot of different ways you can monitor the process and ideally you do it all particularly if you're serious about finding bugs in some given app And I like this just because you talk about like biology kind of mixing into the computer science world They talk about their tool having basically a pesticide paradox Against given software that is if they have a SMTP fuzzer. They run it against a SMTP server It finds 10 bugs you fix the 10 bugs you run it again. They're gone that softwares immune Okay, and all tools are like that and one of the cool things about like my tool and interjecting randomness is It has a seed for that random value, right? So my tool you seed it and there's definitely randomness throughout and if you want to replay that exact session again for a regression testing Use the same seed, but if you want it to be randomly differently seeded differently and you can potentially maybe get some different results. So You know, there's varying mileage varies sort of so to speak on that kind of stuff so Into a little more of my specialty and where kind of some of my research is gone is in generic fuzzing Which is a general-purpose fuzzing. There's another guy that wrote a tool Martin bug node called auto-defe that I'll talk about That he has not released yet, but I apparently intense to do so So automatic protocol detection. That's what I talked about before capture session with ethereal Convert it to a neutral format. That's that's a good thing to do because libp cap files are not all that easy to model manually modify Convert it to some text-looking format. Even if it's binary data, you can do this All right, and you'll see that if you play with my tool and like say you captured a stream of an FTP session But you know you screwed up your login name or something like that You can just change it in the file real quickly to the right pass or the right username. You get obviously different tests Different pass through the code different coverage Depending on exactly how well your fuzzer understands the protocol how much protocol knowledge and how many various Interations through different state spaces that it transfers So that's a nice thing to have the ability to do that and I use that all the time in fact sometimes if the protocol is simple Like in the case of the example, I'll show you with imap fuzzing Fuzzing dev. I didn't even capture a session I just created the us a quick protocol description in a text file the way that GPF expects to see it and you can just fuzz directly from there with it You just sort of give it the protocol knowledge and it's very simple because the bugs I found are pre-login, so I just gave it basically the 12 valid imap commands or something like that and told it to go to town, right? So there's there's there's ways that that can just be really and good good good generic fuzzer should be able to based on any given Capture should be able to fuzz either in the server direction or the client direction, right? Both are important It should be able to fuzz either way pretty easily So attack heuristics. We've talked about that long strings weird strings format strings intelligent randomness we've talked about The only thing about intelligent randomness is Again going back to another this is another this would be another great research project is basically that there has not been any research Kind of trying to find where that that holy line I talked about is nobody really knows where that is how to wait exactly how often there should be faults or where there should Be faults or how the faults are created how much randomness. There's no definite Research there as of now that's basically an art. There's no scientific data at all that I've seen that indicates what method may be preferred So I wait it basically best. I think it should be Remote debugging is a sweet thing that at an auto-defei Does and that's basically like you've got your fuzzer Fuzzing say a server or client whichever direction you want to think of it in and you have this process that is a debugger That attaches to the process, but it's also communicating with the fuzzer, right? So you're getting this back in knowledge. Hey something just happened in signal 11 Let's log it and let's keep fuzzing like there's this cool stuff You can do it with remote debug and really really good stuff Dynamic weighting is another thing I'll talk about and distributed fuzzing is kind of an idea I had that I've never actually seen played out anywhere and you can do this a little bit with GPF It has a super GPF mode that'll basically start, you know as many Instances of GPF as you want and it actually it kind of like file fuzzes your capture file and it sort of fuzzes the Switches that it the GPF expects on the command line So it starts a lot of strange instances against different captures that fuzz normally so it's really kind of wild The problem with that is is it's really hard to identify when an error has occurred if you have a thousand versions of a fuzzer Sending a thousand packets a second and you're not doing some sort of remote debugging or some sort of logging Of when exactly a fall occurs or even if you are if you have some second generation bugs or a heat bug I talk about send a second generation. I'm talking about uninitialized stack or heat bugs I don't know does anybody know what an uninitialized stack bug is raise your hand if you know what that is So there's a few there's a few hands out there that know what what that kind of flaws It's a kind of a cutting-edge flaw which basically goes along the lines of Somebody in their code just says like you know int i at the top of a routine or something They don't say int i equals zero so they don't actually manually Initialize it well suppose there's some routine in code and I've actually found one of these in the wild Suppose there's some routine in code that goes through and there's a path where there's no bug But it just so happens you overwrite maybe by four bytes and you overwrite the initialization of some variable That gets later used then you start again from the top of this program And you're going through down this other path through the code and whoa something crazy happens because that variable was initialized And the code didn't know that it doesn't expect it to be initialized It wasn't written for it to be initialized So there's you know, there's some really cool things That can be found with fuzzers But trying to track down exactly what data caused the fault and where it happened and how and why It's tricky. It really is especially if you've got some of this wild stuff happening, you know And there's some weird bugs and it can definitely be Kind of tricky All right. I think I'm doing it on time. So briefly web and file fuzzing Basically, we'll talk just real quick. How is web fuzzing different than say fuzzing a standard? C type Damon. Well Basically, it's just this if you're fuzzing an HDB server like you want to find an Odey and Apache or IIS Well, good. I'm glad that you're looking at that and good luck to you It's getting harder to find those okay that code sets getting better I think we should still continue to look at it and there probably certainly will be bugs found in it But they're hard to find very hard. Okay. I think I think we can all agree to that now But there's all kinds of these crappy PHP web apps for like my blogger Commer thing and they suck. All right. They are not written. Well at all But the question is how do I find there's these? Has anybody heard of a file inclusion bug? You're looking for there's a few file inclusion. You guys know what that is So it's kind of a pen testing sort of thing trying to find bugs and web apps and A fuzzer that's just basically sending semi-valid data Watching Apache waiting to see Apache die. You're gonna be waiting a while. Okay, so because The web app is having all kinds of problems, but you never see it by watching Apache You need to watch the HTTP return codes coming back from those web apps to see what's going on and I actually have not written a tool that does this But I used spy dynamics fuzzer called spy fuzzer just a little bit and it's kind of slick. It's it's not I'm not gonna say it's the greatest thing in the world because you basically got to define all your tests manually And there's really no good not a real good like but it will if you know of a certain test You want to do like you want to say I'd like to like check this this checkout cart Like what if I buy you know one through ten thousand items of this type without logging in or so You know, you've got this idea how to how to sort of pen test this web app You can automate that with spy fuzzer and that's kind of cool. So it's a little different But fuzzing is actually very similar to normal fuzzing, but there's just slightly different tools And I mean there's a lot of different ways to do this too very similar to the fuzzing we're talking about It's sort of like you have a word document Okay, one way is you could just manually mutate that you could I don't know just change every tenth bite And then give that file to word and watch word and see if word dies if it does you found a bug Right, so there's a lot of different ways you could try to automate that and some guys gave some good talks Sutton green and black hat you can go to iDefense you can grab their tools are up on the website. So You want to learn more about that there should be and I don't think I can't remember I'm trying to remember off top my head. It's been a while since I looked at their tools But there should be some ways to do some more intelligent fuzzing to besides just You know randomly obviously there should be some protocol knowledge. I think they do some of that too so, okay fuzzing metrics Basically, the big deal is Martin Vagno wrote a paper on fuzzing It was it was a pretty good one his paper and the paper by Peter O'Hara Are probably the best two papers besides mine, of course look at applied sec. You'll see my paper There is that but they wrote some really good stuff and his is basically not rocket science But it is it's just it makes sense, right? He's like I want a metric to know how long will it take for my fuzzer to finish Well, if you use true randomness basically the answer is your fuzzer is never finished, right? And that's one of the things I'm studying in my research too is sort of like when's a good cutoff time You know, how can we say that we think we've covered enough or whatever? But his tool is actually very deterministic and what he does is he says all the variables that is the user space pass You know all the all these things those are called variables And I have a list of a really long like ten thousand You know member list of bad strings and I'm gonna take those strings and just put them in place of the variables Right, so I have I have library times variables times a half second or whatever it takes to run each test equals run time That doesn't sound like you know rocket science computer science time kind of stuff But actually it's believe it or not. It's fairly revolutionary in this field There hasn't been much formal study at all in fuzzing. It's basically just been like I send crap I found a buffer overflow. I'm the man. I posted it put bug track, you know all that kind of good stuff so Now there's he does another really slick thing that That's kind of cool. He basically so not not only does he have a run time But he orders that too, which is kind of cool It's basically like if the variable is used in one of these naughty functions got a big list of naughty functions, right? Then we fuzz those first again not rocket science, but pretty revolutionary, you know in terms of fuzzing in the way he the remote debugger Relays that to the fuzzing tool itself. So pretty good stuff You know there are some issues obviously of course with the fact that you know, maybe more randomness could be good for second-generation bugs Maybe Windows Vista says this whole list of naughty functions shall never be used in Windows Vista That's what they said at the talk at black hat They have ways of completely eliminating these from the Windows Vista source code all together So in that case it doesn't really help us all that much waiting weight rise, right? And of course attaching a debugger, which is a must for this kind of stuff could potentially make you missed You know it's sort of like attaching a debugger. It's great for 95 percent. There's that 5 percent of race conditions or something you could miss So Basically some future research my thoughts are kind of like I like to combine Those kind of things that are out there like like Veg knows approach with my tool my approach and even kind of basically the Approach is something like this. Let's do some deterministic Structured stuff up front and if you don't find any bugs with careful inspection then you can sort of open up the throttle, right? You can you can introduce more You know randomness per byte or more more junk basically you can kind of throttle back and let it run longer if you have time to try and find more of those second-generation kind of things and then um Really surprisingly to me. I went to Sparks and Motin Cunningham. They're talk on Sidewinder at black hat great talk They talked about using genetic algorithms in fuzzing and I actually taken a genetics algorithm class this fall I was planning to do research on genetics algorithm But I was happy when I saw their talk now upset because they laid a great groundwork to kind of get started in how some of This might might be used as of yet their tool wasn't super like useful against multi-leg protocols and stuff like that But they had some great groundwork laid on how Basically what they use and for the genetic algorithm stuff. So hopefully I can kind of build on that this year. That's my plan So basically what I'm gonna do right now is give a quick demo and then we'll take take whatever questions you guys have okay, right? This is the only time wireless mics are really kind of handy. So basically if I just Let me make sure I click in my VM here cat all three What I'm gonna show you again is just I'm gonna run my tool against Dovcott and show you some bugs that I found I was Not able to take these bugs and turn them into odays, but it's like a take-home assignment for y'all if you're smarter than me See if you can take these bugs and turn them into odays. Good. Good luck with that. So they're pre-logging and You can see GPF and then I got pre-logging fault one. It's basically just gonna run a capture file that I have and if I cat pre-logging fault one Well, okay that there we go So you can see what a capture file looks like for me in my neutral format. It's it's not in libpcap It's basically like source means it came from the server and You can reverse that easily when you run the tool the size is five and the data is Basically zero five and then probably like a slash r slash n and all or whatever the data actually is there so So that's just what the capture file format looks like it's real simple It's real straightforward simple to modify if you wanted to change that a little bit change the bite change the size. Okay, no problems there So what I'm gonna do is I'm gonna Get ready to run a couple versions of GPF and the switches that I showed you you'll have to check out the read me It's fairly good documentation. I think for a read me There's a lot different little switches that GPF expects But they're not that hard to use and I'm actually gonna be using it in just straight replay mode It's not gonna be fuzzing because these bugs. I don't even actually have to fuzzing to find them It's just when you send a certain sequence of bad things Dub cuttle crash, but I found that based on fuzzing. Okay, so let me start do a U limit So I can get a core dump dash C Unlimited, okay, and then start dub cut the server should be running So if I go over to this is a tail tail dash F of our log mail log We see the entry dub cut starting up. So the question is what happens if I do a PS dash EF grab dub cut What happens if I do a this is the thing you need to do before you fuzz by the way Never fuzz until you've done something like this. You need to figure out. What will a fault look like to me? Will I be able to see it in logs or not? Will I be you know be able to get what am I doing? I need to I'm talking while I'm trying to do so. I'm just gonna put in a pit So if I kill dash 11 dub cut off Who thinks they know what will happen? Does anybody know? Take a guess who thinks they know what's going to happen if I just do this It'll dump core because I said a U limit right and it doesn't depending on kernels and you know EUIDs and there's a lot of different problems that can make create a problem may a process may not dump core But when I do that it does dump core. See there's a core there. Let me remove it So it's not confused with a different one core dot 297 and if I do the PS again, we see that dub cut off is back and it has a different pit, okay So we know that the dub cut watch demon will Restart it and we see in the log see what we see in the log. We saw this child was killed with signal 11 Okay, so pretty straightforward. We know what to expect. We do that before we fuzz, right? Okay, and now we actually Will send it so I run the script here's gpf. It's running You see client server client server sending data sometimes it prints in hex and sometimes it prints in ascii I don't know if you you notice that let me um Page up just based on what it sees if the first few bytes look like they're binary It'll print it in binary if the first few bytes are you know ascii It'll print an ascii kind of deal so pretty pretty straightforward But pretty cool stuff and then you see like couldn't read socket negative too, right? So that means that there maybe maybe was an issue and if we go back and look at our log It's like oops. What was that? We got off died and the login process died so both pre-login. I didn't even use a user in a pass We got two two problems right off the bat and dub cut there. So That's my demo and that's my tool and that's my talk. Thanks I raced through that because I wasn't sure if I was going to make make it through that So do we have any questions anybody have a question? I think there's a microphone you want Go ahead. You want to step up to the mic or ask it from there? Whatever you like Yes, you have to go to the microphone. He said How are you doing your tokenization to differentiate between strings so that you know where to fuzz? Well, that's a good question. Um GPF has a few different modes. You don't have to tokenize right GPS GPF has Lots of different modes and if you see that in the in the read me You'll see like you can just tell it to do some cyclic stuff like just insert large strings or Or just insert format strings and do it only in this leg and do it only in this position You can do a lot of different stuff or you can run it in a dash p mode which I call pattern matching which is tokenization And it'll basically do what you say it'll say, okay This looks like there's a string here Okay, and you have to actually tell it on the command line what a delimiter is so it says I'm expecting strings to be this I expect a delimiter to be a space of one byte long And I expect a line ending to be a slash r slash n so once it knows that it expects that that's the pattern It'll just find that and based on that it'll do different things for delimiters and line innings and it would for data So is that an answer? Okay, great Um, have you reported the uh problems you found to the dot cop people or um, I reported a couple Do you get any response? What's that? Did you get a response? Yeah, actually, um, yes, I did So they're gonna put out a fix these particular bugs. I don't know if they're fixed in the newest version or not I really don't I leave that as an assignment for y'all to have fun with so And did you play with the dovetop pop 3d min at all? Did I what the dovetop pop 3d min? Did you try fuzzing it? Yeah, I did I fuzz that a little bit too and I didn't find any problems there But again, you know, there's a lot of different modes. I've added some stuff since then so I say fuzz away Okay, thanks. Yeah, no problem Dovetop has actually had I found a new a number of problems One of the problems was actually pop through to a reporter that was fixed, but that one's fixed now, so Just uh with genetic algorithms, I mean the key is the fitness function, obviously and seems like you either Crash it or you don't crash it or you get garbage out when you shouldn't get garbage So how do you define a useful fitness function because it doesn't seem like the search space is that Fine grained Yeah, that's that's a great question. That is the problem and a few I don't know if you're a black hat or not But uh, they they talked a little bit about that and I miss that I missed the talk that I wasn't a black hat That's otherwise. It's probably potentially already answered, but I just sincere you're talking about using it I thought you might have a reasonable answer to that. Yeah. Yeah. Well, well, there's a lot of different fitness functions, right? One they they mentioned was basically like a point in the program If we ever get to this point Then then that's fit. So the the code pass or the data that allows us to get closer Excuse me to this code path are more fit than others And they do that by setting breakpoints on everything along the way to that and checking the data So the data could be fit based on that and there's probably a lot of other ways you could you could generate Fitness and that's really one of the things that I'm going to be studying. You're exactly right because Even once you do that it's sort of like well, we found data that was fit and it made it to this point Well, that's great, but it didn't necessarily crash it. It didn't necessarily find us a bug, right? So we got to find a way to work that in right defining data that's fit in a sense that it's problematic So how do you pick those points? I mean like in kernel hacking? There's obviously a bunch of places where Quote-unquote we should never get here. I mean is that type of location that you think about or I mean I mean and obviously that's harder if it's a closed source application Yeah, that's a really good point one of the things you can do is you can pick Potentially going back to that list of naughty strings if you you know throw it in idea pro and you find it Oh, there's there's three stir copies here. If we ever make it can we make it to this stir copy? That's a good question and that actually is a really useful thing to know From a remote does this attack service ever lead us to this somehow, right? So that's kind of a useful thing in that sense Okay, so so that is potentially automatable if there's a set of functions that are potentially vulnerable then you can Effectively Automatically pick those out then use those to pre-define a set of fitness functions that kind of what you have in mind Yeah, yeah, there's probably a lot of different ways. So good question. Thanks Hi, um, how do you get around code that inherently rate limits you? I'm sorry. Can you say it one more time? Yeah, how do you get around code that inherently rate limits you? That right limits me. Yeah that tries to put in delays to stop you Throw in so much data in a particular Time sequence or time frame. So a lot of authentication protocols will actually put in delay and back you off Yes, so is that particularly easy to get around what what the what the issues are? That's a great question Actually a really really good question timing is a huge issue when you're fuzzing because if you fuzz too fast This fuzzer just may blacklist your ip right or the the the server may just say oop. This guy's doing something weird I don't like him. That's a great question And that's a problem you run up against one of the parameters that you feed gpf is timing I default sort of have it a thousand milliseconds per leg, but you can back that way off if you need to Or you can do there's user functions. I talked about there's user functions in gpf Like if your protocol has like an encrypted hash like it expects you to do this do a hash and send it back Like you can you can have a user function do that and you could also have a user function Just maybe every 10 times you have the program sleep for like a minute or something whatever you need to do to keep the timing saying You can do that Yeah, thanks. Good question any more questions All right, well peace blessings and good luck