 Hi everyone. My name is Eddie Lee. I'm from 4-5 Software and I'm going to talk about comparing application security tools. My talk is going to be pretty packed so I'm just basically going to speed talk through this and I might breathe through it. So bear with me. So the agenda for today's talk is I'm just going to give a brief intro to my experiment about comparing application security tools. Go into the methodology and how you can reproduce your own experiments and go into the results of my experiment and then go into conclusions. So what I did in this experiment was actually I used two of the market leading dynamic pen testing tools for web applications. I used a static code analyzer, a source code analyzer, and I used a dynamic test tracing tool. I'll talk about these tools on the next slide. The application I scanned was actually an open source Java-based blogging application. You can find it on Sourceforge. Initially I chose this application because I thought it would be really easy to scan with all these tools. It actually turned out that it was more difficult than I thought. It was more time consuming than I thought. My original plan was to scan a bunch of applications with a bunch of tools and find a bunch of vulnerabilities. But that proved to be overly ambitious. So for today's talk I'm just going to talk about the results for one application. So I basically took these tools and I used them out of the box with little configuration. So basically for the dynamic pen testing tools, I gave the tools a URL, some login information, and I let it run. And in some cases I ran them in manual mode as well. And then I compared the findings from each of the tools. How do these tools work? Well, for the dynamic testing tools, first what they do is they fuzz web input. They basically crawl web applications looking for all sorts of web forms. And when it finds all those forms it'll actually try to fuzz the inputs. Like say if it's testing for cross-site scripting, it'll try to inject some kind of script tags. If it's looking for SQL injection and they inject a single quote. And the way it finds these kind of vulnerabilities is usually signature based or behavioral matching. Signature based for say cross-site scripting, it'll look to see if the fuzz string was reflected back by the application. For SQL injection, it'll look for say certain error messages like SQL exception or something like that. These dynamic pen testing tools also have a couple modes that you can run them in. You can run them first in autocrawl mode, basically point and click mode. You give the tool a URL and some login information and you press go. And the tool will try to find vulnerabilities in the app. Generally that's not very thorough so they also have a manual crawl mode where a human or in this case me, I actually crawl the web application trying to give the tool an idea of what kind of page flow the application had. And then the tool would actually take over and perform autocrawl after that. So these manual crawls are generally more thorough in their testing. With the source code analyzer, these tools actually have a couple analyzers within them themselves. For example, data flow analyzer which basically looks at source code. It knows when malicious input comes in from some source API and when it traces that input throughout the program, throughout the source code. And it knows when it reaches a specific sync API, a security sensitive sync. So in the case of say SQL injection, it knows when there's input coming in from the web and it knows that that input, that string reaches a say database API. Control flow is basically the analysis of a sequence of events in your code. So this may be certain things like looking for a null pointer dereference. If you have a method that returns a null value and then use that value without checking to see if it's null, it will find bugs like that. The semantic analyzer is basically like a super grep that will look for dangerous functions, things like that. The dynamic test tracing tool. This is kind of a new tool. The way it works is actually through bytecode instrumentation on say Java or .NET code. Basically what it does is it injects your compiled web app with monitors. These monitors actually run and monitor the application while you're running your dynamic testing tools, your pen testing tools. So it actually will observe fuzzing input coming into the application and it'll watch to see if that data reaches sensitive APIs in your web app. So you do run that in conjunction with your web pen testing tools. So methodology. Well, basically this is how you would reproduce these experiments on your own, download the source code and build and deploy the application. That sounds pretty straightforward, but generally things don't always build when you download them from, say, SourceForge. But once you do deploy your application, you've got to make sure you know how to undeploy your application cleanly. For example, if your application requires a database, you need to be able to restore the database after a pen test because the first input from your pen test may affect results in subsequent pen tests. So you want to really clear the database between runs. So when you run the scanner in autocoll mode, you just want to make sure the applications don't break. So while you're running your pen test tools for the first time, you basically want to keep checking the web app to make sure that it's still running and nothing is broken. Basically, you want to check certain areas of the web app just to make sure it functions correctly. If you do find that the web app breaks, you have to stop your autocoll testing, go back, reconfigure the scanner and test again, and be sure to clear your database before you retest. And that can be a really tedious process, but the configurations and settings will carry over to your next test, which is the manual crawl mode. So after you finish your autocoll mode, you want to go into your manual crawl mode, which basically means you browse through all the pages of the web app, and then you let the web scanner take over from there. Of course, after you've finished your scans, you want to verify the results. So what I did was I manually verified the results. A lot of times, these tools will say there's a lot of vulnerabilities over here, but they may be false positive, and you just want to weed out those false positives and also make a note of the false positive rate in case you want to compare tools to see what kind of false positives they have. And finally, you want to normalize the results. Different tools report vulnerabilities differently. Say for the dynamic testing tools, they will report vulnerabilities based on a URL and parameter basis, whereas the source code analyzers will report vulnerabilities on a file named line number basis. For my experiment, I actually chose to normalize vulnerabilities based on a source code file and line number basis. Running your source code analyzers. Well, there's not much to it. Basically, you get your source code, you point your scanner at it, tell it where the libraries and dependencies are, and you tell it to go. Just make sure you're scanning the same source code that you use to deploy your application or compile your application in the dynamic testing. Again, verify the results via manual testing and weed out the false positives and record the false positive rate and normalize the results. With the dynamic tracing tool, the way it works is first you instrument the compiled code, you basically inject the code with monitors, then you deploy the instrumented code, and then you tell the application you want to start recording. So after you tell it to start recording, you run your dynamic test tools on it, and all these monitors will start recording interactions between your fuzzer and the code. After the test is done, you stop recording and again, verify results, record false positives, and normalize the results. Okay, here, this is a big table. I know most of you can't read it. This is in your slides on the DefCon CD if you want to reference that. But let me just go over briefly what you see here. On the left, you'll actually see vulnerability information, and on the right, you'll see which tools actually found each vulnerability. The red X means that a specific tool, the vulnerability was found by that tool only in that column. A black X means that the vulnerability was found by multiple tools. So let me just go through the first line to help you guys understand this table more in depth. Here, the vulnerability occurs in the file blogentry.jsp, on line number 16. To access it via the web, you would go to the urlsaveblogentry.secureaction. And to actually exploit this cross-excripting vulnerability, you would manipulate the title parameter. And in this case, the vulnerability was found by tool 1b and tool 2b. I'm not naming the tools here because I don't really want to bias anybody into thinking that one tool is better than the next. These tools will perform differently based on your own environment. So just because a specific tool looks like it performs well on this application, it may not be the case for your own application. So really, you got to run these experiments on your own to see how they perform in your own environment. Again, so going back to this table, what you'll notice is that there are actually more red X's than black X's. What that means is there's not much overlap in findings between the tools. So tool 1b found a set of vulnerabilities, and tool 5a found a different set of vulnerabilities. You'll also notice that tool 1b, in this case, performed particularly well. So some vendors may say, well, my tool finds the most number of vulnerabilities, so we're the best. But if you actually look at the percentages, they'd find about, like, 65% of the known vulnerabilities. So you can't really rely on one tool to find all your vulnerabilities because if you do, you're still missing 35%, 40% of the vulnerabilities in your application. So let's just go into some exploit cases. Here we have in file error.jsp line number 18, there's actually this parameter, and it just printed out directly to a web page. Pretty straightforward vulnerability. They're not doing any kind of output filtering. To exploit this over here, you actually put in some scripting text into the URI section of the URL. Now, this is kind of difficult for dynamic testing tools to find because to exploit this processing vulnerability, you have to put the application in a certain error state. So this is something that actually one of the static tools found. On the next vulnerability we have here, viewResponse.jsp line number 31, this vulnerability was actually found by all of the tools. So this is a very straightforward vulnerability. You see that there's a hidden field, and in this particular JSP, a PRINTA raw variable again. Path manipulation. This is actually, again, one of the vulnerabilities that dynamic testing tools have trouble finding. There may or may not be a signature associated with path manipulation when testing over HTTP. In this case, this vulnerability allows an attacker to overwrite an arbitrary file on the file system. And actually there's no indication of that sent back through the web. This attack, for this attack, basically you put in ..slash, ..slash, and then some file name. You include a percent zero zero at the end for null termination to truncate the .properties extension. And then you can overwrite any file that the web server has permission to overwrite. Arbitrary URL redirection. This is mainly used for phishing. So what happens here is that you just put in, you type in any kind of URL into the redirect URL parameter, and then it passes that string to the send redirect method. And that just sets the HTTP response location header, and users can be tricked into going to other sites when they think they're actually going to your site. While verifying the vulnerabilities, actually, I noticed there were some vulnerabilities that weren't reported by any of the tools. So for one of the pages, I actually did a manual code audit, and I found eight additional cross-site scripting vulnerabilities that none of the tools found. So what this means is that you really can't rely on tools themselves to perform an extremely thorough audit. You really need manual auditing to make sure your code audits are thorough. What we see here is actually a pie chart of all the cross-site scripting vulnerabilities. Again, you see tool 1B here defines about 55% of the cross-site scripting vulnerabilities, but it misses about 45% of all the total cross-site scripting vulnerabilities. Again, you cannot rely on a single tool for all of your vulnerability assessments. All right, conclusions. Well, again, single tool doesn't cut it. Using multiple tools drastically increases the number of vulnerabilities that you find. There's actually little overlap between the tools, so really, you get a greater benefit by running all these tools. Tools alone are not enough. Manual audit really does help supplement these tools. You really can't rely on the tools to help you find vulnerabilities alone. And you really got to run these tests on your own app to see how these tools will perform. These tools really do perform differently in different environments. In my case, tool 1B seemed to have performed the best, but in your environment, maybe it would be tool 3A. And finally, I just want to say, fuzzing tools break shit. I mean, when I ran these dynamic testing tools, I had to reconfigure re-scanned applications several times. It took hours and days to basically scan these applications. So if you want to reproduce these experiments on your own, don't expect these tests to be quick. And that's it. Thanks. Any questions? Go ahead. The question was, what type of vulnerabilities were most difficult for these tools to detect? Well, on the dynamic side, things like path manipulation where vulnerabilities actually didn't exhibit any kind of behavior over HTTP. Those were really difficult for those pen-desting tools to find. The dynamic testing tools did excel in finding things like cross-eyed scripting, say, just because that's what they're designed for. The static tools actually had trouble finding understanding certain constructs in the code. And so it missed certain things because of that. Right. I used a couple of the market-leading web pen-desting tools. Sure. I think you'll need to contact my employer to answer that question for you. That's a good question. Say we were able to fix all the vulnerabilities that the tool found, how long would it take for the hacker to find the next one? That's my guess. In this case, actually, for this specific application, it would be pretty quick, actually. I think there are a lot of vulnerabilities that were not detected by any of the tools. But that's for this application in particular. There were some, but they were insignificant. They did not hinder me in verification of results or anything like that, which was surprising to me. Maybe like one out of 30 or something like that. The question was, what type of features would I recommend vendors to add to these packages? I think verification, something to assist in verification of what they report. So that you can validate that the results that they're reporting are actually true. The question was, how do you get these tools that detect cross-site request forgery? Actually, we're running out of time. We can talk more in the Track 1 QA breakout room. I'll be across the room over there.