 All right, we'll welcome everybody to the pentest is dead long live the pentest I guess we were going to get started despite the fact that we didn't get our Presenter view going so we're going to be missing all of our notes I will also forewarn you we worked pretty hard to cut this Presentation down to 50 minutes from its original black hat 75 minute length So we're going to push through our slides and we expect that there will be some q&a We will ask that you hold q&a until the end at which point in time We will move it to the q&a rune so we don't encroach on anybody else's time All right with that we will offer some brief introductions and then tell you guys a little bit about what we're talking about here Hello, I probably know a bunch of you Probably a bunch more of you have seen me at the cons. I've been a goon for this is year five I've been an information security since 1997 really started doing a lot of Penetration testing more ability assessments I'm sorry vulnerability testing Since 2000 I've done some instructing Minimal presenting so if I'm not very good. I'm sorry So and and these are just some of the cons that that I have actually been on staff for So I'm gonna hand it over to Taylor Hello everybody. I'm Taylor banks Kara, can I have relatively similar backgrounds? We've both been an infosex since about 97 I have also been performing and teaching penetration tests since about 1999 I've done penetration testing for Large enterprise as well as state and local government and I've done pen testing training Well for a wide variety of organizations including large enterprise small medium enterprise federal government and military So hopefully Kara, can I can both bring a little bit of different background although we've been doing this for Equal lengths of time and share with you guys some of our experiences in pen testing some of the things we like some of the things We hate some of the things we love and at the same time Talk to you guys about where we see this this service this industry going. Wow. This is really Not very smooth So quick overview of our talk. This isn't a technical talk. We're not going to talk about zero day We're not going to talk about Black box reversing this is this is the classic network pen test is what we're actually talking about So an overview of what we're talking about with the pen test is dead We're going to get a little bit into the history of hacking slash pen testing We're going to talk about in testing going mainstream seeing it in the movies We're going to talk about tools driven a tool-driven methodology versus an actual Repeatable process and how you go about selecting a vendor will mention a little bit about that and our long live the pen test talks A little bit about how does it fit into the organization? What when would it make sense to have a pen test done? And then the evolution of pen pen testing as we've seen it over the past 10 years and We'll reveal the future of pen testing is going cool clicker So as I've already said something about this, we're not presenting a day. It's not a technical talk Hopefully it'll be fun. Hopefully it'll be informative So I'll just keep pressing through here All right on that note I'm going to offer a couple of warnings and a couple of disclaimers the first disclaimer I'm going to potentially Nay, no, I will most certainly offend many of the people in this audience I want to say this in advance. There are some smart folks out there doing pen testing and rather than mentioning you by name You know who you are right if you've been doing pen testing for a long time You know what a pen test is and you've been providing good and valuable services to your clients You will not be the people that I'm picking on but before warned I am going to lay some smackdown on some folks and again I expect to offend some of those of you in the audience. So be prepared number two I was actually sent an interesting Definition by decode decodes another one of the goons here a couple of days back It was an urban definition for Social plagiarism and it's kind of an interesting phenomenon that seems to happen more in this space than probably anywhere else I've seen You're sitting in somebody's presentation or you're sitting in somebody's class and they begin to tell a story about it penetration test They've performed and you say Wait a second. That was my story So I offer you this right we are we're going to be Filling our presentation with some good stories and some experiences that we've had doing penetration testing if we hear you telling your Our stories we will find you so there are disclaimers. Thank you very much Okay, so part one when we talk about pen test is dead a little bit of the timeline and history of penetration testing In the 70s captain crunch. Okay multicolored boxes 1980s we have things like war games. We remember that movie. We're seeing that movie lot mod Kevin Mitnick In the 90s deaf con all right yay deaf con and then present day I love you a little while ago Virtualization big deal Dan Kamensky one of the big names rock star web 2.0 web 3.0 web whatever And red pill blue pill Back in the day When pen testing started evolving as a service line There were only a few people doing it few pretty smart people. They were typically this elite group of guys You know along black coats Very few organizations were doing it. It was kind of a novelty thing around 99 and 2000 it was just starting to become a little more popular and Information security was kind of poorly understood Typically when you do in particular where you can really see that the shift is an external penetration test if you guys did External penetration testing back in 98 99 2000. How often did you get in? Like all the time right today doing an external pen test a lot of times it's kind of boring Because you find like five ports open right so it's not as fun Some of the initial papers that started to kind of embody or express the mindset of the culture of the people that come to cons like Def Conn black hat hacker manifesto Okay, Dan Farmer and beats of enemas paper and their tool subsequent tool Satan Just some more supporting information but it starts to show an evolution of Hacking penetration testing as a culture and becoming a viable service line when you look at Internet security scanner from from ISS Or Chris class at the time It was one of the first after after Satan it was one of the first automated testing tools Okay, so it started to open up this space and Make it a more viable enterprise service line so Quick story did an external internal penetration test for a public school system in 2000 actually comparative tests so somebody else in the room might have actually been with me on that test. Hope you're out there shouts Anyway, we were doing a public school system and we found an SQL server exposed to the internet that had two nicks in it One Nick was connected to the internet the other was connected to the to the internal network It was on the internal domain and had a blank password for the SA account so As you can imagine there were probably plenty of external manipulations of the data in that database You know what grade do you want today? At the end of this engagement the report that was actually generated for this client Took up two reams of paper there were thousands and thousands and thousands of What internet scanner considered critical vulnerabilities, so this was kind of Early on pen test exemplifies just how really bad it was And what were some of the things that we found in those thousands and thousands of vulnerabilities? Is unicode is the double decode stuff Solaris TTY prompt. I give you guys all seen that that's a that's a great exploits. It's one of my favorites because it's so funny Fruit this was something that manifested in the IX 13 years ago and then resurfaced its ugly head with Solaris 10 Where you'd basically tell that to the machine tell it who you wanted to log in as and That you'd put a dash fruit or dash f root in double quotes, and it just gave you a shell as that user was great Blank passwords saw a lot of blank passwords a lot of default passwords So see these are the kinds of things we were seeing in 99 2000 So early on really those that were doing penetration testing were essentially self-taught In fact, I would go so far as to say that probably the vast majority of those that were doing penetration testing We're doing so because they didn't always wear white hats I don't want to imply it and they were necessarily all malicious hackers But most of us doing penetration testing got there because we spent time in underground communities and you know hackers by night pentesters by day around 2000 we started to see kind of a change or a shift in Pen testing is a professional service and we're gonna work our way from from history through present day But now there was less agreement at the time on on any commonly accepted methodology now We can argue the the need for a methodology and in fact later on we most certainly will but Up to this point in time one of the things that we were facing is that we're no two pentests were alike And there are any number of reasons for this We'll go into some of those as well in late 2000 gentlemen by the name of Pete Herzog Aka idea hamster Released a document that he dubbed the open-source security testing and method methodology manual otherwise known as the ostum The initial version of the ostum right ostum 1.0 and then later ostum 1.5 These were a very good start right there were helpful in a number of ways and it was certainly a laudable effort because there was really Nothing similar available to the professional information security community of this kind now That said the early versions of the ostum really served largely as guidance and while I was doing penetration tests I wasn't going to go and replace all of my own handwritten Methodologies with what I took verbatim out of the ostum, but I did find the ostum to be a great effort and again It filled a gap it filled a gap where again before there was no standardization There was no methodology and there's no way to guarantee that what Kerrick and I might do on a penetration test were Somewhat similar if not completely different So really what we had was a service in search of a methodology, right? We've got a number of organizations out offering penetration tests and we're starting to move towards Finding a methodology, but the problem was in 2001. There was too much competition, right? This was kind of a elite professional service. There's a number of organizations that are doing professional pen testing But we see a lot of job process or job security through process of security in other words Nobody wants to expose the methodologies They're using because then they feel they can't offer their clients some special secret sauce The idea and you know 2000 2001 as long as you were good at what you did and you Provided thorough and complete results to your clients and we're able to help them correct the problems you found Hey, you won But ultimately this job security through process of security really only served to hurt us all Again, not only were no two pen tests alike But in many cases two pen tests conducted by two different individuals or two different organizations were so radically different That you couldn't really feel secure in the results of just one penetration test performed by just one organization So my point here is that hey if it's not a repeatable process ultimately It's not really a pen test. It's a hack and there's I don't want to say that there's not value to a hack Right, there's there is some value in finding and breaking things with no process no methodology But lacking the process lacking the methodology again. We got no repeatable process We've got no way to ensure consistency across retests So progress now we move a little bit away from from history here And I've thrown a handful of acronyms up on the board since 2001 We've seen a lot of changes in terms of available methodologies and available documents that provide frameworks for doing penetration testing Many of these documents have very different focuses. We've seen lots of changes in the Austin document itself This was Pete Herzog's document. This is now maintained by an organization known as isaacom We've got some great web testing frameworks from OWASP We've got guidelines and frameworks from the NSA. We've got NIST documents special publication 800-42 So we've got lots of places where we can abstract information that ultimately help us create Methodology that help us create framework and we've seen a lot of improvements in all of these along the way So 2000 pen testing starting to get by and as I said before Automated testing tools are coming into the market. You start to see things like Internet security scanner This was just before foundstone enterprise starts be came to the market, but you're seeing more and more tools out there Figuratively and literally Because what's happening is as you have the automated tools It's making it easier and easier for people to get into this space come run a tool Give you a report. There's your pen test. Thank you. Have a nice day There was nothing like a pen test framework in 2000 2001 2002 I can't remember the exact year that metasplate came out. It was not long after right around 2002 or 2003 but Basically you you would cobble your kit together out of Exploits that you find on the internet things that you craft yourself scripts and little bits and snippets of code One of the tools that that we used at ISS was a tool that that Robert Graham wrote He would take several different exploits out there that were that were typically good quality and Repeatable he compiled them into one executable and then we would use that executable on pen test and say, okay I want to run, you know the this exploit you know some IIS exploit And then you would pass it the parameters that you wanted and then it would send them So it was kind of a convenient way to carry a set of tools in one little binary But it wasn't as nice as like a framework I remember seeing a presentation by Elias Levy on the concept of a framework at Iowa war games From sands back in 2001 right after 9 11, which was it was revolutionary at the time So What happens with what we used to call them at ISS was scan now? assessments or penetration tests were basically someone would come in run a tool Generate a 1300 page report or you know 2000 pages whatever it was hand that to the client and now they're left to figure out What's important? Okay, the problem with that approach is you leave the client with no guidance They they okay. Well, we'll try and fix all the reds and then we'll fix the yellows The problem is there's no strategic approach. There's no How did how did they determine within the context of their own risk what to do first? How much effort is it going to take and the problem is what they're probably going to end up doing is running around and fixing all These issues and then have to do the same thing next year because they didn't fix it from a process perspective so a Lot of the pen tests that started coming into the market at the time were these scan now Assessments or pen tests and it really kind of put a nasty smudge on on our community So we also start to see hacking in the movies. Okay Everybody recognizes Matthew Broderick here in war games Little sneakers. Okay, some favorite movies there Guys remember this movie, right? Why did we watch this movie? Oh wait? Matrix, okay Everybody recognize that shot All right, yeah What tool is that? Awesome. All right, so swordfish The only reason to watch swordfish And here we go. We actually have a movie based on some real-world events in the hacker community. Okay so How'd you become a pen tester back in the day? So Taylor kind of kind of alluded to that already You went out and read a lot of text anybody ever read the rainbow series? Kind of probably copy of you here to help write it Smash the stack for fun and profit. So there were several examples several texts things out there that you could read You could get in an IRC channel hang out with some people get a little mentoring but really One of the crucial points to make is you need to understand the process the attack Okay, knowing a bunch of tools Doesn't really doesn't really make a pen test. Okay, it's it's really about a mindset I'm gonna stand back up. I can't see anybody on the left side of the room sitting down over here And it feels like a much smaller audience so Around the same time frame right early 2000 2001 2002 we also started to see Commercial penetration testing or applied hacking and countermeasures training and in fact both Carrick and myself were providing training of this ilk In the early 2000s right in the 99 2000 2001 time frame we had some good training available So we're doing pen testing training or we're doing hacking training, but honestly it was good training It wasn't great training There were a lot of things wrong with with what was being delivered around 2000 2001 I don't want to imply that it wasn't helpful and that we didn't we didn't smarten up a lot of people and help Them understand and better identify their own vulnerabilities, but frankly one of the things that we weren't doing was teaching a methodology We were teaching a set of tools and we were teaching prescribed Processes for how to apply these tools in an environment in order to find network vulnerabilities But we couldn't really truly teach a methodology primarily for two reasons number one in order to really understand And follow the methodology you got to be able to think a bit like a hacker No, the whole point of a penetration test right is to to recreate this This environment then we're coming in from the perspective of a malicious attacker on our network If you can't think like a hacker you can't act like a hacker So you can't teach someone to think like a hacker in five days and furthermore again around this time frame There wasn't yet a generally accepted methodology Although I succumbs Austin was available again It was still it was a relatively early draft and it wasn't replacing internal Methodologies so it most certainly wasn't something we were going to stand up and deliver as the word in terms of how to conduct a pen test All right Now so nowadays we see a zillion and one training organizations out offering offering hacking training and The training that we see nowadays goes by a million names, right? It's applied hacking. It's pen testing. It's ethical hacking Unfortunately, I'm going to I'm going to tell you in my humble opinion. I think much of the training we see is truly really crap It really is we've got lots of Certified folks who are out there teaching tools and teaching process but without understanding the tools and understanding the process themselves To be completely honest. I've met I say in the slide a dozen But I will say that at least a couple of handfuls of instructors who are out delivering penetration testing certification Without ever having performed a penetration test themselves Really? seriously All right around the same time frame We also began to see the emergence of hacking books and I will say we see hacking exposed release This was really kind of the first of this the the genre of hacking books and frankly It was a really good book it set the bar pretty high Again largely tool driven tool focused, but it was very well done and and the process Described within the hacking exposed book did begin to lay out the framework for performing penetration tests in and of itself The book wasn't going to turn you into a pen tester But it gave you a lot of context in understanding how you began to go about the process of performing pen tests now since that time again We've seen Numerous other hacking books pop up and while there are a few books that Kara can I both believe to be the notable? Exceptions most of the books that we've seen since this initial rush of hacking books are reworked Rewritten material oftentimes with the same screenshots stolen from the other previously available books So again, we're we're seeing this Hacking go mainstream is is really having its direct impact on the the industry itself, right? We've got hacking in movies. We see hacking training Penetration testing training being provided by pen test or by folks who've never done a pen test and and books that are basically regurgitation of earlier books on the same topics In fact while Kara can I sat at borders writing the previous slide? We decided let's go see what's on the shelf here at borders in terms of security books and sure enough CISSP for dummies hacking for dummies really Really does that help us out a lot? All right now again, I mentioned I'd probably piss several of you off And I kind of hope that I do on this slide, but hacking certifications. Are you really serious, right? Are you proud of that? You know, I'm gonna give you my disclaimer here I've got probably a dozen certifications myself and most of them are probably about as worthless as the ones I'm picking on in this slide, but seriously is a hacking certification any better than a note from your mom I say no I'm telling you guys seriously Who in this room? Who in this room believes that they know somebody qualified to certify a hacker because I damn sure don't and there's a lot of Smart people in this room. So a hacking certification Give me a break All right, so again, you know, I knew I probably aggravate some of you And in fact some of you guys probably have the certifications that I am specifically picking on hell I probably have some of them myself that said those of you who know what you're doing Hey, you're you're one of the dying breed. You're smarter than your peers and hey We appreciate your efforts in in keeping the scene real the rest of you and those of you whose faces turned bright red when I was picking on Those of you with hacking certifications, you know who you are All right So the bottom line though and what Karrick and I are really trying to point out here is that it's not about the tools and learning the tools whether through a training class a series of books or Being certified on the tools that you learned in your series of books does not a pen test make right a pen test is About a lot more than just the tools you got to understand the process you have to Understand the mindset and in fact possess the mindset in order to be able to go out and find the problems that are true Truly going to be those that will be taken advantage of by the attackers that you are trying to protect your clients from So we'll just talk a little bit about some of those tools that have come out Over the years in maps super scanner was one of my favorites. I used to tell people when I taught Ultimate hacking ultimate hacking expert that if I had three tools I could take with me to my desert island It'd probably be metasploit cane and able John the Ripper. Those are my favorites Bone scanners and management tools that are out there, you know Nessus is still a still a great tool I've used on several engagements And then of course there were a lot of other players in that space Some of the wireless tools that have come out air crack in G I mean that tool is pretty much the de facto bomb right now. You can crack a web key in less than two minutes And then of course some of the other tools that we use I started using a lot of virtualization around 2003 ish for ish until then of course I was multi-booting which is so much more convenient but also the advent of bootable distros Were fantastic. I mean backtrack is again a fantastic toolkit. So shouts to those guys So here's actually an example of a Comparing yes tools have improved but kind of takes a brain I did an internal pen is actually there to review the the voice over IP network and I kept trying to you know, guess my way in The way I kind of like to start on a pen test is slow and low low impact So I started digging around looking at different systems trying to guess my way into the few VoIP systems that were there And I came across the documentation for the voice over IP system on some some share and started looking through and oh my god The password to everything was the company name with like ads for a's and and threes for ease now Yes, it's all might have found that but when you're brute forcing something like a password over telnet or whatever It's less effective So kind of illustrates the point that actually just as an aside I noticed when I was there that they were running a wireless network with no encryption I thought wow, that's a bold move and so I asked the guy and he said oh we've got Mac filtering turned on though could you can't change that on the heart? This is a pen test literally like a year and a half ago and an administrator told me that Because you couldn't change the MAC address on a nick. They were all good So how do you go about picking a pen test vendor? Well depends on what you're trying to achieve But by that same virtue I've actually spoken to some clients that have told me well I'll tell you what if I really need to pass a pen test. I don't want them to find anything I'll pick these guys if I really want to know it's got to be secure. I'll pick these other guys So again just something think about so in summary We're saying the pen test as it was is is dead So summary that section we covered a little bit of the history we looked at the lack of standardization There was no real solid methodology or process for how to conduct a penetration test All right, we also saw a torrent of people coming into the space because well everybody can be a hacker I can go buy two or three books or I can take a class. I'm good to go. I'm a pen tester, right? I can get my cert But you can also skew those results by picking a vendor due to the lack of standardization Anybody have this book? All right, so we're going to shift into long live the pen test. Okay, what? Is it viable? Where does pen testing fit into the organization? Well I've actually seen a presentation by somebody I didn't work with directly but worked for the same company I worked for for a while the software security guy It's actually John Viega So he wrote building secure software and was telling a room full of pen testers So pen testing is stupid the the classic network pin. There's no reason to do that anymore It's a dead-end service line imagine the looks on our faces and the sick feeling as you looked around the room I'm sure it was probably highly amusing to the fly on the wall So but the argument is a pen test something that you could you know in terms of the classic network pen is a really something that you Can commoditize every part of yes, certainly there are aspects of a penetration test that you could automate using certain tools But I would argue I mean I've been on plenty of pen tests where you run some automated scanner and it gave you know Hey pass these guys are the bomb you do a little manual digging and bam you own the system I've also I Guess it's a mentality of if you have no software vulnerabilities then then you're solid It kind of ignores the fundamental point that I a lot of times I don't I I've broken into many many Windows networks without ever launching an exploit Don't launch one if you don't need to so it's not an issue of soft of software security It's more of a human issue You know, okay, you can keep throwing technology at the problem. You can go buy a knack. You can go buy IDS IPS and I'm not saying those are bad things. I'm not saying that they don't fit into the overall model but Is it really solving the problem? So Is pen testing dead? We really say no based on the results of the pen testing that that I've personally participated in I would say it's a great way to pop quiz yourself Are these security initiatives initiatives that I have in place are these countermeasures that I'm deploying? Are they effective? Well, and they may be effective within the context of everything you know today But you know somebody like off here, you know comes in and owns your knack Because he did something you didn't think of how else would you validate that? Okay? So one of the funniest quotes I've heard from from from another pen tester. This is Marty sells with ISS We were doing a pen test and he says we're not conducting a penetration test. We're creating compelling events. I Found it highly amusing So again as I said, it's like it's like a pop quiz You can Let me just go to the next slide Is it the point that I was trying to make was somewhere around here? We've lost our presentation view by the way, so I've lost my notes but Essentially the point I was trying to make was that you can use the results of a pen test to To to validate funding to to get initiatives in place to secure issues. You already know you have um So as another example of using multiple issues or depending on depending on a vulnerable vulnerability scanner To identify all of your issues might not be that helpful. I actually did an internal pen for a client where One of the things they really wanted me to do was break into the AS 400 environment I don't know that much about AS 400 so I was looking at ways to get in checking defaults Ran an automated scanner and it actually came up with a pretty good score didn't find much stuff But it did find hmm default SNMP on the AS 400 or a default SNMP community string So what can you do with that? Well, I looked at all the things that all the data returned from an SNMP walk and noticed Hmm look here are all the live connections. What about the connections to port 23? Those might be interesting so I also using a combination of You know knowing that I could sniff on their network They weren't using any anti spoofing technology to protect themselves from network eavesdropping combined Just semi-nasty combined that with now knowing where the connections are coming from to the AS 400. I Was able to capture an administrator Logging into the AS 400 and had complete control of the AS 400 environment So it kind of exemplifies the point of you use something that isn't necessarily a bad thing If you can find a couple of little things you might be able to leverage that into something nasty And that's the kind of correlation that an automated tool it just isn't going to be able to do for you I think ever it requires a brain So interestingly one of the things we find now right after all the things we saw from the earlier session in history of the Pen test one of the things we find today Well, believe it or not. It's the same stuff we found yesterday, right? We still have weak passwords We still have poor architecture. We're still finding system defaults We're also still finding vendor configurations and default vendor devices In fact in a large number of organizations that Karak and I have both done pen tests for we often find devices that the local Administrators don't even have root on nor do they have the ability to modify or patch because they are owned and controlled by the vendor So in essence despite all of these improvements despite all of these things that would make us believe that hey if we write our code Properly if we secure our code effectively, we're not going to need to do network pen tests That's hogwash. Unfortunately the problems we see are not just software problems So again, they're basically the same damn things that we saw 10 years ago. Have we learned anything? Well, yes, our software is getting more secure. We know that for a fact we we can test that we can demonstrate it conclusively but again unfortunately what we're talking about here goes well beyond just Software problems software problems are the bones and we just got done spending 20 minutes telling you guys that a penetration test He is not just a simple bone scan So if a pen test is not just a simple bone bone scan and we're telling you that hey most of the major problems We find are not necessarily or inherently Current software vulnerabilities. I tell you unfortunately There's still a lot of need for pen test because it's the only way that we're going to identify these systemic issues So you know the other thing to understand here is that security is a process It's not a project and you've probably seen one of a dozen of these circle diagrams In fact, I think any vendor that has ever done penetration testing Had to create a circle diagram to describe the processor methodology they used but the idea here is that you know We're not just walking in performing a test walking out and moving on Number one the environments that we're testing in themselves are dynamic and a pen test is essentially a snapshot of a dynamic environment The environment itself is changing all the time and therefore we need to continue performing the processes and continue performing pen tests In order to ensure that changes in these environments and changes in risk profiles within these environments Don't significantly affect the security of the organizations Again as I said before if it's not a repeatable process, it's a hack not a pen test Now there's something to be said for the fact that pen testing is in essence still somewhat of an art form, right? There is still some inherent unique Focus or advantage that will always be imbued into a pen test by the person performing the penetration test But there must be some semblance of a repeatable process Otherwise, we've got no way to guarantee that when we go back and perform a rescan against our organization That we are doing the same things in order to effectively determine whether or not the problems We found the first time around have been fixed have been changed have been improved We start to talk here a little bit about metrics and and and about process and metrics So if we look at the different approaches to pen testing, right? There's the the yesterday approach the the 1999 2000 Scan now pen test right we go in we sit down to the client you dump a 1300 page report from scanner I mean the custom report we generated for you into a client's lap and then they run away You sit and you talk with the clients and you say hey We're not really going to be able to accurately assess the risk of the vulnerabilities We find unless we can really define a risk profile and the client says risk profile. What's that? So, you know again the pen test of of yesteryear was something that was It was driven by a lot more by vulnerable vulnerability scanning process Whereas today we need to to move a bit beyond this and we're now moving from you know Kind of a tactical approach to a strategic approach to pen testing So now you know our goal here or our aim is to be able to sit with our clients and have remediation plans and action plan Matrixes that actually deal with highest impact lowest cost vulnerabilities first, right? We need to help them fix the problems that are going to most directly influence or affect them Immediately before we go on to fixing things like SNMP default community strings on an isolated out-of-band air gap network All right, so again, we talked about metrics and honestly We could probably have a a whole nother presentation or for that matter a series of presentations on Metrics and process and frameworks and in fact we'd probably love to do just that It's impossible to capture what would be in the optimal metric when doing pen testing where the optional the optimal Process, but we know that there are things that we need to focus on and need to identify There's a couple of interesting components here We you know want to think Kirk and I had a discussion and we talked about Changes in the vulnerability landscape During the time when a penetration test is being performed and across systematic rescans and therefore as we're looking at the results of these tests We realize hey, you know 13 new vulnerabilities were released for the systems that we tested since the last time we did a pen test for this Organization that needs to find its way into the metric that we use to produce the results for our clients But we also need to focus on things like again for instance Business process, you know, we need to understand that hey the the risk associated with an individual vulnerability Ties back to what an organization does and what information is affected by the vulnerability that we found Again as I said before You know two pen tests are always still going to be influenced by the pen testers and honestly that's a good thing Right, otherwise a we'd all be drones So despite the fact that this is not a tool driven driven process and although I I encourage and suggest and talk to the points of Methodologies in process again, we know that you know tests are influenced by the individual pen tester and again That's that's an important part of this process We said bone counting is ineffective, you know, and one of the things is let's say we've got findings In a new scan right? We've done a scan for an organization. We go back We do a systematic re-scan we come back and now we've got new findings So what does that tell us right has the organization changed has there have they made changes within the environment? Did the tester change did the tools improve or what combination there of right? Did the tool improve and the tester improve we brought new people onto the team? Oh, and by the way, they they turned on seven new systems since the last time we performed the scan Well, unfortunately the only way that we can really begin to address this issue is with metrics So an example of this Actually had a client we went out there in 2006 and did an internal pen for them where We were able to own total like 60 systems. I felt like it was a complete failure because you have to own them all So the the reason for this actually I'll get into the reason when I kind of summarize the whole thing 2007 we show up Owned everything by like 11 and 15 on Monday So we're explaining the results of you know, the client comes to me and says You only got limited control the first time. We haven't really made any significant changes To the environment what the hell happened so I start to explain well the first time what happened was we came in We find a vulnerability on a system take advantage of it get into the box Rape the accounts off of it and we'd have to shut those off to a machine somewhere else because you couldn't run password cracking on Your local machine because it pretty much makes it a brick So we push that off run John the Ripper somewhere else And it would be recovering the password maybe to take a day to get a password back You'd get about 11 or 12 more systems and bam hit a new brick wall where that password wasn't good anymore It was kind of a byproduct of the company growing through acquisition and Different groups had owned different pieces of this thing and they hadn't really solidified it all together yet So it was it was highly segmented in terms of what chunks we were able to get our hands into Came back the next year What had happened was I had some rainbow tables with me I had John the Ripper had been upgraded to to version 1.7 and it was taking me about 15 to 20 minutes to compromise passwords So and I don't know maybe I got better Might have learned something new a new trick or whatever it was But it kind of illustrates the point that there was no dramatic Change that the enterprise they hadn't necessarily gotten less secure It was more an evolution of the tools and maybe an evolution of methodology or or some approach or technique that I used That dramatically impacted the ultimate results of the test. Okay, so when we talk about a metric systems How do we capture all this? How do we determine if we're getting better? All right, so we come back to frameworks and I threw a bunch of acronyms and Pictures and icons on the board before and so now I'll come back and readdress a few of these You know again, we've seen some significant improvements in the frameworks and the methodologies that we have used and or contributed to across the past Ten years the Austin itself has improved. We've also seen release of a new framework called the ISSAF We've got a couple of frameworks provided Kevin Ory has a pen testing framework publicly available as I mentioned NIST special publication 800-42 talks about guidelines for network security testing now We've even got a wireless penetration testing framework that got rolled up into Kevin Ory's overall pen testing framework again. These are all great starting points and they're all a great set of guidelines We're a basis upon which we can influence the tests that we are performing However, you know, we've had this problem with not being able to see the forest for the trees And so now again one of the things that we need to understand is that because we're no longer just focusing on individual vulnerabilities We've got to look for the systemic issues the architectural issues within our environments Unfortunately, even methodologies might not help us there right when we're looking for architectural problems The specific subset of UDP ports to scan in order to be compliant with somebody's methodology document is not really going to be very useful Again, we talked about move from tactical versus strategic Whereas tactical was our process of just basically identifying categorizing all known vulnerabilities Whereas we move into strategic which is a more systematic approach, right? We're trying to find not only system level vulnerabilities We're also trying to find these architectural issues and we're trying to correlate all of this back to again the organization for whom We're doing the tests business to their their risk right to the the risk profile associated with the nature of the business They're doing and the information affected by the vulnerabilities that we find so This is something that that is really only going to to be Improved through the development of appropriate metrics, right? We need systems that we can use to help us identify and to help us provide Information that ties back to the vulnerabilities to the organ or to the systems that we find and include things like risk associated with business profile We mentioned complexity and and likelihood of attack and this is rather important because again the likelihood of an Attack as well as the complexity of an attack will affect the likelihood of that attack affecting the systems within the organization All right So we move into pen testing in the 20th 21st century and beyond fortunately. We've got five minutes and we've only got 45 more slides All right We talk about methodologies one of the things that's important here is an organic methodology in other words a methodology that is going to improve as the The state of penetration testing improves and one of the real benefits to the documents that we mentioned or that we referenced in some of These slides is that most of these are open source documents or open source methodologies While they have groups of core contributors the bottom line is that they are open for contributions from any and all of us And this is an important part of this process right where there are flaws Omissions or where these frameworks and guidelines are lacking It's up to us to go out and flesh them out and identify what they are missing and where they are lacking in order to ensure That in the future these documents and these guidelines can help improve the consistency of tests performed for our clients We talk about adapting to new technologies and this is something that I think frightens a lot of folks doing penetration tests Right is what happens as we see changes in technology We see things like you know Ajax web 2.0 we've got new technologies like knack and and often times these technologies are deployed within Organizations before the folks that are hired to do penetration tests really have an opportunity to identify how the systems work and how They might be compromised so we got to ask ourselves What can we do or how can we go about adapting to these new technologies? Especially when frankly their technologies we might not fully understand so we say well there are two ways number one The easy way be Billy Hoffman or number two We do essentially what what software developers have been doing in building secure code. We do threat modeling We do attack modeling and we finds right we treat these Environments in the same way we would treat a piece of software that we are trying to secure from the ground up On the topic of threat or attack modeling, you know, this is again Something you will read generally in the context of software development However, at an abstract level the process of threat modeling applies equally well to an unknown or unidentified Technology when conducting a pen test, right? We're trying to identify Essentially data flows. We're trying to look at where is a piece of data who owns or modifies that piece of data Where does that data go and what trace trust relationships or boundaries exist between the pieces of data that affect this system? So can you test without a baseline? Yes Unfortunately, the problem is testing new technologies and testing without a baseline is going to produce produce results that will be more difficult to quantify And more difficult to define or associate qualitative values with right? We start testing new technologies The problem is even though threat modeling can help us identify where the weaknesses are and help us identify vulnerabilities within the architecture or configuration of these systems Until there are baselines for these new technologies relating that information back to the risk profiles in the organizations We're doing testing for is certainly not going to be an easy task Lastly, we ask ourselves the question How can we really provide quality assurance in the context of a pen test and by that? Here's what I mean How do I know if one pen test is good and one pen test is bad? If we don't yet have effective metrics and we're saying that the number of vulnerabilities is not necessarily an indication of The security of the environment, especially if we're ignoring risk profile of the organization How can I look at two different penetration tests and say well this organization or this individual did a good job? And this organization or this individual did not well again We're going to have to come back to process The only way that we're going to be able to come back and define quality assurance or quality control In the scope of penetration testing is through effective metrics and these these metrics are things that again through working with and Adapting the existing frameworks. We're going to be able to define and improve and it will be an ongoing and Ever-changing process Okay, how do we justify a pen test? How do we position this? All right, I got one minute. So I'm gonna go really fast Might be a legislative requirement might be a general genuine interest in the environment. I've got alluded to this a little bit Earlier and clients really kind of fall into those two categories I actually did and it's probably a last-pin test story did an external pen for a company Was able to compromise multiple Oracle servers found PII found credit card receipts found My favorite one was this picture that I found So I actually found a file share and started digging through it and found this series of pictures And there was this woman that was having something removed from her leg. That was this large growth tumor thing So you watch the process now, you know one snapshot removing it and then in the last snapshot the doctor is standing there With the tumor under his arm now you have to ask yourself What's the exposure there, okay a legal exposure and And the probably their biggest problem from an architectural perspective, they had no external firewalls So anybody could have viewed that particular sequence of of photographs Alright, so we make it to our conclusion slide just as we we hit the final moments of our talk again You know Kara can I wanted to focus on a couple of points here? And you know the first half of this presentation was dedicated to what we called the pen test is dead What we're saying here is the yesterday's pen test is dead, right? Simple volume counting is no longer effective while what worked then still works now We've had to evolve and adapt to the process to meet the requirements of changes in technology and changes in security Within and around the organizations that we have been working with Second person of this talk we focused on essentially long-lived the pen test and what Kara can I both pause it? Is that despite the fact that improvements to the quality and security of the software? We are developing despite the fact this will Inevitably help our organizations and improve security on the whole it is not going to mitigate the need for network-based penetration testing In many cases network-based penetration testing remains the one effective way that we've got to identify problems within organizations that go above and beyond simple software vulnerabilities Everything we said may be a lie. Thank you guys very much for coming. I also want to thank DC 404 right we're part of the Atlanta Defcon group and we ended up with seven presentations at this year's Defcon So thank you. Thank you DC 404 and thanks to all of the other speakers from DC 404. You guys all did a great job