 I'm going to talk to you guys today about the CISO's view of pen testing. You know I never thought that I would actually be standing up in front of a room full of people talking about how I like to be penetrated. So yeah, but it's Vegas, so I guess everything goes. I think I have to define pen testing first because there are a couple of different ways that you can slice it. First of all, there's the do I get in, you know, guns a blazing sort of pen test. And then there's the horizontal sort of assessment where you're trying to catalog all of the vulnerabilities at a given site instead of a binary do I get in or do I not get in. And that's whether which one to use is actually a pretty big problem especially when it comes to a pen test that is supposed to happen for compliance or for due diligence. So when I was ordering up pen tests in my previous life as a CISO, I would do either one depending on the requirements. And one of the things that I heard about at Shmucon, I think it was last year, there was a big debate about whether it counted if you didn't exploit a vulnerability when you did a pen test. And so people were saying, no, no, you have to exploit it. When I was asking for pen tests on production systems, I would say, do not write anything anywhere. If you tell me how you're able to get in, I'll take your word for it. But there are some pen testers who felt that that was cheating and that was unsatisfying. So I just wanted to say that it's not about your satisfaction, it's about mine, especially since I'm paying the money. So if I want you to go low and slow or really fast, that's my call, not yours. There are just so many analogies that I just can't escape here, so I'm just going to relax and go with it. So pen testing. I have spent a lot of time trying to figure out what it was when I engaged a pen tester, what they were going to do for me, or to me, or whatever. And in some cases, they were an external organization that was trying to verify my organization's compliance with some sort of security, or simply to gauge whether we had security or not. In other cases, it was for our internal use and I simply wanted a list of everything that we might possibly need to fix. In a lot of cases, I'm really not a fan of a black box test because if I'm paying thousands of dollars, between 50 and 100,000 dollars for a pen test, I don't want that person to spend what short time they have looking at things that I already know about. So what I would usually do is a sort of a semi-crystal box test. I would give them the list of all the things we already knew about and say, okay, the rest of the work is for you to tell me what I don't know. And that could involve anything from social engineering to simply running Nessus. Now I have to give a shout out here to the Pentex Execution Standard, folks, because I think that is such a useful standard that's coming up. For those of us who need to order up a pen test and need to be able to specify exactly what it was, it used to be that I couldn't list for a pen tester everything that I wanted them to do, every tool I wanted them to use, every technique, because obviously it depends on what they found in the first pass. So I would generally just have to talk to them and try to figure out from that if they knew what they were talking about, so then I could trust that they would actually be able to do a good job for me. And especially when you're putting it in a statement of work and you need to be able to enforce a contract that is a statement of work, you need a binary list where you can say you did this or you didn't do this, because otherwise it's not enforceable. Suddenly that turns into a checklist, and I know people hate checklists, but again for the purposes of a contract, you have to be able to make sure it's enforced or not enforced in case you have to go to court. So having something like the PTES as a list and being able to say, okay, you're going to do these parts, these parts, these parts, and here's the time length that it will take will also help the pen testing organization price this appropriately. Because again, when they come in with a bid, it's very hard to figure out from the pricing what they're going to do, how many people they're going to put on it, and how long it's going to take. I have to apologize for not having any slides, I just didn't feel like it, but I would like to call up a stunt pen tester to the podium. So this is Mr. Joseph Sicoli who is just plain awesome. So if I start saying something stupid, you can watch on his face as he starts reacting to what I'm saying. The other problem that both SISOs and pen testers I think have in common is that there is never enough time to do a proper pen test. I have talked to some folks at very, very large MSSP organizations who, the actual testers would be given maybe three days to pen test an app. They would come at the last minute and say, okay, we just signed this contract, you need to be done by Monday. And of course, there's no way that you can do a proper job in that amount of time. That actually, that happened to me once when my management came to me and said, you know, we need to do diligence on this product, and by the way, we're going to announce that we're purchasing it on Monday, and this is Friday, so can you test it over the weekend? And my colleague said some really bad words for a very long time. And then we got to work and we worked, you know, all weekend. Now I can't resist as a SISO sometimes playing with my pen testers. And for example, there was one who, yes, somebody's going, oh. There was somebody who, we knew, I knew him pretty well. He was going to do a pen test on us. And we were going to do some maintenance on our firewalls during the window of the engagement and we thought, gosh, how are we going to keep him from doing this? And we knew that he tended to sleep in late, so we figured what we would do is send somebody to get him drunk so that he would sleep in really late in the morning and that would give us time to do the firewall maintenance before he started testing. Another thing that we thought about doing was what I call port flashing. We would, you know, open something and then the next time he came back with another scan, it'd be closed again. So it'd be open and closed and open and closed. And I really like to see that report, you know, when it came back. I don't know. It was there. What? That's horribly evil. Yes, it is horribly evil. But seriously, you know, in a dynamic environment, you do have to deal with that sort of thing. You know, people are bringing things up and down and you just have no idea. You can't have everybody stop while the pen test happens for two to four weeks or however long you're doing it. So yeah, there will be something that will be on a finding and you'll go back and go, oh, it's gone now. So the are you in yet question? Because I can't tell if you're in yet. That was the sort of, that was the question I always asked, you know. And I would, you know, sleep with my blackberry at night waiting for it to vibrate to tell me that, you know, me and my loneliness and my blackberry waiting for them to tell me that they were in. And, you know, when I got the call, I would immediately have to, you know, get some people on whatever vulnerability it was that they found. But yeah, that was the highlight of my existence is waiting for that vibration, waiting for that call. Now let me talk a little bit about reporting because that's where in my experience pen testers tend to fall down. They want to write about the really sexy stuff and they want to do a couple of screenshots and go, yeah, you know, this is where it was laid wide open. I really love this part. But that's not as helpful to me. I mean, first of all, it makes very lurid reading for the auditor and they get very excited about that. But it doesn't, you know, do a whole lot for me. What I need a report to say is not only what the vulnerability was, how it was exploited, what the potential impact was, and we'll talk about impact in just a minute, but how likely it would be that somebody would be able to do it. What the skill level was that it would take to exploit that vulnerability. And then from there, I would have to figure out, well, who would want to do that? Because the other thing is when I would take these lists of vulnerabilities, these findings to a sysadmin or a developer, they'd say, well, who would want to do that? Our users don't want to do that. And I have to explain that it's not about our regular users. So talking about the impact, the other thing that I really hate is when I have a purest pen tester who is like, oh, my God, everything is so bad. And this is not useful to me because if you list, for example, that we're still supporting SSL 1.0, that's the sort of thing that the auditor gets really excited about. And I have to explain that this is not a big deal. Yeah. Who cares? Debating criticality, both with an auditor and with the staff who need to try to prioritize a fix for this is very frustrating. First of all, auditors generally don't tackle risk. And I know that sounds kind of weird, but if you've worked with an auditor, they are not going to talk probabilities with you. They are going to talk about, it says here in my list that this is bad. And you have this. Therefore, you have to get rid of it. So in some cases, I will really prefer it if you tell me verbally what the issue is and not write it down because I don't want to have to deal with arguing with an auditor over whether this is something that should be fixed or not, especially if it's something that we can't fix or can't fix readily, we have to support really ancient browsers out in the wilderness or something like that. So I just don't want to go there. And the other thing that really drives me crazy is if there is a finding that we're doing something on purpose. So things like telling the user when they've gotten the user name wrong, you know, everybody says, oh my God, that's terrible, you can enumerate user names. Well, I'm sorry, but we have the kind of users who forget their user names. And it is a huge support cost to my organization to have to talk with them on the phone to figure out what their user name is and tell them instead of just telling them that they got the user name wrong and they think, oh yeah, I remember there was supposed to be a one on the end. So I'm sorry. Tell me enumerating user names is bad, I'm going to say, I'm not going to be really excited about that. So there's a big trade off between the things that we do on purpose and that we really don't consider to be high impact vulnerabilities and the things that really are, you know, oh my God, you did what? Sort of things. And I've had, you know, plenty of those too. Some pretty scary things coming in. So by and large, when I get a report from a pentester, I want it to be, I only want it to be the things I really care about. And I'm sure there's some people who are going to argue with that and say, you know, if they're going to do their due diligence as a pentester, they need to list everything otherwise it'll look like they didn't find something. And that reflects badly on them. What do you think, Joe? Yeah, but it comes down to what the client needs. Yeah, it comes down to what the client needs. The other thing is that in some cases, particularly in the public sector, the Public Information Act may allow citizens to request information that would really be sensitive, such as the output of a pentest. Now in some cases, the law allows the particular agency to say they're not going to disclose that information because it has to do with security of the infrastructure. But in some cases, that law isn't there. And so the only thing that might protect the contents of that report are to call them auditors' working papers or something like that, something else that is protected from release. Otherwise, they can, if they're asked for, they will be released. They could be posted anywhere on the internet. And I'm really not a fan of that. And neither is my management. So how am I doing on time? I'm doing good. Okay. So defining and time-bounding pentests or assessments and then figuring out exactly what you're going to do and what you're going to look at and how long that's going to take, I know that in some cases the CISOs can be really unreasonable on that. Like, here's a class B. I'll see you in two weeks. It's not helpful to anybody. Although I like to do that too just to see their faces. So working out that time to engagement and then telling them everything I already know so that they don't go over that material again because the time is very precious and it's very expensive. And then getting the report back, which talks about the things that I really do consider to be critical vulnerabilities. The one other thing that I really appreciate from a pentester is the description of how easy or hard they think it is to fix. And that's another thing that really bugs me about some pentesters who have never actually been on the fixing side of the table where they say, oh, well there's just no excuse for that. And they have no idea how difficult it is in an organization sometimes either to fix something in a legacy environment or an environment that includes third party components that we have no control over. So there are a lot of things that are just plain not going to be fixed. And everybody knows it, everybody understands it. But the pentester who believes that everything should be fixed is not very helpful to me. They need to be more realistic about the business impact of what they're finding and be honest about how much work it's going to be from their perspective. And then I take the organizational overlay and say, well, yeah, that might sound really easy. But for reasons of our own organization, how it's structured or how the infrastructure is set up, that's almost impossible to do. So it may leave the pentester really frustrated, but I don't care. As long as I'm satisfied, that's what counts, because I paid for it. So I wanted to ask for some pentesters in the audience who disagree with anything I've said thus far and say, yes, and tell me about it. I'll repeat it after he says it. I'll use my loudest voice. Yeah, use your outdoor voice. OK, so if I understood that correctly, the question is that you don't feel you can leave off any sort of finding, even if it is on a legacy system or a third-party system that can't be fixed. And so you're asking what I think I should do about that. In some cases, I've seen two versions of reports. One is the verbal report, where they will tell me exactly what steps they took, what the vulnerability was, what they got. And then there will be the four general publication report, which will simply say the equivalent of there was something really bad here. And it might allow access to personal identifying information. So I know, in my working notes, exactly what the problem is. And so I can follow up and see if it is really fixed. But in any document that might have to be released, either to senior management or to the public, it just says something very vague. And actually, state auditors will do the same thing. If they're looking at something and they have a finding, they would often call me up on the phone and say, OK, here's the finding, here's what we have a problem with. But then in their report, they will just say, oh, there was some general badness here. And that's the agreement that we have, both to get it fixed and yet not to make anything too explicit in writing that might come back to bite us. Does that answer your objection, or is that still? Yeah. Yeah, so you're already doing that. One for the engineers, one for the auditors and everything. Yes, sir. OK, OK, good. Very good. OK, so what he said was he felt that my requesting reports verbally without putting them in writing might be seen as a way to skirt regulatory requirements or reporting requirements. And that strikes you as unethical. That's very fair. And I think that in some cases, especially this is really sticky stuff, especially when you're talking about the public sector, because there's the need to protect the infrastructure. And that includes not publicizing vulnerabilities, but there's also the requirement to fulfill any reporting requirements that are there. The thing is, though, especially in legislation, legislation is very, very high level. It doesn't say what you have to report in a pen test. So it's often up to my judgment to say, here's what I think the public needs to know or deserves to know, and here's the stuff that I think they don't need to know. And yet, it may be an internal report that I give to my management and say, here's what they really found, and this is when it's going to be fixed. But there's a big gap in there. Yes, sir, you're in the back. OK, OK, OK, that's a really good question. His position is that as a pen tester, if he doesn't actually document and present everything, it doesn't cover his ass sufficiently. So if that system ever gets breached by something that was trivial to find and should have been fixed and everything, he's afraid that it would come back to buy him. And someone might think, for example, that either he didn't find it, which would really reflect badly on him, or he reported it, but we chose to ignore it. And you want to make it very clear which part of the risk acceptance is ours and which part was you doing your professional job and making sure that everything you found was communicated in a particular fashion. And in some cases, it just comes down to level of detail. I mean, there are a lot of things where I would say, yeah, we have to list that this was found, but we don't have to say exactly what steps you took to exploit it. Now, whether you feel that that's not being forthcoming enough with an internal organization or say with an external product, that's something we can debate in the Q&A room because I have like one minute left. So I will wrap this up. But if you would like to talk about this more, thank you very much and good luck to all of you out there.