 traction here. So our next speaker is Fleur van Loosden. She is the CISO for the ACM, so basically this is the Dutch, the Netherlands Authority for the Consumers and Markets, and she is also a board member at the De Vede, the Dutch Institute for Vulnerability Disclosure. She's going to talk to you about pentest reports and why they don't get used. Of course, for me, this is utterly perplexing because, you know, me and my team, we spent an awful lot of time writing pentest reports, and I'm extremely curious to hear the reasons why they might not get used. So I guess with that, I guess Fleur, if you would like to come up, wonderful. So with this, I guess we are just about on time, so I am going to hand things over to you. Thank you. Melanie Riebeck, of course. Yay, my talk. A CISO approach to pentesting and why so many reports are never used. But first, let me tell you a little something that happened to me last year. We had a pentest at the place where I work, and we had, this was actually a fully digital pentest, but for some reasons I cannot tell you about right here, there was part of this pentest that had to be on site. So the pentesters were actually in our office doing their digital test. And one day, they were walking down the hall and they saw this, our lockers. And these lockers are secured by pin codes. And one of the pentesters jokingly said to my security officer that was with him, you know, why don't you let me just try, you know, just one. Let me try one code on one locker just for shits and giggles, you know. And my security officer unfortunately said, sure, why not? This sounds like fun. And the pentester went off, chose a random locker, went one, two, three, four, enter. And of course the locker opened. And this turned out to be a locker of someone quite important within our organization who did not appreciate this test very much. Because scoping, y'all. But yeah. Apart from, of course, the fact that it's not a very smart move to have one, two, three, four as your code, it's very important to have a scope for your test and stick to it for multiple reasons which I'll get to it in a moment. First of all, I'm just wondering, I'm going to do a poll right here with you guys. Because I've noticed that a lot of us don't actually know this, but managers don't know a lot about IT generally, unless they're IT managers. So what I did was I asked a random group of Dutch managers, which I don't personally know, who are not in IT, what they thought, the word checksum meant. And I was kind of in a friendly mood, so I gave them three options. I didn't ask them right out. I asked them, do you think it means a digit representing the sum of the correct digits in data which can be made to detect errors? Spoiler alert, of course, this is the correct answer. Or do you think it means a sum of multiple vulnerable systems that have been checked for the same known vulnerability over the course of multiple tests? Or they could say it's a way to check that a computer system has not been hacked recently. Now for you guys, I'm wondering what you think most of the managers thought the word checksum means. So I'd like you to raise your hand if you think most of the managers I asked think it's answer A. How many of you think, well, have a little faith in your managers, people. How many of you think that the managers thought it was the answer B? So for the viewers at home, I think about 50% of the people in the room. So probably the other half is going to think the answer C, right? Who thinks it's C? Most people answer C. I'm actually going to give you the answer, of course, but I'm going to give it to you a little later in the presentation. All right? So stick with us. Who am I? Well, I'm CISO for the Dutch authority for Consumers and Markets. We are a regulatory authority in the Netherlands. We regulate, well, Consumers and Markets. So we make sure there's no monopolies or that kind of thing in markets like hospital care or energy markets or that kind of thing. And I used to be the CISO for DIVD, but now I'm a board member. And of course, DIVD, we are a group of volunteers that scan the Internet for mostly known vulnerabilities and try to get the people that are vulnerable to patch or update or fix their systems, as well as responsibly disclosing zero days or helping people disclose zero days in a responsible way. And, yeah, we do that voluntarily. And I have a background in criminology and justice. I have a master's degree in criminology and I used to work for the police in Amsterdam. Enough about me. Anyway, is it fun to work? So let's talk a little bit about how pentests are traditionally set up. First thing I want to say is, of course, you all know this, but there's a difference between a pentest and a scan. Like a scan is automated. And I refer to a pentest as a manual thing. Like, it's actually a person doing the test. It's not a script. I'm not saying one is better than the other. There are perfectly legitimate reasons to go for a scan as opposed to a pentest. But in this presentation, what I'm talking about a pentest, I mean a manual pentest, not a scan, just to make that clear. My experience is that most times when you order a pentest from a vendor, the details are decided and proposed to you by the vendor. So the vendor is going to tell you, you have an Office 365 application here. I would suggest you would test it for authorization, authentication, that kind of thing. So the vendor will ask you as a CISO, would you like this and this and this tested? As opposed to the CISO saying, I want this and this and this tested? Usually a CISO just says, I want my, I don't know, SharePoint tested. That kind of thing. And then usually the pentesters, they do the test, they make a report. And then the CISO kind of translates that report back to the board or management because they can't actually read the report. It's blah, blah, blah to them. So you as a CISO have to take the report, make sure you get the main findings and then translate that in a language, your board or your managers understand to get, to make sure you get funding or, you know, get everything fixed. And this is something that's not mine, of course, as a pentaster. You've probably heard that a lot, but it's very, very true. Your test is as good as your report. You can have the most amazing pentas. If you cannot write it down in a way that people understand, there's really no point. Nothing's going to happen with your findings. So that's something I really wanted to emphasize here, as well as the fact that there is really no point to having a very shiny, perfect report if there is no actual follow-up. Now, that's something that's not your responsibility as a pentaster most of the time, but there are things that a CISO can do to make sure a pentas gets followed up on. So you can write a perfect report, but still if no one does anything with it, it ends up in a drawer. What's the point? So in my opinion, there's some relevant questions here. How do you set up such a good pentast as a CISO? How do you write a good report as a tester? And how do you work together to make sure it gets a proper follow-up? That would be the optimum outcome. So setting up a good pentas as a CISO to me means that you take the lead. So you handle a pentas like you would a project. You make sure there's a planning. You make sure there's a scope that you decide on. So it's not the vendor that comes to you and says, would you like your SharePoint tested for authentication issues or that kind of thing or any files that are hanging around in the wrong places. That's something you need to discuss with the owner inside your organization of SharePoint to make sure that that's actually what you want to get tested. Maybe there are things that happened in the past, like certain incidents that make you doubt certain configuration decisions. That might be something you want to get tested. So instead of leaning back and letting the vendor tell you what you should test, it should be more proactive as a CISO and make sure that you decide together with the owner of the application or the system what gets tested. And you need to write that down. So you make a clear scoping document and then you're not even there yet. You take the scoping document, you go to your pentesters and you say, can you do this? Because it's great when you have fantastic goals or you've written everything down, but if your pentesters actually can't test that because they don't have the tools or the knowledge or whatever, then what's the point? So you're like a bridge between the pentesters and the owner of the application. You make sure there's things arranged for the test to be carried out and you make sure that the scope is clear and it's doable. So you need to make sure you have goals. You need to write them down. You also need to write down what techniques are allowed. Are your pentesters allowed to use social engineering? This sometimes can be ethically something to be discussed. So write down what is allowed, but also write down the limitations. If you don't want them to use certain tooling, if you don't want them to do certain things, make sure you write it down in advance so they know. And a few helpful tips. Add some technical details to your scoping document. So a lot of the time these tests are either white or gray box. They're hardly ever black box because you don't have that kind of time or money when you do a pentest. If you want a true black box pentest, you usually go for bug bounty, that kind of thing. So usually these things are white and gray box anyway. I'm going to get back to that in a little bit. So I like to add some architecture drawings. Of course they are never completely up to date because they never are. But whatever you can give your pentesters as something to hang on to, it can be helpful and it can also make sure they don't spend a lot of time searching for things on the network because they can just look it up. If there are certain settings in your antivirus, your IDS, IPS, that kind of thing, that might go off because usually when you're pentesting and it's not in a developmental stage but it's an in-production system, you don't really want all the antivirus alerts to go off. It's just annoying. So sometimes it just helps to tell your pentest in advance if you do this then that will probably happen. You might want to avoid doing that because, you know, IT people are humans too. So like I said earlier, what kind of tooling you're allowed or not allowing might be helpful. And of course boundaries. And with boundaries, I don't really mean tooling but with boundaries, I mean network boundaries. You don't want your pentesters hacking into your vendors network because there's a connection there or your neighbors or whoever. You want them to stay within your own network. But in order for them to do that, you need to tell them what the boundaries are. So which IP addresses are within the limits of your network and which ones, if they spot that, they have to go, ooh, have to go back here. And involve your IT department, I think, well, it sounds like very obvious, but sometimes people forget. So the IT department doesn't know there's a pentest going on, antivirus alerts are going off, your nice IT admins are tearing their hair out going what the F is going on. Well, surprise pentest. Yeah, you want to avoid that. So I would involve them. I would tell them in advance, we're going to do this test. Is this all right? Do you have any large migrations planned for this period for this application? Might also be a valid question to ask in advance. And even if you do that, like I did, sometimes it still goes wrong. I involved the IT department very, very timely, everything. I even made sure that the accounts that we gave to the pentesters had very obvious names, pentester one, pentester two. And then I always make sure I'm available in the week that we do the test because if things go wrong, you know, people can call me. So I got a call. First, I got a call from the pentest going, our accounts are locked. I'm like, that's not good. Like five minutes later, I get a phone call from the IT department. We locked their accounts. I'm like, why? And they go, well, they we spotted them downloading hackers tools. So we, we, yeah, we disabled their accounts because that's what we do in a normal situation. I'm like, it's not a normal situation. These are pentesters. Yeah, but we blocked their account like we're supposed to do. I'm like, that's awesome. What are they going to do now? Go home? We paid them money to do this. So yeah, then they had to enable the accounts again. And fortunately it was around lunchtime. So the pentesters could grab lunch and then go back to work. But you know, even though you do this with the best intentions, sometimes it still goes wrong. Some things you must never forget. A disclaimer, I don't, I don't really know if this is the correct English word in Dutch. I would say a Freiburgers Klein. So you need to make sure that there's no legal action that's going to be taken to your pentesters if they make mistakes. For us, we usually put in the disclaimer that as long as they follow everything in the scoping documents, the disclaimer is valid. If they do anything that's not in the scoping document, lockers, you know, luck of the draw, so to speak. So in Dutch we have a VOG, if you work for your pentester for a government organization, we ask you to give us this document. It states that you have no, not a criminal record that is related to the work you're going to be doing for us. But it's optional because it's not all, it's only for government. Well, it's not actually only for government, I believe. You can ask for it if you're in a company as well. But usually for government, it's mandatory. Of course, you ask them to sign a confidentiality agreement. You ask them to sign this agreement because if you have a responsible disclosure, this is kind of a wobbly area. You would not always ask for a confidentiality agreement. In this case, when it's a pentester, however, you're paying people to do something for you and you're also giving them information. As opposed to when it's a responsible disclosure, someone finds a vulnerability somewhere in your system and they found it independently. But we're actually telling the pentesters things about our network, our anti-virus settings, our firewall settings, our parameter, that kind of thing. So you want this to stay confidential. So it's a different kind of situation. And when there's going to be social engineering on site, make sure that you give them a written statement with the signature. I do this from the CIO usually. Because yeah, it's kind of a hassle when you have to go pick up your pentester from the police office. So it's easier if they have like a statement they can show your security that they're supposed to be trying to social engineer people in the building, which is why I would give you this tip. And then there's a little bit of boring stuff. If you have a larger test that is spanning multiple days, make sure you have a planning. Like first day, they start here. Their goal is to reach this point in the network or to like get admin, that kind of thing. Make sure you plan it out because you want to have as much tested as you possibly can. You're paying for this test. You want them to test as many things as possible. And of course, some practical arrangements. So you need to make sure that from it, like timely in advance, you give them access passes, password logins, 2FA tokens, that sort of thing, because these things sometimes take longer than you think. And then that's just what I was talking about earlier. Why would you give a pentester credentials or two factor authentication? You do this because if you don't and you just tell them hack this application, I'm going to give you nothing. There's a chance that they're going to spend the entire week or whatever time you gave them trying to get into the system. That doesn't mean your system is secure because an outside attacker has all the time in the world to get in. And your pentester only has five days. So if your pentester had all the time in the world, you probably would have gotten in or she anyway. But they don't, you only gave them five days. So that's why in some cases I give the pentesters credentials not to use immediately, that's why you need a planning. So you tell them day one, you try and break in from the parameter. If you can't get in in day one, write down whatever you found and how you think you would have gotten, maybe you would have gotten in if you have more time. Then day two, user credentials, lowest type of authentication, authorization, see how far you get there and so on and so on and so on. Because you have to assume breach anyway. You have to assume that if someone has unlimited time and money, they would get through your parameter and you want to make sure that all the layers are tested, not just your parameter because it's a waste of money otherwise. Now, for something a little different, I'm just wondering, who knows what this word means? If you know what the word means, you know what it means. You were first, I saw it. Maybe it's possible that you can use the microphone, can you use the microphone and say what it means? Hello. Yeah, go ahead. Another English word for this bullshit. Exactly. Or more, you know, nuanced. It's a language that is meaningless or made unintelligible by excessive use of technical terms. Now, a lot of pentesters in their reports tend to use this type of language. For managers, like I said in the beginning, check some, might be difficult. But what I asked the managers that I also did the poll with earlier, I was like, well, they're here, might as well ask them some other stuff. I asked them, what would you like to see in a pentest report? Now, remember, these are non-IT managers. These are directors, board members, that kind of thing. They're also not representative of all the managers in the world. They're just a handful of Dutch managers. It's just for fun. But what they said was, they at least want the top three findings. They want a suggestion on the fixes of these top three findings. They want a management summary. They want a cost estimate of how much they think you think it might cost to fix this. And they're interested to know if this, whatever you found is being exploited in the wild. None of them wanted the whole technical story. That was an option they could choose. No one chose that option. They do not care. Which doesn't mean you shouldn't put it in there. I'm going to get to that in a second. You should. It's just in a different way. So why are we writing reports this way? Why are we writing reports in a way that managers actually don't care about? Because it's not aimed at managers. We write pentesting reports for technical people. We write them for the IT department. We write them for the CISO. We never write pentesting reports for board or managers, and that's why it's in technical terms because it's meant for technical people. But there's a very distinct language difference there. A lot of us think that we know what managers know. They think certain words are normal in everyday language, which they actually aren't. But we as technical people tend to hang out with technical people. That's our group of people. That's our frame of reference. So that's our language. So we're kind of living in a bubble. And then CISOs are left to translate, but they're not always very good at that. So that's not really an optimal setup, in my opinion, because the people that we write these pentesting reports for are not actually the people that are the ones that make the decisions or that have the budget. Sometimes CISOs have a budget. I personally don't. I don't need it. But a lot of the times the people you write the report for are not the people that make the decision on what to do with your findings. Another problem that you kind of run into a lot is that an internal CISO's opinion is valued surprisingly less than an external expert's. So when an external expert says this is a serious problem, a board member or manager is more likely to believe you than if a CISO says it because the CISO says it six times a week. And the external expert is someone from the outside and, you know, exotic and new. And you've never seen that person before. So if he says that it's serious, then it's probably something you should take serious. And also CISOs just sometimes don't understand the reports or they misinterpret findings and they're just not very good at translating. Sometimes they're very good, but sometimes they're not. And this way you become a gateway between the expert and the management. It's kind of like what you did in preschool where you whisper a word in someone's ear and they have to whisper it in the next person's ear and the next person's ear and at the other end a totally different word comes out. That kind of stuff happens. And as a result, your board doesn't always get to see or feel how serious certain findings are because you're a CISO and you're always yelling at them that things are going wrong and we have to do something right now. And they tend to be kind of right also. They tend to get a little numb sometimes, which is something we need to fix. But hey, I'm just saying. So how do you write better reports? Like I said, a lot of them just want a management summary. A lot of pentesters actually do this. They write a management summary. But then the management summary is sometimes 10 pages long, which managers tend not to have time for. So I would say make a management summary that's maximum one page and make it so that, you know, it's readable to them. There's no jargon, no gobbledygook in your management summary. Make sure there's the top three to five findings. I'm not saying you shouldn't put any other findings in. I'm just saying not in the management summary. For the managers, just give them the top three of the worst things that you found. And what is needed to fix them. And then have the rest of your findings in an attachment in technical terms. Because eventually someone's going to have to fix all these things. So it's important what you found. How can we replicate this? And you have to do this in technical terms because you just can't get there if you're talking like a toddler. So you're going to have to make sure that there's technical information there, but just put it in the back. So that everyone can use your report. The managers can use your report, the CISO can use your report, and the technical people that need to go fix these things can also use your report. And at screenshots and evidence, because, you know, it's just more fun to look at and it makes it easier to replicate, and it scares the hell out of managers. And then the follow-up. Like I said, it's great. You have this wonderful report, you had a great Pantest, lots of wonderful findings, great report, but then the follow-up. This is a job for your CISO. So how do you get the follow-up done? You need to make sure that you let the experts do the talking, if you can. Sometimes Pantesters aren't really keen on talking to board members, but if you estimate that they're, you know, they're good at explaining what they did, have them have a presentation for your managers and your board, where they explain themselves their top three findings, have them talk it over with you before to make sure there's no difficult words in there, and then have them do this this presentation where they explain the top findings that they did, and your board and your managers can actually ask them questions. How did you do this? Why is this important? That kind of thing. And then hammer them on their responsibilities. So tell them, all right, we have this report, we have these findings, you are the owner of this application, or you are the manager, or you are the board. If we do not follow up on this, that's your decision, I'm going to write you an email, a memo, or whatever, and please sign. Sign that you decided not to follow up on these things, or I'm going to give you an advice on how to follow up on it, and then please say that you want this followed up on, like in writing, and then we'll get it followed up on. And like I said, have the experts explain. So this presentation is not finished yet, I'm still going to give you the answer of the poll in the beginning, and some main points, but up until now, are there any questions? I'm just coming back on stage to help with this part of it. So anyone who has a question, feel free to line up at the microphone. Another thing also, those of you who are watching online can also send in questions. So also for the signal angel, do we have any online questions yet? Nope. Okay, great, go ahead. Yes, you said you write your report in, we should write our reports in two levels, business level and a technical level. I'd like to write my reports in three levels. I started with my management summary, which is totally business language, as little technical words as possible, not even say VPN, no the network of the company. And then when I start writing my findings, my individual findings, I always like write a user story kind of thing. So as an attacker that is able to connect to the network, I can steal your data. So you can explain what is the risk with each finding separately. So if a manager starts flipping through the findings itself, they can see, oh there's a risk of people stealing data, stealing money, exposing our secrets to the internet, things that and then I can start on with all the technical details. Yeah, that's very good. Yeah, I would say do that as well. I didn't really mention it, but I kind of assume that's something you would follow up on after a management summary. So it's not a management summary, attachments, it's management summary and then you kind of elaborate exactly like you said, maybe not in a user story, that kind of, that's a taste thing, but elaborate on what does it mean that we found this thing. Yeah, that's perfect, yeah. Go ahead Oscar. How do you deal with the situation that managers push the document, the report back to you as the expert or the IT people, because they are responsible for the infrastructure. How do you deal with the situation where the managers are just pushing it back to you again? Well, they really can't, because what happens is there's the pentesting report, they get the presentation where, you know, there's explanation about what was found and then they get advice on how to fix it and we tell them to make a decision. So I usually, what I usually do when we have a pentest report, I take the most important findings, of course the little findings, like the quick fixes, we don't even discuss those, I just give them to the IT department and I go, please go fix this and they'll fix it, it's quick, dirty, easy, whatever. But the main findings, the big ones, I usually go to the managers and I give them an advice and I give them usually two options. So for example, we had a phishing vulnerability and I said you can put a VPN before this portal or you can put a very serious monitoring on it, or both, preferably both, but give them options because my experience is managers and board members, they like to have something to choose, not too much because then it gets too hard, but give them options to choose from and then you force them to make a decision, they can't really say well, let the IT department decide this because what I would say is they can't make that decision because they need money or they need people or whatever, something is needed more than we have right now in order to fix this. A VPN costs money, a scene costs money, so they can't really push it to the IT department because it's not their field of decision-making. Go ahead. Yeah, Hendrik, I have a small question but it's a bit the position of the CISO and the PEN test inside the organization to give you slightly an idea, in the late 1980s I was in Delft University and I had to approve of with some people on a military raid attack on the nuclear reactor, that's a very rough kind of PEN test, what part of department, who's designing the doing the PEN test, is there what's the CISO, is it general security or is it IT? That's a very good question actually, I think it depends on the maturity of your organization, so the organization where I'm working right now we still have a ways to go, which is normal because we're government, but so I kind of ease them into it, so I say well you are the owner of a critical application, according to our policy your applications should be tested, PEN tested once every two years, your last PEN test was two years ago or never, and then I go to them and I say well according to our policy it should be tested, it's going to cost you about this amount of money, it's going to cost you about this amount of time from your part of the organization, it will give you this information, we're going to expect you to do that with it, so I kind of walk them through the whole process and then I do this a year in advance because I make year plans for my team, so I'm going to ask them a year in advance, would you like us to test your application next year, you should be saying yes because you're owner of critical application blah blah, I've never experienced them saying no, but if they would ever say no, there's two things I can do as a CSO, I can go to the CIO and say hey, this owner of this critical application is supposed to get their application tested because this is the policy, they are refusing this right now, what do you want to do, and then the CIO can escalate it, they can talk to that owner or they can go to the board or they can just say it's fine with me, if the CIO says it's fine with me, the CIO trumps me in my organization, so I would just make sure I get it in writing and I have it stored away, fine, we go to a different application, but if that application gets attacked or we get hacked because of that application that's not tested or that's vulnerable, I would say that's on the CIO and the owner because they made that decision not to test it, so that's how I usually handle that kind of thing, it depends on the organization because there are organizations where it's standard to test every new application or it's standard to test every application that's in development, it depends, but that's just how I handle it. So, Signal Angel, are there any questions from online, can you read it please? There was one question but this already answered. Okay, excellent, there is more space for questions, I'm going to give you guys all still an opportunity to come up, in the meanwhile there actually is one question from me, something that I was wondering, so you mentioned that you would like for the pen testing companies to deliver a cost estimate of what it might cost for the organization to fix the vulnerabilities, given the fact that we're a third party without intimate knowledge of your organization and IT infrastructure beyond what we saw during the pen test, how can we as third parties be able to make this kind of an evaluation? It's actually not something I said you should do, it's just what the managers that I did the poll with said they would like, but you're completely right, it's not very realistic to ask this, but what you could do is have them kind of estimate it, like it doesn't have to be, how you say, Oferta, I don't know names for it, it doesn't have to be like a precise number, but you could say it's probably between this and that amount of money, if they really want to have a cost estimate. In my experience I'm usually the one that does that, so when I write the advice that's based on the pen test report I make a cost estimate for my board and I say it's probably gonna cost you this much, if you choose option A or if you choose option B, because that's the thing, the pen test report usually doesn't specifically say you should do this to fix this finding, it usually says you should do something about your authentication or your whatever, and then it's up to the organization to to make a decision on if they want that VPN for their portal or monitoring to tackle a phishing attack. Okay, thank you, we've got another question here, go ahead. Hi Fleur, so there was a lot of wisdom in this presentation and thank you for that and I would hope to get a little bit more of your wisdom and I know you cannot be extremely specific in your answer but when you are looking for parties to do a pen test for you are there some things that give you good vibes or things that give you bad vibes because one day I will need a pen tester and I would like to have some clues on how to find out the good ones. So I like, I like companies where I get to talk to pen testers, so I don't get to talk to a sales person but I call them and say well I'm going to need a pen test for this or this application. Can you, can you do this? Do you actually do this? Sometimes they say we don't have time or whatever and then if they do have time I always ask can I talk to one of your pen testers because I'd like to discuss with them the what we want them to test and the way we want to test it and I want to make sure that you can do it. And sometimes that's not even a possibility, you don't get to talk to pen testers, you just get a sales person that wants you to jot down a date and when the date is decided upon then the whole pen tester you never see here or talk to, you just get a for you an anonymous report back by someone, someone you never talked to before and there's all these findings and whatever. I don't like that. I want to talk to the pen testers, I want to discuss things with them that's, that's the most important criteria I would say and also as a CISO it's, it's helpful if you kind of dive into that world, you don't have to become a pen tester but it helps if you recognize that a report is sent into you and it's just an automated scan. That's not what you're trying to, to get, like it's fine to use automated tools, don't get me wrong but you want a pen tester to elaborate on the, on the findings they get from the tools. So yeah that's why I said that's what I would look for, I look for are you able to talk to the people that are going to actually do the test and make sure that you as a CISO can see if work is actually qualitatively good or it's just an automated scan. Thank you. Yes. I would like to ask about scope because you were talking about scope and sometimes I like to go a little bit outside of scope because where I was educated in pen testing we were providing network security and not host security. So my question is out of curiosity I'm based in Denmark and we're pragmatic so I don't get into trouble but I would like to hear if that's the same case in Netherlands and then I have a follow-up question to that if I find something which is a little bit of out of scope say the router in front of the environment is really really badly configured would you have it included in the report or would you rather have that in an email outside of the report? So I didn't really hear the last question. If I find something that is out of scope like the router in front of the network which is badly configured would you like that in an email outside of the report or as an appendix? Yeah so if you find something that's actually out of scope which you still want in the report I'll give you the answer what we did with the lockers we put it in the report it was out of the scope we actually put a picture of it in the report and looking back might not have been this wise decision but we did it's a finding so yeah it was out of scope and it might have got me into a little trouble but it's still a finding and I think it's relevant and I think it's important and we should still do something about it so as long as you are not too far out of scope where I cannot defend you anymore as you see so then you're good I will try and you know work it into the report in a way that I still think it's defendable but I think it's also a little bit of common sense of the pentester like it's fine like I said if you find something that's a little out of scope it's different when you actually actively go looking for things that are out of scope that's I would say the limit so any and actually I tell pentesters if you find things along the way that are actually out of scope but you kind of stumble upon it mention it put it in the report that's that's totally okay yeah I was wondering how do you see the role of the CISO in handling the stakeholder management on like your side as a project manager for pentesting I've often encountered parties that are very defensive you know they're afraid that they're gonna be blamed for having their job badly done while in essence pentesting is to help the organization to improve and get better and there is a bit of a shift in that but it's still they feel still feel like a slap on their hands yeah it's very important first of all that's the stakeholder management part is very important that's why it's set in the presentation you mean to make sure that you get your owners involved in the whole process so you go to them you establish the goals what do we want to get tested right now what is the goal of this test and you you make sure that they're involved in the in the end of the process where they get the presentation and they can ask the questions that sort of thing so make sure that they're involved in the whole process it's not just a cold report on their desk after a few weeks that says you've done all this wrong but they are part of the testing process another thing I find very important is that you look at the ethics side of things so if you know me a little bit you know I'm very against fake phishing emails because I think they're ethically very wrong so I would never send out a fake phishing email to do a phishing test in my organization and make sure everyone knows that so also another thing is that's why I said involve your IT department make sure they know that this test is coming because you don't want them to feel sneaked up on as well as if you do a certain certain tests that might make their work harder you need to make sure they know this in advance and also when we come with the findings the first thing there's also a presentation for your technical people so I forgot to mention this you do a presentation for the board and the managers but you also let the pen testers do a presentation of their findings for your IT departments so they can ask questions and they know what was tested and found and I will always make sure that they know you didn't do everything wrong it's not that you don't do your job right it's just how can we do even better so yeah very important I'm going to have you you're the last question and then we're going to finish because you know people want to know the answer to the poll problem the last one yay yay I'm doing auditing and lately I've seen lots of some pen test reports so that they are formulated so that they are kind of kind of risk formulate as a risk for IT and usually for a risk for ISO 2700 and someone thousand one or something else that they are like this is a risk usage so they can kind of be lifted directly to the risk management side of things what shall I take on is that a good way or kind of lazy way of doing it so what you're saying is you're in auditing and when there's a pen test done then yeah I'm just reading lots of pen testing reports and I've seen those that they are like pen test has been done and then it's oh they don't really care what's in it it's just you know we've done a pen test and it doesn't matter what the quality is I mean the pen testing results are formulated as a risk that you have this risk and then you should maybe do this and you can kind of use that in risk there's something to be said for that because in my eyes security is very close to risk management and your risk appetite can be different than another organizations as a criminologist what I like to say is it's you don't have to make your home an unpenetrable fort to make sure it doesn't get broken into it just has to be harder than the neighbor's house because most people that break into homes are opportunists and they just have a look and see the easiest house to get into that's the one they break into and hackers or criminals I must say are very akin to that so they'll likely break into your network if it's easier than other people's networks to break into or if they're apts they're looking for something in specific but that's just a risk factor that you have to may be accountable for so yeah I agree that sometimes it's a good idea to have a pentas and treat it as a risk something you need to either accept or do something about because some you don't have to do you know not all findings in a pentas report need necessarily get followed up on sometimes they are just interesting or mild findings that would take disproportionately amount of time or money to get fixed and the risk what happens if they get abused are very low well that's just something a manager needs to think about thank you so let's have a look remember that we did the poll with the managers and we asked them what it were checksum meant and like 50% of you thought it was the answer of B and another 50 C so you really have no faith in your managers which you were correct in because most managers actually thought that the checksum is some of multiple vulnerable systems that have been checked for the same known vulnerability over the course of multiple tests don't use the word checksum in your management summaries people but the main points I'm trying to make here are when you're a leader when you're a C so you need to lead the test you need to coordinate don't sit back and let the vendor decide what they want your application to be tested for that's your job you need to address responsibility because that's how you get it followed up on you need to write quality reports and be able to talk to a board if you're a pen tester that might be a little bit of a bold statement but at least you know try and see if you can get there and don't use language managers don't understand in your management summary you'll be surprised really kind of use language you would use with a toddler even though they aren't but I'm afraid that's what in our frame of mind you should do they are not toddlers but that's how we would think about it so if you'd like to follow me on Twitter that's where you can find me I have actually a bonus slide that I saved because I asked them another question I still have like one minute or something I also asked them it's in Dutch while translated I also asked them what the word PowerShell means about 53% of the managers I asked think that PowerShell is a popular operating system that hackers use to like disclose vulnerabilities to each other and thanks to Bert who kindly offered me this third option that many chose another 43% or a 14% think that PowerShell is actually a module in Microsoft PowerPoint that runs code in the background but 43% of managers actually knew what PowerShell meant it's a way to automate tasks in computers that will use a lot of op means use so sorry as a lot of people had this right I didn't use it because it's less fun but yeah there you go thank you thank you so much for this really excellent talk so before you leave you need to know that there's another really exciting talk coming up next so we've already heard from one board member here of the De Yves De the Dutch Institute for Vulnerability Disclosure our next talk here at 7 o'clock in Abacus is scanning and reporting vulnerabilities for the whole IPv4 space how day-to-day scales up coordinated vulnerability disclosure so I would recommend stick around and thank you for coming