 All right everybody. Thank you for your patience. We're ready to start. I'm Jill Orhan with the red team Village I just want to say welcome to our inaugural year We're so happy to see so many of your faces here without any further ado Please let me introduce Joe Gray and also Adrian Sonabria to hear to talk with you about the red team framework Is we'll get started with Adrian All right, so I've done some pen testing. I was actually involved with the petez with putting that together It's one of a bajillion people that that contributed to that and And part of the reason we're here is we never really finished that work like we had a lot of ideas That never got implemented. It was one of those things where we've got so many people involved with it like once that momentum goes away just everybody moves on with their life and does different things and and there's still a lot of work to do there and also, you know worked on the defender side and Studied the industry, you know, so I've I've really seen this industry and the job from all angles at this point So about me I'm a senior security architect 2017 Derby con Social Engine capture the flag winner member of the password inspection agency CTF team third place last year in this year No, look on the second place to share besides Atlanta Currently under contract to write a social engineering an OSINT book with no search-press tentatively titled securing the human element so We have all these frameworks. Why do we create a new framework? Well, I mean, why would you create a new framework? Yeah, I think the key for me is is that a lot of stuff isn't working the way it should be working You know, it's it's the way we do a a pen test, you know We hand off the results and we don't really talk to the client after that, you know, maybe we do a 30-minute Chat with them just to make sure everything went okay, but I'd say nine times out of ten They haven't read the report before we've done that and and there's really no improvement cycle with pen testing And what we've seen happen over the years is it it's you know ends up being the pen tester having a good time with the customer's vulnerabilities and Which led me to come up with the term vulnerability But I've actually seen pen testers request to extend the pen test just so they could get an exploit working There's zero value for the customer in doing that, you know Just you know good enough. Yes, maybe somebody better than me could exploit it move on, you know, but I Think the that lack of focus is part of the reason that we're doing this What's going that term the technical Tarzan by the way technical Tarzan? Yes. Yeah So with this we're gonna use a few terms red team purple team blue team Pen testing vulnerability assessment web app testing and and You know, I did a series of talks last year called. It's time to kill the pen test You know, I just just I don't really think it's time to kill the pen test But it you know, it was intended to start a conversation and get perk up people's ears but in preparation for that I asked people what these terms meant to them and And got just about as many definitions as people that that answered the question Especially with pen testing, you know, and I was shocked to find how many people when you say pen testing They think web app assessment when you say pen testing they specifically think web apps Which if you're an older school guy like me, I turned 42 days ago You know, you think network pen testing when when I hear somebody say pen testing, you know And web app was always something like a web app assessment is so far from a network pen test Totally different tools different skills usually different people doing it, you know, if you if you do each one So so yeah, we've got a big gulf in just terminology and language and what people mean when they say words Absolutely, and you know in this industry, especially with people giving talks and I'm not going to hate on other speakers But you know oftentimes we run into people who I'm a red team or I'm a red team or but all they do is consecutive Pen test consecutive vulnerability assessment. They are using offensive red Techniques, but it's not true red teaming in terms of it is more domain admin oriented and less Objective oriented. So let's get into what's wrong with this dichotomy. Yeah, and what he just said I mean, that's that's a good point there. You know, I had one client that was just so frustrated They had gone with three different very well-known pen testing companies and said, please hack our Linux stuff Nobody goes after our Linux stuff. They just go after DA. They go after the windows stuff and And they're just so frustrated. They couldn't get anybody to go after it, you know Because in the pen testers mind or red teamers mind like it's a win-lose situation You know, they want to win. It's like like playing a CTF, you know, you go after what gets you the most points But in reality, you know, the customer wants something, you know fairly, you know, I Mean, yeah, if it's objective-based and it's easier to go through windows to get to the goal Sure, you know, but when the client says forget about the windows stuff go after Linux and you still go after DA That's a problem Just just for my curiosity Who here who in here is a pentester? It does some kind of some kind of pentate something that falls under that blanket and defenders Okay, and what else do we have in the room? Just trying to get an idea of like compliance compliance Okay, sorry man Hey, I mean security is a spectator sport sometimes I we didn't intend to out you I say that because my day job is actually a compliance job, too. So That was like 60 40 Absolutely And I would just venture to say there's a lot of the reason behind that is there's no me me cats for Linux Yeah, I mean somebody should like make me me dogs or me me pink ones So I don't know if you can see this from the back of the room but but generally, you know part of the reason I put together that that That talk time to kill the pentest last year's I felt that the design was fairly flawed Not only is it not an indicator of an organization's risk, you know, because I mean they're especially a large bank or something like that It's just best effort, you know, you got 20 hours 60 hours 80 hours to do the pentest There's stuff you're never gonna touch. There's stuff you're not gonna get to I mean Just to interject you can't get an understanding of the environment to actually make the informed decisions that a Well-informed a well Executed adversary would do because depending on which industry report you trust you're gonna have a dwell time of three months to seven years So you're not gonna find out enough in 40 hours to be able to do anything of that magnitude Or the flip side of that is you have a better understanding after 20 hours on the network Then then the people that are running it. Well, which is which is a different issue And not because they don't spend time with it because they don't look at it from the perspective that you do You know, they've never run the end map on their own network. So, you know, you're finding stuff That's that's just never occurred to them to look for it Also doesn't simulate adversaries, you know, how often do we do we launch ransomware simulated ransomware? During a pentest that's what's actually hitting people, you know, or sending BEC emails, you know Trying to convince them to move money from one account to another We're not actually using the attacks that are impacting people when we do a pentest Instead like pentest has become its own kind of isolated thing and red team where You know, we use the social engineering toolkit because it's there, you know, it's it's good. It works You know, but we're not approaching it from a standpoint of okay What would the bad guys actually do sometimes because those tools don't exist or it's just not Easily feasible to do that during an assessment like running ransomware, which I think we actually absolutely should do You know, I love that an NCC group client Actually asked for them to make a neutered version of want to cry and run it in their environment Like I think that's something that everybody should be doing because it's what's hitting them. Absolutely I mean you can easily in various languages web-based like JavaScript or even a compiled language Execute functions that would actually copy the file and attempt to encrypt it because Let's be real. You don't need domain admin to wreak havoc. Can you send an email to yourself? Can you send an email with an attachment to yourself? Can you install something or you know, can you encrypt a file? Can you change a file's permissions? Those are all things that could be far more devastating the DA if if you I mean if I can get to your crown jewels and Email them to myself. Who cares if I'm DA. I've got everything I want. I'm going to cripple your business And the other thing we have to understand about about red teaming pen testing is that As a business logistics play a part, you know If the guy who's really good at a B&C isn't available because he's doing another engagement somewhere somewhere else You might get the junior pen tester You might get somebody who's less experienced or you might have somebody doing a web app test for the first time They're not going to tell the client that but that's who's available. You know, there's a calendar. Everybody else was busy You know, you're getting one person in their particular skill set You know, so consistency is super difficult with with this kind of thing when you have one or two people doing the assessment And it tries to prove and disprove. What do I mean by persistent negative here? So it's the idea that You could do perfect on a pen test pen tester comes in doesn't find anything You're fully patched and you can still get popped tomorrow that organization can still have a breach tomorrow because It's never gonna this assessments never gonna cover a hundred percent of things, you know, so it creates this false sense of accomplishment or security And I mean just to summarize the last two bullets, I mean, it's inefficient Back to Adrian's point. You're only gonna see a small segment of the of what could be done You may be focusing on web app when someone's coming through and you've got RDP open Or you don't implement training and Someone paid me and I'm fishing everybody or calling people something to that effect. I mean, it's not efficient But at the same time it doesn't make money for companies to admit that They are not the end all be all I mean for the first time I went to the vendor floor black hat this year and I mean It was all advanced persistent Blockchain enabled AI ML military grade Super women dying synergistic Garbage So, I mean, there's no money in admitting that you need a relationship with a feedback loop such as like UDA That can actually take something like this into account look at true business objectives things that could impact your business and actually confront the problem head-on because It's kind of like one of my complaints with like the medical industry. We're all about symptom management Here take these pills. You're good, but the side effect of that pills gonna make you come back three months later Oh, you need you need this you need this and it's symptom management. It's not trying to find a cure for the problem Yeah, yeah, and root root cause that's that's definitely and some you know So so I would like to say I've seen huge improvements in pen tests Over the last five to eight years, you know, you do see a lot more reports getting into the root cause of issues Which is often in people and processes Whereas, you know the traditional pen test report would just say missing this batch, you know had that vulnerability and The customer ends up fixing the symptoms of the problem rather than you know Why was that patch not installed in a timely manner in the first place? Why were those vulnerabilities present not discovered? So one of the things I you know the the company I started started experimenting with this a lot It doesn't exist anymore since been acquired and stuff long story, but We said, you know So speaking of inefficiency that you know two parts of a pen test that just don't have a lot of value for the customer Running some kind of vulnerability scanner, you know or whatever Tools you run to gather the data in the first place to figure out what you're going to go after That's a huge chunk of time that you charge in the customer for it And then on the back end of that writing to report another huge chunk of time So you get this little bit in the middle. That's actually doing the pen test and in some cases I've seen that's as little as two to four hours of actual work on a 20 to 40 hour engagement that you can actually call pen testing, you know So I started thinking how can we expand that how can we get rid of these two inefficient bits on the on the Front end and on the back end on the back end or reporting a lot of that's automation a lot of companies are doing a great job on Automating is as much of the pen test writing as possible You know some even using portals, you know where there's no word doc no PDF But you know a nice portal where the results go, you know great work being done there But on the front end I thought what if I just use your scan when I come in, you know You're doing scans already what why am I going to run Nessus, you know And you pay me to run Nessus for eight hours when you've already got them completed and a hundred percent of the time I found misconfigured scans and like well, there's part of the problem right there, you know I found in one case they were scanning all their web servers and Somehow and how they copied and pasted the domain names into it And was missing from comm on all but one of them So they're getting just enough results that they thought the scan was working But didn't notice that 13 out of the 14 websites weren't getting scanned at all because of a typo I would say there's a business in Columbia. That's probably mad about that too. Yeah, yeah business in Columbia Yeah, that's you know, but But yeah, that's huge like why aren't we checking to make sure that they they're running a scanner correctly in the first place We're gonna come in do a red team. It's like kicking a baby in the face to teach it martial arts like it's it's Pointless, that's not how you learned so basically in that let's look at the contrast of pen testing versus red teaming so The ponage-based part of a pen test. That's that whole technical Tarzan mentality of users are stupid. I'm super lead I'm gonna come in I'm gonna bulldoze you and I'm gonna give you a report and you're going to enjoy it and that's about it Pintests largely as of late have became a tool for compliance The second that PCI mandated that pen tests are required then vendors started salivating and how do we commercialize this? So as a by-product, I mean you can go out there and find companies that are gonna give you excellent in-depth thorough pen tests You're also gonna find what John Strander first to as the pen test puppy mills And ultimately cash cow. Why not exactly? So that's one of the big problems in changing the industry is is you know people are gonna buy it regardless of how good you make it So, you know, there's some of that why bother element to it Absolutely, and this I coined determine that last bullet the digital melatonin to help management sleep better at night We've had a pen test. We've had some really good hackers coming in trying to get in We stopped them in their tracks Meanwhile, they're missing the apt unit coming in via their exposed RDP because that poor was not within scope in their statement of work So with the red team You're looking at the specific a specific actor or a technique or an IOC based on something and It's based on an objective something like I want to see how long it takes my Sock to detect someone coming into our our enterprise From a web from the website pivoting into finance That is an objective you can measure with a red team engagement How long it takes to get DA or if you get caught giving DA is not an objective That's outcome and that that's a huge that's been a huge missed opportunity for years that I see done a lot more unfortunately And what I'm referring to is detection like like involve the incident responders Involve the whole security team and seeing if they can detect the attacks that the pen testers are using huge opportunity I mean, it's either that or wait till you actually get hacked and see how you do You know, so if you're gonna pay the money to do it, you know Have everyone try to detect what the pen testers are doing talk to the team beforehand and say You know, maybe let's pause after each day you tell us some of what we did and we figure out what we missed You know that and that's part of building this this loop the cycle where you're actually learning from the pen test instead of Oh, they found some vulnerabilities. We're gonna pass those Because those are vulnerabilities, too, you know when they're When they're sweeping the network password spraying and nobody notices that's a problem like that should there should be alarms Getting set off everywhere for that. You know people should be freaking out So absolutely and and measurement is as you will see later in this presentation Measurement and detection is one of the key steps within our process So we're gonna go through a few myths that we've heard through our travels and years of experience Again to spell them. So you want to kick it off? We've probably covered a lot of these. Yeah, we've probably touched on a lot of these So so we might want to accelerate through this, but but yeah, you know, so so yeah the digital melatonin You know, we talked about yeah, no, it's not an accurate measure Measurement it's a best effort. It's whatever the pen tester happened to touch, you know So again, that's the good thing to do coming out of the pen test, you know, which IPs, you know How much of the network were you able to cover? What did you touch? What did you look at really closely and what did you just skim, you know Try and get some kind of good idea of how much your different assets were looked at which ones are missed Maybe the next time the pen testers come around have them focus on the stuff that was just skimmed in the previous pen test This myth According to Adrienne's colleague Haroon Muir Pen testers emulate other pen testers. They don't emulate adversaries and that was from 44 con, correct Yeah, that was the first keynote for four four con in 2011. He said that and here we are almost a decade later It's a good talk. I suggest checking out the video on YouTube. It's called pen testing considered harmful And to think about this before we move on to the next myth This is a perfect example of why those of us working in the offensive space Need to actually study defensive techniques and what other attackers are doing So you need to know what what your Opponent if you want to use the us versus them mentality or the blue team or sock whomever, whatever you want to call them I mean anything What are they looking for? They want you to emulate APT 29. What are the IOC is associated with that? What are the TTPs? Because IOC indicator of compromise and TTP are two completely different things IOC That's kind of like after, you know, you fall and break your arm and skin your arm. That's the blood stain The act of what caused you to fall break your arm and bleed out on the sidewalk is the TTP So Understanding that you need to know like from the social engineering perspective. What persona are you getting in to do this engagement? How many people in here have played with infection monkey? Free tool called infection monkey. Yeah Yeah, so I earlier I mentioned, you know, yeah, if the bad guys are running ransomware through an organization We should be too open source out there Brilliant easy to set up all the instructions are on github. I think Garda core makes it and gives it away for free MIT license But infection monkey is basically neutered ransomware that you can just deploy and Let it tear through an organization You can give it a list of passwords to try It'll try some common vulnerabilities that are out there to to spread throughout the network and It's best thing I've seen to simulate an actual ransomware worm getting into an environment and great great tool for obviously you want to you probably want to talk about the Organization before using that in a pen test, you know, because that's not something you traditionally see in a pen test But all the clients I ever talked to where I use that tool in an engagement. They're like, yeah You know their eyes open real wide like both goes on. Yeah, I'd like to see what would happen if ransomware we're in here without any negative results Absolutely. So yeah infection monkey is a good tool. So We've been talking kind of about why pen test aren't as great as we are led to believe But we're not going to say that they don't deserve a seat at the table They do serve a purpose in life if nothing else for things like HIPAA PCI and other compliance frameworks Additionally, it could be a stepping stone in assessing your own maturity to be prepared for a true red team assessment If you can't withstand someone coming in and getting DA, then why are you going to go for something a little bit more advanced? so so again Red teaming pen testing similar disciplines Not the same You could say red methods. I really like this one though Black box testing honestly from my perspective and based on insert industry reports here Based on that old dwell time statistic. We've talked about There's no such thing as a black box internal test they've got time to look around they're able to Check configurations look get p-caps Maybe even log into your sim look at your diagrams They're understanding what's going on The only thing you should be considering in this perspective as a black box test And maybe even not even a black box probably gray box would be your perimeter Yeah, I think it's important to explore options and present options to you know for the pen test team and for the the buyers You know Understand why they're having it like do they want to scare the crap out of management so that they get budget for something? That's still a thing these days. Unfortunately Unfortunately, I I call it the oh shit moment getting to that point where key Stakeholders believe that yeah, they're they're really our problems You know because there's there's still this feeling Well, nothing's bad happening. Nothing bad has happened yet. So whatever i'm spinning on security right now is is fine But um, but yeah for a lot of pentas That's still the purpose of why they're having it is to try and convince management that yes, there is a problem Yes, there are things we have to fix You know, we we need more tools. We need more people And hopefully the pen test will help us accomplish that goal So if that's their goal then then fine, you know work with work with the client to help them achieve that goal You know as quickly and efficiently as possible. I actually put together a A a different kind of assessment. I called a breach impact assessment And actually designed to do it for free is is kind of a loss leader just Just to show people that they have gaps in detection, but we'd run a bunch of automated tests that would simulate a post breach activity or post intrusion activity Lateral movement exfiltration stuff like that a lot of that stuff can be totally automated You can just sit down with them in the conference room and explain what you're doing We'd go through each of 13 tests. Okay, you know now what i'm doing is i'm i'm pulling in zero day malware off the web So if you have anything looking at the wire for that kind of stuff This is a test of that or i'm exfiltrating data out And in clear text if you have any kind of d.o.p. You should get that And we had a bunch of ginnies in there like just grabbing icar off the internet like Somebody should get some kind of alert Hitting the kill switch for want to cry Your mssp should get the alert 100 percent of people's mssp's failed all 13 of those tests every time we did this assessment And it was very effective having everyone sitting in the same conference room. They're looking at their phones waiting for it to blow up You know, we're going through each of these tests explaining what each of them does and and nothing on their phones Nothing at it's it's very very effective and you accomplish that oh shit moment In 60 to 90 minutes versus a whole pen test after somebody reads the report So introducing the actual red teaming process This is a 10 step process Um, I couldn't think of two more steps to get it up to 12 Um, honestly, we could actually go down to eight because retesting and purple teaming would actually be optional Oh, that lady gaga's long just hit my mind the paparazzi We were talking about that earlier. Um, but anyway, so just to walk through There we go. Um, just to briefly look through this. Um, Basically, we're scoping identification of the threat model baseline security assessment re-scoping learning phase execution Measurement debriefing and then the optional retesting and or purple team assessment so during scoping We're going to talk about the objectives to find success for the client talk about time of course money Um, and then the number of systems involved and of course start the conversation About the rules of engagement and what we like what kind of stuff do we want to utilize? The the assumption with this would be that most Most customers that would be having this conversation would be in the mindset of i'm scoping this for a pen test That's why there's a re-scoping phase later where this information is refined Um, adrian, do you have anything to add with the scoping? Okay. So with the the threat model Um, from my perspective, I would say it's going to be be based on the client base Who is your customer doing business with? What sector or industry are they in? Are they affiliated with any governments? Where are they physically located? What are they doing? That's going to drive your threat model if you're dealing with a financial services company Their target is going to be a lot different than say a manufacturing company I I think a lot of these stages as we go through these uh should be should involve the client more than traditionally they would have You know From both sides, I think threat modeling should be you know should be informed by what's actually happened what's going on what could happen um, and just uh Really useful to to walk through them with it and talk through them with it because You know, especially on the pen testing red teaming side if you don't understand how their business works really well You're probably not going to understand what they're worried about what their their own Equivalent of threat modeling is you know, especially if you get into manufacturing and stuff like that like I I've learned a lot sitting down with clients and going through this kind of stuff. So it's it's uh hugely valuable even if it's just One hour two hour Go-to-meeting whiteboard sessions something like that So then we have to assess a baseline security model this can be derived from the critical security controls um at a minimum I would say to be tall enough to ride this ride. You probably at least need the top five Uh, and also if you have the top five, it's estimated that you're more secure than approximately 65 of other companies in the world Um other things you can look at would be NIST special publication 853 But you also want to assess where it is your previous testing What are you what are you basing this off of? What's the vulnerability management posture if you're not patching things there's really no sense in doing this um And do you plan on monitoring do you plan on? executing incident response with this um, and if you if you don't have certain levels of maturity then Your money may be best fit elsewhere I'd say 80 90 of the defender of the buyers for pentesting services are lacking the top five all five of the top five Um, so you start to ask yourself. Does it make sense to even do a pentest? You know and and they'll tell you they'll yeah, I mean There's this stuff over here where we haven't even looked at yet. We're not we're not patching it We know it's deficient, you know, like they know where a lot of the problems is and what they really where the problems are So in a lot of cases You know, they just need help figuring out how to do some of that stuff, you know, and there's not anything I would I see a lot of companies, uh You know have on the on the menu of services Uh, that fits that like help me do the top five You know, and then let's talk about a pentest like like what what is that? Assessment call, you know, and and there are companies that do this stuff that just do this general consulting And uh, you know again, uh, experimenting with my own company what we came up with is, uh, What was a subscription service? Where for for different rates, uh, you could get four eight 16 hours of our time every month We would do a baseline we'd figure out, you know, which critical controls have been implemented Basically the maturity of the company how far down this, uh, down this path They were and we create a roadmap for them And you know, we we wouldn't really hard hold them to that roadmap, but we would check in with them once a month They most of our clients ended up using probably 30 to 50 of those hours for incident response related stuff We wouldn't fly out or anything, but uh, we get on the phone with them help them understand You know walk them through some of the incident response stuff Build a plan for them if they didn't have an incident response plan but basically, you know, we found You know, our clients found a lot more value in having somebody they could pick up the phone and call Year round then to take that same amount of money and blow it on a single pentest once every year So for for that 80 to 90 percent that has done none of the top five critical controls Maybe something like that makes a lot more sense than you know, let's do a red team. Let's do a pentest But that's that's all people know to ask for so that's what they reach for so I was gonna say like If a 64 ounce, uh, coke is the only thing on the menu Yes, you know, I mean guess what people are gonna order Maybe it's not the best thing for 100 percent people to to order but Exactly and we discussed scoping. So here's rescoping. So we're refining the objectives. We're focusing the scope Um on the following. So we already kind of talked about time and money But now we're talking about the execution time frame. When do you want this done because a pentest You can schedule that for next week for this because of the learning phase that's about to follow this You can't because there's a certain amount of stuff that you're going to need to know to properly Emulate that dwell time So you're going to need that and you're going to need to refine the number of systems within your scope There's a good chance that it's either going to go up or down. It's probably not going to stay the same Um, then from that do they want you to explicitly do something? So Are that I don't like the idea of them confining and saying no social engineering or no web app or None of this, but if there's one thing in particular that really concerns them that they want you to highlight It's good to get that out and the other thing that changed with this is Depending on the sector. It may even be worthwhile to solicit input from one of the uh iSACs So financial services energy retail any of those because You have group knowledge of what's going on across your entire industry Yeah, iSACs are great. You know, I wish they happened sooner But if there is an iSAC for your your industry, absolutely like a lot of them have Thread and tell that specific just to your vertical just to your market And um, and yeah, yeah, I mean someone else is going through the same problem that you're having they're having the same challenges and I can't tell you how often I've seen amazing solutions to stuff and I've asked them Have you shared this with anybody? Like like this is great Like like it's easy. It's quick quick quick. It's cheap. You know, it works for your market You know, you don't have to hire additional people Like you should write a blog post or something about the my company won't let me, you know and You end up running into that over and over and over where people have had to come to these Eureka moments individually, you know Whereas that they could have benefited from more sharing and iSACs kind of solve that problem because Most of them are private, you know, you have to be in that vertical. You have to be vetted first To be able to share and and become members So so I like seeing that and there's tons of them now. I think in the early days. There was just a financial iSAC But yeah, there's at least a dozen now that I've seen Yeah, it's like an aviation iSAC. There's I think there's a maritime iSAC that About as many as there are villages. I was just about to say if there's a village food. There's probably an iSAC Yeah, probably so so after this we'll set up the red team iSAC Totally kidding. Um, so we don't know what any of these acronyms mean. We just right Is that what they meant by that's not what any of this means? Um I guess so So with the learning phase like I said earlier depending on which industry report there's going to be dwell time So there's ways you could do this get data from the sim It could be data dumped out It could be just giving you access to the sim as the red team that's actually going to engage in this That way you can understand What's considered normal because honestly depending on how detectable you're trying to be You need to some concept of normalcy for your target organization So you might want to look at previous pen test reports Might be worth something might not Either way, um packet captures Net flow other monitoring tools get a feel for what's going on because The adversaries may not be using these same exact tools, but that's ultimately what they're trying to do They are trying to learn Um score diagrams get configurations Um interview the administrators the security administrators the network administrators system administrators HR anybody you can interview Yeah, so this is uh, this was a big thing You know when we set up that subscription service with clients that we did is is we would just go through configs And just all day long every day anytime we looked at a config we found serious serious issues And you know just like head forehead slapping like no wonder, you know, you're going to get hacked tomorrow you like You know, it's worth going through What's a deny all So with execution, we're not going to get into telling you how to execute things You you already have the the things on the table for you Here's three examples of what you could use for technical frameworks to actually perform the execution Um, we saw this as an opportunity to avoid reinventing the wheel Um, we're just trying to kind of refine the tire Yeah, the tire or the car itself to help it go a little bit more smoothly Maybe the shock absorber Um, I'm not mechanically inclined. I apologize, but basically No two race tracks are the same you're going to set up camber and stuff like that differently no two Companies are the same. So it it's okay if no two pen tests or red team engagements are the same You know, it's a custom service. It should be that's why we have all these apt numbers because they're not the same So this is probably the most important slide of the entire presentation This is getting past that whole Um, technical tires in beating your chest because you're so late and they're so dumb Great you got domain admin you go at access. What does that mean to the business? They didn't detect you. Okay. Is it because you're so super stealthy or is it because they don't have the Capabilities kicking a baby in the face. You shouldn't be proud of that exactly so These are the specific data points that I would recommend so time to detect I'll share a quick anecdote with you. I was doing a fishing engagement against a grocery store chain When I do mine, I don't use automated tools when I fish I send them out in batches of six to ten Um Come to a social engineering talk of mine at some point. I'll tell you more about how I actually do it The tldr is uh about halfway through I got blocked At the time that I got blocked I had a six percent success rate on scoring creds Well I checked the stats the next day. I was like, uh, it's gonna be bad again I went from six percent to 42 percent because the action taken By the network administrator who blocked me was to forward the email and say do not click this link With the link still in the email correct And not blocking it going out so The time to detect for them. I mean was about two and a half hours Okay, that's it could be faster, but it could be far worse And that's not an isolated case He actually he told me this story when we were eating lunch like an hour two hours ago And I have the same exact story except it's it's even worse than that It's the network administrator shared it and seven forwards later, you know, this was a company that hired out, you know, basically you know EOD type company, you know hiring out People do shady things and shady companies and you know stuff like that Somebody in iraq opens it and puts in his password twice and we get it, you know, it doesn't have a great grasp of english you know and and uh forwarding the forwarding the weaponized emails is Yeah, and and that that's going to happen over and over and over again, right? So we also want to look at the quality of the report Um the accuracy of the report and then the efficacy of the actions taken because honestly Forwarding an email with the link in it without blocking the link or doing some sort of mitigation It's not really an efficient thing to do Uh and for them it didn't work out in their favor. So these are all things to be graded on Um, and then finally we end up in the debriefing phase that I'll let adrian address Sure. Yeah, um Yeah, and I mean the thing to remember here is is um, you know, a lot of this should get covered along the way You know, I think there should be more touch points. Uh, you know, we talked about doing the the threat mapping with them You know some customers at the end of each day, they might want a debriefing, you know, that that's not a bad idea Uh, things are going to phase in the pentester's mind as well Um, but definitely I This is the most important part and honestly when I was uh for all my years of pentesting This is the part most often skipped I would often run into clients at conferences and things like that and say, oh, hey, how are things going? How's it? You know, what happened after we dropped off the the report? We haven't even opened it yet. You know, this is six months later. They you know other things going on You handed them a 64 page report. Of course, they haven't opened it It's a 64 page report, you know, like half of that are screenshots You know, you got to make it consumable. You got to make it make it actionable Um, and a lot of that, you know, I think getting into the detection stuff helps a lot Like how could you have caught us? Like what's the obvious things that you could have done that would have busted us along the way? You know, so this is I I disagree with you. I think this is the most important slide Maybe we can just compromise and say the whole thing's important No Stay tuned after this we're going to have an MMA match in the hallway Um, but anyway, we're going to get the statistics from that measurement phase and then the recommended actions And because you're emulating an actual adversary, you don't have to go and research everything about how to detect this Because you're reading this from a threat intelligence feed to find out, you know What's going on the IOC's and the TTP's you can just reference to that and say, okay Well, here's here's what you can do to remediate that because undoubtedly some vendor has already written a blog post To try to be first to market with it Um, so and then the other thing is I like the idea of a qualitative score Low medium high I wouldn't say yes or no because it's not a binary thing I would say that with the whole actions taken piece that we used as an anecdote That would probably be like a low or a non satisfactory But there's no sense in trying to establish an actual numeric value to it. Stay very qualitative in nature So high medium low severe critical whatever whatever term you could go with the traffic light protocol version Red green yellow blue whatever And then here come the optional steps as well. So you're going to let them retrain adjust and retry Um, you're going to have to be a little bit sneakier about the way you retry Because you're still going to have the information you might need to do another bout of the learning phase But they're going to be watching for you. So you might want to like catch them off guard at some point in some way shape or form Yeah, so um, I'm gonna well disagree a little bit on that last slide with the with the qualitative stuff So with the breach impact report that we did There were it wasn't comprehensive But these 13 specific tests that we would do that simulated attacker actions that they should be detecting We tell them how many each of them was pass fail Either you saw it or you didn't and there are all things that you should have seen Uh, does that cover everything? No, but these are all 13 of these tests Are common things that you should be able to see that you are going to see from adversaries And and that's one of the one of the core problems You know, we probably should have mentioned up front with with pen testing is it's very very focused on the preventative and not the detective And that kind of shifts the mindset to Just spending money time and effort on that preventative layer But then once somebody gets in You haven't you haven't trained that at all you haven't tested for that at all and you're not prepared for that at all So that's why you know in in my opinion, you know, somebody of the public breaches we see are handled so badly They haven't thought about pr crisis communications Uh, how to talk about it Um, how to respond to it You know and the the initial reactions, you know are generally the wrong things to do Try and hide it pretend that it didn't happen that kind of stuff and that just exacerbates the problem so um Yeah, I you know, you got to think about it some of the stuff I think you can score but you know if it if it's something that's uh more subjective, you know, like like How well do you patch? You know, that's never going to be past fail. You're always going to miss patches You know, I think what gets important is how do you handle when a patch is missed? You know, do you have a plan for mitigations for that? And absolutely should be building to be able to defend an enterprise with unpatched stuff because it's never going to be a hundred percent Which leads us to purple teaming which basically would be A red team engagement that may have the red team and the blue team in communication with each other or even in the same room Um in the past with red team engagements I talked to joe vest He was he was the technical editor of red team field manual and what him and his team did At minutes before they were bought by spectre is they would start and they would do something And as they continued through their engagement, they would get louder and louder creating noise And then finally if if the client didn't recognize that they would just play thunderstruck through their speakers I'm personally a fan of rick rolling and I was recently just introduced to the klingon rick rule I like that even better Because you know rick ass will alone you any of this pixar movies, but one Oh Go ahead And another reason that we this is another reason that we did the You know the the um subscription the monthly subscription You know because you know imagine if you're only improving your security based on an annual pen test Versus you know if you're sitting in the same room with somebody else and you've got these smaller constant iterations So I mean we I think ultimately we need the equivalent of What dev ops does for Building applications and writing code But for security, you know if we're improving every week a couple times a week a couple times a month If the attacker and the defender are sitting in the same conference room You know in the moment the attacker gets some kind of success the defender is learning from it and they're fixing it And that that actually happened with clients when we did that service, you know they they would We'd be going through configs and fixing it in real time with them on a You know just sharing their screen And on a go-to meeting something as simple and humble as that did a lot more than an annual pen test did for So here are some supporting frameworks. I mentioned them earlier. These are just links to them Um the bottom one, uh, if you can't see it very well, uh, it's from o wasp I'm sure we're going to share the slides, right? Absolutely So when we put our twitter handles up on that slide, just make sure you take note of it because that's where it'll end up Um my upcoming speaking engagements, uh, I'll be a defend con in seattle hacker halted Which I have a coupon code for anyone who wants to come for free Uh, you can do it and then I'll be doing a talk and a two-hour oscent workshop at the wildest hack and fest Only thing I have coming up right now is is virus bulletin me and haroon mere are doing the closing keynote And it's going to be kind of a a takedown of shady marketing stuff Like like awards that vendors buy stuff like that Sorry in advance for any pain that causes So, uh, I'm offering my oscent training through the oscension. I've got some upcoming training environment training opportunities I'm going to do in person Probably right before derby probably right before hacker halted I'm trying to get to dallas philly in boston this year. My format is it's a one day oscent course followed by a four hour ctf Uh targeting the local businesses and like I don't mean like your local gas station I'm talking like oscent against nearby fortune 500 companies or publicly traded companies that have enough of a surface that you can collect stuff Uh, I got stuff coming online Uh, here's that coupon code to hacker halted if you want to come in for free I like free stuff Uh, the keynotes this year we have kc ls of bug crowd Paul acidorian from security weekly jenny radcliffe from the united kingdom Other speakers include jeff man um ginsburg myself marcel lee He didn't submit this year right Or did you I don't remember I did it didn't get accepted. I'm so sorry Oops That must have been that meeting I missed Awkward moment right here I'll tell everybody to submit CFP will open up for it again and probably like I think february Um the recon ng training i'm offering two two hour recon ng uh ng sessions. It's the version 5.0 the new one um You'll need some api keys. So uh sign up sooner rather than later if you want to do it use those coupon codes Um, I'll get you in cheaper and I'll get an email to you telling you which Api keys you're going to need The cool thing is i'm not going to tell you to pay for anything The only optional api key that costs anything is have I been pwned and that's 350 a month so any questions Yes, what role do you feel white carding has in red team engagements like Being given access to certain parts within a network To more efficiently use the time that you have during an assessment Do you want to handle it are you already sure? Yeah, no, I think that's huge. I think that's uh this whole assumption of uh pen test not being legit If you have to earn every step of access like I said before it's you know an 80 hour engagement It's a 20 hour engagement You know, I I think you're increasing the value when you make certain assumptions like let's assume you've gotten in Let's let's give you an account. Let's uh You know we think somebody could get that far. So why don't we start you off that far? Yeah, absolutely. And that's that's where that breach impact assessment with those automated attacks we did came in so valuable It's because let's assume the worst has already happened and you know Could you actually detect it and respond to it in any kind of reasonable amount of time? Yeah Yeah, so no, I think that's super important and we need to get over No, let's make it super legit and like, you know, because If you want value out of that all of a sudden you're in like 30 40 000 range to have enough time to do that properly We got a question over here a couple of them actually a few we got a question pod um, so One thing I've noted in the environments. I've worked in is that nobody wants to beat down Or have systems down or impacted that people are actively working on Do you think there's any benefit to adding systems that are capable of being targeted and actually locked down or infected That are not going to impact their daily operations Uh Did someone say they couldn't hear is it okay, so the question was is there any um Value added to testing Systems that should not be taken down um I'm conflicted with this Each of the different but you have a system of sock that you can actually same thing in like the hr departments or any other divisions That would actually allow the teams to know setting up simulations in other words Right, right, but you're you're setting up systems that are non-production, right? But standard configuration. Yeah, I I personally like that idea My one flaw that i'm going to see with it standard configuration Most companies don't have one of those well, no he means the same as the production I I agree and that's what I'm I Fundamentally, I agree. So that this is the whole concept behind behind Uh cyber ranges The whole idea behind the cyber range is that you you make a copy of production that you can do anything you want to Like like you can do proof concepts in there. You can test out new products In there you can do Anything you want in there it's non-production, but it's it's a very close copy of what production looks like Obviously once you get into ot and you get into like skater stuff and you know Stuff you can't just throw into a vm that gets a little bit more difficult Um, but absolutely as opposed to leaving it untested Yeah, yeah, no, absolutely. You should do that. Yeah, there's a lot of value in that Actually, I really like the framework now Is there any reason why the framework shouldn't be applied to more than just red teaming? But include for example, you know any customers reaction to pentesting compliance check and whatever You know Yeah, yeah, I mean I mean I mean forget the word Red team and and there is a document that we're working on Um, but yeah, no it applies to more than just red teaming and it's it's meant to be You know an a la carte menu where you can pick and choose and customize something out of that So, yeah, if you can make that apply to compliance, you know, and absolutely it does apply to a normal pentest Then yeah, yeah, and that um, yeah that that Feedback loop does need to be better defined. Yeah, I agree I was wondering based on your experience for the enterprises How often do you think that this red teaming? Activity should be performed and how long? Ideally would it take Yes Yeah, I mean it depends on how much you're going to cover how much you hope to cover with the with the assessment I mean so so that's that's a huge problem in that that's uh, you know scoping properly doing scoping and Somehow calculating the number of hours that you need to do it Um, it is pretty tough. I don't think there's an easy answer to that right um The velocity of the learning phase would have a lot to do with that as well And then depending on what the actual objective itself is That may drive that as well so So the problem you run into is you need the kind of details that you gather during a pentest to properly scope the pentest so you So, I mean we've asked people for nmap scans or for existing vulnerability scans. Let us look at those And uh, and then we'll tell you how long we need You know, so I I think that's a legitimate and again that gets into, um, what'd you call it white carding? White white white carding. So, I mean, you know, they they might Object to that but I mean Thing is if you already know the vulnerabilities, you know, you can just help that much quicker anyway So yeah, I think you should start. Well, obviously you're going to get an nda in place So that's that's the other hard problem, uh to that Is you're talking about looking at sensitive data before you've even they've signed a contract You know during the scoping phase so so that that can be a little tricky just with the The the contract process just the process of getting them signed on so So yeah, if if you can hack that process where you've already got a contract like an msa in place And and an nda in place before you actually give them the statement of work And insert this phase where you're going to go over vulnerability scans together and information about the environment Maybe even do the the threat mapping phase before you actually Scope and sell the red tape I think that's how you have to do it because doing it blind it's going to be wrong most of the time Either that or make it easy to extend it, you know Tell them hey, it's going to be between here and here. We might use less money, but You know get more time pre approved or something like that, but it's a tough problem Yeah, this is more of an observation for my experience in this. Um, I run an adversarial team for a fortune 500 retailer um, we do pen testing and red teaming and most Red teaming is pen testing on the market. We talked to a lot of people There's only a handful of organizations out there that my experience Understand the difference between red teaming and pen testing We outsource our pen testing and then we kind of do our own red teaming internally We differentiate the objective based adversarial assessment And the best time for a red team is during a pen test. Um, so My experience is blue teams Love pen tests. They despise red teams Um, because they don't know how to catch adversaries Um, because when you go out there and you talk to red teamers a lot of times You talk about methodology when you're onboarding them and they talk in it and they go through the whole process Like that's a pen test You're like no, no, it's like I know a lot of guys in the underground Who do this for a living in like real life on the bad side and like they don't use nmap They don't use vulnerability scanners I said, so if you know vulnerability scanner or you talk to your we asking their tools if they use If they give us lots of tools nine out of ten times they disqualify themselves It's like, yeah, no, thanks. We're good. So Really that your pen if you're doing a red team you need to really understand what the difference is because there's a serious difference No, that that's insightful I I think that's fantastic I I never That I'm I'm still Best time to do a red team is during a pentest is still Flying around my my uh my skull. I bet you called a lot of good names by your blue team counterparts. Yeah No, but that it's I mean and that goes to if you go back to that 2011 talk by haroon mere I mean, that's one of his major points is pentesters emulate pentesters not adversaries, you know, like like there's this Uh, I mean there is a methodology to it. It's not really written down anywhere, but it's pretty basic It's it's you're gonna come in you're gonna run nmap You're gonna run nessus or some other vulnerability scanner And if you find anything juicy, uh, you're gonna try and exploit it And if nessus didn't find it your pentester is not gonna do it In a lot of cases because they're using that as their guide, which is a terrible terrible guide Like most of the stuff that has default creds you'll never find You know like nessus doesn't look for all those things Um, that's why you go look at the global asset inventory on showdan.io The good stuff I do find in nmap and nessus is just where it tells me a web server's open Like a way back in the day. I wrote a script that would take a screenshot of every Uh web interface web console that I would find then I go through that directory of screenshots And uh and look for config consoles and stuff like that and that I knew had default creds You know all those are going to show up as informational on your vulnerability scanner Not not with a 10 not with a 2 not with a 1 with a 0 with Informational so yeah, that's that's a really good point. Um Yeah, that's What's that? Oh man, that's just mean A d r an actual test of dr is punishing enough. I know I like the way you think Any more questions? Yeah Companies started to acquire bridge and attack simulation tools. Yeah, do you think red team can depend depend on these tools or Is it more of a red blue team tool? um Yes, I think uh, so so um Partial answer. I think yes attack simulation tools Uh our blue team tools and I I think they're huge for the industry It's the first time we've had any kind of standardized tooling that actually tests all the other products that we're using Uh, and I think that that it's hugely important to use that stuff some of it I think it's you know, I see less value in an attack simulation tool that's Uh cost, uh, you know six digits and it uh, you know You have to hire one or two people to configure it and run it and stuff like that I think at that point it starts to lose value Uh, you know, I think it should be pretty simple to do it um to to to have that value but um Blue teamers need to be able to do what the pentester and the red teams are doing to be able to even know if they fixed it correctly Again think if they're coming in once a year You only get one shot a year to see if you Fix something correctly. That's horrible Like you you need to be able to run that test hundreds of times if you need to dozens of times to make sure that You know password spraying past a hash, you know that there isn't a box in the network that you fix because it's not In the domain, so it's not getting the group policy Like you need to be able to test yourself and find those outliers at the same time that breach impact assessment that I mentioned As pentesters or as consultants we use that one of those tools In our engagements, and I think that's that's another good use of attack simulation tools Is for pentesters to to automate some of that stuff, you know, because you can just let that thing go let let it run uh, and You know that's more value for the customer Where you're not having to build human time to do that. So there needs to be a lot more automation in pentesting basically So yeah, good good stuff any other questions Questions concerns. Oh, there's one If you have current clients that are pentest, you know, they want pentest on how do you Get them how do you Get them to want to do red teaming What's your what's your pitch to them from a social engineering perspective offered is a new introductory service for the same price One of the things I'm I'm going to take away is that you've you've basically given me permission to interact with clients and say Hey, why don't we why don't we get a little more dynamic and change this up and have a conversation at the end of the day You know like I like what you said Um, you know show me where you detected me and I'll show you where you missed I I think one thing you could do with that is commission an internal anonymized um test amongst clients Uh, where you ask for specific metrics like was this detected? How long did it like what time was it detected? um, what steps were taken basically that The measurement phase Ask for those compile it into a report and then give it to the customer for free and say hey Based on the clients we do business with this is what we've determined Based on this in comparison to the dbi our is tr the yada yada yada reports. Um This is how our customers collectively stack up. So we are implementing a new thing and here's what it is And that's that's the approach I would take So I I I'd use that breach impact analysis or something that looks like that I don't actually call it a breach impact analysis by the way because bia is business impact assessment, which is something totally different And uh, I just couldn't find anything else that sounded right that didn't share that acronym but um Getting that that that was part of my goal in changing How we did these assessments is getting that oh shit moment at the very beginning You know, so that's why we intended to do that as a loss leader You know, we intended it to take no longer than you know an hour or two hours Um, but showing them that an attacker in their environment can do all kinds of stuff without getting detected Uh, I think it's a great motivator to to sign up for a service like that or to to do a larger engagement Yeah without having to charge them for An engagement to sell them on an engagement Yeah So I think the takeaway is a lot of stuff that would really motivate and convince Uh companies to do more of this type of work can be automated and done really quickly I mean, we could have gotten it down to where we sent them a link and say click this link And uh, it'll all happen in in a browser and give you a score Like it could be automated to that point to where it could be done in like a uh Uh just just a link that you send the client in a web browser Assuming they open that link while they're on the corporate network at least Open it at home too I think we had one more question So you have a client that says they have unlimited budget Oh a limited I thought you said unlimited I've I've got a card unlimited at the frequency. So I mean that that's why The conclusion I came to is don't do monolithic assessments, you know, and and we offered a You know, it was a thousand What was it? Yeah, a thousand two thousand or four thousand four eight or 16 hours per month This uh a subscription service that we offered Which is you know the way a lot of companies budgets work There's a certain amount That you hit where they have to get permission Depending on how how high the buyer is in in the organization And there's a certain amount where where they don't even need, you know, like like that's almost petty cash for them, you know, so by Making it monthly, you know, it's an invoice that they sign off on every month Like four grand is is totally doable for a lot of these guys. So instead of that 30 40 000 shocker Um, you know, you can do a lot of the same stuff but spread it out throughout the year so that they're constantly focused on security constantly Uh improving throughout the year instead of in one big engagement So with a limited budget, you know, that that's that's a conclusion I came to is is break it up and let them spend that that budget slowly And they they really do feel like they get a lot more value out of it I would also consider the business model customer base and uh sector for that as well So some industries are a little bit more sensitive than others like finance They may want it more frequently Because they're attacked more frequently. Um, if it's a large multinational organization They may need it more frequently than some mom and pop place in podunk montana um, so I mean everything on the internet In theory is there's equal scrutiny put to it But then when you start putting two to two together and you find out who it belongs to and what What there is to gain from it Then things change When when you're working with them all year, I mean you develop a much better relationship with them Uh, you know people on your staff know their network as well if not better than their own people You know, you become that truly that trusted partner that they turn to for stuff So you end up getting more and more work out of that anyway So when when I think with that we had nothing but positive results from that subscription service. Yeah Any other questions concerns complaints grievances thoughts opinions or otherwise That's all folks