 Welcome everyone to the Blue Team Village Incident Response Panel. We have an awesome group here for you today with unique experiences and perspectives on security incidents and incident response. I'm really excited for today's discussion. But before we get started, take a look at the poll we posted on the Blue Team Village Twitter earlier today about ransomware and whether or not victims of ransomware should pay the ransom. We are going to get on that topic in a bit and we would like your input in the poll. Also, please post questions for the panel in the Blue Team Village Discord in the Talks Track 1 channel, which we'll be monitoring. My name is Russell Mosley. I go by Smokam on Twitter and I'm a lead with the Blue Team Village. I've been a blue teamer for my entire career, building and managing security operations at small and mid-sized companies. So I wear a lot of hats. I've done everything from sysadmin, I've been an IT director, a security ops lead, a compliance auditor, and today I'm a CISO at a rapidly growing government contractor. All of my experiences involved developing and managing small teams. And as a result, I've had the opportunity to do a lot of different things in the security field and build decent security programs without a huge budget. I'd like to introduce my co-host Xavier Ash. Xavier. Good afternoon, everyone, or good morning to where you're at. So my name is Xavier Ash. I go by Rubix1138. My IR experience is a little varied. I was an IR team lead at Bit9 slash Carbon Black. And also I was able to start the first IR team at SunTrust, now truest. So 28-year veteran, been around for a while. I can remember when my IR plan was, you have to scan every floppy by hand. So that's me. Cool. Next I'd like to introduce Litmoose. Moose is frequently posting to Twitter, comments on exciting things that she encounters doing IR. And this week it was jump boxes that looked like flaming toilets, I think was one of the highlights. She refers to herself as many things, including Florida Woman, a DFIR violinist, and recently completed a cross-country hacker road trip with cats and stales on board. Isn't that right, Litmoose? That's fairly accurate. I did remote DEFCON. We got to see a few goons. It was nice. So I go by Litmoose on Twitter, also known as Moose. My name is also Heather. I'm currently working at CrowdStrike. I'll dox myself today, which is not a thing I usually do at DEFCON. But I'm very proud of where I work and what I do. I work with a lot of companies of all sizes. I come from a background for financial giants as well as manufacturing giants and have been doing this long enough to where I think I've been awake and can age myself in the 60s if I add all of my cumulative IR experiences together without sleep. So I'm looking forward to retirement. I don't know when that will be, but I do love what I do outside of work. I've heard cats and I play violin. And so I'd actually like to pivot and introduce somebody that I've worked with a little bit, 1099. One of my favorite people in the world, look, I'm sober right now, which is a problem. But Ryan, I'll go ahead and pass it over to you. Well, hello, all you cool cats and kittens. My name is Brian Moran, and I go by the incredibly clever Brian J Moran on Twitter. I have, I worked primarily in the Air Force. That's where I started my career, worked for a three letter agency, did that for a bit, got out of the military, bounced around a couple of private sector companies and eventually back in 2014 started working completely for myself. I also started my own company and I am, I am an IR consultant. That's what I do probably probably about 60% of the time. The other 20, 30% is probably primarily focused on training and development plans. And the other 10% of my time is primarily spent trying to keep the ID for fit going where it's all about just trying to get a little bit more active and hopefully we can all be on this little blue dot a little bit longer. Cool. Yeah, I certainly was going to mention Brian's Twitter seems to generally be filled with DIR fit and comments on rum. You left the rum out. Well, I have some right here. Oh, funny, you should mention that. Better than me, Brian. This year from virus before we have our toast virus. So he says he's only been a blue team or for about a year and a half with a background in red team and he's currently doing what I would call kind of red blue and innovative network defense for a large financial organization if you want to elaborate welcome virus. Yeah, so I work at Stripe currently. The threat detection team doing, I guess, I guess the word for it is adversary direct engagement and threat hunting background wise started in the in the dark gray into the pool as a young kid. My first death counter 17 fell into penetration testing and then red teaming like full spectrum for that stuff and that's around the world doing that for a while. Kind of drifted backwards pen testing and then surprise had a kid and you know turns out that gal eventing around the world breaking into buildings and random countries and risking getting arrested is not a good way to be a dad so had to pivot for a while. I did a stent doing mergers and acquisition security at Salesforce and found out I am not bureaucrat. Now, now I'm in blue team so yeah. Cool that's that's awesome really looking forward to sharing some of your perspectives that we heard in our in our planning meeting yesterday. So before we get going we wanted to carry out the the DEF CON tradition of first time speaker drink. Everybody have their glass ready. Awesome. Cheers to IR. All right, so to kick it off. We're in a unique situation here in 2020 right with COVID forcing work from home for for so many people. So I wanted to start with a question for our full time incident responders moose and Brian. What types of incidents are you seeing the most of right now, you know, like what are the trends and attack methods persistence and anything else you've observed lately in your work. Yes, I can go ahead and start. hasn't changed too too much. The volume has been a huge update. I'm still seeing a ton of ransomware attacks. The serious thing that that I'm seeing more of now is actually I would say more intrusion via remote code execution has had an uptick recently with a lot of the box out there for ability scanning. I would say that those are getting hit before people can patch just a little bit faster than I used to see. A lot of the end game, or what I'm seeing now is either encryption and then extortion, or X though and then extortion. And I would say that the volume has just gone up versus what it was like a year ago. Yeah, I would 100% agree that the exact same stuff that's happening now has been happening for the past probably two to three years has been the general trend. The volume of it is just immensely increased and probably the part of that is the bad actors have nothing else to do except stay at home, just like we have nothing else to do. So, you know, the idle hand idle hands will play I believe is the is the phrase that's used there so the one thing that I have seen more recently an uptick of then I have over the past year or so is actually 0365 and not necessarily 0365 intrusions it's it's users clicking on links they shouldn't click on and then getting access to stuff that they shouldn't be able to and then the attackers getting access to stuff that they shouldn't necessarily be able to get so it's not necessarily 0365 intrusion is just phishing which is kind of directed more towards with the overall goal of getting some 0365 access. Interesting. I wanted to ask, you know, a lot of companies have focused a lot of their security on their networks, and, you know, perhaps not as much on their endpoints and cloud services. And now that, you know, many of their workers are remote, they don't have the option, you know, to do monitoring and, you know, to on their network, and maybe aren't as well prepared to monitor from their endpoints and their cloud services. So would you agree that that's the case and you know how is that how's that impacting what you're seeing out there. So I would say definitely does. But the copy out to that and I already kind of talks where I work is I'm getting calls, because they're not necessarily using an EDR to begin with or or maybe there's not as much focus on the endpoint. And I think that what's neglected in cloud is that there's this perception of security that's not really there. You're, you're just working on a server and somebody else's data farm. And so you still need to put the same protections on it and monitor like, you know, your VPC flow logs and look at what what VPC you're putting stuff in or what segment rather. And you would monitor it just like you would something on your environment. And I'm seeing that being so quick to set up that sometimes those steps are missed. And then I'll get a call. Yeah, I think the, as we talked about a little bit last night, one of the things is the data that you need to actually respond to a breach of some sort, it's always been there. It's never really been used or even, you know, thought of being looked at because before it was, oh, well, we're just going to block everything at the network layer because everything is controlled. But that is everything spidered out like all the logs are still there all the access like all of that still there it just never was bothered to actually be configured properly in the first place so that's why we're seeing what we're seeing. So is is with with a more distributed, you know, I'm hearing more endpoints and more cloud. Has that really changed the way that we're doing IR now. I would say yes and no. So we're still asking for logs that are representative of the same data, instead of asking for firewall logs in this case you're looking for that flow might be. So, you know, we're still asking to run scripts to an endpoint, which, whether or not it's in a cloud or on it's going to be capturing the same artifacts we're looking at the same things on when it's that we're looking at on an AWS Linux server that we would look at on in a server environment. So, I would say that about things have changed in slight ways and how we capture the data so you know converting an ABS snapshot versus, you know, the MPK versus I'm going to run it to it. Those little minor changes are there, but I think just getting creative about, you know, I'm still caring about the same things I'm still caring about how the threat actor got there. So, from your original access point, I'm going to look at off logs. I'm going to look at, has there been any expel file creation file deletion. So the same questions we've been asking forever we're still going to keep asking. It's just that creativity around in a different environment now. Unfortunately, those logs are generally there for companies that rely on Office 365. You know a lot of times you just have to have the right licenses to go look at them and cloud app security and other places right. So what happens when an organization hasn't really invested in, you know, logging on their endpoints or, you know, anything beyond the defaults that come with Windows and, you know, you have, you have a situation where you have, you know, people getting fished or getting malware, and there's really not much to go on what do you what do you do in that case I mean, you know, generally, you know, you're not going to have the easy access to get a forensic image and conduct analysis on the device because it's, you know, somewhere far away from you maybe on a on a slow link. How do you deal with that? Well, I think the, the, the old way of well, you know, we just show up and collect data from somebody's computer in the event that something happened like obviously that is no longer, no longer going to be existed. So you have to get you have to get creative with essentially trying to get the same people on together like on a call like this and figure out what you actually have access to not necessarily what you what you have because more than likely in the event that that someone like like myself or Moose gets called in like stuff has already happened that's really, really bad like we're the we're pretty much the last people you want to see in the event of an incident. So you have to get creative with essentially figuring out what you have access to, and then how far back that goes if it goes back at all, and then try to build out from there because a lot of times when you when you know we show up somewhere like the first question is like Do you have net flow? Do you have logs? Do you have X, Y and Z? And a lot of times like the initial part of an incident isn't even actually responding to something. It's figuring out what we actually have access to and then trying to get creative with ways to get get access to that data, get it as quickly as possible and then turn that into usable information that we can use as quickly as possible. So what are some things. Go ahead. Sorry. I was just going to say I think one of the boons right now is that so many people have invested in cloud security so it almost negates the problem. A lot of times a threat actor is to the point of getting into a lot of devices we can think about and I almost want to turn it over to virus for this one, but if you think like an attacker, you have a better chance of getting the data you need. I just want to turn it over to case recently where you know I couldn't run the scripts I wanted to but I looked at what they did and I said well let's just replicate what the red team did or what the threat actor did in this case and let's run to it and I got the data I needed. So that's sometimes you just got to look at the problem a little bit. I think another curveball that I started to see is, you know, for companies that have used wise terminals, you really can't take those then clients home and so we're seeing an increased use of other devices like Chromebooks. And so, wanted to see any advice out there somebody that's listening is going to be doing any type of instant response involving Chromebooks used as, you know, thin clients. Is there any advice that you can give to those folks out there. Looks like we lost for a minute. Well, so, so I know that there's been some research into the essentially the data you can collect from Chromebooks. I know Jessica Hyde from Magnum Forensics has done quite a bit on that and I believe this past year she's given a few a few talks on that so so there is data you can get from you have to get clever with it but one of the things that's interesting is, if you use Chromebooks more than likely using some sort of Google instance and Google has a ton of data about you whether you use Google or not Google has a ton of data value. And because you're able to get all of that data back from them like this and admittedly there's a lot of stuff to go through. But it's not necessarily like Chromebooks are incredibly secure like I love Chromebooks I recommend Chromebooks to pretty much everybody who gets click happy with any kind of links like just use a Chromebook for a while. And hopefully that's going to eliminate a big portion of your tax service like it's going to not going to complete eliminate it, but it's going to take a big portion of that away. But it's essentially at the end of the day, you know, we're still trying to look for the oddity outlier information of, of data access, you know data expel stuff like that and with the stuff that you can get from Google specifically like all of that's there it's just a matter of being able to get it being able to process it being able to digest it. What are your thoughts virus on, you know, doing IR with with cloud and distributed endpoints without the, you know, without the hardened network of tradition. In some ways, I guess I haven't been around in the blue team ecosystem enough to know the difference. Like I've always like the few roles that I've had that are hardened blue team I've always had the power to at least have a say in kind of what our telemetry looks like. And I mean coming from the red side I've frankly never really bothered wanting network stuff it's it's always, you know anything that's a network artifact I only care about it in the context of what the endpoint did in terms of interaction and so it's just usually easier to filter on down and kind of pinhole what I want. I have definitely within the past year I've dealt with some vendor breaches, which are all public so I can say that but I guess I won't say exactly what they are so you know I won't get fired immediately. And it's kind of interesting one of the nice things about I shouldn't say nice but one of the interesting things about being kind of being tangentially related to a breach that gets released public is suddenly you have like a legal stick to beat them with to actually get IOC's back from the breach as opposed to just reading the news and having a Twitter opinion like the rest of the world. And from what I've seen it definitely like the game for the vectors in terms of the game for getting in doesn't seem to have changed it's usually not the server. It's usually it's way easier to get on a laptop and then because no one knows how to network in the cloud, like everything is just grabbing the AWS token or the Azure token off disk and then going to town. And then usually from there it's a oh by the way either there's no logs or there's a ton of logs but nobody's ever actually triage them to find out which logs are useful and which ones aren't so there's no logs. And then it's usually going back and being like oh well after they took this token from your machine then they directly logged in with this super weird user agent that no one's ever heard of install your data and you did nothing. At least that that's that's been the trend from what I've seen the Chromebook thing. There actually is I don't have a lot of specific details that I can share but I do know that there has been some publicly released stuff on Google implementing a kind of audit D type thing. That is that is going to be very similar to like OSS security center in terms of accessible data so I think that's going to be a game changer with all of the EDR tools that are out there that are hooking in the audit D as kind of a funnel for data I think that will be really cool. And then also anybody who's looking to do if I are on Chrome locked devices in general I would actually route you to some research from about eight or nine years ago from Mike Osmond and Kyle Osborne. They did some offensive research on basically how everything in Chrome is a browser and the browser doesn't know the difference between a file system link and a remote link. And while Google has obviously tried to address the security concerns that they found with some of that research you can actually use a lot of those vectors to gather debug information about the Chrome browser environment and that ends up being almost equivalent to system logs on a Chrome locked device. And you started your comments there with you know first getting on the laptop and then pivoting out from there. What's the what's the most common method for getting on the laptop. It's the usual fishing or I've seen some supply chain stuff recently and there was a there was a Ruby supply chain bug that came out relatively recently that I know affected a bunch of shops. I've seen some slightly less talked about in the media because the go ecosystem is so blurred in terms of the concept of standard live versus some person stuff on GitHub but I have seen some bugs there that have caused some supply chain issues. There's little known fact, I have seen well, it's a more known fact that OS X attacks around the rise but it's a lesser known fact that Pearl is still on the OS X boxes so there I have scenes kind of some like, you know, oh I haven't seen that attack since like red hat five but you know that Pearl subsystem is still on OS X and way more people look at Python OS X the pearls so there's a little more of that. That's great. I'm curious Brian, your observations, you know, how they compare to viruses feedback you know and what you're seeing. Yeah, I think it's so I don't think the methods have changed the methods haven't changed and honestly, the methods don't really need to change because they still work. So if they're still effective there's no reason to to burn some you know some zero day exploit or even even go the route of looking for like RCE or anything like that when if you can get somebody to click on a link and get the information you need and that still works like that's what the attackers are going to do because you know, again at the end of the day like the attackers are are human just like us like they're, they will do just as much efforts as needed to get them the goal that they need and you know you have to be able to account for that. The tools I haven't heard yet so far for telemetry is you know cloud based DNS. So a couple providers that where you can get that from a centralized source. We're seeing more people, you know, take that on as we get to a more distributed environment, or is that still an afterthought to like oh yeah, we can't get good DNS telemetry because everybody's at home using their, their home DNS. Does anybody got any experience with that. I do. I've had some that have it, but not very many and with the caveat of saying that if they're calling me something's gone terribly wrong, or or counter, we do do security assessments and environments that might something to come back with because they've set up a cloud so I will say that I have a little bit of a bias that answering this because even though I don't see it as much. It's usually because they're calling for help. Yeah, I mean, I control is kind of is kind of a craft shoot but in terms of telemetry like, man, like passive the combination of a low level passive DNS service and OS query seems amazing for DNS to me. Yeah, like so so I'm going to say pretty much the exact same thing that boosted like I would like to see it more. However, a lot of places don't see it and the places that call us more than likely they have that there have been several shortcomings that led to calling people like us in the first place so typically we wouldn't see something like that because we're not you know we're not the the security architects or anything like that like we're something horrible has happened everything's on fire I'll call them in and they're hopefully going to calm us down. So before we pivot to another topic. You know if you were called in to give advice to a company today and you know they wanted to spend some additional budget for security with people working from home. You know where would you advise them to focus. I can tell you where we have an entirely remote workforce right now, and there is some scatter in terms of operating system layout and like pretty much 100% of our EDR effort at this point is done using OS query that's pretty public information I don't really feel like I'm giving away a secret sauce saying that and at least from a telemetry perspective even with specific regard to DNS I can tell you it's pretty amazing like we have other issues but DNS telemetry is certainly not one of them from my perspective. I think the one piece of advice I could give is don't ignore any from I see a lot of people spending very heavily in one arm rather and then maybe ignoring the other. So they'll focus just on them or they'll focus predominantly on the network side. And I think that the balance and depends in depth is still very very important. And I would say, you know, to borrow something that we all as a team talked about yesterday. People are trying to eat the elephant hole. So don't do that. Don't keep your number flat either. So my top advice usually to people is, you know, make a list of priorities in your environment what would hurt the worst that one public and start segmenting know your your group commissions, you know, run some of these red team tools run, run bloodhound sharp pound in your environment, get a better picture of what it looks like, and then make a decision, but make an educated decision, not just, you know, a flat blanket decision because not all businesses are the same. And not all businesses have the same needs. I think before spending any money, the first thing that a company truly has to do is take a look at what they already have the people they already have, and then try to build a solution from that like like just just like a lot of times when I go to places like they don't even have something basic like a network map. So then I look for the attackers, when essentially they dump like IP dot txt to some system and then I have an actual network map that I can go on because the attackers have left their forest fortunately. So that's literally the only way insight that we have to what's actually on the network is what the attackers have done. Yeah, you can't you can't be just spending money while you have all of these gaping holes that just have never been addressed like you have to do a serious on an honest and truthful evaluation of what you currently have already before you make a decision on what you're actually going to do to try to improve. And related to that you know skills versus tools right should you be investing in your staff and in their skills, you know training sending them to conferences like Defcon, you know where they can learn from the community and what others are doing, right, and increase their capabilities instead of buying tools. Thoughts on that anyone. I have strong feelings. So I think it's both. You really need to invest in your people, but don't overtax them so much that you're not giving them the tools to work with so that they're getting aware. You don't want to have them working with the bear bear nothing and then getting mad at them when they miss something. So, you know, take the team you have and really build them up, but then also listen to them on their level of what they'd like to use and what they'd like to have and use that training of sending them to conferences and coming back and gathering that information to inform your decisions. I think that it really needs to be that full cycle of your investing in their education and then going to conferences and networking and letting them come back to you with a solution. And that is going to sometimes be investing in tools for them. I agree 100%. You know, you really want to listen to your team, you know, listen to their experience, right, you know, get them to get out there and participate in the community and learn what works and, you know, and take their advice on those areas, you know, and figuring out where to where to invest in your team. I wanted to talk a little bit about IR teams and folks who want to get into IR. So, you know, I think the two related topics, but a lot of people have asked if we could give, you know, what's your best advice for people who are interested in getting into the incident response field. And that's for anyone who wants to go first. All right, I'll go first. So, so my, my personal opinion on how to get into IR or really kind of any info set, like specific sort of doctrine at all is just to simply be curious, like, ask questions, try to research stuff on your own figure things out. Like, you may not be the best person at assembly or or firmware architecture or or or IR or mal reversing or anything like that. But as you find questions and, you know, find things that you're curious about and want to research and learn more about, like, you'll generally kind of find an area that you want to go down. And at the end of the day, like, you have to kind of decide if you want want it to be a typical job where you work eight to five every day and show up and you get to work and that's it. Or if you want it to be a career and you actually want to want to be good and want to actually, you know, be better at what you do and be more effective and be more efficient and network with people and all of that. And it's not really it's not really a lifestyle decision so much as it has to be an internal passion and desire that you actually want to do something. Anyone else want to comment on that one. But I will all chime in and say everything that that Ryan said absolutely. But just know that there's no wrong path and that to can do anything in your spare time that you're passionate about and turn it into IR. The differences is is what I typically see in good IR is you don't stop at the first answer. So, and you don't stop at the second answer either. It's the stubbornness and that drive of being severed to where, okay, make yourself of a laugh, image your own devices, break them, like see what you can do and see you see what you're seeing and use that to train yourself. And it really is something that it takes a lot of training and that there's not just you're there one day so I'll admit myself. I get imposter syndrome a lot. That's because every day I have to Google. There will be something every day that I don't know the answer to or I haven't seen before. And I am researching the hell out of it and saying all right. But anyways, I'm just going forward with this and, you know, sometimes it feels like I'm banging my head against the wall but that doesn't go away. And so if you're okay with that and you have the personality that has that drive, you're going to be great. And being like listening to this right now. It shows enough that you're you're willing to listen to the five of us ramble for an hour. You're probably a good company and and you should probably just keep pursuing us. One of the on that line, you know, talking about getting into our talk to the more along the lines of the type of work and particularly about the stress. You know, you're, you know, typically ir is your, you're, you're more like a fireman. You're dealing with sometimes some high stress situations. And so any comments or suggestions for the listeners around, you know, kind of dealing with the excitement and the stress that can come along with being part of an IR engagement. Well, I think one of the things that you have to take into account is so first of all, like from an IR perspective, like you have to you have to try your best to be calm. Like you, you can't overreact like everybody else, like let everybody else overreact, let everybody else run around and and scream and yell and all of that, like you have to be the calming voice. However, that doesn't mean internally you're not screaming and yelling to because you absolutely are you just have to portray yourself as having that calming presence. And you also have to essentially not necessarily not necessarily have the right answer, but you have to have a answer because the path to recovery always starts with one step forward and it doesn't matter if that step is even down the wrong path initially like you just have to go forward because the the takeaway that you can always have no matter what is somebody out there has it much worse than you currently do. And you always have to remind yourself like as bad as everything is like there's some company out there that got owned, and they are in unbelievably an even worse situation right now than you are. I think that everybody in IR is a little bit of an adrenaline junkie, and that you're kind of lying to yourself if you're not at Brian didn't you go like face jumping or something at one of the cons that we've been to recently. Yeah, I jumped off the stratosphere in Vegas a couple of years ago twice because I wanted to so. I was going to take the question and ask each of you individually, you know, how do how do you get away from it how do you deal with the stress, you know. So Brian goes jumping off the stratosphere in Las Vegas. How about you moose, you know how do you get away from this what are your breaks. Well, it's two fold right like everything I see on a daily basis. It's somebody else's worst day. I don't work for their company. And I am, I had to realize this about myself and every job I've had so I'm not always that I keep I used to be that tech used to be medical. All of these things have a similar like mind in that it's really high stress, but it involves helping. So I think to me, my release is in seeing that step forward in that question and even if it's not happening right away, being able to reassure that people that I'm working with that it is truly happening that at the end of this their environment will be better than we left that it started at and then if that doesn't do it. I have cats in the violin and on the first day I'll just shut everything down everything gets unplugged and I open up the violin case and I walk away. And I think that's really important is that you don't have to eat sleep and breathe this you really should walk away sometimes. And that's what's kept me alive. And how about you virus, I mean it sounds like you certainly have some of the adrenaline junkie in your, in your experiences in your past, breaking into buildings, you know, so what do you do for breaks. Oh, shenanigans. I mean, I guess the first thing is like, don't get stressed. Like that's that's a big part of it. I mean, I've made a lot of I've definitely steered a fair amount of my career decisionary of like, you know, when I finish an arc I go. I like that. Or was it just stress in the bad kind of way. And then I use those takeaways to just not do that anymore, you know, kind of quantify how much my time is worth because if I spend the whole time when I'm not at work, stressed out about work, then I'm essentially working for free. So if I factor that into my paycheck, I'm like, oh, this isn't that worth it. So that's definitely the first step is just do stuff that stresses you out you know know what know what your zeal is. But on a practical level like I played music for like 25 30 years, semi professionally so you know I do that I kind of stand out. I cook I dry aged meat I make my own gin I you know I like scotch I do a lot of drugs. I play with my boy and I'll stand up for my desk and I'll work on our problem and just say hey let's go for a rock and he's three so that's amazing for him it's great he just runs around. But yeah, it's I mean, you know, I think I think a lot of it is just you got to be holistic about your approach and if you love the chase you got to love the chase and you got to love your chase you got to love the chase that you're in and know where it is and just not lie to yourself you can't, you know, do it and I mean, there's definitely an undercurrent if I'm being extremely frank and extremely personal about it I do think people that I think there's a fair amount of people on the blue side specifically who have a very I don't want to say ethical, but they have a very empathetic attachment to the world slightly larger, maybe than most red teamers do where at some level they're kind of in it because they want to they want to fix something they want to make something a little better and. I know I mean, I wouldn't call myself a complete break but I'm definitely on the side of like, yeah, I like money. Long as I moved the needle a little bit and I got paid I'm not too worried and maybe that's a whole other conversation. If we could get back on the topic of, you know, the IR field and our teams, we had a question in the BTV discord about, you know, what are your top recommendations do's and maybe some don'ts. You know, we're growing your your sock and your IR team, you know, if you're you're at a company and you know it's it's growing and you know you get budget to have some full time folks doing security operations and you know, you know skills and tools ready to do incident you know, where do you want to focus, you know, what what skills do the folks need to have. What are some of your preferred tools, and I know it depends a lot but just in general, you know, where would you focus Brian, why don't you go first. Well, I think it goes back to it very much does depend, but it also depends on what you currently have already, and essentially what you have the ability what what you want to essentially try to strive for like, if you want to go a very very cheap route as much as possible you use something like OS query or use something like like velociraptor and, you know, try to essentially keep it on the cheaper side but you know at the end of the day it's still going to be able to work. You have unlimited budget like you just line up boxes, you know, like crazy and then you have all the tools and everything but, you know, to be completely honest that that's not that's not feasible for anybody like like companies will not just blindly give you ridiculous amounts of money to spend on people's and people and tools and resources so you have to try to make whatever you have work with what you can and that that's one of the things that I stress like it doesn't necessarily what matter what the dollar amount is it's trying to make every dollar amount as much as possible so that way at the end of the day you end up with a truly better result. If you have a very high budget, great, like you can afford expensive tools but do you really need those expensive tools. Do you really need an expensive person to come in maybe for like, you know, three six months or something like that and help train and make the team make the team better to already use the stuff that they already have. In that case you could argue very much that bringing somebody in external for a little bit and helping them, not really as a staff fog more of kind of like an overseeing mentorship almost like that is a much better investment than just blindly buying you know a million dollars worth of random tools. So it truly does depend but it's like everything else like you have to make a business decision about it and essentially try to maximize every dollar and get the best value for that as much as possible. You know, and the we've used the word a few times in this panel telemetry. You know, so from my perspective, one of the first things that I would want to do is, you know, is set all that up is set up, you know, a central a central logging system, whether it's some whether it's you know something much simpler. But, you know, for me doing incident response. You know, those are the keys right I mean I've heard it said you know logs are everything. And that that would be one of your first steps would be to make sure that you, you know, you're, you're gathering enough information so that, you know when an incident happens and just in general, you know, having logs is has so much value for for it support for everything that we do in security. Yeah, first of this is really, it's like a three step. It's like a it's like a three three layer moniker. So, I think the first thing that it's important to grasp is that a program we talk about a program whether it's an IR program or security program large or blue team program a program is business process. That's what a program is it's not blinky boxes it's not even the talent in your bullpen. It's it's documented process that that is the definition of your program. So the first rule is, if you're going to build a program, build a program that means encourage your people to. And this kind of leads into the second thing which is tribal knowledge is the enemy. Right. So like the first the first job of everybody on the team is to take everything on their head and inject it into business process. It's your job. It doesn't matter if you are, you know, the self proclaimed bottom rung of the team or if you're the epic gangster rock star at the top. Your job is to take whatever's in your head that applies to the program and to build the program and the more that you encourage people to do that and the more that you reward them for doing that. It's like a self incentivizing loop right that that's how you balance balance burnout and all this other stuff because it's not it's not focused on one person or one tool. And then like I said the second thing is, and I think I feel like the way more common on red teams that it is on blue teams and I kind of don't know why. It is extremely common on the red side to not use tools that are touted as being the latest and greatest thing just because you don't know them. Right. I mean as a red as a red team or I feel like there's this moniker that is accepted more and I feel like it's true on both sides. So I kind of don't understand why it's a lobster that we're like, you know, it's the old ninja moniker right we're like not every ninja uses every weapon right you have your weapon. And it's got your epic moves in it and you know it like the back of your hand. It's almost an extension of yourself and that's the point. I don't really think that's different on blue team. I mean, case in point, the team I'm part of a strike that kind of got built about a year and a half ago. We were all very new to the company and was, you know, granted, we were very fortunate to have leadership with a lot of experience. And so they reached out and, you know, there's a lot. There's an extremely high amount of experience and talent on my team. That's a little bit a skew for most. I'm not saying that to brag. I'm just saying, you know, a lot of teams don't have that in their tool bin. But the first thing we did was take what EDR we had and rip it out. We ripped that almost everything and then built everything off of bare bones telemetry because those were the systems we could use better. That yielded better to our kung fu. And then from there, the third thing is build like tool your kung fu. You know, take your brain dump, put it into your program and then hypothetically start from scratch and go, what tools do we need to do this? What's written better? Frankly, every deviation from that is a waste of time and money. So you said that you ripped out your EDR and you built up bare bones telemetry of the things that are most useful to you. Can you elaborate on what those are? I mean, it's literally just stuff that's built into the OS. I mean, we use OS query at scale so that because we use the fleet manager concept in order to kind of achieve orchestration goals. But our detection system is entirely homegrown. Our response system is entirely homegrown. Our orchestration elements and the way that we do log tracking is entirely homegrown. Even our case reporting, I mean, we're kind of sort of bubbling around the hive right now. Honestly, it's all Jupyter notebooks and Python. It's all just kind of manual tooling. And at the end of the day, if I take that and I put that in the ROI bin, we spent way less on human time writing some Python that we would ever spend on third-party utilities. And to be honest, most of the third-party utilities are trash. I mean, I won't pull any out just because I don't want anybody to kind of have to be under the boot because I said something. But like, man, there's just so much money spent on these tools that just don't even do what they say on the side of the box. I just, I don't understand why that flies. I mean, that's just, you know, I don't know. Say what you will about Red Team or Blue Team Infra. At least the Red Team stuff does what it says. It just doesn't say a lot. Blue Team stuff, man, this seems to be a problem. The cereal box nutrient facts are just wrong. So Rubix, you've built an IR team, right? I'm curious, you know, your thoughts on all of this. Yeah, you know, it was, I always go back to the, you know, kind of using the fireman analogy. So IR for, you know, a financial services, there's a whole lot of downtime. And so figuring out how to practice and be ready for the big event. And giving them enough work of a real, you know, I guess, you know, smaller level incidents to be able to stay busy and relevant, but then also, you know, plenty of time to prepare for, you know, bigger stuff. And so, you know, lots and lots of practices, you know, tabletops drills and, you know, encouraging our Red Team to really give us some good scenarios. And so I think that if you are, you know, not balancing that, you know, if your guys are all, or folks are always spending time, you know, doing, you know, we talk about being really busy that they might not be honing their skills enough. And so, you know, finding that balance between, you know, doing and training and practicing is, was very key. I think the one thing I have to add to that, and everybody, everybody else is given really good advice here. And this is a part that I'm really passionate about because it's one of the reasons that we need to be companies. Don't silo your people and listen to what they have to say. If you have somebody on your team, it doesn't matter what notch in the team they are. If you're calling on them for some kind of incident response, whether it be a soft person, somebody who's doing this admin work, someone who's a network admin, you're trusting them with that role. Trust them if they say they have a passion in response as well. Let them play in other fields because there is not a silo an incident response for our attackers. So there really shouldn't be a silo for our responders. And one of the reasons that I work where I work is because there are some days where I'm not incident response. I have the freedom to be renting stuff. I have the freedom to help build our tools if I want to or scripts. Some of the most brilliant scripts I've seen today were built by my teammates in response to the Citrix vulnerability, for instance. So knowing where your strengths are on your team and building those and then listening to them and what they say they need. Because if you're the one building, that assumes that you're at a higher level position. And really we are strongest in that position when we have people under us who we trust. And I think that's really key is to my pastor right now. So that's what I keep looking over for is any time I'm on one of these they write something. So anyways, but listening to those people that you've hired and obviously you trusted them enough in the first place to hire them. They're smart people and more smarter, not because we know something. But because we know who to go to and we need certain knowledge. And I think leaning on on your teammates, no matter what level they are having that kind of group decision is still important. Yeah, I think that's, that's really, really great advice, you know, that that more companies should heed, you know, to try to try to cross train their folks and and give folks time for for training and for research, right, and and have budget for tools and devices right and lab time. I think it's I think it's really important. So I wanted to, I wanted to pivot and and talk about ransomware. So we had a, you know, we had a passionate discussion about this yesterday and I'd like to try to rehash some of that for for attendees. And to kick it off. Earlier today the the blue team Village Twitter. We posted a poll. And you know, we were looking for people to give us their opinions. The poll question is your organization is the victim of a ransomware attack. Should you pay the ransom. And about 89% of the responders said no, and about 11% said yes. I think, you know, in summary of our discussion yesterday is it's it's not that simple. But, you know, for the benefit of everyone who wasn't in our private discussion yesterday. You know what goes into the decision of paying the ransom or not. And I think Moose started first yesterday if you want to go first again today. Sure. This is this is the most like, let me tell you right now this question made my blood pressure like go up a little bit because for me when somebody says, do I pay the ransom on on any call I have or even if I've been in person, I look at them and say, And that's still kind of my answer is that I don't think you should make that decision for your business in a vacuum. And I usually recommend that people talk to either their internal console or have external console involved in that decision making process. Not necessarily because these are if you look at it this way, the way I like to frame it because I'm not going to say yes or no spoiler. I'm not going to say yes or no, because it really does depend. And that's not my decision as you responded to me. But I think that the most intelligent things I've seen done was calling on experts to do this every day because if you think about who hit you. You're not their first hit. You're not the first people they probably ran some they hit other companies. These are professional criminals that are going after you almost all the time if they're exploring you for something. So get a professional. That's kind of my opinion. Okay. Brian, you, you kind of immediately had a, you know, from the heart response that you then elaborated on. Could you share that with with all of our research. So, so as so as Brian Moran human being for my first response is always no. However, that is simply me as a human being because I don't feel that it's right to pay criminals for doing crime. However, we'll preface all of that with put on the, the consultant hat, the business process hat and all of that. And honestly, the answer is more than likely it's, it's usually cheaper to pay the ransom than it is to deal with the cost of everything else involved. Like, it's never, it should never just like Lou said, like it should never be the anybody on the infosec team ever it should never ever be their call. It should be legal is called because legal absolutely needs to be involved, not just from the perspective of is this okay with the company. Also from this perspective of, you know, you're potentially looking at paying somebody in another country like are there sanctions against that country like right there you could violate even more laws just by paying the ransom. So, so now you're in now you're in double because all your debt is gone and now you're getting sanctions or and fines and investigations levied on you because you are literally, you know, potentially financing something that you shouldn't be financing. So, it's not our call to make at the end of the day like you know we can we can respond and say, here's x, y and z here's our thought process but generally from the overall business perspective it's usually much cheaper just to pay the ransom and be done with it. Then to deal with any of any of the public scrutiny deal with any of that but you know, we're not lawyers I don't think any of us you're even going to pretend ever to be a lawyer like even if we say it a holiday and express we are definitely not lawyers by any stretch of the bean so we can have our opinion and we can give you thoughtful evaluated opinions but like our actual thoughts in the day truly don't matter because it 100% decides on legal and what the course forward is from them. Thank you. You know I think I think a lot of folks have that reaction of no you know I don't want to pay criminals but there there is so much to the decision to be made. And virus, you came at this from a different angle than than I've ever really thought about it and that angle was about ransoms and kidnappings and kind of relating ransomware to that type of crime. Could you could you rehash that for everyone. So I basically just still everything. I tried to kind of get across our joint meeting as ransomware is a crisis the breach that put it into the incident. So, if you have a flat out answer on whether or not you should pay the ransom when it comes to ransomware you're doing it wrong, because that's not how crisis management works. And I don't understand why as even an industry infosec would try to solve crisis management when every major Western power, every branch of every military and military contractor that's worth their salt every political, you know, wing contractor I mean crisis management is a well understood science that has been researched by every tactical engagement and business risk school in the world forever. And the fact that the thing that's being ransomed is a piece of electronic capital versus human capital does not change the game. I just don't understand why people would insist on coming up with a cat answer for that I think it's stupid. And while I think that, you know, the response to a ransomware crisis because it is a crisis will likely involve legal. I don't necessarily think that legal is the first is the end all be all shop on that it probably isn't most shops, but you know what the world is bigger than the law. You know, there, I could tell you all day long about, and all of you could do about every major insert fortune 500 company here, where they have a crisis management team that has nothing to do with computers and I guarantee you the first stop they call when the CEO gets kidnapped by, you know, random activists in South America on a business trip is not the lawyer they're like the third. So I don't know why it's different because it's a computer and not them. And then after you handle the crisis, however you handle it, maybe it's paying maybe it's not maybe it's using shadow currency, I don't know, right, then you handle the incident which is, if you have ransomware you were breached, you'll handle that that's your incident that the ransomware that's not an incident that's a that's a crisis. With the shift of a lot of ransomware actors, for example, maze to not only just destroy anymore but also steal the data threatened to do more stuff with it than just destroy it. So how has your way of dealing with ransomware has in my advice. So, I'll go ahead and speak to that and that I have given a little bit of advice, you know, working with a legal team, of course, if it makes sense for the company and what they're backing. So all of that included so assume that from the get go but you know a lot of these groups that do extortion, they're saying that they have something and usually they'll serve up what they have. And if there is at all 24 hours that you can buy your IR team to investigate if that claim is true or not. And by the time you can figure out if what they say was exposed is true. I mean, more to that. So if they've gotten more than they say they have, you know, take that time and lean on your investigations team to see that expert piece and see if you can see that creation of that in your system. And that's a very hard thing to do in a short amount of time. Especially because it assumes that you have full visibility every endpoint on the network to have that kind of visibility to begin with is incredibly difficult for your team, let alone outside team. So I would say that the good things that come out can come out of that, not that I'm negotiating with Eris, so to speak, that are withholding your data and extorting you for money. But you know, sometimes I've seen outside council get a lower price, so that if you are going to pay, you're not going to pay as much. But, you know, time is valuable to having that on your side is important. I think one of the things that is kind of hugely important, but also at the same time incredibly underrated is the way that ransomware work ransomware is a business. You know, just like any other business ransomware at the end of the day is just a business and I have I don't recall seeing personally and I'm sure it's happened but I don't recall a case personally where the actors have claimed to have something and they truly did not that they were bluffing because if you're running a successful business like you're not going to lie about what you have and everything well unless you're and Ron and stuff like that but but for the most part you're not going to actually not tell the truth about certain things. So if they claim that they ex filled data and you had you as a company have no way to prove that like number one you're already in way worse problems and having ransomware like you have so many other problems that you need to deal with. But if they have if they claim that they have something they more than likely do and then you have to evaluate the cost of all that well what if that data goes public what if that data goes goes even more publicly widespread what if that data is then sold to other places and and everything like that and you have to take all of that into account with the entire process and so you know we often try to look at it so much it's like a pure like, like, essentially is there a correct answer or is there not a correct answer is any yes or is it no and it's it's never just a yes or no like there are so many factors that you have to take into account all of that through it and we in the in the information securities field like we only see a very small portion of all of that so you have to involve other people you have to involve other people in the process but you know, just, just like it was said like, if you have ransomware like you have, I guarantee you have at least five other problems that besides just just how ransomware got there like you have a lot of problems that need to be addressed ransomware is the big you know the big public facing thing on all of that but there are so many problems that need to be addressed internally and then you have you as the practitioner have to take into account. All right, here's my little take on the ransomware here's all that and then you know essentially that decision goes to somebody else but ransomware like I've never seen anywhere where like oh well we got infected with ransomware and that's the only problem that they have there are many many other gaping calls that they absolutely have I can guarantee that. And on that note you know we've we've unless somebody else wanted to say something else on ransomware virus were you about to speak up. I was just going to answer Rubik's question. Okay. If I understand the correction correctly it was in a nutshell does introducing extortion into the ransomware topic change the game. Either that would be no, at least not for me because I've already given my two sounds on how I think that you know the orchestration of the conversation on ransomware is wrong. In terms of have I seen it change the game. I would say yes, and I would say the way that I think it is changing the game is that counterintelligence is a thing counterterrorism is a thing black on black kill squads are a thing. They have always been a thing in blue team they just haven't been very talked about. Because until words like hacking backs that are being introduced into the info set conversation those those are mostly gray areas to a lot of professionals. My experience is that the reintroduction because I have seen it before especially dealing with like that the old fields way more common than what I've seen in corporate space and now I'm seeing it a lot more than corporate space. I think those areas are getting a lot less gray and they're showing up on the radar of a lot of risk teams because they're starting to realize that that's how you fight this. But, you know, again, is my prescriptive advice that there should be more of that now. Awesome, thank you. Any other comments on ransomware. I will say that the other thing that's that's I've seen what is ransomware used to be kind of a singular program that executes and you know they've got his programmatic way of doing destruction is that now when you hear of ransomware it's it's a full on incident they have a number of backdoors they have a number of accounts right you can't just go and contain it and say I've cleaned up the ransom we found ransomware and five machines. We've reimage those five machines were good right now they're going to have you know spend some time in your environment, stolen accounts. Have other ways of getting back in and so you really have to do a full incident responding to find out all the things that the attacker got. It's no longer a simple case with like we've used to deal with some. The only other thing that I would add that I this is the hill I die on because I'm one of our say Linux enthusiasts where I work is most people associate ransomware with windows. And while that's the case that doesn't mean you should ignore your Linux environment and never. So make sure that you're building that up and they're looking at it and and don't don't ever assume that right. Don't ever assume that it's just, you know, if I have access to your environment and I'm a bad guy. If you get me an easy to the point. So, so kind of have that in the back of your mind. And don't like go to college and only on your systems and don't make it easier. Sorry virus, I didn't mean to hear you. I just never heard anyone say that ransomware is a Windows thing I'm like they're been ransomware on mobile phone boot ROM since like 2001 where who are you. I hear it. I do I hear it I I hear that this usually only hits windows and look at the windows environment and all that. And sometimes that's that's absolutely true. But you should be scanning for lateral movement to everything. What about Max I thought Max were immune from viruses. Yeah, so we've already gone over time a little bit, but I think we have about 10 minutes left. And so I wanted to kind of reward the folks who have stuck around through all of this very, very serious conversation on incident response and try to try to end on a lighter note. And so I was going to I was going to ask each of you if you could, if you could share one good story about like one of the craziest things that you've seen. And if you have a response or, you know, in some of your own actions virus, and I will remind you that this is being recorded. So who wants to go first. Anybody got one queued up. I have one kind of queued up a little bit. So, so this is actually a combination of two. So, so one is with the worst thing that I've ever seen an environment. Was there was there's a firewall company who is a very, very popular, very, very known firewall company. And for whatever reason, they somehow implemented all of these firewall rules that were that that were okay they weren't great they were just okay. And then somebody put in a triple any firewall rule not a double an actual triple any so it was any any any. So if you find out from speaking with someone who will rename nameless but is on this panel, that company actually allows you to have as many as five enemies. So, so not only is there a problem with five any is like the even the option for you to be able to put that many any is in something like that doesn't make any sense like what are they possibly thinking like we're just going to make sure no matter what that you can actually find some way through our firewall. So, the, the hybrid of that is is it was with the same client. It was, it was kind of kind of a cool story because we had a chance to talk about it. I believe last year in New Orleans because the actual indictment came down. So, they were a part of a breach that was essentially targeting the aerospace industry, and the attackers did all kinds of stuff so we were able to actually go through essentially like their gen one through gen five of their malware including like the last ditch absolute effort where where it was incredibly sophisticated because it was worth burning something that was amazing on it and when I found out like that was just incredible it was amazing like actually did a little dance because like all right now we found this now we have to find out how we're actually going to get rid of it. But the way but the way that we're able to track back to exactly where it was is again because attackers are human beings and they make mistakes. So for the most part they bounce their IP through through other countries and everything was just fine. Except we had a total of 11 instances over the course of I believe is a three month period where they forgot or they just did made the connection before it actually made all the hops. And we had 11 times where they actually able to pin down pretty much to the exact building and a certain company that rhymes with Rhina of exactly where the attacker was exactly where what they were doing and exactly what they were after. And you know that was that was amazing because we had we literally had gigabytes of logs of all this access in and in and in and in and then we happen to find like oh well here's this web shell that they had put in. The only person to access it was this person we know that for sure and 11 times they just made a mistake and we're able to pin down where they were so it literally takes just one time of messing up. And like OK now we can actually build an investigation and we turned all that over to law enforcement and unfortunately it took them about five years to actually go through the process because they had so much stuff to investigate primarily for our case. But it was it was also kind of disheartening to see like flash reports get put out that is essentially the summary of our report that we had written and just has an agency logo at the top of it like it's like that's my stuff. Don't do that. All right so who can go next. Oh man, I guess I should follow just because Brian at the time I had told you I saw five enemies on a firewall and I was like what in the hell. And it was like a horror for me because I was just like it was at the top to you so if you if you think about firewalls where you've ever done any kind of architecture on top down. And it's any source any destination and the application and user. And I don't know which one I left out location and service. There are actually six that can be set and they set one is deny and I didn't even know why they set one is denied because I was just like it basically just works. That's the answer I got by the way was it just works. But my, my worst story is actually recent. I have a lot of empathy. I see a lot of stuff in ransomware but one of the coolest things that I've worked a couple of was not too long ago there was assault staff vulnerability that came out that allowed for Basically, the long story short is that allowed you to get root to anything that was in the next expose that was assault master. So salt is used salt stack is used to manage and administer a Linux environment. And it's a really cool thing I actually really like it for administration purposes. But there was a vulnerability and if there was a crash, you can get root into anything that was in the next words. And I just happened to see it in a couple of cases. I thought going into this, oh man, I had root on something automatically that was Internet exposed. What could I do. And I spent a long time looking at this environment thinking, I'm going to find something cool. And they got in and they used it just him to mining box everywhere using the minion service. And that was literally it. And so I want to tell that for a story because I lived to for the cool like when am I going to find an O day and I've seen them like don't get me wrong I've seen some really cool stuff from these and like, you know, some really cool tactics I've seen people absolutely stop out servers to where I have to go into an allocated space and get really creative about carving things out. But this one was one where I was I was going in going I'm going to see like something absolutely crazy because you automatically had root access and you could do all of these things. And that was not the case. And I've seen so much of that over the course of doing IR where I'm just like, you know, you have this level and you work to this level. And I think that's something to keep in mind just when you're doing these things is, you know, I was super happy to go back to the client and say I have great things. All they did is run up your cloud bill. But at the same time, like, never assume anything because I was thinking like I'm going to boil the ocean and find something and I did it. 2005 I was on a plane coming back from a job. And we're talking about war stories. And he tells me a war story from the week before, which is why he was late to the job that we were on. And I got what you do. He goes, I was doing IR on an old Windows XP network that was locked down by this government thing because it was a defense contractor. I'm like, oh, well, how'd the bug get in? It's like old Java bug, super like 265 day, right? Like old Java on Windows XP. Oh, it was a vector. Facebook. Yeah, I was around there, you know, running Facebook and put something got affected. Oh, that's interesting. Why did it take you a week to clean up? Well, didn't have a lot of ambient room on the operating system to install tools. Why? Oh, it was a drone control station for Predator 3. I had a similar story that the moves about, you know, tracking down a what we what we realized was a O day for a printer. And, you know, we thought for sure that this was a very advanced threat actor burning an O day for, for this situation and found out it was just to run a where server off of our printers. Yeah, because they have so much storage. And so that, you know, again, yeah, really excited. All right, yeah, this is going to be like going to make me famous, right? No, no, no, they're just running where. So, but they somehow managed to get a no day. And, you know, that was what they spent it on. Well, I, I certainly don't have any, any stories as exciting as those that the rest of you have shared in my somewhat somewhat boring career in info sec, not getting to go around and do incident response all over the place. But, you know, I just want to take the opportunity to thank all of you for accepting our invitations and coming here and sharing your experience and, you know, helping all the attendees, you know, here who are here at the blue team village to learn from you. I've learned a lot from you. I think this is, you know, this is such great valuable information and getting, you know, a variety of people's perspectives, you know, to help us all, you know, learn, learn new things and different ways to approach problems. It's awesome. I want to thank you so much for participating. And for, for the attendees who were here in the BTV Discord, we're going to move over to the voice talks chat. Anyone who's available, panelists who are available to pop over there and take some questions like hallway con. So, I guess that's a wrap. Thanks everyone. Thank you.