 Hello, ladies and gentlemen, boys and girls. How are we doing today? Wow, we're still hungover, certainly not still drunk. Not at all. Let's start out with the disclaimer. This was not reviewed by my current, past or future employers. They have no insight into what I have been talking about this, nor that I was intending to, so don't hold them accountable, please. Basically, I speak no one, nobody speaks for me. Various other legalese in there. So, who am I? Yeah, I'll drop a couple F-bombs here and there. I apologize. But it's nice to be able to give this talk because so many years ago, I actually started out as a SOC analyst in a security operation center here in Las Vegas itself, worked there for many years, transferred to being the technical lead for an incident response team at a large research laboratory and been downhill ever since. So this is going to talk about why is this important? Because in my current day job, I get to talk with vars and vendors and they're talking about evolving to the SOC of 2020 and this tool will solve all your problems and there's a hiring shortage, air quotes, etc., etc. And I just keep throwing up the bullshit flags. So let's talk about this. Let's talk about how we've gotten into this situation. What can we really do to improve and what things should we be working on and when those vendors come in going, hey, this is a great tool. You should look at it. You should buy it. You know how to see through that. So what is security operations? Right? What are we looking at here? What is the real meaning of that first? Let's set the stage for what we should be doing and then how we should be evolving to. I look at security operations as being that where the rubber hits the road for your organization, where your policies, procedures, what you're looking at is all those tool sets that you have bought, those operating systems that you're running, those applications, databases, etc., what they're feeding into, what are you looking for to find the badness? What are you looking for to find when Tim's been sending those harassing text messages to your secretary or surfing for porn or down a landing mount, etc., not that I've ever done that. This is where it all feeds together, right? If you're not getting the right logs, if you're not getting the right information, how do you get that visibility? Normally we don't, right? That feeds into when we find something, how we're going to do incident response with it, right? Oh, this is involving a certain person. Now we need to do a forensic investigation in regards to what they've been up to. That should also feed into our security awareness program with our employees. And here's the things that we're seeing. Here's that phishing attack that our CEO got yesterday about, hey, I need to change my W-2 deposit from, yes, from some lightcoast.com account because I'm locked out of my domain account, right? Security operations and computer network defense. We can call it a SOC, we can call it a CSER, we can call it an S-SER, throw whatever three or four letter acronyms you want in there. Fundamentally, this is the group of folks of individuals or one individual in some places, right? That's looking for the badness and trying to put the fire out and then figuring out what we can do to make this better so it isn't as painful the next time. Though we sometimes forget there's three key pieces of this. People process technology. What do we tend to focus on? Because we're tech hacker geek nerds. And we don't like talking to people and we hate writing down process. As we have, some of us have evolved, have moved up the food chain as we're into sucky management positions that we don't get to do all the cool technical stuff, we need to focus on all three. How do we leverage each of the pieces? There is no silver bullet, right? No matter how often our CISO is looking for that, right? Or no, we don't have a CISO, let's talk about the CTO. There's certain necessary capabilities that each organization should have. Two varying degrees of how big they are, what the resources they have, what their threat profile look like, what critical infrastructure sector they're in, et cetera. Right? Looking at the events and what they're happening, trying to correlate those to get some quantifying idea of what those events are looking like, leveraging that threat intel, leveraging that indicators, putting those together to get an understanding of what your threat profile looks like. And when the shit hits the fan, incident response, put the fire out, clean it up, what can we do to make it better and try to prevent it from happening the next time, right? If it's more in depth, do we need to do digital forensics on it, right? Is this some commodity ransomware we've dealt with a million times or is this something new targeting us? Is this something that we can do some analysis on and share with the rest of the threat intel and incident response community? And then just supporting the tool sets, the things that we have. Hey, Tim, how come we haven't gotten a report from the DLP system for two weeks? Oh, yeah, because we're not doing a performance monitoring yet and the drive filled up. Derp. Those audit and compliance controls. Why should we sit in a weekly meeting going through spreadsheets? Why can't we leverage the technology and tools to do these cross checks for us, right? The intent of audit and compliance was, okay, kids, you have to be this tall to ride this ride because companies didn't want to do security. So let's set something up as bare minimum to do. Well, with the tool sets we have, what things can we do to have this do it for us? And then certainly some insider threat detection. How many people are stealing source code? How many sales people are stealing the entire customer database for the next company they go to? Not that that ever happens, right? Key piece, vulnerability management. What volums do you have trying to correlate that with which ones the bad guys are trying to target? And what things do we need to get the network server desktop folks to fix priority, right? And then outreach. As all we're collecting all this stuff, as we have this ability and understanding, how do we communicate that in a way that the business folks understand it? The pointy head managers understand it. The C levels and even sometimes the board and for some of us, even to our customers, right? Because all those capabilities because Matt did a great job at putting another pyramid together. How do we build from that? How do we need those different pieces in getting that understanding, right? And if you're looking at the Sands top 20, if you're looking at the NIST and all those other standards out there, this is pretty along the same lines. It just gives you better pretty pictures and another good way to display it to the pointy head manager that you have to try to explain it to. But let's try to group those together and Matt did a great way of doing this, right? So when we're building and trying to mature our security operations programs, this gives us some guidelines and some tool sets to help evangelize this to the folks that we need to get those resources, whether it be money for tools, cooperation with other teams, headcount, et cetera. And Wendy did this great one. I really like this. She just tweeted us out about two weeks ago. I'm like, oh yeah, this is going into talk because we all need some kind of pyramid, right? But yeah, fundamentally, knowing what you have and what is it doing. How many times I sat in that sock not too many miles away from here and I'm dealing with first and second tier sock analysts are like, oh yeah, that's normal. And I started digging into it anyway and ended up being one of the biggest incidents for that quarter. Controlling those changes, understanding those threats and who's tacking them and using the products effectively and also monitoring their performance. Because there's that capability chasm and here John Lambert did a really good way of identifying those different areas that a lot of security operation centers or security operations capabilities fall into problem wise. Not having executive support fall into various traps. You don't have the right data or you have too much of it. Basically, how do we as teams stay poor? How do we keep running around trying to put out fires instead of taking a couple hours every Friday afternoon and doing something more strategic to help us in the long run? So there's different types of security operations, right? First one, bare bones beginning ad hoc when it hits the fan when there's a certain individual or two from a certain three litre agency that's packing a weapon and knocks on your door going, we need to have a conversation, right? Not that I've ever experienced that. Yeah, the organizations that have evolved to a point of, hey, we might have one or two security people within IT that do this to, we actually have a full fledged C-Cert whether it's dedicated or assigned people to full fledged security operation centers. In this table here, I'll outline the organizational model kind of the size of the organization and we'll touch on how to quantify that in a moment. And then geographical scope. And this is directly from the MITRE 10 strategies of World Class Cyber Security Operations Center. I will reference this book a couple different times even though I don't agree with everything it says, it has some really good information in it. So let's talk about that evolution. How have we started? How did we evolve? How are we still hunched over on that crappy ass desk? Well, we started as cavemen, right? We might not be talking about Wesley Dale quite yet, but we're certainly looking in that direction, right? This is that ad hoc when it hits a fan. This is that IT guy that goes, oh, I don't need blogs. This is not knowing, but we're starting, right? We're trying to figure out the wheel. We're trying to figure out how to get fire. And then we kick out the person from FireEye. Because, you know, the IT guys, this is, this is so easy a caveman can do it, right? But, you know, Irish, there's no evil bit on a packet. So we just let all of it through, right? This is the organization that doesn't know what they don't know, right? Meanwhile, all their data is being ex-filled. And some foreign countries making product based on their IP that, you know, they didn't have to do any R&D for other than figure out how to compromise your network. So let's go to the Middle Ages, right? Let's go to the somewhat organized ad hoc as needed when we need to go conquer Jerusalem or need to go conquer a local nation state. We need to go put out the fire and then we disband when we don't need it anymore, right? So when it does hit the fan, we might have some process and procedures. We might have some things going on, but it's all old from the last time we did it. That distributed co-managed. I look at this as being the explorers here in the United States, right? Let's go explore what's out there. We have an organized team doing it, but we still need some local help. Do we need an MSSP? Do we need somebody on retainer to do instant response, do forensics? What did you know, thinking of Lewis and Clark and what things they needed to do to make their explorers so meaningful in documenting all the birds they found, all of the nature that they found, right? How do we do that within our security operation center where we kind of hit this level, right? Have that ticketing system and document every little piece and then leveraging that as a resource that we're searching for the next time something comes up. Oh, hey, we've had this ticket, four or five tickets with this same source IP because it's been beaten on our network. Leveraging the tools and the data that we're pulling together so that our exploration, our voyage, is that much more meaningful. Then we have some sort of C-cert capability, right? We're in that organized wagon train across the desert. Hopefully we don't get stuck in truckie for the winter, right? But it's a group of individuals working together in any organized, some sort of organized way, right? For a common goal. Hopefully don't kill too many of the natives while doing it. But you also have those security operation centers that are like the gold rush, right? They keep running to every place they hear that there's gold and they dig and dig and what do they find? And they completely run out all the land that's there that we completely devastated looking for that one nugget. But then we can do security operations at scale. Let's get a tier one, tier two, two, three. Let's get a bunch of tier one folks doing the same process manually over and over and over again just like sewing. Just like the seamstresses that died in that fire. Because they're going to burn out, they're going to say F this, I'm going somewhere else. So let's try to put some automation into it, right? With those things that we can do, that we do every day, how can we automatically create a ticket and pull in a threat intel from our feeds and from internal and connect all the tickets that have that same IP address in it. Let's try to make it easier for ourselves, easier to make those determinations on how risky isn't how much do we have to take care of this, right? So I mentioned the MITRE book and the process, certainly determining optimal manning levels I worked with multiple clients in regards to how do we figure this out? Well, certainly there's no firm answer. There's no real equation to figure this out. But I have put this together based on MITRE, based on my own experiences as an analyst and working with multiple clients and working in multiple places for many years and how can we get at least into the ballpark of how many headcount, how many people we need and how do we leverage them. This also you need to look at how you as a company are growing so you can project how you need to grow your team, right? So first thing, just like the pyramids, just like the sans critical controls, you have to know number of what you have in your organization, what you're trying to monitor. At least the ballpark, right? Including the number of sensors you think you're going to need to use, how many cloud providers, domains, et cetera. Internal, external, because you're going to be getting the logs from that. Then what's your scope, what's your threat, who's targeting you and how? What compliance and regulatory obligations you have, what's your geographical soap, et cetera. Add all that up. If you have less than a thousand, then you probably don't need a 24-7, but depending on your threats and your risk plan, you're probably going to need something, all right? And if you have many thousands, then you definitely need something. If you don't, I really like to know how you're dealing right now. And I'll buy you some alcohol to deal with it, right? So specifically for analysts, specifically folks that's going through all the data, good ballpark number is 1 FTE per 50 to 75 devices, IPs, domains, et cetera. This certainly depends on the number of logs you're getting from those. And the intent is to handle all those critical alerts you're getting after turning your tools as well as process and procedures. So optimal situation, right? Number for ratio from tier one to tier two for basically every two tier one analysts between one and four tier two. And if you're doing FTE for a 24 by seven security operations center, it's more like five to one or three to one in a sock because you need shift operations, you need scheduling, you need floaters for when people are out sick or say F this, I'm leaving the company, I got a better job, et cetera. Or hey, the half the sock is going to DEF CON. Not that that ever happens, right? So for engineers, you know, there's the folks that we have that are doing the sensor tuning or soon the sensor maintenance, basically the stuff, all that tool sets, platforms that we're getting the data from, about half FTE depending on the deployment and also the tool sets you're using, right? If you're using certain Sims that start with an A, you're probably going to need a few more, right? Other ones, potentially less just because of what's involved in keeping them running. If you have your system administrators broken out from your engineers, then your typical ratio one to one or one or two to assist admin engineer types depends on your organization depends on mature you are depending on your separation of duties and to your fellow says admins out there, a belated sys admin day. So placement in your organization for your security program because this is going to directly relate, right? Do you have the wherewithal? Do you have the resources? Does the individual that you report to know the balance of the confidentiality risk, et cetera, right? Or is it going to be like, no, don't turn that off. We need that for such a business thing. We don't care. So you have the conflict of interest. Yeah, we might have a CISO but who does that CISO report to? Does he report or she report directly to the CTO? You still have that conflict of interest, right? Do you actually are mature enough or as big enough of an organization that you have a chief risk officer that looks all of this? And some organizations, hey, they might have security in the sock underneath legal, which is awesome, but are they tech savvy enough to understand what you're talking about when you come and tell them that your website is a cross-site scripting attack going on? So common conversation is do you need something internal or do you need a contract? Just contract it all out. Give it to an MSSP. Certainly, it could be useful to augment your internal capabilities for your smaller, less mature organizations, but, you know, MSSP services are commoditized. There's those common issues you deal with when you talk with certain organizations of how come you don't do this? Oh, well, you didn't buy that package. You didn't look through the terms and conditions, T and C, so we don't do that. So, certainly if you're in the market for getting an MSSP, understand what they're going to be providing you. There are the big ones, there's a small ones, and then there's the one, MSSP is in the middle. I kind of, I try to, I call those the boutique MSSP's. Those tend to fit most clients, most folks I work with a little bit better in regards to their capabilities and being able to pay attention to you and your needs versus just shooting alerts via email or to your ticketing system. Key point is that last bullet there, you, I need to manage that relationship and nurture that relationship. If you're not talking with them on a regular basis, they're not going to care. If you don't hold them accountable, they're certainly not going to care because they're just collecting your money. Oh, I don't need to buy an MSSP, I don't need to sign it up, I'll just do it myself. How well has that gone? Certainly, if you're the Facebook, Google's of the world and you have a shit ton of developers and that's all they love to do, go ahead. I've seen some of their projects that they use internally that they started 10 years ago and then the guy left and nobody's maintained it since. So what's your software development cycle look like? What happens when the folks that you maintain that get hit by a bus or leave the company or retire? I got my stock, I'm out, bitches. What is it about those open source tools that are already out there? What is it about Greylog that you don't like? Why can't you use it in your environment that you're going to need to create your known for brand new? Why can't we just fix what you don't like and improve the open source project that's out there? Tied to that is also how do we staff our sock? So that MITRE book heavily encouraged that we only needed developers in the sock. I call bullshit. Sorry. It's that diversity piece of getting different individuals that have experiences in different areas within information technology and information security that in the middle of an incident go, hey, I was a sys admin on a sun system and this is what it does. This is how we can go and find it. This is how we can fight the badness, right? No slam against developers but most developers don't run their own systems, right? They don't have that expertise. It's having different individuals at different perspectives in to deal with that fire fight sooner and faster and also come up with better ideas on how to prevent this from happening in the future based on the background experience that we have. We need real and valuable metrics. This is how we use to justify that security operation center we have. This is the baseline that we use to figure out how are we going to evolve to the sock of 2020? If we don't know how well we're doing now, if we're not monitoring the performance on that DOP system to realize that the drive filled up, how are we supposed to know how we need to evolve to? So I did a Google search. This is the first image that came up in evolving to the students. Thank you, marketing. All right. Because that's a good portion of it, right? There's a whole lot of companies out there trying to pitch this out. They're really going to make a difference. So the first thing I've seen is having some robust monitoring and detection, right? But we have Span ports. All right. Wait a minute. Span ports were never intended to be a permanent fixture. They were only supposed to be a diagnostic tool, right? They're only a sample of the traffic and they're the first to die when that device gets overloaded. And when I'm in the middle, too many times, I've been in the middle incident and I'm trying to figure out why our devices are not triggering on the badness because the device is on the line on me. Basically we're missing activity. If we can't see it, we can't fight it, we can't put the fire out. When does this normally become a problem? When the organization runs out of Span ports. A little too late, right? But, all right. So the packet brokers, there's a couple other terms out there. Basically it's a middleman for your network traffic. So instead of putting multiple devices in your network, in line, you put the packet broker there and then make a copy of all that traffic and filter it however you want to your monitoring detection tools. So it's a force multiplier for the tool sets that's monitoring your network, not just for security but also for performance monitoring. So this is a tag team effort with your network folks. Some of these also give you IP fix, DNS, other data out of it. This also lets you use less tech, less platforms. So they're in a prior life, particular organization got a quote for, I think it was, 35 boxes this company wanted to install in this environment. You can imagine the price tag on that. Brought in a packet broker, we were able to cut that in half. That quote went down significantly. That vendor was not happy-ish, but at least they were happy to are getting the opportunity to deal. And we were able to help that organization out by getting better coverage for their environment and into the tool set they wanted to deploy. Not all packet brokers are created equal. A ton of marketing, a ton of foot out there. The UX UI experience with some of these are horrible. Some abilities you can only access through the command line, not through the UI. Also, check the security posture of these devices themselves. Have they updated their firmware lately? What's their software development lifecycle? Certainly do a third-party risk assessment before you buy these platforms. Now, internally, who's going to come up? And certainly the cost of them. Quick picture and how those are normally deployed. Let's talk about data integration, right? Getting that data into your monitoring detection platforms. Hey, I already have X platform. It has the data feeds coming into it. Great. How are you using those feeds in your Sim in your other platforms? You able to leverage those as a pipeline? Is it stove pipes into each of these tools and you're not sharing them back and forth? They're open. Need that situation awareness? Provides better mitigation decisions? Basing it on real data, not that gut feel that we tend to do with insecurity. These tools are not intended to be myopic. Why do we keep trying to deploy them as such? And then from that, let's automate and orchestrate using the real data. Address that low-hanging fruits or you're less of a target. I just call it automated fruits. Whatever. This is no different than the Python, other scripts I did 10, 15 years ago. Right? But now we have a platform. Okay. Awesome. Because it's relatively easy deployment, I can have a tier 1, tier 2 analyst go in and do some clicks and create or update a playbook. And the next place we go to or the new person that comes into the team is going to be familiar with it. We can't say that about that Perlin-Python script that I wrote on a Sunday night after a couple of Bourbon Bear laid doubts. Right? When you're looking at these, be cautious. There's been plenty of acquisition. There's a ton of buzzword bingo out there. Do a POC, do a POV, look at the acquisition space and who's looking at it and what's going on. Oh, yeah. How many times did we see this on Sunday day before? Right? The premise is to take all that information, your logs, and come up with that user account or that entity account and try to identify that this is risky or there's a major threat. All right? Yes. It is useful in ways of if you're mature enough and you have a threat hunting or you have a wolf team that go digging, this gives them some indicators to go chase down. Just in if you're not mature enough to create some of those pretty advanced rule sets in your SIM, this will help simmer some of those things up to the top and help correlate some of those log entries that, you know, don't really quite make sense. Good case and point is, was working with a client. They had an alert come up that one of their employees, one of their employees was in South America trying to log into all their accounts. Okay. The analyst tried saying, oh, false positive, this product sucks, get it out of our environment. Wait a minute. This is a question of risk, all right? Is that employee actually in South America? Well, call HR, yes, she's on vacation. Okay, good. Second question, why did she ask her? Because in that particular country, it could have been stolen and quite likely could have been. So is that question of risk when you're looking at these tool sets, it might be less risky for your organization, but potentially not for others. And that's where we need to look at for these tool sets of it just helping giving you indicators of rabbit holes that go down into to make that assessment. It can't make that judgment call on risk for you. And this is getting thrown on a lot as well, right? Is it really machine learning? Is it really artificial intelligence? Or is it that startup that a friend of mine worked at, that they called MLAI by just counting how many times an end user would log in? They didn't do really other potential math after that. So definitely dig into when you're talking with these vendors to cut through the bullshit, cut through the marketing buzz. The perception of false positives similar to UBA, UEBA, what's that question of risk in those alerts that it's providing you? Because when you have those things alert, it's not like an old IDS alert anymore where this is bad, this packet was bad. These tool sets are trying to correlate all these different things together to go this kind of fuzzy. This might be bad. So it's a question of how we're going to assess risk. So let's take that MLAI and look at all the network activity from your span port or your packet broker, right? And then look at that end point and look at that traffic to figure out how bad it is. Be very cautious with some of the vendors in this space. Some of you folks I'm sure have already gone through the sales pitch with a certain company that only hires very pretty dude bros right out of college that give you the hard sell that have completely drank the Kool-Aid any inkling that they are not making a sale they completely flip rip the gear out of your data center and they will never talk to you again. Even if you see them on a conference show floor they will turn the other way because you didn't drink their Kool-Aid. What does that say about their product, their sale cycle and what they can really do if that's their marketing approach and sales approach. The thing about I have been able to use a lot of looking at this traffic over a long term to try to figure out what's really bad. What is those protocols that we're not expecting? Why are these two DNS servers talking to each other over open VPN on our network? Are the tool sets we're using our IDS our IPS the MLAI all these other tools are they working as they're intended and supposed to? How do we take the scientific method to prove that that IDS is feeding into our SIM and we're getting the right alerts and we're taking the right automation tools measures to stop the badness, right? So there's some companies out there that are doing this threat simulation platforms is what I'm calling it, right? Basically you establish some fake endpoints and some systems internally there's a similar system outside your network and they run attacks in between and does your tool sets adequately detect through the kill chain as they're simulating them what you need to see to find the badness when it's the real bad like anything else there's some thud be cautious, right? How are they integrated? Is it really a separate platform or is it a function of some existing tools you already have internally just haven't leveraged yet and then deception which again just like some of the some odd years ago and there's some stuff we've done on our own in regards to deception and sending up honeypots and fake documents and fake email addresses and that sort of thing here now you have a vendor that's providing a platform to do that for you and monitor for the badness and activity again like everything else we've been talking about there's a whole lot of thud out there what would be useful to you and how mature you are as an organization to encourage this vendor to push it out and maintain it in an easier way with less resources so how are we doing how do we measure ourselves how do we say that we need to go in this direction or that direction right there's those optimal characteristics and capabilities like we mentioned earlier in grouping in different areas there's that organizational model how we're structured do we have the right process procedures to implement all this how do we get there how do we have that capability internally we can run with ensa the c-cert self-assessment the first GCS also has a maturity survey certainly leveraging the sans critical controls we have the NIST ISO EEC so as part of our security program as part of these process procedures we're putting together we should also put in cross checks for us to check if we're doing that on a periodic basis depending on your organization and your threat profile finding somebody to bring from an external agency to make that assessment and make recommendations on how you should improve hopefully with an organization you already have a report with that already know you taking that measurements and giving you a model of hey here's where you at and from our conversations and your threat profile that roadmap to your director of security and your CISO and they go and fight for budget and headcount and here's our plan on how we're going to mature as an organization so where do we go from here we have our log processors we have our dirty sock we have the pen tester come in every so often what's next through the ages we've evolved but we certainly have not matured as an organization as hackers what can we do to help with that we need to ensure the basics are complete back to those triangles do we have that foundation to build from or have we jumped ahead be willing to hire be willing to invest in those hires be wise to the hiring in your local area to your needs and capabilities and what's available in your local market because if you're trying to go we're going to deploy this tool but there's nobody available to run it and maintain it does that make economic sense for your organization does that make proper sense in regards to risk for your organization and is it good enough or are we trying for that perfect solution so many times we're having conversations not just what last week week before oh reddit got pwned because they were using sms2factor and sms2factor sucks and nobody well for them they made a risk call and they said hey it's good enough certainly they probably have potentially have the right monitoring pieces in there to go are these sys admins getting targeted if it's good enough have that monitor there to be able to detect and see is it not good enough anymore and we need to add other mitigations other compensating controls so some takeaways for me rambling for the last few minutes be careful look past that buzzword bingo into marketing use that real data to try to determine what's useful for your environment your risk posture what's that size and scope for your organization when do you need to step aside and go hey I'm in over my head when do I need to bring somebody from external help us with this certainly with that hire to your realistic needs and be willing to hire for this why have that position open for six months hire somebody now you can train them up in that six months right a couple different references I've already mentioned a couple of them quite actually all of them looks like thank you for joining Matt John and so first the other organizations thank to all that have come before us that have worked in socks and have helped us evolve to where we are yes so if you have any questions let's meet outside in the hallway next to the bar thank you very much all have a good day enjoy Defcon