 It is actually now my pleasure to introduce you, Eric Capuano. I appreciate that, and also the line that you guys were in to get in here was super humbling for me. Thank you so much for taking time out of your awesome DEF CON to come and listen to me drone on a little bit about security operations on a budget and some tips and tricks that I've learned in my time. As Ming said, we go back a little ways because I'm actually a member of the Pack of Hacking Village in the Wallace Sheep here. I run the Honeypot Project, so if anybody here has seen that or if you haven't, please come take a look. We've got a really cool deal going on over here. So I'll dive right in. Heart rate is 101 right now, so let's see how that fluctuates throughout the talk here. Oh, clicker. Okay. All right, so a quick introduction. So I am the SOC manager for the Texas Department of Public Safety. I'm also a cyber warfare operator for the Texas Air National Guard. In my limited-side free time, I'm also a private consultant. As I mentioned, I support the Pack of Hacking Village here in the Wallace Sheep at DEF CON each year, specifically emerging threats, honeypots and deception systems, cyber-patriot instructor as well. And that's something that I think everybody, if you're in the industry, you should be doing that. Pay it forward, teach the kids. We need more talent in folks in this industry. I'm a blog author, not often enough though, Security Against Obscurity, blog.ecapano.com, and I'm a hobbyist teacher student. So if you notice, there's all these things over here, and it all revolves around coffee. A few disclaimers, right? These are just my opinions and mine alone. There is no absolute truth in any situation. Your mileage may vary. And to cover my ass, all products, vendors, figures, and potentially anything else in this presentation are completely fictional. And also, the only wisdom is knowing that you know nothing. Guys, I don't have all the answers, okay? I just know a few things that I figured out in my journey. So here's the million-dollar question, right? Why is enterprise security so hard? And then why is it so damn expensive, right? Quick history lesson. 114 years ago, this guy, right? Marconi, he wanted to go out and demonstrate to the public that he could securely transmit information completely hacker-proof, right? Anti-APT, zero-day-proof, secured transmission of data. Well, just like today, most of those promises were empty. And this guy Maskaline, who's actually, Neville Maskaline was actually a magician, right? This guy wasn't a hacker, he wasn't any kind of computer dude, but he was a magician, and he said, I'm going to embarrass the hell out of you. And so what happened was, this guy, he's like, hey, look, I can broadcast things over radio, totally secure. Trust me, because there's a padlock picture next to the radio tower, right? And this guy over here is like, nope, what I'm going to do is I'm going to hijack your transmissions, and I'm going to replace them with my own, and I'm going to talk all kinds of shit about you in Morse code during your proof of concept. So he totally rocked this guy during his POC, just to demonstrate that security is hard. So here's the question, okay? 114 years ago, this is happening, why are we still getting it wrong? In my opinion, honestly, what it boils down to is a mindset. It's not stuff you buy, it's not, you know, magic boxes that you put in your data center, it's a mindset that we've got to get right. And slides, you got to get those right too. Okay, what does it not come from, right? Padlock.png on your website, magic box in the data center, next gen magic box in the data center, or spending millions, right? There is not a direct correlation between how much you spend and how secure you are. Checking yes on your compliance audit. We're good. It's a mindset. Okay, some common problem statements, right? These are things that you hear all too often. I can't hire good people, right? There's this major talent shortage, apparently. If that's the case for you, please leave me a business card, because when I post a job in Austin, Texas, I end up getting about 20 times the applicants that I can afford to hire. So I have too many qualified applicants. So let me know if you need help there. I can't train the people I have, right? We're going to talk about some ways to do that. I'm too busy responding to incidents to improve incident response. That's one of my favorites. And my budget has a lot of zeros, but they're on the wrong side of the decimal. This is a really common one, and this one drives me crazy. My users are dumb, and they keep clicking and opening and downloading stuff, right? Next, next, GenAPT signature avialist firewall is too expensive. It's true. It is. It is too expensive, but the good news is you don't need it. If only there was an affordable way to solve problem, right? Open source solutions are not feasible because of reasons. If anybody wants to debate that, please see me after the talk. All right, so going into what I think is the foundation for solving this problem, it's people. You absolutely must hire the right people. And then when you have them, you need to take care of them to keep that team. Because if you can't get this right, nothing else is going to matter. This is the foundation for doing this. So I know you guys have seen this before, and if you haven't, this is something to consider. When you post your job, right, you're hiring key words, you're hiring certifications, or at least that's what your HR department is looking for. We've got to get away from that, okay? Because key words do not fill sock analyst seats with the right people. So your cybersecurity job requisition, right? This Venn diagram illustrates that the people that perform security jobs often do not overlay with your job requisition. Bridge that gap. Stop hiring based on degrees or certifications, okay? I do not have a degree in cybersecurity. I do not have a degree in computer science. But I do this stuff night and day, and somehow stumbled upon being pretty good at it. Same story for every single analyst in my sock. Look for passion, look for drive, look for someone that has the ability to learn, right? If you guys have seen CISSP Googling, I mean, it's a bit of a parody, not to pick on the sysps, right? But what it really is here is that this industry is driven entirely by certifications. When you're hiring just based on the certs that someone has, you're doing it wrong. Instead, take a chance. Take a chance on the newcomer that has no certifications, has no experience, but in the interview expresses a passion. It says, hey, I go home at night and I build a malware reverse engineering lab just because I want to learn how this stuff works. That's the person that you want on your team. Once you get your Rockstar team, you absolutely got to take care of them and you've got to give them the tools that they need to succeed, right? Especially for someone like me. I'm a state government. I do not have infinite budget. I cannot pay what private sector can pay. So I have to make up for that in other ways. So first, enabling effective security operations communication. You've got to get it right. If you're still using email for security operations, you're going to have a bad time, all right? You've got to come up with something that can keep up with the speed of operations because your team is never going to be any faster than the method by which they communicate. So when we migrated away from email-based communication, we actually achieved a rate of about 90,000 useful messages per year between our analysts. It's a lot of traffic and that's not just analysts talking to one another but that's integrations. That's chat operations. My analysts are able to blacklist IP addresses from Slack. That's pretty cool. If you're on call and you're at the movies with your significant other and you find out you're getting attacked by a phishing campaign, it's kind of neat that you can just pull out your phone and handle the situation. But at the same time, it raises the situational awareness for everyone on the team. Train your people because HackerTyper and the North Threat Map are only going to fool your executives for so long. Training is a must. So how do you get it, right? Training is so expensive. Well, guys, it can be done on a budget. Send your folks to DEF CON. Send your folks to SANS if you have the budget for it. And I'll go over a few more training options. Documentation. The least favorite thing among developers, engineers, analysts, but it's a must-have. If you're not documenting your procedures, if you're not building repeatable standardized procedures for the things that your analysts do on a day-to-day basis, you're not going to have a good time. So it doesn't matter how you do this, but crowdsource it. Each one of the folks on your team, put them in charge of a program or a tool or a system, and they need to be the one that develops a documentation for others to follow, because everyone on the team knows something that the others do not capture that. Tribal knowledge will be your Achilles tendon. So some ways you can train your team, right? CTFs. Send the team. As a team, participate in CTFs. If you haven't known, cibrary.it, fantastic resource, completely free. I wrote this a while back. They're well over 400 now. Free courses in IT and security. Send your people to conferences, especially ones like DEFCON, B-Sites. And if you've got the budget, again, SANS is pretty unbeatable. You've got to train your people for worst-case scenarios. Now, this one, I could have a whole... Actually, I do have a whole other talk on just this point alone. If you're not training your team for the worst-case scenarios, they will not be ready when that day comes, because you cannot train for something you've never... You cannot effectively respond to something that you have never encountered before. So tabletops are a good place to start. If you don't have the resources to actually simulate an incident, then do tabletop scenarios. What if? What if we were breached today and someone moved loudly through the network and now they're X-filling, you know, MySQL database across the network. What would we do? And find out if your team is prepared to work as a team effectively in the middle of a shitstorm. If you do have the resources, simulate incidents. So a DFI or simulation is as close as you're going to get to actually performing as a team when the heat's on. And, like I said, tomorrow around this time, I'm giving a talk on how you can do this yourself to build an actual incident response simulation platform. But one of the best parts about this is it can be done very inexpensive if some open source concepts are applied here. Okay, and then this is really more kind of your administrative, your managerial, but lead, don't manage, okay? Because it's micromanaging, it's a morale killer. You've got to maintain the morale of your team. They're already in a very high-stress environment, working long hours, dealing with incidents in the middle of the night, Sunday mornings. Morale's a big deal, right? So what I do once a month, you know, the first Friday of each month, I let my team go half-day. We go to the local pub for team building. It's things like that that improve morale, that keep people happy, that keep a good team with you. Once you've figured all this out and you can keep a good team, that's when you can move on to the more technical side of these things, how to do security operations just a little bit better without the latest tool that leverages artificial intelligence to take zero days in your environment. So, be less busy with this one simple trick, right? Love it, it's my clickbait, sorry. By shifting from a reactive to a proactive posture, right? And that's one of those buzzwords that we love to see and hear that everyone uses but no one can explain. So how do you start? How do you achieve this, right? Well, first thing we have to do is identify where are your actual threats? Like where are you most vulnerable? Not where the latest headline or vendor tells you that you are. And here's some extreme examples. Fansmitter, who remembers this one? Transmitting information across air gaps using the CPU fan, RPM. Who's seen this in their environment? Raise your hand. Exactly, so why is an industry, do we care about things like this? When this hits the news, we all go crazy, right? Everyone starts tweeting, oh my God, they're ex-filling data with CPU fans in a laboratory at MIT. Not in the real world, okay? So we need to stop getting all crazy about the theoretical nonsense and focus on things that apply like word macros. Now here's an extreme on the other side of the spectrum, right? Is anyone familiar with APT Squirrel, Cyber Squirrel 1? Now this is real, these numbers are real here. Somebody actually ran the numbers to find out how many major outages were caused by animals. 927 major information system outages caused by squirrels. So you've had zero incidents caused by APT or fansmitter, but we've had over 927 across the nation caused by squirrels. So should we invest in anti-fansmitter or anti-squirrel? Again, extreme example, but just to show you the differences on both sides. You've got to be aware of where your actual threats are. So think critically and apply this mindset to all of the incidents that your team is handling. When you encounter an incident, whether fishing, we'll call it fishing, how frequently do you see this type of incident? Does it have similarities to other recent incidents, right? This is how you're racking and stacking to prioritize the things that you're encountering. Now let's say you're just seeing a rash of word macros that are coming in, infected word macros that are dropping ransomware in your environment. Well, what type of in-house proactive controls can you put in place to stop that? It doesn't mean rush out and look for the latest anti-word macro product. There are actually things you can do in-house. So let's apply it to a real-world scenario. So ransomware, right? Real-world example. So last six months at DPS, well, we ran some numbers. We saw over 140 ransomware campaigns and we've got a user base of about 10,000. So if one campaign went out to 400 people on average, then you can do the math on how much exposure we have to this problem. So it's real. This is definitely something that we were concerned about. So we did a little bit of research, right? And I just want to say this threat research, it's not been shared with the public until now. Where was the ransomware coming from? It was coming from emails, right? Email attachments. Really advanced delivery mechanism. And fortunately, we have some in-house controls that we could use to mitigate this threat. But what else do we want to know? What else do we know about the delivery? What else do we know about the threat? If they're macro-enabled documents, we can actually control macro execution on our workstations. So we stood up a CA and we required digitally signed macros now on our environment. So now we've stopped malicious emails at our boundary, at our perimeter, and we've also mitigated the execution of macros on our workstations, and it cost us nothing but time. So we've, knock on wood, 100% mitigated this threat without spending a dollar. If anyone's curious about how you can go about doing that, actually have a blog post, you can go and see step-by-step, even from selling your executives on this to actually implementing the plan and rolling it out. Yes. So we found that that eliminated about 91% of the ransomware that we were encountering on our network. The remaining 9% were coming in the form of Billy's resume.pdf.exe.js.wsf, et cetera. So it's fooling the average user that's not paying really close attention to their file extensions to running executable code thinking that it's something else. So what we did was we looked at these rogue file extensions that were coming in and we noticed, hey, wait a minute, Susie and HR does not need to run .js files on her workstation or .vbs files. So how about we just change the default opener for all these weird file types to Notepad so that instead of double clicking and running script, she double clicks and sees script and no harm is done. Another completely free solution to a very real problem. So that was a good win because with all that money I saved, I was able to hire another analyst. So let's look at one more that I'm sure everyone here has probably dealt with, right? Fishing campaigns. So this is one that I think a lot of teams are losing a lot of time over because we're still playing IP address and URL whack-a-mole, right? Quick, blacklist the URL that's never going to be used again. There's probably a better way to do this and you don't have to buy a special tool to do it. Any kind of IDS, IPS, including a free one like Snort or Suricada can do this next thing I'm going to show you. So let's take a fishing URL. It takes us to this really cool-looking Facebook site that's obviously not really Facebook. How can we proactively stop this with next-next-next-gen anti-APT technology? Well, let's just look at the source code and that doesn't look good at all. I'm sorry. But if you can make this out, the source code of the page actually has an entry for the tool that was used to clone this website. So, HTTP website copier. I don't know if anyone analyzes fishing pages often, but this one is very frequently used and our attackers are way too lazy to pull it out. So let's capitalize on that, right? I can write a really quick IDS signature now that's going to detect this string and now proactively stop not 100% of fishing sites, but a hell of a lot of them and it cost me nothing and it saves my team a ton of time. So again, it's finding ways to proactively prevent incidents, leveraging things you already have. Cool. So moving on, we're getting a little better at this security thing, right? Thinking proactively, what can we do to reduce our tax surface without spending a ton of money, but at the end of the day, guys, security does cost money. So how do we get money, right? You need executive support, but there's a right and a wrong way to go about getting it. And if you're not getting the executive support that you need, there's a chance that you might be selling it wrong. So how do you do it right? Instead of asking for it, convince them why you should have it and it will come. So example, stop using the fear tactics, guys, okay? Don't go into your CISO, your CIO's office. If you don't give me this much money, we're going to get ransomware. He's heard it before and he doesn't want to hear it anymore. Instead, what you need to do is you need to put the stats in his face on how much awesome you're already doing with the nothing that you have, right? That's not common, actually. Most IT folks go in and say, well, I could do that, but I won't because I don't have budget or I don't have people. Okay, well, that guy's going to continue to not have budget or people. But if you just go ahead and kick ass with the little bit that you have, generate some stats and go and put that in front of the executives, all of a sudden they're seeing value. Okay, I'll throw a little bit more money your way. But get away from the fear stuff because it's counterproductive. Also, showing your executives what you're doing and showing your users what you're doing. Publish a newsletter. Honestly, it sounds like a waste of time, but I'm going to tell you that's something we started doing that has paid dividends. Publishing a newsletter to our 10,000 users about what our team actually does to protect them. We get a flood of thank you emails every month after we put that newsletter out. And we honestly, we put it out just as like a trial, like let's see what happens. People actually give a damn, like seriously. So put that out because now you're winning friends, you're gaining influence, which trickles up to executives, which trickles back down to you in the form of budget. And it turns out that your C-level folks do like honey. So securing all the things without spending all the money, because budget doesn't always come, but security must continue. So how can we do this on a very small budget? Well, first of all, we're going to talk about breaking bad habits, building your own tools, ditching bad solutions, starting with the Pew Pew map in your sock. So here's my advice on how to do this. When it comes to the stuff that you already have, right? Stop suffering from contract renewals. Let's just renew this because we've had it for years. We just we just we need it. Let's just renew it. But hold on a minute. You're spending a fortune on the things that you have. Are they performing the way that they should be? And I've got a few examples for you, but stay away from these statements here. Well, we've always used it. So just renew it, right? Or why have we been paying for X when we're only getting Y? And then the big brand romance is right. If you're in love with big brand and that's why you're you're hanging on to it, then you're living in a silo. When you're assessing a problem that you're trying to buy a solution for, absolutely ask yourself the question. Like, is this actually a problem that exists? Or is this just something that you've manufactured that someone else has manufactured? You know, my my CISO comes to me all the time. Hey, we should buy this new anti-APT. And I say, okay, but we have these other four tools already in the environment that do what you're asking. Do we really need to buy this? And then ask yourself, are we even using the tools that we already have? Because that is a huge issue right now. How many tools are in your environment? And how many people on your team know how to use all of those tools? That answer is always a frightening one, right? So instead of buying more stuff, use the stuff you already have. But when you do need to buy something, do your research. Don't talk to vendors. Gartner, it's a love-hate thing, but it's a better place to start. You know, NSS Labs, read the reports, find out how these tools actually perform in real-world environments, resist the marketing. And then always demand a hands-on demo in your environment, because it's just like clothing. If you don't absolutely love it the first time you try it on, it's not going to get any better after that. Also, buy what fits in your environment. Nothing sucks more than buying a device or a component or software and then finding out that it doesn't integrate with anything, but that manufactures other stuff. Big Firewall Company sells a threat intel platform that, guess what, only integrates with Big Firewall Company's firewall. Bad move. That's how they're trying to trap you into their ecosystem. Most importantly, why buy what might be free? Ask yourself, is there an open-source tool that can solve this problem? The answer is probably yes. I'm going to ask you guys to consider something, right? Why is it that security product, why? It costs millions of dollars, but the social engineering toolkit that Boris just used to break into your company is free. Slippery slope, right? What if I told you there was a free tool that could block Tor exit nodes, blacklist phishing URLs, automate analysis and incident response, pull hundreds of open-source threat feeds, deceive your adversaries and warn defenders in the early stages of attack and almost anything else you can imagine? Has anyone used this tool? You probably have. Python. Come see us in the Honeypot project where we're actually doing a lot of those things with Python. Constantly assess the tools that you already have. For every noteworthy security incident that your team is handling, you need to be holding your solutions accountable. Are the indicators of this attack new or old? Because if they're a year old, but your sensors did not detect them, you need to be making a phone call to attack. Why the hell did this get through my million-dollar IPS when I found open-source intel on Twitter from a year ago for this attack? That's embarrassing and that shouldn't be, you know, that's not something vendors should be getting away with. So ask yourself, what systems had visibility over this threat? If it was an email-born threat, obviously your email security appliances had visibility, maybe your IPS had visibility, and then obviously endpoint agent had visibility. So those are three solutions that you should be holding accountable to why did this incident happen and hold your vendors accountable for it. So I'll give you guys some really quick examples. I'm going to glaze over these on how we've walked the walk at the Department of Public Safety in Texas with said vendors. So scenario one, big brand email security appliance, right? Big brand usually means big marketing. We were paying $234,000 annually for these systems. Terribly difficult to use, configure. Stability sucks, they don't cluster. Zero bells and whistles. I mean, it was just basic email security stuff, right? $234,000 a year. I replaced them with a solution, also a big brand, but very little marketing, and that saved me about $200,000 a year. Way easier to configure. Oh, and it came with attachment sandboxing. Pretty cool feature for email. And actually, our in-house testing showed better security performance, win-win. Scenario two, perimeter UTM next-gen firewall. Now, this is a big brand that's actually a marketing company that sells firewalls. $750,000 buy-in. $274,000 a year with a 4% increase annually. Terrible support. Absolutely terrible, terrible support. Matter of fact, they once held support for ransom, and I was a month out from my renewal, and they said, we'll help you as soon as you renew. False negatives on threats with year-old OSINT. Hey, I just used that example, right? And then usually the response from support on said incident is, oh, good catch. That's going to be in our next signature update. I'm glad that I pay a quarter of a million a year for these. So migrate to almost as big brand with no marketing. $36,000 buy-in, $40,000 annually. Magic, awesome greatness, we're so happy. I'm saving a fortune, guys. What does that equate to? That equates to more SOC analysts, because if you recall from the beginning, people, that's how you do this right. So then let's get to the fun part. Where's all this free stuff, right? You're telling me to use free stuff. Where is it? It's on Pirate Bay. I'm just kidding, but try GitHub, okay? But there's a few of you out here, and you don't have to identify yourself, but you're saying to yourself, open source has no place in an enterprise environment. Well, I've yet to hear an argument that actually sticks, because I have a single DevOps engineer on my team that's able to maintain about 14 open source projects that brings a ton of value to my team, but I'd be glad to have that conversation with anyone that feels strongly. I will say, though, if you're totally against open source, when's the last time you looked under the hood of the commercial product that you have? What do you think it's running on? Open source stuff. So, you know, at the end of the day, yes, there's a lot to be said about the support number you can call them when things break, but when you're doing security on a budget, you're not going to be able to avoid using open source, nor should you. It's a fantastic resource to use. This is a good one, right? You would not get cloned a SIM, would you? Right? Why the hell not? Because guys, think about this. In the context of InfoSec, right? Hacking was built by communities of smart, passionate people exchanging free and open information, and the good news is that still exists today. You just have to tap into it. So, here's an interesting thought for you, right? If you think back to the Cold War, right? One school of thought about why Russia lost the Cold War is, well, we outspent them, right? You know, they basically, they went broke. Well, consider this. What if, during the Cold War, our weapons cost money, but Russia's weapons were free? Do you think it would have ended the same way? Think about how that applies to cybersecurity today, right? We're spending millions of dollars on defense. Our attackers don't have to spend a dime. We're not going to outspend them. So, it's just another way of looking at leveraging open source tools that you're evening the playing field a little bit. So, there are so many great open source projects, guys, that I could have spent this entire talk just listing them off, so I won't, but I will cover a few, and then I will publish these slides later on. But one that's certainly noteworthy that everyone in this room, if you work in a SOC, if you manage this, a team of security analysts, you should check out this one. The Hive is an incident response collaboration platform. It is phenomenal. It costs $3,000. That's free with an F, and it is phenomenal, and it will allow your analysts to collaborate on incidents, automate analysis, track observables, you know, all the magic stuff, that $140,000 a year platform that I could name but won't, does, but it does it for free. And here's just a little screenshot of the Hive. It's actually been updated a few times, so it's even cooler and better than this. And by the way, if anyone does use the Hive or is interested in Hive stickers, they actually ship me some to hand out after this talk, so I have some swag from these guys. The Hive is also highly integratable, like I said before. How does a tool fit into your environment? Well, this one does very well. Here's some other things, right? SIM. So, if anyone here is familiar with Alien Vault, Alien Vault has a freemium SIM. It is free $100,000, and it does all of the things that your $2 million SIM does. Asset discovery, vulnerability scanning, HIDS, NIDS, behavior monitoring, and it integrates with a free threat intel platform. Pretty cool stuff, right? If you're doing security on a budget, that's a lot of capability for nothing. Log aggregation, right? We all love Splunk. Splunk is awesome. Splunk is not cheap. So, if you're on a budget, Greylog Elk Stack, which Greylog actually runs on the Elk Stack, Grafana, there are free tools that can do all of those things. For HIDS, antivirus, OSET, Clam-AV, these are phenomenal tools that, guess what, a ton of those enterprise-grade, super-expensive platforms run these tools underneath the hood. Vulnerability scanning, open VASNIC 2, WAPIDI, the list goes on. There's so many of these. Intrusion detection, Suricada and Snort. They're very similar, one is a fork of the other. Firewall, PF Sense, Indian. Again, I'm not going to say that you're going to want to replace all of your systems with these tools. I'm just saying that in the absence of the budget to buy the enterprise-grade stuff, these things will get the job done. IOC sharing, right? Alien Vault OTX is a personal favorite, but anomaly stacks, bisps, there's plenty of these things. If you're into the really fun stuff, Honeypots, right? Check out ModernHoneyNet, Kauri Kippo, RDPI for remote desktop stuff. Tons more in that area throughout research. Again, I know this is a wall of information. I'm kind of blasting through it really quickly, but I will publish these slides. Also, if anyone's interested in open source tools in any other area, let me know. I've got a long list of them. Forensics, if you're into that. Google's Rapid Response Platform, Live Collection, Magnet RAM Capture. These are all free. Every one of these tools are free. If you're paying a ton of money for some sort of CBT to train your user base, FishMe actually publishes free CBTs that you can load right into your LMS and train your entire company with. That would probably save a lot of people a lot of money. I probably don't need to tell anyone in this room about Cali or Metasploiter, SETK. And that's all I got for you guys. So if you have any questions, if you want to share intel, feedback, any of that good stuff, tell me that the talk was terrible. Or find me on Twitter, and I'd love to hear from you. Like I said, come see us at the Honeypot Project. We're doing some really cool stuff over there. And also, if you're interested in building your own incident response training simulations, I have another talk at this time tomorrow about how to do that. Way more technical. So thank you guys. We're going to spend what? 10 minutes for questions? 10 minutes for questions. We've got 10 minutes for questions. All right, so you have a question. I'm sure there are plenty of questions. Please. So he's asking about what sort of metrics do we report up to executives? So that's going to vary based on obviously what you've got in your environment now. So if you already have some sort of IDS component or vulnerability scanning, email security, something that's producing logs that you can generate reports from, that's an excellent place to start. Don't fall into the trap of, hey, our firewall stopped 400,000 threats today because eventually they're going to ask you what that's based on, and then they're going to find out that that's a pretty empty metric. But if your team, like my team, actually we write custom IDS signatures every day. What we'll probably average anywhere between five to 15 custom IDS signatures a day, that's a real metric. I report that up to my executives because that's stuff that says, hey, look, the team that you pay for, that I hired that runs this suck is producing high quality things and we're sharing them with sister agencies. That's a metric that hits home with my executives. They're like, hold on a second, we're making this stuff? Yes, you bet. So metrics like that have a lot of weight. But until you get there, do your best to report on, hey, because we recently fine-tuned our email security appliance, we had an increase of threats stopped. We increased by 30%. Things like that, that's what your executives want to hear about is what are you doing with the stuff you already have? Then we'll talk about giving you more. Executives, they want to see value before they invest. It's not the other way around, which we want it to be, right? No, invest. Then I'll show you value. No, it's got to happen the other way, which sucks for us because you got to do a lot with a little to get there. Anybody else? Sir. You bet. I love it. So he said one of the arguments about leveraging open source in the enterprise environment is the support, right? There's no 1-800 number to call it. This thing breaks. But here's another cool thing to think about. So the Hive, that's one of my favorite projects. So when we started using the Hive, one of my analysts, she found a bug and we put in an issue in GitHub and it was fixed in 48 hours. Do you know any commercial vendors that will fix a bug in their tool inside of 48 hours just because you asked them to? No. Because you are just one of millions that are sending them money. Yes, we'll put that in our queue. Thank you. Open source projects are open source because anyone can contribute. If you don't like the bug, you can fix it. You know what I mean? So yes, there is a difference between having a 1-800 number to call at four o'clock in the morning on Sunday. That is sometimes a saving grace. But I will say that open source can compete because if it's open source, that means there's going to be tons of, in most cases, there's tons of documentation publicly available and there's a repository where you can submit issues or submit a patch, submit a bug, fix yourself. Sir. Okay, great question. Very real problem too because he got sick a couple of weeks ago and we were feeling the pain. His question was, I mentioned I have a single DevOps engineer maintaining 14 open source platforms and he does a phenomenal job. What happens when he is not here? If he gets hit by a bus, am I in trouble? Yes. Luckily, if you go back to one of my recommendations about documentation, I do not allow anyone on my team to maintain all of their TTPs in their head. If you have things that you do on a daily basis, routine maintenance items, incident response items, that needs to get documented so that if you are not here, anyone can pop open your procedure and do what you do in your absence. So that guy, while he is really good at what he does, he has documented all the things that he does. All his systems, his diagrams, all that are publicly accessible, publicly, so if he is not here and we need to restart a system or we need to go in and change something, we have got the information we need to do that, but the real solution to that problem is not to have a single point of failure also. If you only have one highly specialized person on a team, then you have got a choke point. Ideally, you would have two, but I, like the talk covers, I am doing security on a budget so I have got to make do with the single engineer. So I compensate with those procedures for continuity purposes. How much time spent documenting? This is going to be a stab in the dark. I would say probably about 10 to 15 percent of their time because I do not say, hey, stop work and go write documents. What I say is, while you are working, keep your documents up to date because the best way, the best time to be documenting something is while you are doing it. So maybe you are just adding a few extra minutes to the task because while you are doing it, you are jotting down notes over here about what you are doing. So I would not imagine it is a ton of time. Yeah, no, but the return on investment is significant though. It is significant. Sir. He is asking, in my portfolio of open source solutions, is there an area where a commercial product would be recommended? So, yes, I would say there is always going to be areas that it is going to be better to put a commercial tool and not necessarily because it is better, it is a better tool or it is faster or more next-gen or whatever. But for instance, for an agency my size, like I said, I have got over 10,000 users and we are spread across 460 remote locations. So I have a very big network that is literally the size of Texas. So what it boils down to is I need a very reliable infrastructure. So if I am looking at perimeter firewalls, IDS, IPS and things like that, I am dealing with 40 gig throughput in the core. I am dealing with 10 gig throughput at the perimeter. I need to have things that can handle that sort of traffic without issue, without hiccup. And then if there is an issue or hiccup, I need to have a vendor that I can be hanging from the ceiling until he gets it right. Because it is just too critical. It is too critical for our infrastructure, right? Would I be happy to put an open source tool on that perimeter? Absolutely. But at 4 o'clock in the morning on a Sunday, I would much rather call brand X to come in and fix this. So, yes, there is always a time and place for a great product. Credential management. Are you talking about, say, for my security team or for my entire agency? For the security team, we leverage LastPass. Phenomenal tool. They leverage a perfect forward secrecy. All the encryption is done client side. So you are not sharing your secrets with LastPass. They are simply holding them for you. We had started off on a local, like key pass type thing where it is reading, writing from a flat file, sitting on a network share. It is okay from a security perspective. It is terrible from a collaboration perspective, because what happens when two people are writing to it at the same time? The best tool that I found, there are others. Don't get me wrong. There are others. Dashlane is one of them. But the best tool that I found for the job is LastPass. They do a hell of a job, especially for teams, so that teams can share highly sensitive information without trusting your stuff to a third party. It is pretty good stuff. Absolutely. So, my DevOps engineer is not only maintaining the applications that we use internally, but he is also maintaining the underlying infrastructure. So, yes, the ESXi, the route switch, everything. Everything in that environment is under his... Now, luckily, though, being a cybersecurity team, we've got in-house network engineers. We've got in-house, you know, sysadmins, or previous, former sysadmins. So, that's one of the cool things about a security team that you guys absolutely need to tap into, is that almost all security people were previously something else in IT. They came from sysadmin. They came from network engineering. They came from development. So, leverage those tools. Don't just look at your people and be like, oh, you used to be a developer. That's cool. Well, now you're this and you're just this, so do that, you know? No, tap into that. Hey, it used to be a developer, right? Can you help us automate this thing that takes us all day long to do? Because that's what you did. You're a developer. You're still a developer. So, leverage those tools. Leverage those experiences. You bet. So, he's asking about that newsletter and what sort of information garners the most attention and appreciation from our users. So, what we found is, obviously... So, our security team, we see a wide breadth of things. All kinds of things. Things that may not really impact the everyday user, right? Like, really unique threats that no one's gonna see or notice. So, obviously, you wanna write about things that are applicable to everyone. So, write about the things like the IRS tax scams, right? That's something that everyone needs to worry about. Grandma needs to worry about it. Everyone needs to worry about that because it's indiscriminate. It applies to all of us. And so, things like that, that are gonna have higher impact are the things that are gonna affect your audience. Don't write about, you know, the latest zero day in Adobe because they don't care. They don't care. You know, as far as they know, as long as they're updating and patching, they're fine, right? Which, to an extent, is true. But, write about things that, look, no amount of updates or patches are gonna stop you from getting an IRS scam email. You need to know what that looks like. You need to know that that's a real problem. So, that's the kind of stuff that we tend to write about. Ma'am. Awesome. Awesome. Okay. You got it. Absolutely. So, her question is essentially, how do you make time for your team to train? If training is so important, right? Well, I stand by what I say. Training is important. Therefore, it's a function of your job. Therefore, you can do it at your job. You must be, you must enable your people to spend time training without requiring them to do it on their own time. Now, I will caveat that with, you will not succeed in this industry if you are not dedicated and motivated enough to train in your own time. If you're a nine to fiver, and that's it, and you go home and you just leave work at work and you don't care, okay, well, you're just gonna float and kind of coast right where you are for the rest of your career. That's fine if that's what you're into. But, I will say that I, specifically, I make time. I let my people do training at work. I got a couple guys do an OSCP right now. I bet if anyone here's done OSCP, when did you do it? You did it at nights and weekends, you know, many of them, right? No, if I'm, if I'm asking you, or if I'm enabling you to take OSCP, I'm gonna give you time during your day to do it within reason. So, enable your people to get training. That's critical. Half my team's here at DEF CON right now. And has been. So, you know, taking time away from work for training. What kind of validation environment? Okay, great question, right? Because always, always, always validate your tools, especially in security, especially in forensics. You know, don't just, you know, get clone that repository without, you know, giving it a once over, making sure that it's got a reputation. So, how we validate our stuff, usually it boils down to this. Between those of us on the team that have experience with a particular tool or platform, we're personally vouching for it. So, the Hive, for instance, I had been working with it for some time before I even brought it up to our team. Hey guys, let's use the Hive. I had already looked at the code. I had actually already pushed a commit to the repository. So, I was pretty familiar with the code base. I said, okay, this is a trustworthy, it's a stable platform. You know, so there has to be validation. I'll be honest with you, I don't have a structured validation plan aside from just, hey, what's the reputation of the repo? You know, who are the developers? You know, how frequent are the commits? One thing to definitely look for is how frequently do they push code to this repository? If it hasn't been touched in years, you need to be aware that it may have issues and that they're not going to get fixed. So, how fresh is the project? You know, how frequently updated is a big one for sure? Sir. So, if you had a small team, what would maybe the top tools be to implement? Okay, I imagine you're starting with nothing from a security perspective? Okay, if you're starting with nothing, well, you probably have some things like maybe a firewall and some other things. Okay, the very first thing, the most critical thing that you need to have is a sim of some sort because you have all these systems already, right? Just in a native Windows environment, you've got domain controllers that are generating millions of logs. You've got firewalls generating millions of logs. Maybe you have an email appliance generating millions of logs. If you don't have a central pane of glass to look at all that stuff, you're not going to be able to find anything anywhere. So, the first thing, log aggregation. Have all of your things sending logs to a central place. That's priority number one. Gray log, elk. Phenomenal tools, extremely powerful. Okay, and then for sim, OSIM, Alien Vault OSIM, it is a completely free sim platform. They hold back like two features that you would otherwise have to pay for and you wouldn't even know they were gone, those two features. I mean, 147 capabilities to that you'd have to pay for that you've already replaced with gray log effectively. So, that would be my top two right there. Log aggregation and sim because you're also with OSIM, you're getting vulnerability scanning, passive asset detection to learn about all those things in your network you don't know exist. All kinds of cool tools. The list goes on. If you want to know more about it, meet with me afterwards. I'll tell you some more. Wrapping up. Okay, that's all I got, guys. Thank you.