 We'll try. And we are live and on time. So this is a vlog Thursday. And we're going to talk, I have a couple of guests. This is number 257. I have a couple of guests here, Jason. And I want to call you, Jason, the Cookie Monster Slagle, because that's what it says in Slack. I don't know why, but we're fine with that. And Ray, who's been on a channel before, but I needed some subject matter experts for this. And these guys are always fun to talk to. We have a lot of group discussions on things. And, boy, there was a long thread on a burnout. And it is in direct relation to the whole log4j and all the other vulnerabilities. And, you know, let me know where to start with it. But Jason is a fellow MSP. Just anyone who's wanting to connect with him, I've left link to his company. I've left link to his LinkedIn, same with Ray. Ray is kind of a former MSP, trying to be a fully former MSP. I think we describe it in Ray Run's OIT VoIP. And definitely we're all people involved in this industry, but we want to just have a group discussion, just about this topic. So I do a lot of videos on getting people started in IT. We have my homelab show. It's all in, and I engage with so many people that, hey, I want to get a career in IT. We're going to maybe discourage a couple of you. Maybe, at least we're going to be realistic here about it. So that's, I think, a really important aspect is that it can be a little bit of a stressful job. And I did put a disclaimer in the description. We are not at all mental health professionals. We're all tech professionals. If you feel more stress over this topic or goes beyond the scope of this, there's a few links you'll have to go down below to deal directly with mental health resources because, you know, it's tough for some people in here. And, you know, if you feel like more than just rage at quitting your job, something more beyond that, then there's some links down below. We're just going to, like I say, we're going to keep it focused on tech. And yes, someone has already commented that there's some beards in here, and Jason's definitely the winner on the beards. So. Yeah, it's a, yeah, it just kind of happened. This is No Shave November 2020 edition. Yeah, it's just, it's kind of like how the pandemic started and never ended. It's, you know, Shave November one day, and then here we all are. Like, that's the amazing part. Like, he has No Shave November 2020. This is me like working on it for like three years at least. So, Jason sneezes and he's like, Papa Smurf. He's like, bam. Yeah. Yeah, that's a, I don't know if I'm jealous or not because I like how little care it takes. I just kind of quit shaving a few months ago too. And this is far as it grows. It doesn't go any further. So I've not actually really done much with it. I'd spend trim to once. I trim it to keep it out of my mouth when I trim or it doesn't grow in fully. And this is like just the default, right? This is, if I don't, if I don't touch it for a year other than one trim, this is what it looks like. So it needs trimmed. I got to go get it trimmed. Every new CVE that's 9.0 plus Jason's beard goes under an inch. There we go. There we go. That's a good one. Go ahead though. Jason, let's start with you in the intro. Tell people a little bit of your background and how you got to where you're at. Yeah. And so, I mean, we're a MSP in Slido Highland area, but I mean, I've got, it's been like a long and winding road to get here. We were in ISP. I left, kind of spent some time in Secure, or in Linux, Unix administration for a couple of enterprises. Ended up at a startup, finally came back and we just kind of pivoted a couple of years ago from being more like a boutique consulting firm to trying to, you know, chase that recurring revenue and moving more to an MSP model. So I've been doing security. I was actually laughing. I've got up there on that shelf, the fifth edition of the CISSP book. That's how long I've been like, I should go get that. It's been like at least 10 years that I should go get my CISSP. So I mean, I've been doing this for a bit. Cool. So you're also, I know a Linux enthusiast and you're also like being hang out with some of the Huntress folks and jump on cybersecurity issues and discuss them and maybe write some code here in here and patch a few things. So well-rounded in that space. Yeah, I've been, I've worn almost every hat and IT at this point. So from San Administrator to Linux guy, I've never been a Windows admin. You're not missing much. No. I started as a Windows admin and went to Linux and I'm happy there. Like if people ask, how do you run MSP if you're not a Windows admin? I'd say it's really easy. I hired some smart people like this. I mean, I can muddle my way through some active directory and I understand a lot of the concepts, but man, it's just, once you really get deep into Linux, it's a happier place, I think. So yeah. Yeah, I've been using Log4J, you know. I've been using Linux since before Windows 95 came out. I started with Slack where 1.1 when I was like 14 years old. So it was like kernel version, like 0.99.3 or something like that. So it's been a bit. I failed Slack where I tried, I never got into it. I always, it was never. Then Red Hat, I think it was, how many is it? Five floppy disks for Red Hat, the first one that came out. So that's where I got my start. But what about you, Ray? Were you a Linux? Because you've been a lot of network engineering background, obviously, in voice systems, but what else? So I begrudgingly went into Linux, typical Windows admin network guy. So, but you know, can't count that for anything. A ton of network ops. But because I did a lot of on-prem PBXs, VoIPBXs, Elastic Asterisks, all that stuff, I had to get really comfortable with like Debian, Sentos, you know, the rail derivatives, you know. And then today we run tons of Ubuntu. So that's my happy place. Now more comfortable there than I am in Windows most of the day. Yeah. And you still keep your network engineering skills sharp, you know, managing VoIP systems and stuff like that too. So I mean, you did a couple of videos you can find on my channel about the attacks on VoIP systems and what had to be done to mitigate that. That's a fun, complicated topic. And I imagine those people, I think the debrief on it finally, people pretty much were up 24-7 sorting that out, trying to make voices, trying to make phones ring. And that was the craziness because like these aren't, these were major enterprises. I mean, we were talking about like some of the names that were backing these people up were had significant resources and absolutely had DR plans and everything go, and it still took them by storm. And it wasn't even like, you know, and that's like one of the things I wanted to talk about with like on the tech burnouts, I was like, you have to be comfortable with the fact that no matter what you do, it's gonna hit the fan. I mean, you can plan for the best, but you have to understand even despite your best plans, some stuff is gonna go sideways. And if you haven't been burnt out between the Kaseya and Heartbleed and the voice-meaning, which is what we're calling all the voice attacks and log4j and all the other 15,000 things this year, you haven't been paying attention. Yeah, and we're seeing it now, the cascading failures of it went wrong at Amazon. And it was kind of crazy reading the debrief. I mean, we've now built systems that are so scalable. It's crazy, but it's also when something goes wrong, the compounding effects like a ripple goes across the tsunami shutting down servers. I mean, we had two AWS outages within what, within about seven days of each other, roughly. There was another one. I'm blaming Jason for that because Jason says all year long, friends don't let friends put them go on AWS East. And a week later, AWS West goes down. Yes, that is true. And it's funny because we're in US West 2, like all the resources I run are in US West 2 and it's been very reliable. And then the network at US West 2 just dies for, that was at least a pretty short outage. Yeah. Yeah, it's certainly, it all is leading up to the discussion of today is the hair on fire things on that. I'll let Jason lead on that. Cause he has a video he posted on his channel talking about this as well. Just go ahead and what's your thoughts on the hairs on fire problem we have sometimes in cybersecurity? I mean, it's difficult, right? So there are certainly instances we have where the hair should be on fire, right? I feel like log4j is actually honestly one of them, right? Like this is an incident that I was trying to think of when the last one of those came out that was like that, probably shell shock, right? Like we can go back and that was what, 2014, right? So we're talking six, seven years that we have a vulnerability that's in like a core piece of functionality that is accidentally included in a bunch of other things, right? So we'll call it core service. And it very much warranted a lot of the attention we gave it, right? And from security people as a guy that kind of plays as I was joking with somebody the other day as a guy that kind of sits in this weird world where like I do security stuff and or an MSP but I also speak to a lot of other MSPs via vendors and stuff and because we're an MSP I have to speak to small media businesses, right? It's really hard kind of to balance that because the security vendors and the security people are for sure always gonna be up in arms when there's a huge security issue, right? And I think part of the problem we have right now is the MSP community is watching the security experts and when the security experts get all wound up the MSPs are getting all wound up and that's probably not the reaction that you wanna take because not every single vulnerability is earth shattering to an MSP and the one that came out yesterday the what, 2021, 44, 790, the Apache one, right? Yeah, Apache Lua, Apache Lua. Yeah, we're sitting there and everyone's getting all worked up. I see that tired meme of the train hitting the bus and pops up and it's like, oh, it wasn't you? Oh my God, here we go again and I look at it and I'm like, yeah, this is a CVSS 98, right? But this is a not event, right? And it's like the last thing we need is MSPs right before Christmas to run around with their hair on fire going, oh my God, everywhere I run Apache I have to patch it. I'm like, this is, yeah, we have to figure out a happy balance there or it's... My favorite is Tom sent me that same meme three times and I think it had like, it had a log4j 214215216 Yeah, over and over again. Yeah, it is the log4j, I think it's legit. The other one in, at first I see, I seen and this is me overreacting to, I wanna see a bleeping computer because they said vulnerability 9.8 found in Apache. By the way, it's Lua. Like there's this little, you gotta read the detail. And by the way, Lua is not popular. Even me, I think I had asked him one of the Slack forums like, is there even, I mean, there's lots of things written in Lua, but is there any functionality that's really popular where Lua has been turned on? It's a mod for Apache that's disabled by default. And it wasn't enabled in any of the servers we have. I'm like, Lua too though. Yeah. Apache, yeah. Right. And that's the trick, right? Like, what do you do? And I think that's the problem with this log4j stuff, right? If it was, like, if it was look for this package, look for this software, look for this thing, but we're talking about stuff buried inside other softwares, right? It's not just, does it have Apache? It's now, does Apache have Lua mod? It's not, do you have Java? It's just, do you have Java? And is it using log4j and is it the affected version if you can find it? Yeah. The scary part. Yeah, and because it specifically had to be Apache with mod enabled, these circumstances had to be met before you could even potentially exploit it. So if all those circumstances are met, yes, it's a 9.8, but by default, none of those circumstances are met. And so it really was kind of a non-issue. We can go to sleep on it, so just to an extent. Well, CVSS, actually it controls for that, right? And so there's a couple of different parts of the CVSS score, and one of them is like the environmental score, and that is intended to talk about how widespread and environmentally this is. And no one ever posts that, they always post the base score. The base score is always the sky is falling. Yeah, and I think, I don't know if we got it, if you were able to get to the analysis and we had a... Yeah, I saw them getting posted. Yeah, there's a couple of things where you can look at some of the, what's trending on there. And I haven't really had time to play with it to come up with, are the scores averaging up over time, over years, or things like that. But it's definitely, once everything's an emergency, nothing's an emergency, it's also what we're almost getting to. Yeah. And that's a scary part, right? Like it gets to the point where like the MSPs, and that's where the burnout happens, that you don't even know what to be scared. Because, and that's what I've always tried to like put on my technicians and my support teams, whether it was MSP, whether it was VoIP or whatever. But it was, when you're talking to the client, you can't talk to them like the world is going to end. If they say, I have this problem, you're like, oh my gosh, that's gotta be horrible. Like they're gonna freak out. And I'll give Jason, I gave him a lot of hard time this past week. And I was like, when the security, I did. I said, this is your, well, to be fair, I also blamed John Hammond and Wes Spencer, which none of them deserved it either. But the point was like, appropriately, the security community is addressing and bringing up these things and you have the media going nuts. And then the MSPs are like the client in that doctor relationship, where the MSPs are not the ones that are the experts on what's going on. You know what I mean? So they don't know what's going on. And what was the response have, the response you always get from a security researcher when something happens and you say, well, should I be scared is, it depends. It depends. It depends. Because it does depend. It doesn't help anybody. And that's why we've been talking about actionable steps instead, right? Instead of saying, if the doctor says, oh, you're screwed, you know what I mean? Or we don't know, we'll see. That's one thing. If they say, okay, here's the regimen we're gonna go after. Here's the things you're gonna look at. And I'm not putting it all in the security researcher. This is planning MSPs can do. It prevents the tech burnout. Because otherwise, if everything is, it depends, you don't know how to resolve it. To your SOL, absolutely. But I harken back to a lawyer, right? If you just casually ask a lawyer or something, they're gonna, everything they say, maybe they'll tell you something, but they're going to preference it with this is not legal advice. And if you talk to a doctor outside of the doctor realm, they're gonna tell you, this is not medical advice, right? Some point or another, there's professional liability that we're attempting to, if I tell you it's not a big deal and you get popped by it, what's my liability look like there, right? So I think that's where some of the, it depends comes from. So can I give you, so my counterpoint to that and keep in mind, all three of us are good friends. It's not, you know what I mean? But the three of us are all public figures, right? We all have a lot of stuff that we put out online and I completely get where you're coming from, Jason. My problem is that if you're not comfortable giving advice, don't give advice online. Don't get in front of a camera and say, this is what's going on. If you can't qualify it with response, it's just adding, and I'm not saying you because you were one of the heroes, you know what I mean? All heroes don't wear capes. Some have giant beards. You were one of the heroes along with Huntress that go out and put out information and help build the scanners and the scanners that were so nice that some other major corporations copied it with that attribution. But like you were, you know what I mean? So it's, I'm not trying to get my trends out. Yeah, you know, but can you imagine if like the answer on the scanners was like, it depends, you know what I mean? Yeah, so some of it from my perspective comes down to like which audience, for me in particular, and I can't speak to the general industry, for me, it depends on what audience I'm talking to, right? Because the things I'm gonna say to a security audience are different than the things I'm gonna say to an MSP audience, which are different than the things I'm gonna say to an SMB, right? So maybe we need to be better, those of us that play in all three what realms. And I think the people you mentioned there of all three of us for sure are playing all of those realms, right? Like it's maybe, maybe John and Wes aren't directly talking to SMBs, right? But we both, or all three of us play in the world where we're talking to security researchers and we're talking to MSPs. So maybe we need to be better about qualifying who the audience is. Because from a security- Some of my videos too, it's like the, who is this for is, I've said that a couple of times and I probably should say it all the time, especially on the security top, it's not probably not a bad idea. Who is this for? Who is the target audience for the information I'm about to spew? Cause I think that does, like you said, it's the, it adds a lot of context to it because I would never tell my clients, my clients don't really know what log for J is and nor should they care. Are you patching whatever thing is allegedly on fire time? Cool, we're happy. Yeah. And that's what I, you know, one of the things I love hearing, like it depends, it's kind of like my soapbox thing, right? Trigger. And you're absolutely right. I have a couple of soapboxes that I may have talking to soapbox manufacturers, but like one of the things I talked to, like when you talk to a sales person, you're like, you know, oh, well, give me this and we're working on a process as IT professionals. How many times have we said, okay, well, let's walk through your process and the client says, well, it depends. And we put our engineer hat on and we say, okay, well, let's, let's whiteboard this out. And if you can break it down into individual pieces with actionable steps, that's the engineer's job. Right? And that's what we're supposed to do. And that's why I say, I don't, like I think that when you say qualify the question, right? Like, yes, the audience has to be, you know, as a security researcher and I'm not a security expert. Absolutely. I'm a network engineer with the CEO hat on. But so when I listen to these guys talk, I have my pen and paper and a lot of times I go back and Jason was the first guy when, you know, we got the announcement from the ISO about log 4J. Jason was the first guy with the minutes. I'm like, can we, can you evaluate my systems real quick, please? You know, and he did me the favor of logging in and we had an announcement going out within an hour. But as the engineer side, we can approach these things and say, okay, well, we need to have a software assessment. We need to know what we have, what versions we have. And this is all in your RMM or whatever you're using for monitoring. These are the software packages we have. What can we live without for a week? What can we live without for a day? What, if there's a vulnerability in this, do we have to leave it on because we'll take the Sony approach and leave it in payout, whatever we got to pay out after the fact. Believing it on is less critical than being without business. And the MSP that's not a security expert can do that. So that when there is a breach, you can say, okay, these are the category of, we're good, leave it alone. These are the category of, we got to shut down. These are in the category of, we can afford to nuke and pave. And these are the category of, we'll see what happens. You know what I mean? And that is, that's ubiquitous. That is an absolute need to say applies to everybody. Well, after you beat me, I actually did give like a checklist. Like here is the flow chart I would follow, right? I will say this generally out to an MSP audience, right? Like when something like this comes out, right? Like for step one, is it internet accessible, right? So if you have a vulnerability that can be hit and triggered via the internet, you need to patch or shut down immediately because you have hours, if not days, right? Like days or hours, right? Thing number two, can it be triggered externally, accidentally, right? So let's say you consume data from somewhere else and you import it like in a batch process. If you can trigger it via that, right? You need to stop doing that or patch before you can do it, right? Number three is regulated industry, right? Because the risk you're exposed to is a function of how likely you are to get popped and how much it's gonna cost if you are, right? In a regulated industry, it's gonna cost a lot even if the likelihoods no, so you patch those. And then after that, it's like, ah, you know, I can probably catch it in detection and I can wait for the vendor patch and live without it, right? So that would be the general advice I would give in all cases here. And that's completely reasonable, right? Like Jonathan's asking, you know, with big alerts like Logport J, or actually I'm reading the wrong one, he's saying, you know, how do you balance urgency without wanting to rush, waiting to have a measured response? And the answer is I think what, you know, what Jason just said, you know what I mean? It's, you know, balance the stuff out. I think priority one is in all MSPs or IT professionals, whatever should do this, have the person that's gonna be in charge of when a vulnerability gets announced or some security incident gets announced or any item of interest comes across your table. This is the person that's gonna go review first. They're gonna go look at the real write-up. They're gonna go look at exposure. They're gonna go look at what the details are and determine a game plan for scoping it out. And then you can determine a game plan for going against your clients. And like you said, because they always go with the base score, right? They don't go against, you know. Yeah, it's a 9.8. It's not remotely exploitable, but it's a 9.8. And it only affects people with Tomagachi watches like, you know, whatever. I feel singled out. You know, yes, I'm old, but, you know, but that kind of stuff, you can do that in advance and there's gonna be this stuff where you say, hey, we have to shut down. And that's part of your BDR strategy anyway, is how long can we be down? How long can we, does it take to get back up? How far back of data can we lose, right? RPRTO. RPRTO, yeah. These are standard things, right? And it's the same thing here. Yeah, I think a lot of it, in here you say the words, but you should have threat modeling. You build all your threat models, you get these understanding on this. We were talking about this a little bit earlier, but you just have to have an understanding and hopefully not the day log for JKM out before, but you should have something in writing and play these out through your head. Play them out if you had this. And you know, could you live without this software for one day, two days? What is the risk factor? What is it like to restore? This is something you should always be going, taking some time, walking through your DR plan. And making sure you've thought about it, because it's really hard if you haven't thought about it all when an incident happens. It's hard enough when you have what you think is a great plan, it's still kind of nervous going through it, going, all right, that server literally bursts into flames that worst case scenario is here and how do we walk it back? How do we start restoring it? And you hopefully have got some of that thought process ahead of time that you've gone through. There's plenty of templates out there, lots of so many free resources. Let's Google it a little bit. You'll find to start, I'm sure I remember the name, there's a couple sites I remember to have, I think it's, is it part of CISA? They have some templates you can download that walk you through it. There's, I think SANS has a few of them, there's so many. Yeah, I mean, there's a bunch, a bunch of other MSP resources too that are available for IR plans and stuff like that. And table top, you popped it up there. I think that threat modeling was Jason's response that actually set me up on the edge for the project. 100% was. I said, dude, you got a threat model and you're like, that doesn't help me. I don't know what that means. No, I do know what it means, but like, you know what I'm saying? But if you're an MSP, you should know what that means, right? Like that's, I don't know. If you're an SMB, I'll forgive it, but if you're an, in 2020, in 2021, in 2022, if you're an MSP and you don't understand security enough that you don't know what the word threat model means, I might suggest you're in the wrong industry. I'm just gonna say it. I don't disagree. But there's different levels of operational maturity, business maturity for MSPs. It's very easy to say, like, that's the whole fire of the client because they don't know what the hell they're doing kind of thing. As MSPs, we've taken, I should really stop saying we, but I still consider myself an MSP. As MSPs, we've taken the onus of protecting the client. If you've taken their revenue, you have a responsibility to do it. As experts, and again, the experts to get in front of the mic in front of the camera, put out stuff on LinkedIn, put out stuff on Medium, they're doing stuff out there. If you're gonna educate, you gotta educate. It can't be I'm only gonna educate the people that understand me because that's not education, right? If we're gonna, and Huntress does an awesome job, and you've done enough of these with Huntress, right? With amazing jobs with Huntress doing those. Huntress has table tops out. Jonathan has table tops out with Ninja 1, Ninja RMM if you go back in Google. We did the Jurassic Sox stuff. We did the Rogue One stuff. The information's out there, and yes, MSPs have to be educated, but I think each of those has a place where we do it in different fashions so that people can consume it at the level they're ready to consume it at. You know what I'm saying? Yeah, yeah, so I'll take that. I mean, but it's not like you can't figure out if I come onto a webinar and I tell you that you need to do threat modeling and you don't know what threat modeling needs in your MSP, it's one Google search away. So you wanna hear a funny story about that. So you guys know I'm, you know, and I'm sure the audience knows I'm one of the mods on Reddit on RMSP have been for a very long time. Somebody posted today, I'm not gonna call him out, but somebody posted today said he has a couple hundred clients, he has 25 to 30 remote desktop clients because everybody's working from home now and they're using RDP to get there. He did have it secured with Duo, Duo MFA, but he said, he or she, I don't know, they said they wanted a way to make it more secure. They understand RDP is inherently insecure. They wanted to know if there was a remote desktop gateway type thing, which anybody has done this for 30 seconds knows you Google remote desktop gateway and that's the thing. You know what I mean? It was like, it was in the post and Kelvin to our, you know, Everett Cyberdrain Lime Networks. Another great person to follow if you wanna educate yourself. He wrote this really nice post explaining things and I went back to Kelvin. I'm like, wow, because my hair was on fire, trying to respond in the nicest way possible and I was struggling because it's like, you have the answer, you just gotta Google what you said. And Kelvin was nice enough to put in those words. And I get it, I've been on the other side where you're like, that's such an obvious thing. But you know, at the same time, we took that job of getting in front of the camera. We gotta be comfortable with explaining it. Those are the ones where I would post to let me Google that for you, link in some Facebook group and I'd get ripped up for it. So. It's, that exists for a reason. Yeah, it does. And let me just tell you, I mean, you both know my wife. Sending your wife will let me Google that for you, Link. It's not a bright idea. Oh, I know all about that. Yeah. I've sent a few of those to her too. She's never pleased. Yeah, I mean, but that's kind of the thing, right? Like we gotta, you know, and that's why I wanted to have this conversation because Jason and Tom are usually the guys that are saying, you know, this is going out like Tommy did a log for Jay. He did a couple log for Jay videos, right? Jason, you know, it was funny, Jason yesterday. Oh, I should probably post something. We were talking about making a video and Jason posted something, you know, right away. And I thought that was really awesome. But yeah, I mean, scoping it out, right? Like one of the things we do in this year, we started like quarter four of this year is, you know, everybody knows me because I have my documentation and I'm the documentation guy. I have thousands of documents in our KB. But we started scoping them now where this is the intended audience. And I think Jason hit the nail on the head when we talk and say, this is this, this is this. Maybe saying, this is the intended audience. If this is below your knowledge or your skill set, reach out to your IT professional or reach out to these resources, right? Like, you know, Tom put the link, the back doors and breaches link there. We talked about the other tabletops. I think we can do a better job at doing that. So MSPs aren't just seeing a bunch of links and saying, and I, you know, internal IT, corporate IT aren't seeing these things and saying, oh, damn, especially with Christmas Eve coming up, you know? Yeah. Yeah. And I tried to be reasonably clear on the log for Jay. For, you know, day one, we knew it was bad. It was so being triage should really determine. We, we seen the score, but is it really this bad? Oh, man, it was, but it's also that is more the exception than the rule because the next one, we have to go back quite a few years before we mentioned something like shell shock. And I, what was it? The other ones was it poodle? How was that? Yeah. So poodle. Yeah. Yeah. Poodle was just asked to sell my time. Yeah. Poodle or shell shock and Heartbleed had amazing names and amazing logos and log for Jay, log for Shell. Like, I think we could do better. That paint, that paint logo was amazing. I mean, Gossy did it. Gossy the dog did it, man. That's my favorite version of the logo of that one. But I, I tried to keep it brief because there was from my audience and something I've talked a lot about is Unify. And I actually was in talking directly to the people at Unify on their security team. And they right away knew it was a problem because of the way they processed it. So they were within 24 hours, they, well, I think about 12 hours, they had a patch out, which is great. And I made sure I did a video on that. They had another patch that was a follow up, but no one, it took almost two days before someone actually could weaponize it. They haven't just closed how, but someone did figure out a way to completely specifically attack a Unify controller based on public ports being open. I could, yeah, I could 100% do it. Yeah, we, yeah, Jason could definitely do it as well. I seen some of Jason's work. So it wasn't weaponized on day one, but it was important to have it patched on day one because Jason by day two of not sleeping in untold amounts of caffeine, I'm sure Jason's like, oh, I can do this. Yeah. I mean, it's one of the, it's, so, you know, this came out and it's like, at first we didn't realize it was limited to Log4J2, which I mean, this is like the one time in history where you're going to be able to say, hey, I'm using End of Life software and it saved me. A lot of these packages are still using like Log4J1, right? And so I'm running around going, oh yeah. So VMware, yeah, VCSA is totally Java, right? It's almost 100%, right? Basically you can assume if it's Java, there's like a 75% chance of choosing Log4J, right? Cause you're using Log4J, LogBack, or Apache Commons Logging, right? It's going to be one of the three. And I think the thing that limited some of the exposure here was that a lot of places are still using Log4J1, too. Yeah. Well, would you follow Tom, security through what? Security through antiquity. My friend works for a pretty big place. And he says, we're so safe because of security through antiquity. I'm like, aren't you open to everything else? He goes, yeah, but no one seems to be exploiting these whole things. Yeah. I mean, I run free BSD at home. But we're protected from Log4J. There you go. Yeah. I mean, we have code, we have code that's still one too. Like I, you know, I have some unique perspective on it because I am a Java dev and I employed two Java devs, right? So it's like, this is well connected to me. We have code using Log4J, so we had to fix code and we, and we're going through it. I'm like, oh man, we're still using Log4J1, too. I didn't even know it was end of life until this whole thing came about. But which brings a whole like, we have vulnerability scanning for the dependencies we use, but those vulnerability scanners don't flag end of life. Yeah. So do you remember I told you, I said something to you because we were talking about people that wanted to get into dev, right? Like anybody that got into IT two years ago, they wanted to get into programming. Now everybody that gets into IT wants to get into security. Maybe not this quarter or this year, but traditionally they want to get into security. And we were talking about like first languages use and stuff like that. And my daughter went to, she started computer science when she first entered college. And I said, why do they always teach Java? Like, what do I care about my application running on my Windows PC and my toaster? Who cares? And Jason said, oh, because this and this and teaches you this and this and this. And now the Java is everywhere. I got to worry about my CD player having Log4J. So, you know, I'm just saying, we don't really need this stuff. The best is Gidra. The thing that the threat researchers are using to look at malware is vulnerable to Log4J. Yeah. You can literally craft malware that lets you take over the security researcher's PC. And that's pretty meta. And I'm a gray log user. We trade logs stacked on top of Elastic, which of course is vulnerable to it. So now my log ingestion platform can be my undoing. Your tax service. Yeah. Was it connect-wise control on Linux as Java-based? Is it not? It is. But it is not. That was a big concern. If you do a patch of commas logging, yeah. Right. But we didn't know that until you started deep diving into it. So when this first happened, this was now, Kaseya level event because the agents that were out there, this is a, oh, do we need to call everybody to shut stuff off? Or do we need to make it out on site visits? Well, then I started calling that out and got told by the vendor to stop. Well, that vendor has told me a thing or two, too. So. I love that. I love you, Drew. Thanks, NSA, for leaking such a great tool. This is a good tool. It's just some of the things they jumped out there. It's become quite popular. So. Jason Kyle saying from, and I was wearing my MSP geek hat all day long today, but he was saying anyone who uses Java should be retired. I don't know if he's trying to tell you something specifically or. Again, I feel singled out. Yeah, and I'll admit this is a misconception and I see Gracie here posted about, you know, to keep your taxes as low as possible, avoid using applications. Well, we didn't know that before and we don't know what's next. We don't know what a egregious unsanitized code is hanging out there that's going to be the next one. It's not, anything looks great in hindsight is 2020. And I've seen a lot of people, you know, throw that out there, but any of these, you look at something like shell shock, not a Java problem. And there's always going to be something. We always need more code auditing. We need more people. That's would be a better answer. A burnout, a burnout like answer to this is that like, essentially this comes down to third party risk management and supply chain problems, right? Like, you know, to take this into a language that people aren't going to make fun of, let's talk about JavaScript, right? Like if you do install react type react up to make yourself a website and you've pulled in 400 node modules, 400. Any one of those node modules could have a vulnerability and as a matter of fact, in 2019 or 2017, somebody found an abandoned one on GitHub, re-registered it and released a version of it that had specific code targeting one crypto wallet to steal cryptocurrency, right? Like, and this is an event, it's like a eighth party dependency to like react, right? So how do you control for that, right? It's like, you wanna talk about, well, I avoid software that uses log4j. It's like, okay, do you avoid spring? I mean, if you're doing Java, are you avoiding spring? Because half the spring stuff is bringing in log4j. And that was the thing, right? Like the thing that's been caught up now for the last couple of weeks is people talking about Sbomb. And I would imagine many of the people watching this are like, Sbomb, what the hell is Sbomb? You know, software bill of materials, which is exactly that. It's if you have, you've built that application, JavaScript or Node or whatever, having that list of packages with the list of versions, which is there. I mean, if you design the software, you have it. But how many vendors, application vendors say, this is what we have. And how many MSPs are qualified to say, start looking at not only the main application version number, but now all their sub modules and version number and all that. I mean, that is a monumental effort on its own. Yeah, I'm still worried for it because I think it helps get us an inventory. At least, you know, does this product even use the thing that's in question? Cause if it doesn't use it great, I can just scratch it off the list quickly. You can just leave it. Yeah, it doesn't even, it's not built on Java. Great, just not even, just throw that aside and don't burn it. But I'll go back to what I said earlier. Gray log is going to tell you that you use an Elasticsearch. Right, they're gonna tell you that Elasticsearch uses this and this uses that. I mean, I have a 13,000 line Java program that we maintain for one of our clients, right? This is one of the smaller components of this. It brings in a hundred libraries, right? Like, how do you- On the vendor side, I'll push for the vendor side. When this first came out, my second message after messaging Jason was, hey vendors, you guys need to put out a message. You guys need to like make the MSPs because all the vendors that I'm friends with, we all serve the MSP and the IT community. We need to go put out a, we are or we are not very, very quickly. Not we're gonna look into it and 16 days later we give you something. We need to give you that peace of mind. And for the record on the S-Bomb thing, I'm a hundred percent for it. But at a macro level, your RMMs can tell you what applications you have installed into and can tell you what application. At bare minimum, you should know the main applications and what OS versions and that stuff you have installed. So you can at least look at that and say, when you have that, if it's a software that I can live without it, okay, fine, you find out that from the vendor, it may be, you know, whatever, or, you know, like in Tom's case, right? And a lot of us use great, great log. But, you know, then it's a, do I shut down great log? Do I keep it going? You know what I mean? But have that decision made in advance. So you don't have to do it while you're panicking because when if you do all your panicking, you're gonna make a mistake. And how many of these decisions you're gonna have to make across your clients, right? Most MSPs have 10 clients, 20 clients, 50 clients. You know, if you're one of those, like per desktop, you may have a thousand end points you're monitoring. If you have to make a thousand decisions, you are gonna be burnt out. You're gonna have litter damage and be emptying bottles of bourbon. Not saying I'm against that, but I'm just saying. You should be doing it because you want to, not because it's the only way to cope with the security problems. But SysControl 2.1 is the thing. I'll mention too, because we, one of the other big things that did happen here in 2021, it feels like forever ago, but it was the dependency confusion. That was a really interesting problem with our supply chain that was brought up where we found out that we could publicly find namespaces that were internally named. And the pecking order was, look externally for this particular library, find me the newest version externally. And if it doesn't exist, I'll pull the internal one. That basic order of pulling in libraries was really a crazy attack vector. I thought a really interesting write-up from earlier this year. That whole supply chain thing, right? I mean, but you have that with Orion was the same similar issue, right? It was like, they got into, what was it, an intern uploaded or left a password open or something like that. There's been so many this year. It's just nuts. That was 2020, man. They're all blending together. They're all blending, but that's the thing too. It didn't come out, SolarWinds Orion with this issue. It was SolarWinds. And back then and Central Enable was also part of SolarWinds, right? MSP manager or whatever it was was also part of SolarWinds. So now it's all, everything SolarWinds and plus every free tool you've ever downloaded to do an ARP scan or a packet capture or looking at your NetFlow or whatever. And so all my SolarWinds stuff is broken and that leads to tech burnout too. I mean, it's frustrating. I don't think you really work in tech unless you contemplate some non-technical job at least once a week. Like I could be a farmer or anything. It's a frozen banana stand, man. Right. There's always money in the banana stand. I mean, one of my buddies grows avocados in California in addition to owning an MSP BJ. And it's like, every day he's like, I could just go back to the avocados. I don't need this. I get it. I absolutely get it. Well, I'm the first one to tell you, like you could not drag me kicking and screaming back into being an MSP. There's weight and I'm not saying you shouldn't be. I'm saying I completely get it and my level of neuroses with my understanding of security, I'd have like most MSPs are like four of the five people are techs. I'd have like nine of the 10 people are securities people and then maybe one billing person. And that would be it. You know what I mean? It's tough, man. Literally my hat's off to MSPs everywhere. Yeah, it takes a, the intrigue keeps drawing me into it. It's, you see this with anyone who's really in cybersecurity. I mean, how many hours straight were you up, Jason, working on how to completely weaponize a log for J? I mean, you were, I see you post like, I won't sleep and I'm like, Jason's still posting stuff. Yeah. So, I mean, so the Friday, I ruled out a bet at like 730 went, oh, like started seeing some things. I'm trying to remember where I saw it first. Started digging through things was like, oh crap. We worked with Huntress. And so full disclosure, Caleb's, the Caleb did 90% of the work on that tool. I had an initial POC that he used and provided some peer programming guidance when he got stuck, but it was almost entirely his work. And it was John's idea. John had been up all night working on it and had gone to sleep. We were up, we finally got the tool working and released at 1226 a.m. Saturday morning. And that was up until three something, maybe 330 working on it again and then it was back up at seven the next morning, right? So that was an entire week. I mean, I was at a conference earlier in the week. That whole week was like less than six hours of sleep every night and it still continues, right? It's like, I get, my brain won't shut off, right? It's like, we've got other, here's another way we could do it, right? I've got, I'm sitting on a, I think other people have done it now. So I'll talk about it. I'm sitting on a payload where I can full take over Java without ever spawning a sub-shell or writing to the file system, right? So that was a terrifying part. It's watching John, right? John Hammond posts these mine scrap, minecraft pictures. Like, oh, I got this and I got this. And then, you know, seeing Caleb post stuff and then seeing Jason. And every time like I walked away and I came back, I see Jason, oh, it's even easier than I thought. I can run it completely in memory. I don't have to install anything, which means it's not going to get caught. And I'm like, stop it. You're making it worse. You know, to the vendors out there, if you send an email to me or somebody I know saying that you can be your whitelisting, ring fencing, bullshit marketing term, say, oh, sorry, I just tried and asked for on this. No, you come on back to your topic, bullshit's a good word for it. It's gonna stop this. Then I'm going, I'm triggered, right? You talk about your soapbox, right? I am going to as many, I will, until my death, I will sit there hacking at something to prove that it's possible and that you're here for it. And I appreciate you for that because that, because that stuff looks alluring, right? Like, and that's, that really comes down to also like the MSPs, I consider myself security savvy. I'm smart enough to know I'm not a security, like I'm not a security expert, right? I have smarter people than me do that. And I surround myself by a lot of smarter people than me like you guys. And so, but like I sit there and I look at this stuff and when they say this tool would have stopped everything, that is very alluring to MSPs. And MSPs love the shiny stuff, right? Like I've never met an IT person ever that's like, you know, you talk to 10 MSPs or 10 IT people and none of those conversations were about tools, right? Oh, what AV do you use? What EDR do you use? What MDR, what OPP, when IT, what whatever, what do you use? And it's all, it's all tool talk. So when somebody says, oh, we could have prevented it. Ah, like, you know what I mean, that's scary. And it's almost a marketing play. I've always told, you know, I've had this from a, even all the way down to the SMB when we're meeting with a client and they're like, well, the other guy that's given us a proposal said he guarantees nothing will happen. I'm like, just walk him out the door. I'm never gonna tell you that lie. I won't put that in writing. Well, he said, I said, good luck. I said, yeah, go with them. If that's how you fill up with it. I said, I can't in good conscience just write, like, I guarantee you'll never be breached if you use our tool stack. Go tell him you're gonna drop your cyber insurance then because he's guaranteeing you're not gonna get breached and watch how quick he walks that statement back. Oh, no, no, keep that insurance. But like, that's the thing too, right? Like, and I think MSPs could relieve themselves of some of the burden. You talk to a security researcher, they'll tell you the same thing. There's no way to prevent it, right? I think one of the best analogies I saw which one of the slacks was, you know, it's like saying you can prevent a hurricane. You can't, you can prepare for a hurricane, but you can't prevent it. MSPs, I think generally understand that, right? I've never talked to a Corp IT. I've never talked to an IT consultant or an MSP that doesn't understand, you can't prevent it. You can prepare for it. How many MSPs or IT professionals have talked to their clients and had that conversation? Yeah, you're talking to a different group of MSPs than I am because there are a lot of MSPs out there that are saying they can prevent it and truly believe it. You know, I wish I could see that. Over tell ourselves, people, man. I've heard, I can't believe I'm competing against some of them sometimes. When they're like, well, the other guy said, I'm like, why would he say that? It's not based in reality. But that's an easy sell too. Okay, they're saying it'll never happen. They'll prevent 100%. Are they putting in the contract? Like they're going to cover your costs. They're going to cover your insurance premiums. And if something goes sideways, put your money in. It's a limited liability look like, well, Pete, no one reads the contract, right? So it's like, they'll tell you whatever they have to do to get you to sell. And the contract limits their liability to basically a month's worth of service. And yeah. That's forced measure right there. It's like, I don't know what to do. Cue South Park terms and conditions. I've agreed to a lot of things. I'm not sure why, but I feel one day it's all going to come back at me. Yeah. You know, it's a man. And like that's the thing like, and there are ways to make this better. I mean, I can honestly say, when I say, I surround myself by people smarter than me. I'm not kidding. I'm an MSP geek and I'm in, you know, IT pool party and MRU and, you know, a couple of different groups with these two gentlemen, you know, and so, and the hundreds of people. And so when stuff happens and obviously Reddit and when stuff happens, I do use that because honestly other MSPs, other IT professionals are very quick to share. It's not all going to be nuggets of wisdom. I'm telling you right now, much of it will be a dumpster fire. Where's your dumpster fire thing, Jason? You know, a lot of it, but you can generally start filtering out what's important with not. When the logport J-HAP stuff happened, we immediately jumped on. I started a thread, said start posting here and inviting vendors to say something. And people were saying, okay, well vCenters affected by it. And, you know, elastic search is affected by it. This is not, and this is not. And that central repository made it so much easier to go back and look, you know, surround yourself by those learning resources, you know? And it can make it easier for you guys. Yeah, I threw the link up here for MSP geek and I'll also mention like Reddit, our MSP, have any public discussions because that way when one person says something, you usually find just like we talked about earlier, some corrective replies, not just let me Google that for you replies, but some actual, so you can see the back and forth discussion to come to a common understanding of a best practice on a topic. So this community community is really something I think helps with the burnout one. You're not alone. All of us are dealing with this at some levels. Turner Nightmare was called Turner Nightmare because it didn't just affect one person. It's a recurring dream. It's a recurring dream. Okay, so going back to that, you bring up, you brought up a good point. Being surrounded by others that have also felt your pain, right? Like when I worked at the police department, like the people that worked at PD with me, you had a certain understanding of the job that was much, that was different from anybody else outside. You knew, you went through these shared experiences in the MSP groups. It's the exact same thing where, there was somebody talking yesterday or there was a post yesterday or this morning or when I saw it, where he was frustrated, they were frustrated because their client kept asking for immediate quotes and then refused to sign or kept disappearing whenever the quote needed to be signed so they could get started. Everything was, hurry up and wait. And that's frustrating. We've all been through that. That is super frustrating. And that person came back after a bunch of us gave advice and it was like, they were like, actually I think he did identify as he, he goes, yeah, thank you so much. I appreciate it. He goes, I was just frustrated at the time I needed an event. Thanks for listening to me. And he goes, I already know what I'm gonna do. So, which I'm assuming is fire the client, change your RMM. But, you know, but like, the fact that people to get you, right? Like that's an important thing. So I wanna take advantage and call out while we're here, you know, since we are the experts in how to handle this stuff. I guess I wanna start with Tom. I'm taking over Tom's show. I want to go ahead and then we're gonna go with the bamboozled beat. No, I'm kidding. What should we call it? No, Tom, first, can you describe a time where you were so frustrated you were seriously considering exiting IT? Like what led up to that? And how you handled it? When I worked for other places, well, many times the corporate BS of working enterprise IT is very real and was very painful. I have a handful of friends that work at very even Fortune 500 companies. And they wanna quit sometimes because of the bureaucracy on top of, the pressure for bureaucracy on top of that. So I've had a couple of those times more so than when I worked for myself. And I think in one of the reasons I end up working for myself is so I can constantly be defining the parameters. And I've always had some good contracts and done well and had a diversity of clients that if a client I think is, that's usually where it comes down to, will taking on this client cause me grief? Or sometimes I don't know that until after I took them on. And I even have a video on my channel that's very popular called Why I Fire Client sometimes. And it just becomes a not worth it type thing of, I'm not starving to death. I'm not going to, it is not that I lose this client who may be paying me good money, but at some point my sanity, cause I don't wanna be that person that's trying to rage quit my own job, especially when I'm the owner of the company. And I try to really compartmentalize it. And matter of fact, we just turned down a very large bid because I realized some of the requirements in there were like, it was so great. Everything was so good looking with it, but dealing with the people, I said, the money I'll make on this is wonderful, but I'm not gonna take it on. It's not worth the stress. It will be something that I know I'm gonna feel terrible about. And so yeah, it definitely happens for a time to time. I do my best to kind of compartmentalize it and just not take on something I think is going to kill me stressfully. That's been my, one of the takeaways from running a business for 18 years that where I'm at, it makes me so much more comfortable. I'll take less money for less stress. It's a better trade. I'm not trying to chase the most money over it. That's why like YouTube so much, it's just easier. I just gotta talk about it. I'll be doing anything. What about you, Jay? I don't know. I don't know that I'm legit considered leaving IT. I mean, you've done teaching, right? You are now, right? The three of us are business owners. Like you are now officially the president of your company. Congratulations on that. What are we doing today? So yeah. Yeah, but you're gonna tell me you've never been so frustrated. You're like, you're almost at that level of like, I'm out. Well, I mean, I've been frustrated enough with IT companies I've worked for that I've been out, right? Like I've definitely quit for like, this just isn't worth the stress reasons or, but I mean, much like Tom, we, I don't, for me, it's all about risk these days, right? It's like, I'm letting clients go because I don't want to deal with the risk. And I don't want to deal with the fact that like you don't want to spend any money to prevent something from happening. And you want me to drop everything and run spend 24 hours a day on it when something does happen. It's like, I'm not the fire department. Like I go, go, go find somebody else that wants to put up with a headache. I mean, I've, I've often, and you know, obviously you and I have had this conversation a non-zero amount of time. I mean, I've given serious thoughts to whether or not I want to be an MSP just because of the stress that comes with that life, the stress of owning the risk of all of my clients. But I don't know, I love IT. I don't, I can't see myself exiting IT. I can see myself redefining what I do in it. See, you guys kind of stole my thunder. So you suck because you're intelligent enough to have figured this out. You know, when I started the MSP, I'm fortunate enough that this is my seventh business. And I, so I know what I like, and I grew up with entrepreneurs. So I like, I had a lot of lessons I had figured out for me before I even started making mistakes myself. But when I started the MSP side of business, I learned early on there were clients, I mean, within six months, there were clients I was just not gonna do business with. I would rather not have the revenue than have the revenue and have the headaches that went along with it. And I've said that over and over. And that wasn't like, I started the business and I had 10 employees. That was me. And then I hired my first employee at month six. But it was, I learned that when I was smaller, I had less time. And I could not devote 100% of my time to a needy client that didn't understand and want to get to the shared goals I did. I'm not saying I haven't been frustrated over the years. Especially when the Casayas stuff happens or not that I was using Casayas and nothing against them. But, or like the Voip DDoS stuff right where they took out bandwidth for a couple of weeks, that's absolutely frustrating. And especially it's more frustrating when you can do nothing. Like that stuff is mind blowing. But, you know, you learn to manage the stress and that keeps you at a general happy place, right? I'll share when I was, yeah, exactly. If it clients who revenue you can't lose, grow enough to be able to lose it. I used to get scared when I had that big MSP client that was 20 grand a month, 30 grand a month or whatever it was. And I was like, I need like 10 more clients to make up so that if they have to leave, I'm okay with it. I don't want to have somebody be able to put me against the corner and be thinking, yeah, absolutely. Like I don't want somebody other than me making the decisions for my MSP. I don't want the client making my decisions. So, you know, you'd make those little decisions and it makes your life much easier where you can breathe. And I know I can already hear it. I can hear you guys, oh, but if I'm just starting out, I can't afford to not take the business. I'm telling you, you can. You have less freedom. I can stop what I'm doing, go fly my drone, go play some call of duty and then get back to work because I have a full staff of people that are doing business, you know, taking care of things for me. When I was a one person, two person, three person shop, I was doing everything. And then I really didn't have time, much less for a needy client that was keeping me from going to make sales and going to other stuff. Your time is more valuable when you're smaller. So, you know, when you grow, then you can dig around like, you know, Tom and Jason and I do. Stay on the square all day and get on you too. I hope that's a joke or Richard is attempting to trick me there with that statement. Rich, I love that. You don't understand my market. My clients are different. That's the running joke, right? Yeah, and I'll address this right here. I'm a firm believer. I believe player customers are raising rates until they stop being dumb or calling. It doesn't really work like that. Matter of fact, my employees at one time, I remember they were like, Tom, we will all chip in a percentage of our paycheck to never hear this person's voice again. And I'm like, that's a good sign you should probably fire that client. I listened to a lot to my staff in their interactions, but at some point, it's not even about raising rates. The fact that they can call you and that you'll even talk to them as opposed to saying, sorry, I am just not the right fit for you because there's a satisfaction because it almost is this constant abusiveness. And we had a couple of clients. I'm like, I don't know how, I don't know who deals with them now. God help them, but we're just happy they're finally not telling the lies. Like I used to, when I was at the police department, I really enjoyed my job at the police department. I worked with really good people. I had really, you know, I had lots of great friends that worked there, but I was miserable because of the bureaucracy. I was miserable because in any service-based business and especially public service, when stuff goes sideways, you're working. There's no, that's how it works, right? If there's a hurricane, you're doing 24-hour shifts or if not more, you know? And so there was a lot of stuff that I was miserable that when I wanted to spend time with my family or whatever I couldn't do. And I say this to all the people when I interview them in a hiring, I would say, you know, I worked and I would get up, I worked midnight by choice, I would get up at night and I'd have 10 reasons to not go to work and I'd have to come up with a reason to go to work because of all the other stuff that's stuff pressing down. So when I started this company and the MSP, the same thing, I said, I'm never gonna make this business, the business where you're going to work is gonna be more stress on you. That's not gonna, nobody's ever gonna be like, yay, I can go to work now. You know, everybody has their own stuff going on in life, but I wanted to make it so that we weren't the additional pressure on you. You know, we were a safe place to come work. So one of the things I instituted very early on when I was at MSP was I would let MSP, I would let my staff nominate clients if they wanted to, they wanted us to fire the client or there was a red flag client. Not that we would fire them immediately, we'd bring it up at a staff meeting, we'd discuss the pros and cons, we'd weigh the revenue, we'd have to replace the revenue, we'd actually look at the data, look at the tickets, is there a problem person or is it a recurring thing, have we tried to address it? We would look at it logically, but if we came to a consensus, that client's gotta go, that client went. It's staff first 100% of the time and that made the staff much more comfortable knowing that I was always gonna look out for them. That wasn't a question. Yeah, I mean, I've had clients that are like verbally, I lost a client in a very public fashion. I had, we shared a Slack channel with them. They were verbally abusive to my guys. And finally, I told the person there at that company like publicly, look, I'm not gonna let you be in like an asshole, sorry, no, I'm sorry, in a butthole to my clients, to my employees. I publicly said that and he fired us and you know what, life went on in the stress of all those people that hated working with the guy and the unreasonable expectations and all the other things that came with it, they went away. And it was a couple thousand a month lost revenue, but in the end, that lost revenue, it wasn't worth losing the employees that he was verbally abusing for. No, I mean, you have, Steven Spalding said it perfectly. People don't leave because of work, they leave because of management. I mean, you have, IT is already hard to begin with, right? We're already a service business. They're already calling you when something bad's going on, even if you're doing QBRs and do VCIO and all the other fun stuff, but like they're calling you when something's broken. That's already, you're starting off on a negative relationship, even if they're a cool client. So stuff happens, but if they're gonna start berating you and they're gonna start being a problem person, I'm not saying fire everybody, that's not my thing either. You can talk to them. There's certainly clients where I've had a person, one of their staff members where I've said to management like, look, this person's verbally abusive. I've asked them to stop. I need to limit their exposure. Either they go tickets through somebody else or they only communicate by email, but I'm going to have to, I have to make the decision either they tone it down and we figure out a workaround or I have to let you guys go as a whole and I don't want you to, but I will. And most management have always been receptive to that to most of my clients. And they'll realize, if you have a good relationship, they don't want to lose you either. It's not like you're not putting them against them. All you're saying, let's work together. And if they say no, go on, there's other clients. There's seven billion people on the planet. There's hundreds of thousands, there's millions of small businesses in the US alone. There's other clients out there. Don't sweat it. Yeah. And it's a lot of just that triaging back and forth. We had a client that the owner was the one we had to avoid and even the managers would kind of joke about it. We loved dealing with the manager. Like, even if the company's nice, I'm like, how does this guy employ so many nice people? Cause he is unpleasant. And they're like, yeah, he's kind of unpleasant here too. We like when he leaves and I'm like, well, we don't like when he calls because he would just go and like you said, these weird tirades of calling people's names and stuff. And I'm like, we're just gonna, I told myself to just hang up on him when he calls that way. And then he'll, the weirdest apology ever gave when we threw it in a fireman was he goes, well, I was kicked in the head when I was a kid with a horse. And I'm like, what does it have to do with anything? Why are you calling my employees by name? Like it's cussing at them. I'm just like, I didn't even know how to react to it other than to tell them, look, you can't call anymore. That's my answer. You either your manager calls or you just lose our number or we'll fire you as a client. I don't know. But I don't even know how to respond to an apology saying, I was kicked in the head by a horse when I was a kid. That was the only response, not like context beyond that. I love Matthew Fox's most nerdy geeky response to a bad client. No, but the Apache mod, you know, A2M mod just replaced their words with like nice words and problem solved, you know, or Jason's swearing. I'm usually the guy swearing, replace Jason swearing with nice words. I'm trying. I'm doing my best, but it's hard. I mean, can somebody build a Lua mod so we can, you know, replace bad speech? Oh wait, not Lua, my bad. Maybe Java? Maybe Java, there you go. We could pipe it through log4j and create actions on it. What could possibly go wrong? Man, substitutions. Substitutions. That's a great idea. We should be able to substitute words in our logs. You remember that hacker that was a patching, like open, I forget if it was open WRT or, he was patching firewalls that had some kind of RCE on it. Yes. You remember that? And he patched a bunch like, where's that hacker when we need him, right? Could he start like using like log4j to patch itself? He posted, he had a Twitter account and he posted some anonymous stuff. He was calling himself like the internet sanitizer or something weird. People got mad at him. Like if you had a few people pissed off that he fixed there, he would have created a firmware. There you go. It was called the Brickerbot because he was bricking things with firmware. He was, any time he found out it was firmware, he wrote brickers for him that would just brick the firmware. He's like, they're never going to fix it. So I am. Yeah, there's, I've seen it. So there you go. Streamyard doesn't have a seven second. It does have a delay. It's different per platform you send it to. It doesn't have the beep though. Oh no. It doesn't have a dumb button. Yeah. I'd be bad about pressing it. So. Yeah. I'd be messing with people all the time. Like I'm the last person you want that to happen. How long did you stay up to? I don't think we established it in the beginning. Oh no. I'm good till four, three 34. I know it's three 30 now, but. Whatever. Yeah. I'm off today. So I'm just hanging out. You got a promotion and you're taking time off. Lazy, man. Lazy. I'm going to finish Christmas. I have a six and a nine year old. So did you take that opportunity to explain Java and log for J and explain anything or no? No, but my kids like Minecraft. So the fact that it was a Minecraft exploit. So John Hammond's YouTube thumbnails are interesting enough that I think my kids would watch his videos just because how cool his thumbnails are. Yeah. Speaking of thumbnails, I saw you, you put my OIT cookies on one of your thumbnails for your blog last week. Those cookies were so good. Did you send cookies again? Somebody sent cookies to the office and they weren't like named who they came from. No, no, this had like OIT on the cover. Yeah. It's in here. Hold on me, grab it. It's right here. I know we sent to you, Jason. They're probably somewhere. Somebody else sent us cookies. Yeah, I know those cookies. Those are good. These are. Do you know who didn't get those cookies? Well, if they show up, I'll ship them back to you. There you go. Along with the Toa of Satan. The medium we sent to Matt over and over again. Yeah. Along with the Toa of Satan. Yeah, the Toa of Satan. I'm not ready for that again. I know there's about 100 people in the live stream. People have some cybersecurity questions and stuff like that. Shoot away in there. We don't mind answering a few questions and things like that too, so. Yeah, so I love John Hammond, but I can get sucked down the deep radical. So that's my thing. I'll watch John and I'll watch some of his stuff. And I try, like I tried doing the Avenue Code. I did a few days or whatever. It is so far above my head. So it's like he says something and then it's pause and then I'm Googling for 35 minutes. And then I come back. And I don't think I've ever actually gotten through a whole video of his, you know? Oh, the Avenue Code stuff. I have the opposite problem. It's boring. Well, when are we gonna see your live streams, you know, hacking stuff? And I've begged the Huntress guys and Jason too. He's like Huntress and Jason. You know, I've begged these guys like open your, you know, your incident responses live stream. I mean, you have to filter some stuff. Of course, but like to see that inside process, like that fish tank. Well, see, I've gotten invited to it a couple of times, right? So I have seen some of it. So it's fun. It's, I mean, you know, burnout wise, that stuff doesn't burn me out, right? It's the, I get burnout on the business problems and the crappy clients, the cybersecurity stuff. I mean, if it happens all the time, I'll probably eventually burn out. But that's not what causes burnout to me. That's a good question. There you go. That's a really good question. Too late to learn? No. No. There's a lot of people, I think that start out late in some of the cybersecurity stuff. And I don't think that's an issue at all. Starting out late in there, there's a lot of people pick it up much later in life. The craziest thing is one of my friends, he worked in a factory, like automotive, different automotive level jobs all the way up until he was about mid thirties. And he's now my friend who's managing 158,000 endpoints, manager, Sim and Sock team for a very large company. Like it's crazy. He's just never thought of himself as a computer guy. And he jumped into cybersecurity side of it, did some college courses, got some intern jobs at a small place locally and now worked for a big nationwide company. And he says, I don't know, just kept clicking with me as he went up and he loves his career and everything. So I would say, like I said, I think he said he was 36 or 37 when he started and his entire previous barely any computer pretty much just worked in automotive. And that's it, nothing technical background. He didn't switch from one technical side to another. So I don't think it's ever too late to do that. Be prepared to hustle, right? Like, you know, if you want to do it, you don't change because you think it's a good, easy way to earn money. Cause if you're going to be successful, a lot of your educational learning is going to come outside of your nine to five job. Yep. You have to have a lot of passion for it. And he is, at least if there is a qualifier, I will say about him, he's a little bit intense and it is something that lends yourself to intensity is that all the extra reading and things like that. If you're just looking for money, I've told people to work in finance or something. That's just money and things like that. From like 20, my eight year old says he wants to become an anesthesiologist cause they get paid a lot of money. They don't do a lot of work. Yeah, my wife is a pharmacist and it's just a job, right? She's not super passionate about it. She said if she had to do it again, she'd go be an audiologist because it's the same amount of schooling, the same amount of pay, but they work like they work office hours and never have nights and weekends. Yeah. Now here I see here, and I don't know if you've had the chance to look at this, Jason. Are you feeling with how crowds are? I am not. So it's really slick. I have a friend who's dove into it. I set it up and played with it some. It's basically specifically for firewalling things like maybe WordPress or more public facing infrastructure on Linux. They have a lot of plugins and it's all open source and it's all open source threat intelligence done in a really interesting way. And I haven't had a chance to really dive in with it. I'm looking at it. Yeah. Is it plugged into like PF sense or something? It plugs into anything. They got a long list of things it plugs into. Like directly you would load it. Picture it working very similar to, I was at a blocking tool for too many logins. Login. Brute blocker. Yeah. It's like that. Yeah. Okay. It's like one of those, but really interesting on the threat intelligence side. And the founders are someone who used to run, I think over in Europe, a very large hosting infrastructure. And they started building it there. And he said, why don't we open source all this and have more people crowdsource this whole concept It's really slick. I think it's a great idea. It's going to come down to, of course, like anything. How can they sustain it? They have a lot of systems in place to keep you from triggering like someone false positive stuff or slamming it. They build up reputations with people and everything else. It seems really cool on the surface. I haven't dove enough into the technical details, but enough people, they have a quite a few people using it. My immediate thought is, how are they going to monetize this? They want to keep it not monetized. They will monetize it by selling, I think their business model is, it's free for everyone, but if you don't want to participate, but it's free as long as you're participating in the model. So you are giving them threat intelligence data as well as taking it. But if you want to be a one-way street where you never give them any intelligence data from filtering, you can buy a feed from the crowdsource. I think that's their plan for monetization. He also has a, from listen, let's do an interview. And I thought it was interesting that he doesn't really want, he's not trying to grow it into this constant, I'm going to go get a bunch of VC funds, run this until I big enough clients and then slap everyone with some hedge buy. They seem to have more than that. They don't want to do that. That's actually the problem they see with most of it. That's why they're building it without investors. One of my first question for MSP tools is what is your exit strategy? Yeah. And they're an open source strategy. They're doing everything open source, but it's a little bit different of a concept there. Interesting. I'll have to look at it. Yeah, it looks like it's like a SAM rate. So it looks like it's collecting log data and then they're essentially processing it. That's cool. Neat. Yeah. I learned something new. Hey, something else to poke at. How do you decide between commercially supported products versus open source? And me and Ray, we're going to do a video sometime and maybe we'll include more people of how to vet the vendors, but being an open source advocate, the same question I'll ask of a closed source company, have they gone through a code audit? This is something that's very, if a product is through a mature life cycle, they're going to go through some type of auditing and this absolutely applies as much to open source as it does to closed source. I think Bitwarden, we've brought up a couple times as the way it should be done because they're so transparent with their code audits and everything they do. They have an entire Wiki dedicated to each company, even like what company audited, the results of the audit, links to the PDF files and everything else. It comes, a lot of it has to do with that. It's not like just because someone says, well, this stuff doesn't have a breach. I'm like, did anyone ever poke at it? Cause sometimes you find out, it's like, it's no one ever looked at the code. It's not that it's here. And when was the last code audit, right? Well, somebody that goes through this stuff, right? Like you will find bugs everywhere, right? So it just, yeah, just no one's looked at it 100%. And audits are no guarantee either. So I have a different perspective on that question. It comes down to who you want to support it, right? Do you need to be able to call the vendor when it breaks, right? So I run Sentos because I'm never gonna call a Red Hat because they're not, the person, the level, I've been using Linux so long that like they don't have anyone that can help me, right? Like if it's to the point where I need vendor support on that thing, then the first eight people I talk to aren't gonna be able to help me. And it's gonna have to be somebody that has access to the source code and yeah, it's probably not gonna do it. But VMWare on the other hand, it's so complex and there's so many moving pieces that 100% of the case, even if they had open source vCenter, I'd probably pay for support on it. Well, so what are you gonna do when Sentos goes end of life? Rocky Linux. Yep. And especially on the hypervisor topic, I'm a big fan of XCPNG, I've talked about on my channel and they're an open source product I like and you can buy support, you can get it for free. And that's one of the other reasons we're choosing them is they have very clearly laid out documented support plans, like here's our plan, here's your number of tickets are covered in this plan or this plan or what are SLA agreements, they've got it very much. Okay, I like the way they did it because you go to the.org to get the open source version and you go to .com and it's the same version but they let you buy the support contract for it. They make it very clear our .com is our business side and we fund it with all this and Oliver Lambert's head of the team on there. And yeah, I like that business model that's available and it's also a factor, I like Greylog. Also, Greylog is open source with a enterprise support package I can buy. So you the homelab user want to play with it, go ahead, I got a whole video on it, go ahead and download it. You the enterprise business user go, I'm ready for this, okay, buy a license. Get the upgraded support, have them handle standing it up for you. Zavix is another one I use, same answer. Zavix, get it for free, buy the support or maybe a lot of times just paying for the consulting to configure it. Zavix has got so many knobs and buttons that it's a great tool but it's a bear to set up because of what it does, it's kind of an analogy for those not familiar with Zavix, it's kind of like an RMM tool but more for Linux. It can do Windows but it's a great monitoring tool. And Riley Chase, I told him he should do some videos on it. He's done a great job of integrating Zavix. By the way, he also hired Zavix for the level of integration that he has. He uses it to auto load balance and restart servers and give statuses of all of his Hostify instances that he has for the Unify hosting. It was a really easy integration because you can create actions items. But there's plenty of open source products and obviously Red Hat as a company is an easy example of you can have Linux or you can get Red Hat because you're not at Jason Slaco's level of understanding Linux and you need that kind of support for patching and understanding and setting up products. I can imagine Jason calls, you know, Jason calls Red Hat and starts arguing with the guy because they're giving the wrong answer. Yeah, or the one time, literally the one time in my last career or the last job I ever called Red Hat support, I got through, I got to the second level. They're like, you need to restore from the backups. We had a corrupted LVM metadata and I totally pulled it off the disk and restored it myself. I'm like, screw this, I'm not restoring this box. That's some work. I am not an LVM expert. I can screw it up. When it works, it's fine. I just back everything up because for a lot of reason that's one of the little things the Linux never clicks on my head well. I can do ZFS all day, but LVM is always a little bit. Oh, I'm the opposite. ZFS is like black magic to me. Oh, I love ZFS. Yeah, I love it. I usually use a free NAS or true NAS when I'm doing that these days. Yeah, building all servers. I think this is always a loaded question. And I actually like the fact that Huntress has addressed this in a very general way that no solution will solve all of your problems. There is no specific endpoint vendor. And we all have our choices we have because we like the product, because we like the dashboard, because we thought it was easy. Maybe the sales guy was nice. He sent me cookies. There's actually, sometimes I'm just saying. I always feel like I don't, I can't always 100% because someone says, why do you like this product? Or the other ones I said, I simply, there's not enough hours today to check every single vendor on a particular one. I just chose this one because once I started using it, I felt like it clicked in my head and it worked. I mean, there's obviously, once you, I'm gonna talk more about the top tier vendors when you talk about endpoint protection. Like I like Huntress from the day I started using it. I say, cool, this product works. Is I actually found it intuitive. I found working with the Huntress team great. And I also met them through their outreach of covering topics and blogging about security. That's what even let me, oh, what are these guys blogging? Oh, that's a really interesting topic. Oh, oh yeah, they make a cybersecurity tool. That's cool. That's kind of the arc of where it got me. And I've stayed with them because it's been effective and it also doesn't drive me nuts with false positives. So what's your thoughts on that guys? I said it in the chat. Oh, okay. I was gonna say, that's the one we should use now. No, I mean, so like going back to like. I'll agree with that. You know, go and put installs on everything. It's like the Java of AV. Yeah. It's like three billion devices, 20 years later. Exactly. You know, but like that's one of the things I like, you know, when I was started, this whole thing of like complaining about security guys that are unclear when they're talking. I think Huntress goes the opposite direction. I think they do an amazing job of saying like, you know, do this, use, we use, we're integrating with Microsoft Defender, but we understand these are the limitations. These are the this, you know, not going and saying you should use this specific tool because no vendor is gonna say that unless they're me and they're stupid. But like, you know, but they're very clear on what limitations are and what you should look for and stuff. And I think, you know, what your qualifier should be, right? And I think when you're an MSP and you're not a security savant, that helps a lot. That helps to know, okay, well, you know, even if Jason or Tom or John or whoever is not saying, I need to use this one tool. They're saying, these are the things I need to look for. And then I can go look for a tool that has those things. And that's more effective in my opinion because you're training them forever. Yeah, integration levels matter too. There's other, there's so many factors that go into it. Cause once you started with one particular stack with your arm and you want things that also integrate and tie in with the other tools you're using. So, so many decision factors. Yeah, I mean, there are tiers, right? Like, you know, at the top, I think you've got Sentinel-1. I think Sentinel-1 is complete. I don't think anything touches it. Honestly, somebody that's done AV bypass, but you pay for that privilege, right? So, you know, then there's like the third tier ones. There's a bunch of them. I won't name them. And they're all roughly equally effective. And defenders, honestly, in that tier three group, right? Like, and then there's like tier two and tier one vendors, right? So it doesn't matter. Just pick one that does well enough. I mean, we're talking a couple percent detection between like tier two and Sentinel-1, right? So is it worth three times as much money to get like an extra 4% detection? Or are you better off spending that extra money on something else that can allow you to get it some other way? Well, you're saying Defender was giving you a hard time when you were trying to prepare for your... Yeah, dude, Defender was hard. It was hard to get by. It's, I mean, like it's something that does exploit development. Defender is like my nemesis right now because man, it's all over everything. And it's, it's deleting. And then I use a Dropbox or some sort of file sync to sync between the PCs I work on. And the PC at home is nuking stuff out of the stuff I'm working on here. And I'm like, oh my God, how do I stop this? And yeah, it's, it's bad. How crazy that the operating system vendor gives you like a complete way to protect the operating system they sell, like, you know, if only they would have done this 40 years ago, you know? It's crazy cause they went from joke to not joke. Like it was, it was the, oh, it's just Defender, you know, the integrated one. And I, and they left that market space open but Microsoft really has come the other way. And I'm hearing that, you know, from a lot of cybersecurity people that, especially when they were trying to deploy something, they think, oh, I got this idea. I'm gonna test the scenario let's have a clean window system. And actually when me and John Hammond working on a demo, you know, he didn't realize this when I set the server and I forget which thing we were trying to exploit. Me and him did a video together and I couldn't exactly what the exploit was but Defender stopped them and it was hysterical that he's like, he's like, this worked the other day but you have Defender enabled now it doesn't work. Yeah, and I can't get it to turn off, right? Like I turn it off and then it turns itself back on mysteriously and yeah, it's. It's more persistent than the stuff Hunter's looks were. It's, yeah. Yeah. Oh, lots of people there. People asking, I don't really do much with this. I see, you know, people asking me like Grafana on some of the self hosted dashboards. I never really get into those. I'm not a big pretty dashboard person. I even talked about when I use Greylog, I have a minimal amount of dashboards. Mostly I'm hunting for something there because I'm like, how many GNDIs do I find in my logs and things like that. So I'm not a big, I don't know if you either when you guys use Grafana or any of those. Yeah, it's cool. It's, it looks pretty. Don't get me wrong. I just, I have Tesla installed and it uses Grafana on the back end. Yeah, I mean we use, I used other commercial products before the MSP. We always had dashboards for everything. The dashboards we use now come out of Salesforce. They're not fantastic, but they do the job. I would love something more complete. It's just like so low on my totem pole for like priorities. Yeah. Yeah, I mean like for us, the same thing, we look at the RMM dashboards. We're looking at the action items on there. I don't bother to try and export it out. It's, you know, look at things. And matter of fact, before you really dive into what EDR should I buy, the bigger question is, what's patched? Is it patched? Like that's, I see people jumping all over. I've even, I think I was, I don't know where it was probably one of our discussions. We had something more along the lines of, I said, really focus on these end points first, because that's where the bad stuff is happening. Cause I've seen people that think the, oh, I'll just buy a firewall that'll magic all of us, right? That is a- And then everybody goes to work from home. Yeah. I mean, we're also worried about the log4j stuff, but like no one's really talking about the, the two windows chain domains or domain admin privilege escalation that is arguably way worse if you don't have the November patches installed. Yep. That's why it's a year of the Linux desktop. It's, you know. Yeah. And it's, and it's quite the aggravation patching. Is it any of a moment, if something's read in our ARM dashboard almost always cause we're waiting on the client because of their work schedule to get something rebooted after you loaded a patch. We, you know, probably in its servers, of course, and some of these escalation things and we're trying to keep on top of those. That's actually where you end up spending a little bit more time as an MSP. And as you should, you should keep these systems up to date. For reasons unknown to other than Microsoft, there's Microsoft does not put the effort, I think they could for the scale and scope of their company into making these patches not require constant reboots. I would just, Oh yeah, they totally could do that. They take the easy way out and say, rather than restart the process, that would allow us to basically write the file, right? Like rather than, rather than stop the service, write the file, start the service, it gets a access denied cause files and use rights into that like do on reboot log and flags that you need to reboot. Like, and that could be 90% of the time unless you're updating the kernel. Like in even that Linux in on redhead, I can update the kernel without rebooting in almost all cases. Yeah. That's kind of why I like Linux so much. It makes patching so much easier. It's, I have on all my Linux stuff, I have all the unattended upgrades, installing everything, I have it auto restarting services as needed. I just set that up. I had some people, aren't you worried cause like my forums and some of the other public things I have facing directly for me. I'm like, aren't you worried about breaking? I'm like, no, I would much rather have it for some reason. Engine X didn't start or my Docker container failed to start. Then, oh, there was an egregious hole in the system. And now Tom's forums are spreading mailware. That's a way worse area. So you can, you don't want to auto update. Like, as a service provider, we have certain uptime requirements. Like there are certain packages on purpose. We hold when we're not auto updating because we package, we update those specific ones during specific maintenance windows. You know what I mean? Like Linux lets you do that. That's not that cool. Yeah, you have a strategy, yeah. Yeah, it's very scriptable. So, yeah, live patching kernels. Who knew we could get there? Solaris did, Sun, Sun Microsystem did 20 years ago. You know, it's really weird to things like ZFS and all the other crazy things that come out of us. I'm like, that was just such a think tank of smart people. That is, yeah. Asked you by that little collection of knowledge. Like we're going to design the file system that somehow you're going to be using 20 something years later. And that's a lot of forethought. Yeah. There's the new systems. It's been, I mean, so you do you run it on Linux? Cause it's like a third party citizen on Linux less so these days, but it's like pretty, I run it on FreeBSDE. It only since they've switched it in all the more over my head legal arguments and debates that went on about that, since they've moved to the two point O version, it's actually being developed pretty much in parallel now. I think on BSD and Linux, that's what's allowing TrueNAS to do their TrueNAS scale. And that's going really well. They're in a release candidate too. I think came out today or yesterday for TrueNAS scale, which is going to be a game changer cause they're pushing a lot of clustering into it. So they're using not Seth, but the other one, Gluster. So you're going to be able to build your Gluster clusters. On top of ZFS. On top of ZFS. Yeah. Gluster is a lot less complex than Seth is. Yeah. So I'm a reseller for 45 drives and they're really big on Seth clusters and things like that. So they've got all kinds of things they've worked on for that. Cause they worked on it in ZFS with Linux combined with, I think they're going to probably switch some things to a bunch of some things to Rocky cause everything was sent to us with them. But they worked with the, what is that called? It's the dashboard, the web interface. They're building web interfaces for it. What's that Red Hat one? It's the Red Hat web management. Oh, satellite, it's Catello. Forming. It satellites the upstream version or the commercialized version of it. Cockpit. That's what it's called. Cockpit, that's it. Yeah. They've read the upstream of that. Yeah. That one is kind of neat, but they're starting to get into all that. I think that's going to be being able to build these type of failable clusters. Like, I like that there's competing cause I don't, we don't need a monoculture. It's funny, but they're both using ZFS underneath but they're clustering 45 drives is mostly staff. But it's going to be interesting cause our new true command product is going to be able to manage and scale out your cluster clusters and give you, you know, redundancy across them for, you know, and it's really targeting these companies cause we slurp up more data than ever before. So now it's going to be. We use a free NAS for most of our beam repositories at the moment. And it's like, I've got, I've got enough systems now that I've considered moving them to Gluster or or stuff on the back end just to get redundancy there. Yeah. It's, I'm excited to start diving into all that as it all comes out. So ZFS is just magic. We, I built that on petabytes server and one of our new clients, I'm excited cause they're expecting to fill it up roughly every six months with a petabyte with their amount of data they're producing. I'm like, awesome. So we're going to be back. So. Is that a, how's that? So now we're completely the topic. It's like we're on tech bar. We're so far off the rails we can't even see them anymore. Oh yeah. Well. How's the performance of that? That's been my one complaint for, for sequential storage that stuff performs well but I've had problems trying to do like random IO on it. It's not bad as long as you start breaking out all the VDevs enough time. So there's 16 drives in the cluster. So we broke it out into six, six separate ones. So we made them all 10 wide for storage efficiency. And that's 10 wide is fine with EFS too. You can probably go up to 12 wide without a problem. But that's where you start. That's how you solve some of those issues on next. That is always the debate. Yeah. I'll do like six wide and then just, and then just stripe them together essentially. Yep. So you can get that, still get some decent random. I just did a whole series of benchmarks on that. It's, it, there's not a lot of people doing as many benchmarks on those scalable systems. Me and Wendell from level one are the only two people that I've seen posted on YouTube. And so it's been kind of, kind of been fun. It was Wendell, he, like me works, we actually ended up working with just some of these very large systems of storage, which is just fascinating. The fact that we can store that much data. Yeah. Of course, where do you back it up to? I have no idea. Well, and then if you're using like 14 turbine drives, it's like one fails and you're like, please, please, please don't die. Yeah. Yeah. And we have them all on a Z2. So you can hopefully resilver without another one dying. But yeah, there's a risk. There's a whole risk. Yeah. Like all my, all our beam stuff, it's like I'm maxing out at eight turbine drives right now and I'm only striping six wide, thoroughly doing RAID, RAID Z1. So, yeah. Oh, is there any other questions people have here for rewind this down? So I think, I know Ray's got to go soon. So we'll talk forever and be off topic forever. But yeah. I don't think we've ever ended at the time we thought we were going to end. No, I mean, I can just drive up and have a beer with you and look at all your cool toys if I want to keep talking. So. Yeah. That too. All right. You're both Tesla buddies actually. Both of you guys drive Teslas. Oh, okay. Yeah. Yeah. So the, there's even a charger here. So if you drive here, you can charge at my office. So. Yeah. We'll do one at the new office. So once you get our building built out. Absolutely. Well, thanks everyone for joining. And while you're here for your listening and posts to even hit the like button, helps the YouTube algorithm know that you like this content. So thanks everyone for joining. And hopefully we talked something about tech burnout in the beginning, wandered off into a few other topics. Hopefully you have a better understanding. You're not too old for cybersecurity. We need more of you in cybersecurity. So we're less stressed. If everyone was in security, like Ray said, if we have nine employees in security and one doing that, we need more of you. So. It would be nine burnt out people and like the one billing person feel fine. Like it would still be. It's only nine people that are here on fire. Yeah, fair enough. Yeah. That's all right though. All right. Thanks everyone. See you next time I've left in the description how to get ahold of these two gentlemen connected with them over on LinkedIn or their companies and learn more about what they do. And thanks. All right. Awesome. Thank you. Thank you for having me on. All right. There's the end button.