 All right, Tom here from Orange Systems, and I'm here with Dave from Huntress, and we're going to talk about how some of these MSPs get breached. I mean, sure, we see it in the news, and we're not here to name drop. We're not here to do anything more than provide some education based on the knowledge that, well, Huntress, you guys have been involved in a lot of these investigations. So, and you probably see a couple common things, am I right? Oh, yeah. Yeah. There's definitely some reoccurring patterns. Yeah. So we want to talk about those reoccurring patterns. They may seem obvious if you work in security. They may seem less obvious if you're living and working in MSP. As a matter of fact, we started to get off topic before we started to hit record because we talked about life as an MSP. There can be some real challenges because of all the alerts and everything else. You can lose sight of these. I narrowed it down to seven common reasons these MSPs get breached. So definitely some challenges in there, but yeah, what do you think, seven's a pretty good top number, right? Yeah. Seven's a good place to start. I mean, it's always chasing the, like we were talking about earlier, there's no, you can't get it to zero risk, right? Right. You can't get it to zero. But you can try to get as close as possible. And I think with some of these tips here, I think it's going to be a good step forward. So these are the seven most common. This is not like, do these and you're fine. These are just the most common reason things get breached. That's the best way I can describe it. So if you're doing all these and then you have more you do, awesome. You're ahead of the curve and you're probably not someone who's in the news or part of these, uh, other side of a debrief when you're, uh, going through a breach and everything that's, and we'll start with the first one here. It's 2020 and are we really still seeing lack of two FAA and privileged accounts? We are, you know, and, and people are tired of hearing everyone tell them they need to FAA. I'm sure they're like, yeah, yeah, I know we got it. But you know, this is, um, by far the most common thing that we see, right? Like it boils down to the human get attacked, right? And a fishing or, you know, something of that nature. And then someone gains control of that account. It's a privileged account and it doesn't have two FAA on it. And that's the most, that means overwhelmingly the most common reason that we see things breached in mass, like, uh, you know, an MSP tool, a backend tool, something of that nature. Um, and it's, it's unfortunate because it's, it's a pretty easy problem to solve these days. Yeah. I think pretty much every major software has it. The challenge might be that some of them, if they're legacy, they haven't forced them to do it. I know like if, even when I sign up, uh, in any of the tooling we use any new user add, it's like, it's a non option. They, you have to turn it on, but I think some people, cause the legacy stuff to like, ah, you know, we just haven't bothered upgrading it yet. I know it says that a reminder that I click okay to, and then I get my work done. So, um, but it's, this is a really simple one. I don't think there's any major player in the MSP space and that has a tooling product that doesn't support it. So it's low hanging fruit and easy to turn on, although we know it's inconvenient that I'm not going to argue them and wants to say that, but it's still there. I seen this one in, uh, Reddit, someone talking about, uh, a debrief on there. So API service account no longer needed, but still active, uh, we tie, you know, the tooling all gets tied together through API calls, but if you eliminate a piece of tooling or eliminate a piece on there, but you still leave this hanging out there. Um, so this one's an interesting one. Yeah. And this comes kind of down to just sort of some best practices of just sort of like auditing accounts, you know, not even just API and service accounts, you know, as well as, as, you know, we'll talk about this in this, in this segment too is, you know, actual people accounts, just making sure that you don't have accounts that are laying around that, um, you know, we're not being used anymore or maybe they were set up a really long time ago. You know, maybe you've gotten integration from something, you know, two years ago, five years ago, whatever, and it still may be set up in a less secure manner. Maybe you go back and update that password to something more secure or something that too. But we've seen a few of, um, um, even kind of recently where there's been some, you know, some account that was created for some service, that service was discontinued or, or, or no longer, or maybe they changed the way that it was done and that old account was still there. And, um, you know, maybe there's some commonalities, like a vendor always required that account to have the same username. So attackers learn that and they can just brute force that one username. They don't have to guess both username and a password. And unfortunately that, that's been a way that we've seen folks get in before. Yeah. I've always liked that, um, you know, Google and the G suite service, they have that reminder as a security health check and you go, did I authorize all these things to talk to my Google account? You know, maybe some of that should be implemented occasionally with some of the other tooling. Yeah, absolutely. Uh, you know, it's, it's, I'm starting to see that in more and more services that I use, you know, I see, um, uh, uh, those kind of like reminders of like security health checks. A lot of social media, uh, kind of does that now. A lot of your, like you said, the G suite account, they do end up with a bunch of the, um, uh, you know, the sort of like, Hey, is what's going on here? Can you, can you kind of do a health check? And, you know, one good thing to do is develop a procedure of like, you know, maybe every quarter or, or every six months, you know, list all the things that need to be checked, you know, go through and just do that as part of your kind of regular security audit. It should help eliminate this as being an issue. That makes sense. And this is that other side of it too. Terminate employee account left active. Yeah. Um, there, we were actually involved in one like this. Um, there was, uh, uh, a guy selling, um, access to an MST on the dark web. And that was kind of what this case was, right? It was a person who was terminated and many months later, that account was still active. And you know, he's on the dark web selling that account for like, I don't know, it was something very inexpensive, like $600 or something. I remember you guys said that right up. The person was, um, just unintelligent about how they were doing it. The fact they were selling access to an MSP for $600 was like, what? Like that's just. Yeah, you know, it's just trying to make a quick buck and, and, and you know, a lot of folks, you know, it's a shockingly low number, right? Cause if I'm an attacker and half the work or maybe more than half the work is done for me for $600, I mean, an MSP is far more valuable to, to me as an attacker than that. So that's a, that's a minimal investment. That's not really an issue at all there. Yeah. Cause you talk to this MSP, we have, you know, so much control over our clients and so much in with many customers. So you pivot, this is why the attacks MSP are so high. We are high value targets because we represent so many targets, one compromise and many end points potentially ran somewhere, not just one company, but all the companies we have access to. So $600, that's a drop in a bucket, man. Oh yeah. I mean, in terms of that, that's, it's a, it's a one to many business strategy as we'll call it there. So that works out really well for them. I've actually liked that they started referring to the ransomware people as cartels. I'm going, that's a good word for it. Yeah. I mean, there's a whole economy and stuff that's running in there and they're pushing that pretty good. So yeah, I like cartel. We can go with that. That works for me. It's fair enough. This one, RMM agent from previous or acquired MSP, we, when we do takeovers, I scrub and look hard because we do not know necessarily what they loaded. Cool. If we find common tools, but I also don't leave it at the assumption that that's what happened is that they only had, you know, one tool on it. I may have more, but this is obviously a pretty big hole that can get left in there. Absolutely. We've seen both of the, you know, there's previous and acquired. He's two separate incidents here and, and, you know, and unfortunately I've seen both, right? Like in, you know, kind of a simple scenario, right? Like you take over a client from an old MSP, didn't get around to removing that tool yet. You know, maybe they didn't audit it correctly. And then that other MSP gets compromised and then your, your now client gets, gets hacked because that old tool was on there, you know, they get ransomed or whatnot. And same exact scenario for an acquisition, a rather large MSP had inquired kind of a smaller one in the region they were trying to break into and they just hadn't, you know, I mean, unfortunately in that case they were in the process of going through all that, but you can understand, you know, acquisitions and whatnot. That's, that's complex. It can take a while to, to merge everything together there. And unfortunately, one of the assets from the acquired MSP was compromised during that phase and ended up causing, you know, all the customers that were part of that acquisition to get, to get ransomed. So it's, it's tough. You know, you got to get those, you got to get those tools that have that level of control out of there. When you, you know, you may have a very well put together security controlled environment, but when you're bringing in some external stuff, you know, you've got to bring that up to your standards just as quickly as possible. I mean, that's just got to be one of your most important things to do because if that, especially if that acquisition or something like that, or, or even, you know, if someone was being, you know, I don't know, less legitimate on you taking a customer away from someone, there's a window of vulnerability there, you know, is a mode of opportunity where they know that they can probably sneak under the radar. So getting those tools out of there as soon as possible is just, it's paramount. Yeah. And I think one of the problems you have when, you know, acquisitions are really common in the MSP space because getting clients is rather difficult. So frequently larger MSPs buy smaller ones to keep getting bigger. They kind of swallow them up. But that also means the smaller MSP. Well, if they have 20 employees, they may not keep all 20 of them because there may be some redundancy. So you have this other attack vector could have been, well, hey, you know what, go ahead and I'll sell those credentials because I just got my job cut. You know, there could be some managers. Yeah, absolutely. So you're helping yourself out for risk. Yeah, there definitely could be some of that, you know, that scorned and bully type thing going on. I haven't been privy to as specific as, beside the one we just talked about a while ago. I haven't seen that being as super common, but it's definitely a window of opportunity. You just want to, you want to slam that thing shut as quickly as possible. Yes. When you take over, get rid of all the old stuff as fast as possible. Seems obvious, but yeah. And this kind of relates to that because not practicing proper lease privilege, and this is rampant in the enterprise because I can't ever figure out why someone in a C-suite has the level of access they do because I bet if you put the word why next to everything that they need access to, that would be enough to stop. And someone in a C-suite just says, well, my title is such that I should have access to firewalls, even though I manage finance, but this the same thing applies to the MSPs as well. Yeah, absolutely. That's the term I used to bounce around for that was executive privilege, right? Like it's because I'm an executive, I need that privilege and I experienced that when I was an IT director. I experienced that when I worked in an MSP and it's tough. And hey, at one point I was guilty of, I mean, we're going to rewind quite a bit, but I have logged in with my account being an admin or my account being an Office 365 administrator. But you can't do that anymore. Whatever your normal account is, your normal account, your fetch and all your email and stuff with, your normal account you log into your domain with, they should not have administrative credentials. You should be escalating to that externally. It should be a very well thought out process of like, I need to log in as this other account to do this action. That way, if for some reason you do manage to fall victim to phishing or something or your credentials are harvested, they probably don't get right into that privileged account. So and then don't share the password between your between your unprivileged and your privileged account as well. Yeah, I've seen that too where they this while just use the same password, but I'll put admin underscore and then my name on the admin one. But for convenience, I'll just. Yeah, absolutely. It's you got to kind of do it in the in the inconvenient method, right, for the security. But yeah, the least privilege is a that's just a really common thing. We see that in environments where oh, everyone's a local admin. Well, you know, it's another issue that happens too. Yeah, and even here, you know, as an MSP we take as we hire people in. I don't just throw them to the wind and have access to everything they have access to things as they're trained on it because there's also the oops factor giving the new guy high levels of privileges. Well, he could oops something out of the MSP world as well. So you're protecting yourself from that and it protects yourself from the just the fact that they as they're new. They do they really need it here reducing your threat surface as much as possible by that. Does the sales guy need access to the RMM tool or does he need a read only level of access just to see who's in there. Yeah, absolutely. And that's what happened with the one we talked about with the terminated employee that that employee was not employed by that MSP for a very long time. I think it was only a few months and that, you know, like it seemed it seemed as if really early in his tenure there, he had a crazy amount of access. I mean, full blown access to to create accounts in the like cloud hosting utilities and stuff. And that that just seemed a little yeah, that's out of protocol. Yeah, you know, that's that's probably a good point. I mean, you want you want employees as I hired them to be useful as fast as possible. But also you got to mitigate risk and that's a big one. The same with I've seen and talked to other people that have their sales people in there or, you know, we're like, why does your sales guy have access to RMM? We just like to look things up. I'm like, there's probably better ways or read only ways to do that. Sales people are wonderfully social, but they're not always technical. We'll just. Yeah, absolutely. Yeah, no, we've all met them and we know we know and love them, right? But yeah, absolutely, at least privilege is something that's if you don't need to do it, then you you shouldn't have access to do it. And if you only do it every now and again, that should probably be in a separate account that is, you know, that is something you have to deliberately switch over to to perform actions that may be destructive or privileged. Yeah. And I, you know, there's actually a good seminar a while ago that Microsoft gave and they talked about some of the ways they handle things internally, you know, at the Microsoft scale, but it's like they temporarily elevate privileges on your very strict circumstances with verification and then as soon as the task is completed, they de-escalate that privilege back down. I mean, that's an extreme use case, but we are talking about when you start talking with these large MSPs, 10,000, 20,000 endpoints ransoming 20,000 endpoints is a disaster and you've seen some of these happen at scale. Yeah, absolutely. And it's a lot of it can be contributed to at least privilege as well because it's it's usually an account, you know, obviously, you have to have privilege to do those mass actions and things like that, depending on the account. So, you know, protecting that for just kind of reducing the number of people that have that access to the minimum required to efficiently, you know, get your job done. And that's a that's a win right there. Yep. MSP tool with vulnerability left unpatched. I think that a lot of the cloud ones are pretty good about it, but sometimes you still have to actively say update the endpoint agents. And I know people are worried it's going to break something. It's going to have a problem. I think Microsoft, I blame them because we all brace for impact whenever there's Windows updates. So maybe they have this mentality of updates break a lot. But I overall, I want to see the endpoint agent updates on most of tooling we've used have generally gone pretty well. And I also, you know, truly try to keep them up to date. But yeah, this is definitely another one out there. Yeah, this is getting better with a lot of the stuff shifting toward cloud hosting. You know, it's definitely getting it's definitely getting better. As far as, you know, like some of the self-hosted ones or whatnot, you know, I've seen some folks that, you know, we're helping with or whatever and they're using an on-premise version of an RMM or a remote control utility. And, you know, they're on some old version. And it's like, hey, you should probably look into this because somewhere in those patch notes, there are some security vulnerabilities that were patched. You know, this isn't super common, right? Like this is nowhere near as common as hacking. We're going down the list here. Yeah, yeah. It's nowhere near as common as hacking the humans, but it's definitely just good to kind of inventory the tools that you use and figure out how those vendors disseminate information about vulnerabilities. Do they have a security-focused page or they have a mailing list you can sign up for? Like, how do they give that information out? You know, and if you don't know, reach out to your rep for that tool and ask them, how do we get information from you about that sort of stuff and make sure that the right people are doing those. Again, if you kind of roll back to, we talked about executive privilege, you know, if the CEO is getting an email and ignoring it, that's not really helpful, right? Like, so make sure the right people are getting notified if you've got a security person or a lead tech or an engineer that's focused on handling, make sure those tools are up to date. You know, make sure there and then know as quickly as possible. Absolutely. Password re-utilizing for easy lateral movement, I know, I have, there's a couple MSPs we've taken over from and I like, it's kind of cringey, but someone will say, well, I'll get the password from them. Oh, you're using this company? And I'm like, they always use the same password and you're like, yeah. And I'm like, that is, that is such a no-no. Don't, don't do that. Do not use the common, even what they'll do is they'll use the same part of it's their company name, underscore administrator day, which create that account and then use the same password across all their clients. I mean, sure that's easy. Yeah, I'd say, but it makes it super easy to attack and, and you know, this is usually, this is usually stacked on one of the other ways that someone gets in and it just makes the attack so much more devastating and so much more efficient. If you're sharing local admin passwords or domain admin passwords and usernames across the entire client base, then that just, if, you know, if someone gets into one of the clients or if someone gets into one of the tools, it makes it very easy for them to spread because they're going to try that first, right? I mean, that's like, there's already pre, they've got pre-built scripts they're ready to go to start knocking at doors and the credentials that they already have in their hands are the first ones they're going to try. You know, so yeah, it's, it's, it just makes it, it makes them much more efficient attackers when they've already got the credentials. Yeah, it allows them just to run through and just burn through every other client under like, oh cool, all I had was a remote access to what the gave me access to the screen, but it turns out you guys have the same admin password everywhere, so every client gets the R dropper. It just makes it super easy to do and it's, it, you know, part of our, you know, part of our slogan here at Unstress is we're making them earn every inch of their access, you know, and the kind of methodology behind that is don't make it any, any easier, you know? Yeah. Yep. Even if it's just that a little bit more difficult by doing this one or two or three or a little extra things, just don't make it easy and then, you know, let, let the hack, the hackers want quick easy targets, right? They want those, they want to be efficient, they want to knock through it fast. Don't be the low-hanging fruit. I mean, that's, that's kind of the takeaway there. Oh yeah. It's the low-hanging fruit gets picked. I mean, that's as simple as that. Now, that was all we had for the slides. Those simple seven things are the most common ones that you guys are seeing and that's, that's impressive. I mean, I really loved your work. I think John had published it, the hiding in plain sight, the really advanced malware and that's, you know, that's really cool and people think that's what cybersecurity all of it is. But until you get into the field and you're like, oh yeah, these debriefs were no 2FA fish passwords or company name 123 was the password. And yeah, we had a ton of fun with that hiding in plain sight and actually there's two separate blog articles on it because we found that first half and then, then we moved over and found the stuff that was all the cool, like DNS over HTTPS. John and I had a couple of late nights tearing that one apart. And then, but yeah, I mean, that's the cool stuff. That's the you know, that's the like, you know, Mr. Robot, Swordfish, Techno Music and the background hacking kind of stuff. But, but the reality of is that it really just comes down to, you know, just some really baseline security practices that I don't think anything that we said today here is like revolutionary, right? No. You shouldn't be blowing any minds, but it's all very simple things and that's really what it comes down to. Most of the attacks that we have been, you know, involved with, you know, that we've been able to, I don't say involved with, we didn't actually do it, but you know, involved in the kind of cleanup and remediation and assisting our partners and prospects with that. Most of those could have been prevented and that's what this education is about. Prevention starts with education and that's, you know, these should all be pretty simple to go through audit, you know, make these changes and hopefully harden your environment a little bit and just, you know, move yourself up that tree of fruit to not get picked as easily. Yeah. I mean, trust me, if they see someone with 2FA and someone not, they're going to go, I'll go with the 2FA one and the unfortunate thing, I mean, we had a tradecraft Tuesday that I flew out to you guys and did where we kind of think like an attacker, we actually didn't even publish in that video everything that we had found. Now, the table talk that me and Kyle did, we're like, wow, it's like, I was a few steps away of getting a cell phone number of a person MSP like, this guy has RDP exposed. He's got his name on here. I mean, I can't show this in the slide deck. This is, right. Absolutely. You know, there's so much low hanging fruit out there. You just got to close those holes up. Yeah, absolutely. It's just all about, you know, making those, those simple changes, you know, there's small iterative changes, you know, nothing, you know, none of these things should like, make you have to like rethink the way you do business, right? Like, these are all things that are probably options that just need to be turned on, things that need to be audited. I mean, some of it's just general housekeeping stuff. Yeah. You know, it's a lot of, you know, we've seen a lot of attacks attributed to just, you know, bad housekeeping and that's, it kind of hurts a little bit when something is, you know, it's easily preventable, right? Like it's, that's what makes it, um, that's what makes it the worst. You know, hey, if it's, if it's some super elite zero day vulnerability that you get hit with or whatever, there's nothing you could have done about it. Well, you know, fall back to your, your disaster recovery plans, you know, go through those motions, get that done and just, you know, go to bed knowing you did everything you could and it's still happened. But if you're just making it easy for them, then, you know, it's, it's uh, what can we say, right? Like what could we say to make that any better? Yeah, he just, yeah, there's nothing and that's unfortunately what a lot of them are, but hopefully, you're not watching this after you've been breached, you're watching this and go, I at least can do one or two of these things in a seven, hopefully not all seven. If, if you have to do all seven do them, whatever you need to do to get secure. Yeah, yeah. We don't really like seeing any more of these breaches. It'd be better if there was something else to talk about in the news, but yeah, absolutely. That's, that's the, that's the call to action, right? Now take some of these, you know, things start, even if you're like, oh, I know I've done that, but maybe you haven't done a watch, double check it, you know, take some time while you're, while you're sifting through one of your tools and just spot check stuff kind of as you go. And if, yeah, and if you make security sort of like an everyday part of your mindset and your posture of, hey, while I'm in here, let me look at this, let me see what's going on. Then it's not as, it's not as large of a task, as monumental of a task, if you just kind of continually do it as you go throughout your day to day. Yep. All right. Well, if you want to learn more about Huntress Labs, just huntresslabs.com. They're always in, check out their blog, they post all kinds of stuff that's not marketing, that's like these deep dives into technical things, especially to check out that hiding in plain sight. Yeah, definitely check that out. That's, that was a lot of fun, part one and part two. And of course, everyone knows what to find me, there's links below. I'll leave links to Huntress Labs and hey, thanks for joining us and hopefully this was educational. Thanks. Yeah, great. And thank you for making it to the end of the video. If you like this video, please give it a thumbs up. If you'd like to see more content from the channel, hit the subscribe button and hit the bell icon if you'd like YouTube to notify you when new videos come out. If you'd like to hire us, head over to laurancesystems.com, fill out our contact page and let us know what we can help you with and what projects you'd like us to work together on. If you want to carry on the discussion, head over to forums.laurancesystems.com where we can carry on the discussion about this video, other videos or other tech topics in general, even suggestions for new videos, they're accepted right there on our forums, which are free. Also, if you'd like to help the channel out in other ways, head over to our affiliate page. We have a lot of great tech offers for you and once again, thanks for watching and see you next time.