 Tom here from Orange Systems and on July 2nd of 2021 a very large-scale ransomware attack was deployed specifically using the Kaseya VSA software and the ransom affected roughly 1500 businesses and 60 different IT providers and this is why they were referring to it as a supply chain attack So we'll get to that a little bit later in a video, but let's start with yes There was a 70 million dollar ransom put on here and the our evil ransomware group and their happy blog Yes, that is what they call it is where they post how much they want in ransom now This is a pretty tragic story that actually starts not on July 2nd of course but with all the planning and everything that went into this attack and I'm gonna start here with Victor somewhere this year the DIVD NL which is the Dutch cert organization will share a story about how we Nearly prevented an enormous supply chain ransomware attack which potentially led to the single largest ransomware spree in history and Failed and if you don't know who Victor is or familiar with Dutch Institute for Vulnerability disclosure where Victor has a write-up and publish it will get to and why I'm bringing all this up is very important so you can dive into the history of Victor over at Darknet Diaries episode 88 And you'll actually find that Victor spent a lifetime Disclosing properly and with full intention of getting things fixed not exposing but getting them fixed properly vulnerabilities found in many many different products and He's part of the guild of the grumpy old hackers, which is episodes 87 and 88 of Darknet Diaries I just bring those up I like to offer up some of the further reading of where I first heard about and really some insightful information on how this Guild works and what they've done to help bring the bar of security and raise it up But the other reason that's so interesting is because the CVE is signed and when they were actively engaged with the CVE Dates all the way back to April and they have a whole case update right here on their blog And all this will be linked down below and this is actually where the problem starts There was a vulnerability found in the Kaseya VSA product now. What is Kaseya VSA? We could probably go back a little further for those of you that don't work in the IT industry We are a managed service provider me as in my company And that's one of the tools that is used is the Kaseya VSA now. We use a competing product from SolarWinds Yes, I'll talk more about them later because they were in the news as well a few months ago But Kaseya VSA is among the tooling that is used in my industry so we can manage many systems at scale This is why they refer to it as a supply chain attack now supply chain means they get in To how the systems are being supplied services now They didn't at least it does not appear to get into Kaseya Directly that we are aware of right here on July 6th of 2021 But what this means is they were able to use the Kaseya VSA and it comes in two different flavors It has the option of being Hosted by Kaseya that via their software as a service or licensed to host it on prem Either way you're going to rely on Kaseya for updates and this is why we roll back to this you know Dutch notification over here by Victor they had a flaw in the product and that flaw in the product and a flaw in a tool that works at that scale means That ransomware can be deployed at scale now They have a whole you know breakdown of what they did they were doing responsible disclosure as I stated That's the way Victor works and the judge cert team works. So they had assigned it and Really got engaged with Kaseya who they said was responsive But being responsive and getting it done is two different things It is very hard to write secure software that problem is a scaling problem as well as Software becomes more complicated. It will undoubtedly have more flaws This is why you need to engage third-party pen testing agencies to poke at the software Look for the flaws and hopefully fix them and then once you're notified of these flaws Hopefully fix them in a timely fashion. What is he timely fashion? That's gonna be well up to the lawyers Honestly, it's not up to me if Kaseya knew about this flaw so many months in advance Was it that hard to fix or was it something that they were almost getting ready to patch and the ransomware our evil group found the flaw before it got patched These are some of the questions that we still don't really have answers to but obviously this is something very Concerning and one of the problems in the industry right now is just how fast these companies patched We as an IT provider myself as well We put our confidence in these tools that we use to help manage systems That means any of these tools having a security flaw. It needs to be patched and done right away Not delayed not well, we'll get to it. We want to do a few product releases and updates first something else Security has to be a priority because these incidents are absolutely Something that happens at scale now for those you that may have followed my channel previously or all the way back in 2018 when I covered this Yes, I've covered Kaseya VSA because in 2018 it was used to do crypto mining now This is Kyle CEO of Hunter slabs back in 2018 in the write-up of house Kaseya VSA had a flaw that was exploited and then used to deploy crypto payloads times were simpler than they Just wanted to borrow some CPU cycles to you know mine a few extra coins Come that back today and Huntress was leading the charge on getting us all notified and very detailed breaking down What was happening here? They were looking at it from the outside and of course came to the same conclusion and everyone's now in communication But they have a similar write-up They actually posted where they found what appears to be a potential authentication bypass and the potential for sequel injection Now they did this by looking at the logs of the infected systems And so they go, okay, we see through this log through this here Now something a threat actors do is actually destroy a lot of logs. So this is not easy to obtain Huntress had to really reach out to a lot of Engage with I should say a lot of different MSPs to find one that had good logs because part of the whole process is Don't show people how you did it makes it harder to patch And that's the way these different ransomware groups work But Huntress was really on it for getting on top of things and identifying which IP addresses were used to send the commands How this whole thing broke down and really helped get the word out They really led the charge on this and gave really detailed updates step by step each of the way This also is something to say it was doing on the side over there They seem to be a little bit less forthcoming about how they were doing it, but they did shut down their servers They didn't recommend to their people to shut down their servers. We have a recent statement. I'll leave a link to below from their CEO You know usual PR statements saying we have lots of customers and only a small number of them were affected type of statement because that's what you expect from our companies It's a it's a PR thing. I love it or hate it. It is what it is, but This type of attack is obviously very scary very concerning for people like myself who manage and have to rely on these different tools in our industry And this is one of the things I want to talk about the way forward because this is something that really concerns me We do not have the best responsiveness in the industry to these type of attacks It's not that it's not gonna happen. That's probably the one thing is true It breaches happen is how those breaches are handled Was it a problem you knew about since April or was it a wow no one seen this coming? That's really clever type of hack and that's where things get a little bit fuzzy And the way we help mitigate against these things one transparency These companies should be having full penetration testing done on a regular basis with their products They make the money to do it pen testing quality pen testing I should say is yes very expensive It is also one of those annoying things that the people who work in the business admin side go hold on We keep paying these people not to find things are they really useful and you know business citizens are made around that Yes, sometimes they do find things and that's the whole goal We want these security testing people we want the pen testing companies to find these flaws Get them fixed and keep these software tools secure This is a problem that was faced earlier and I covered a video with connect wise and they Mishandled essentially their attitude from being told they had some flaws and one of their products by Bishop Fox They had a video I did on that and come here. We come all the way here in 2021 We have a completely different response from connect wise or even participating in bug bounty programs That is kind of what we want to see in the market We want to see these companies go from hey I guess we should probably get in this versus downplaying it all saying it was only some small amount of customers if you're That's small amount of customer if you are to date the fewer than 60 Kasey a customer You know, you're just one of the 60 of them that represents 1500 businesses this is not small scale and please note I don't really have a number for exactly how many endpoints that means But obviously a small mom and pop shop getting infected with ten computers is terrible a thousand systems or 2000 or even more affected at a single large company Which we know this has shut down some pretty sizable companies a Swedish grocer was included in the list on there as well This is a much more serious problem because it's not about which business the scale and scope of these businesses And of course if you're that business you are deeply concerned and you're asking the questions What do I do about this? So I'm hoping that this is just one more I hate to use the word wake-up call because it's not a cliche and But sometimes it is a wake-up call and sometimes it is companies like connect wise who go alright This is our new security posture. We're participating in hacker one we're going to really take this seriously and Finally, let's talk about solar winds which was also a supply chain attack But a slightly different one now This is where in I believe fully because solar winds came out better after the incident What happened was solar winds specifically solar winds Orion because solar winds In a similar way to Kaseya is a company built by acquisition when you build your company by acquisition You slap your name on all the things so it's solar winds Orion solar winds MSP Etc. Etc. The solar winds Orion product was attacked But that then caused an audit of all internal things on solar winds, which is great They re-upped procedures revamp things and you end up with a better product And hopefully Kaseya does the same thing Kaseya VSA is the specific product. They have confirmed was attacked They've engaged with mandiant fire. I and are going to be working on a plan and remediation Now when they engaged with solar winds, that's actually a great thing Fire I was a little bit interesting how they became engaged on December 13th of 2020 fire I actually announced that they were part of the solar winds attack Well, they more specifically announced that they were attacked in December of 2020 and it turned out solar winds was a source But the source went all the way back to November 2019 I'll leave a link to the video where I break down the whole timeline Because the solar winds attack was obviously interesting in scale and also interesting that it went on for so long unnoticed But it also had a very different goal the goal of the are evil Ransomware gang is to collect money So specifically they're asking for 70 million like I said the goal in the solar winds incident was espionage Which means is quietly and stealthily as possible You go through systems and collect information with the goal being espionage not the goal being deploying ransomware So their roles technically supply chains attack the solar winds one went up the supply chain to attack a specific product They knew was deployed by many companies Which does include fire I which led to their breach which led to some really good debrief But the our evil gang is part of the ransomware as a service And yeah, they're just out to make money They matter of fact if you dig a little bit into them The reason we're aware so much of the happy blog is where they happily post all the different Ransoms they're asking from many different companies. They've been around a while. They're not stopping this And I know there's going to be some people we need to solve this cryptocurrency problem And I seen a CEO make some hints of that from kaseya, but honestly that is great I live in a real world where that exists I don't know that the world's going to shift dramatically against cryptocurrency and Stop the transactions from happening once a product has been out there and these attacks scale up like they do partly Yes, is certainly an enablement to be able to move that kind of money anonymously But I don't see any easy solution outside of Massive changes, but so many places are adopting cryptocurrency that those massive changes are going to be harder and harder to implement every day to get traceability Atom not saying there's no efforts that should be put towards it But I live in a world where I know that exists So we know we have to create security because honestly if you want to talk about the perfect world It would be the one where microsoft updates just work and all the software updates just work And we don't even need these RMM tools because computers are just magic and there's never a problem with them But back to the real world that we all live in All right, leave your thoughts down below on this and hopefully the recovery process which I know has begun will go smoothly um One little comment on that if you spend a little bit time on reddit r slash msp You will notice that I know one thing that's hampering some of the recovery Process is the scale and scope some people go. Hey, just flip the switch for backups That's great until you have thousands and thousands of them and you tax the backup servers and the actual restoration process is Always a little bit longer than you might expect but should be something to really consider It's not just a matter of rolling these back. It is the You know the time it takes to do it and finally someone may ask isn't there software that will magically solve all these problems That's a mixed answer because it really depends on whether or not you Excluded everything antivirus and firework exclusions and trusted apps per the casay are right up if you completely wait list Everything casay it does to allow you to get things done You now kind of neutered a software that you may be using to protect you on it So there's not like a magic this absolutely is the way to do it because Everything in security is a balance We use a series of different tools to stack them on top each other to help provide a comprehensive protection stack But with any of these there's balances I can turn off the computer and claim it secure But the reality is I had to use our As I said we're solar winds user We had to use solar winds to deploy a lot of mitigations against the Just recently posted printer problems that thank you microsoft for not Doing that properly and then causing lots of time and mitigations to fix the printer nightmare issue That was last week's problem, which by the way is still this week's problem It's just that this is also a problem for a lot of people in the industry And my heart goes out to all the msp's that were affected by this obviously Many of them were doing things right. They were multifactoring things And also a shout out to many of the people in the industry including huntress and all the others that participated in one Documenting this and getting the word out and other msp's that are been helping out everyone So it's been cool that we've seen a lot of community support And hopefully kaseya at the end of all this comes out better and as a solid program And maybe we'll even see him on hacker one that would be pretty cool. All right, and thanks And thank you for making it to the end of this video if you enjoyed this content Please give it a thumbs up if you like to see more content from this channel Hit the subscribe button and the bell icon to hire a sure project head over to laurance systems.com And click on the hires button right at the top to help this channel out in other ways There's a join button here for youtube and a patreon page where your support is greatly appreciated For deals discounts and offers check out our affiliate links in the descriptions of all of our videos Including a link to our shirt store where we have a wide variety of shirts and new designs come out Well randomly so check back frequently And finally our forums forums.laurancesystems.com is where you can have a more in-depth discussion about this video and other tech topics covered on this channel Thank you again, and we look forward to hearing from you in the meantime check out some of our other videos