 Tom here from Orange Systems and security is much more than product. It's people everything else is just tools We use to sort out the data now choosing those tools that will help sort out the data and all the people behind the tools is a complicated topic and this video is not to take product a and product b and put them together and tell you the Ultimate winner and the security Solution to all of your security problems like marketing people like to say I want to walk through a security incident here That we had on February 12th of 2023. Yes, it was a Sunday. Yes, we responded and yes Sunday security incidents suck, but hey, that's when these things happen sometimes we have to be prepared We have to be ready and I want to go through the process I want to go through each of the detections how we dealt with them how the tools dealt with the detections And I want to serve as a data point not a decision point just a data point So you can understand some methodologies we use when looking at the tools when looking at the processes We use to secure our clients now. This is not a sales pitch to use any of those tools, but I will disclose Yes, I'm a reseller for both of these products I want to always make sure my bias towards anything is very upfront and clear so let's dive into the incident and walk you through what happened and How the response from each company went because there's a lot of nuance in here They think is very important and it really highlights the complexity of security Now we buy Huntress direct from Huntress and we buy Sentinel one their visual inch version through Ninja one That means we communicated directly with Huntress for this investigation We communicated with Ninja one to talk to Sentinel one. So there's a delay in response I want to highlight that just so you know that when we talked to Ninja one They related to Sentinel one who related it back to Ninja one who related it back to us So there wasn't as much real-time communication as we had through Huntress and that's basically why and it's not something That is a ninja one policy per se This is the way when you buy through a reseller Sentinel one has you communicate through that reseller that you bought it from those not just if you bought it through Ninja one we previously used to buy it through packs eight and The same process was followed for that now This client is a co-managed client and this co-managed client has partial coverage with us As in they have an internal IT team and they have decided that they only want us to load our tools on their servers But not their entire network not their end points and we don't have any visibility into their firewall We didn't have a password to it This is a very niche co-managed deal that we have and this is why we have certain Limitations that we're going to hit on how deep we can go with the investigation because we just don't have the extra Visibility but we're going to pull everything that we have to kind of walk you through this now This incident is kicked off when we receive a notice report from Huntress that says incident high this is a Threat and Huntress has extremely low false positives therefore when we see a notice from Huntress We know it's something bad and something we need to address immediately Now I've redacted some of the information here But let's zoom in to specifically the remediation instructions because that's one of the first focuses is what is it and how do we Remediate it how do we stop this from going further because they have the instructions of what to remove and we did follow these Instructions and remove these things they give us some indicators of what it was and a Virus total link that kind of leads us down the path of what is this file and what might it be doing? So let's dive deeper into that so we have our virus total link which shows 15 out of 70 detections and Here it is 15 out of 70 and this was actually a file that was seen all the way back in 2022 of 10 19 Was when the analysis was done a virus total bring that forward to February 28th of 2023 We re-ran this again and the virus total link if you want to see it currently is down in the description below but now we have the details and a couple more vendors have flagged it now one of the vendors that still hasn't flagged it is Sentinel-1 as of February 28th 2023 and I will note that it does say right here in virus total This may differ from the commercial off-the-shelf product The company decides the particular settings which the engine should run in virus total And this may be a precautionary measure and as I said it may be different than our vigilance instance We have of Sentinel-1, but as of right now Sentinel-1 has not flagged anything We have no indicators from Sentinel-1 no notices no tickets and we did open an investigation With Sentinel-1 after we deleted the file so we could start digging into why they didn't see something and Hunter's did Now what is this file FRP is a fast reverse proxy It allows you to expose a local server located behind a nant or firewall to the internet Sounds like something you really don't want to find in any of your local servers, but hey It was there now this file is available on github so you can take a look at it yourself And it was popularized by threat actors so we'll find and dig into some threat research that talks about that and Huntress doing some background investigation So we engaged with the huntress team and they actually said well this file looks like it was here on logs We have going back to 2212 18 so that was there quite a while So this is not a new file that just got created getting created and getting detected are two different things So let's walk through that next so we do see there was an event data on here and that was shared with us from Huntress and we're like okay This is a little more information that they shared with us on this But then I engaged with Dre and this is at 6 30 in the morning the next day because we've deleted the file We can't find any other indicators. We've scanned the systems We couldn't find any other connections, but you know the question is now how did this get there? What happened and I like that it started out with I'm sorry this one has eluded us That just shows that even though Huntress is still the only one who has found this file That's a really I don't know just a good sign at the beginning because they're like hey We should have found this sooner and it just speaks to the way Huntress engages with us And we'll also engage deeply on this topic with me and I said hey since this file was there before What made it trigger the investigation and at 6 33 a.m. Trae answers I believe we pushed a non system 32 detector to look the file wide But let me confirm with our detection team to see what prompt us to look and from the detection team we have a Detector for commonly used names outside of system 32, but when log in wasn't one Then we added when log in to start to detect things like in Tom's case here So I really good response so 6 33 a.m. To 10 33 a.m. To get a relatively concise response It was I said we've already deleted the threat. We're not seeing any more triggers any more indicators We're all on high alert watching things We're engaging with the client on this and letting them know we're we're trying to find the source of this But with the limited visibility we have we're doing everything within the power we can but I have a good answer from Huntress Now let's dive deeper into the research side of this and Huntress sent me these links I also did some googling myself But these are really good articles here threat research from June 2022 from deep instinct everything as I said is linked down Below and FRPC stands for fast reverse proxy client the downloaded FRPC is configured to connect yet another attacker controlled server creating a tunnel between the attacker and the compromise system So this is a report that shows that this file was in use back in june of 2022 in this Now we have the threat highlights report from with secure May of 2022 and fast reverse proxy client and gaining a road access to their victim systems Then going on to enumerate a wider network and later on moving before deploying bitlocker as a form of ransomware and an unusual step the attackers also are sending their ransom notes to a local printer, which I highlighted because I thought was kind of funny and Once again, this is that tool being used It's not that the tool is anything more than a tool But it does show that it's a popular tool amongst threat actors to enumerate the network to move laterally to create a series Of connections because they want to identify Anything you're going to do to stop whatever their plan is usually that plan is going to be ransomware But they may want to disable backups, but they want to also maintain a persistence. So well, they can stop you from Stopping them essentially though. This is a big indicator compromise. It's been documented several times This has also been documented by Florian Roth. They have the Loki and Thor scanner signatures This is publicly available and Florian Roth do follow great security researcher easy to follow on the different socials this indicator is back updated all the way from 2022-11-3 so we have all the way back in November of 2022 Detect FRP fast reverse proxy tool often used by threat groups So we've got a lot of research that we compiled to really Send over to Sentinel-1 to let them know. Hey, why didn't you trigger on this now? One of the things I did as well This is the Sentinel-1 deep visibility system that allows us to take the SHA-1 of that particular FRPC and Figure out how they see it or if they see it with the Sentinel-1 or was it somehow blinded? Turns out Sentinel-1 was not blind to this. I was able to use their tool to see This and the connections it was making so yes, indeed it is a reverse proxy source IP and Destination IP of the same because it's proxying the connections now I couldn't find any indicator that there was any external IPs used or there was another connection that spawned off of this that Showed me another connection and it will get later that yes They can't see not just internal connections, but external ones But it's looping back to 3389 from these other ports This could be because of there's one more component or another tool missing where this was a tunnel that was created So we're seeing these tunnels and don't have any more information because we can't look through the firewall logs and dive into the Networking to say where were these going, but we couldn't see through Sentinel-1 anything related to external access from here Or that it was actually accessing and connecting to or just binding to the port 3389 So I thought this was interesting that we had this here But there's not much else I have besides that but definitely Sentinel-1 can see these TCP connections and can see This tool running so this was in their deep in deep visibility So we sent this information over to Sentinel-1 and regarding the hash This was how they replied to us we sent it and it's February 13th They responded February 14th, so we opened a ticket the next day Monday February 14th We have the response regarding the hash It is considered a risk-ware and was not deemed fully malicious based on your reputation But then they go on further Initiated a new scan and increased reputation would expect reputation detection in the future I globally blacklisted the S1 cloud in meantime, so if that file is found again, it will be flagged now once again This is internally their cloud versus. I don't know how they provide it to virus total But this file is now blacklisted in the cloud instance that manages our systems And we did search all all throughout our systems and all their client systems I should say with Sentinel-1 because you can search just for the hash We did only find this and this particular clients network on a singular server for So it hasn't appeared anywhere else and no other indicators and that directory hasn't come back since we removed it moving further And this goes all the way to February 21st because we sent them the evidence going well your behavior analytics should have picked this up That was our statement we made to them and This is the response Sentinel-1 had that took about till February 21st heard back from Sentinel-1 They saved the following the agent is not designed to monitor or detect traffic opening TCP sockets It is possible the file was dropped on endpoint prior to installation But can I confirm since file events are seen only deletion February 12th and retention only goes to January 14th So they didn't really acknowledge much on there. They see me delete the file They were able to identify it's strange because they identified it that morning of running and actually don't see it in Sentinel-1's logs running prior to that and creating those TCP connections, but still they never flagged it We did delete it now I want to show you what a Sentinel-1 false positive looks like now back to us having vigilance So they investigate these false positives, but they absolutely can trace all the connectivity I bring it up with screen connect because well we use screen connect We connect to our clients with it and when there's updates or changes or sometimes when we're modifying something using screen connect in our System it can absolutely behave like someone doing something nefarious So they flag this and this is like the quote graphic how I can walk through the processes I can walk through and I've blurred this out, but you see this is a behavioral indicators This is there why we flagged it we show the network connections So absolutely when there's an external connection I can dive into the Sentinel-1 deep visibility and see this I can see it in post after They've done the flagging themselves with their team working on it and this was the notes on this particular Incident it's happened afterwards on February 17th 2023 on our screen connect client where it says vigilance mitigated false positive Action taken unquarantine resolve comment mitigated threat verified as false positive in order for an excessive entries in your lawless villain vigilance will Not yet exclude this detection, but may reconsider and matching events So this is what a false positive looks like I can show you that they have all that visibility into the things and their behavioral analytics Does get things wrong, which is part of the reason we use their Team to help us sort these out so they can get sorted faster with less pressure on my internal team for this But we've dealt with this before we've been using set in the one for a little while now Some of you may ask why I'm using two products that seem to overlap with each other on a Venn diagram Everything we do to secure things before an incident seems alarmist But if there's an incident everything we did before we'll just seem inadequate Defense in depth that turn gets thrown out there a lot and as it should with security I believe having two of these gives me a better sense of having a good picture of what's going on But it is tricky doing security. This is not an easy task to do at all in conclusion Due to lack of visibility, we still don't know how the file got there Our best guess right now is that it's a failed install of some type But we have read no other indicators on that server or any other servers the clients that we monitor that there's been any further Potential compromise next security is complex and threat actors are ever evolving their trade craft in clever ways I say it like that because this is where I don't think AI beats out people trade craft Encompasses the overall the way and methodologies that this is done the people at Huntress the teams there spend a lot of time Engaging with the greater security community sharing their own Intel and also having this Intel at the ready So when they examine something they have a large team of people Internally and externally in the community that they can engage with to understand the threat landscape and how it changes Threat actors are very clever They are often looking for new and interesting ways to get into systems because they're very financially motivated So you have to stay engaged with that community look at what's going on talk to your peers and Huntress has really proven again And again to be a team player on that space They are huge at giving out a ton of information whether your client or them or not They publicly post a lot on their blog and they stay engaged and you can follow some of the threat researchers that work there maybe they'll tweet things and have these discussions because they want to Understand what needs to be modified and what's going on in the world of security so they can keep tuning the tools They use to do the better job of doing it such as the adjustment that led to this getting flagged Next your business is not too small to be attacked. It's just probably too small for it to make the news This is a company with an internal IT team But honestly a company even 1,000 people may not even make the news if they go out of business and often some type of Security incident does lead not the day of the incident or even a week after the incident to a company going out of business This is because it sets off a domino effect a chain reaction of events When a security incident costs a lot of money that money has to come from somewhere a business make it a loan that loan Maybe something that puts extra burden on a business But they're able to sustain longer and then they may close This is something I want people to really consider these events are very damaging But it's damaging especially to small businesses that may not be as well equipped or have as much sophistication to mitigate these issues And it's just not going to be newsworthy of other than maybe some local paper pointing out that a small Manufacturing company with a thousand people went out of business and this could still be as I said a long time after the event If they were to get some cash infusion to keep them going. All right That's all I have to say leave your thoughts and comments down below head over to my forums for more engaging discussion Let me know what you think about this. I'm curious to hear to community thoughts Am I being too harsh on something a one and their behavioral analytics should have picked this up because it's a TCP connection I don't know. I'm just want to throw this out there to show that this is a very Tricky topic. It's security is very hard. It's a lot more than just detecting a file or seeing something There's a lot of complexity to it Let me know if you want to see me do some more videos like this because these deep dives are kind of fun But I will admit they're also kind of challenging to put together because you have to be very concise on it All right, and thanks