 Hello everyone, welcome to this special Cube conversation. I'm John Furrier, host of theCUBE. We're here in Palo Alto. We've got some remote guests going to break down the Fortinet vulnerability, which was confirmed last week as a critical vulnerability that exposed a zero day flaw for some of their key products, obviously FortiOS and FortiProxy for remote attacks. So as we're going to break this down, it's a real time vulnerability that happened, has discovered in the industry, Horizon3.ai is one of the companies that was key in identifying this, and they have a product that helps companies detect and remediate a bunch of other cool things you've heard on theCUBE here. We've got James Horseman, an exploit developer, love the title, got to say I'm not going to lie, I like that one, and Zach Hanley, who's the chief attack engineer at Horizon3.ai. Gentlemen, first thank you for joining the Cube conversation. Thank you. So before we get into the whole Fortinet, this vulnerability that was exposed and how you guys are playing into this, I just got to say I love the titles, exploit developer, chief attack engineer. You don't see that every day. Explain the title, Zach. Let's start with you, chief attack engineer. What do you do? Yeah, sure, so the gist of it is, is that there is a lot to do in the cybersecurity world and we made up a new engineering title called attack engineer because there's so many different things an attacker will actually do over the course of an attack. So we just named them an engineer and I lead that team that helps develop the offensive capabilities for our product. Got it, James, you're an exploit developer. Exploiting, what are you exploiting? What's going on there? So what I'll do in a day to day is we'll take end days, which are vulnerabilities that have been disclosed to a vendor, but not yet publicly patched necessarily or a pocket exists for them and I'll try to reverse engineer and find them so we can integrate them into our product and our customers can use them to make sure that they're actually secure. And then if there's no interesting end days that go after, we'll sometimes search for zero days, which are vulnerabilities in products that the vendor doesn't yet know about. Yeah, and that was the most critical. Those things can be really exploited and cause a lot of damage. Well, gents, thanks for coming on. We're here to talk about the vulnerability that happened with Fortinet and their products, zero day vulnerability. But first for the folks, for context Horizon 3.ai is a new startup, rapidly growing, they've been on theCUBE, CES, Nehal and team have described their product as an autonomous pen testing, but as part of that, they also have more of a different approach to testing environments. So they're constantly putting companies under pressure. Let's get into it. Let's get into this hack. So you guys are just kind of like, I call it the early warning detection system. You're seeing things early because your product's constantly testing infrastructure. Okay, over time, all the time, always on. How did this come about? How did you guys see this? What happened? Take us through. Yeah, sure. I'll start off. So on Friday, we saw on Twitter, which is actually a really good source of threat intelligences these days. We saw a person release details that Fortinet sent an advanced warning email that a critical vulnerability had been discovered and that an emergency patch was released. And the details that we saw, we saw that it was an authentication bypass and we saw that it affected the 40 OS, 40 proxy and the 40 switch manager. And we knew right off the bat, those are some of their most heavily used products. And for us to understand how this vulnerability worked and for us to actually help our clients and other people around the world understand that we needed to get after it. So after that, James and I got on it. And then James can tell you what we did after we first heard. Yeah, take us through play by play. Sure. So we saw it was a 9.8 CVSS, which means it's easy to exploit and low complexity and also kind of gives you the keys that take them. So we like to see those and they're, because they're easy to find, easy to go after their big wins. So as soon as we saw this come out, we downloaded some firmware for 40 OS. And the first few hours were really about unpacking the firmware, seeing if we could even to get it run, we got it running a VMware VMDK file. And then we started to unpack the firmware to see what we could find inside. And that was probably at least half of the time. There seemed to be maybe a little bit of obfuscation in the firmware. We were able to analyze the VMDK files and get them mounted. And we saw that their operating system was compressed and when we went to decompress them, we were getting some strange decompression errors, corruption errors, and we were kind of scratching our heads a little bit, like, you know, what's going on here? These look like they're legitimately compressed files. And after a while, we noticed they had what seemed to be a different decompression tool than what we had on our systems, also in that VMDK. And so we were able to get that running and decompress the firmware. And from there, we were off to the races to dive deeper into the differences between the vulnerable firmware and the patch firmware. So the compressed files were hidden. They basically hid the compression. Yeah, we're not so sure if they were intentionally obfuscated or maybe it was just a really old version of that compression algorithm. It was the XZ compression tool. So what happens next? So take us through. So you discovered, you guys tested, what do you guys do next? How did this thing? I mean, I saw the news hit heavily. You know, they updated, everyone updated their catalog for patching. So this kind of hangs out there. There's a time lag out there. What's the state of the security at that time? Say Friday, when it breaks over the weekend, potentially a lot of attacks might have happened. Yeah, so they chose to release this emergency pre-warning on Friday, which is a terrible day because most people are probably already swamped with work or checking out for the weekend. And by Sunday, James and I had actually figured out the vulnerability to make the timeline a little shorter. But generally what we do between when we discover or hear news of the CV and when we actually pocket is there's a lot of what we call patch diffing. And that's when we take the patched version and the unpatched version and we run it through a tool that kind of shows us the differences. And those differences are really key insight into, hey, what was actually going on? How did this vulnerability happen? So between Friday and Sunday, we were kind of scratching our heads and had some inspiration Sunday night and we actually figured it out. So Sunday night, we released news on Twitter that we had replicated the exploit and the next day, Monday morning, finally, Fortinet actually released their P-Cert notice where they actually announced to the world publicly that there was a vulnerability. And here are the mitigation steps that you can take to mitigate the vulnerability if you cannot patch. And they also released some indicators of compromise but their indicators of compromise were very limited. And what we saw was a lot of people on social media, hey, asking like, these indicators of compromise aren't sufficient. Like we can't tell if we've been compromised. Can you please give us more information? So because we already had the exploit, what we did was we exploited our test Fortinet devices in our lab and we collected our own indicators of compromise and we wrote those up and then released them on Tuesday so that people would have a better indication to judge their environments if they've been already exploited in the wild by this issue, which they also announced in their P-Cert that it wasn't a zero day being exploited in the wild. It wasn't a security researcher that originally found the issue. So what's the unpack the difference for the folks that don't know the difference between a zero day versus a research note? Yeah, so a zero day is essentially a vulnerability that is exploited and taken advantage of before it's made public. An end day where a security researcher may find something and report it that and then once they announced the CDE that's considered an end day. So once it's known it's an end day and once if it's exploited before that, it's a zero day. Yeah, and the difference is zero day people can get in there and get into it. You guys have saw it Friday on Twitter you move into action. Fortinet goes public on Monday. The lag between those days is critical time. What was going on? Why are you guys doing this? Is this part of the autonomous pen testing product? Is this part of what you guys do? Why Horizon 3 AI? Is this part of your business model? Or is this was one of those things where you guys just jumped on it? Take us through Friday to Monday. James, you want to take this one? Sure, so we want to hop on it because we want to be able to be the first to have a tool that we can use to exploit our customer system in a safe manner to prove that they're vulnerable so then they can go and fix it. So the earlier that we have these tools to exploit the quicker our customers can patch and verify that they are no longer vulnerable. So that's the drive for us to go after these breaking exploits. So like I said, Friday we were able to get the firmware, get it decompressed. We actually got a test system up and running, familiarized ourselves with the system a little bit. And we just started going through the patch and one of the first things we noticed was in their API server, they had a diff where they started including some extra HTTP headers when they proxied a connection to one of their backend servers. And there were, I believe, three headers. There was a HTTP forwarded header, a VDOM header and a cert header. And so we took those strings and we put them into our decompiled version of the firmware to kind of start to pinpoint an area for us to look because this firmware is gigantic. There's tons of files to look at. And so having that patch is really critical to being able to quickly reverse engineer what they did to find the original exploit. So after we put those strings into our firmware we found some interesting parts centered around authorization and authentication for these devices. And what we found was when you set a specific forwarded header the system, for lack of a better term, thought that you were on the inside. So a lot of these systems they'll have kind of two methods of entry. One is through the front door where if you come in you have to provide some credentials they don't really trust you. You have to provide a cookie or some kind of session ID in order to be allowed to make requests. And the other side is kind of through the back door where it looks like you are part of the system itself. So if you wanna ask for a particular resource if you look like you're part of the system they're not gonna scrutinize you too much they'll just let you do whatever you wanna do. So really the nature of this exploit was we were able to manipulate some of those HTTP headers to trick the system into thinking that we were coming in through the back door when we really coming in through the front. So take me through that impact. That means remote execution. I can come in remotely anonymous and act like I'm on the inside system. Is that, and that's the case of the kingdom as you said earlier, right? Yeah, so the crux of the vulnerability is it allows you to make any kind of request you want to this system as if you were an administrator. So it lets you control the interfaces, set them up or down, lets you create packet captures, let you add and remove users. And what we tried to do which surprisingly the exploit didn't let us do was to create a new admin user. So there was some kind of extra code in there to stop somebody that did get that extra access to create an admin user. And so that kind of bummed us out. And so after we discovered the exploit we were kind of poking around to see what we could do with it. Couldn't create an admin user. We're like, oh no, what are we going to do? And eventually we came up with the idea to modify the existing administrator user. And that the exploit did allow us to do. So our initial POC took some SSH keys adding them to an existing administrative user and then we were able to SSH in through the system. Awesome, great, great description. All right, so Zach, let's get to you for a second. So how does this happen? How did we get here? What was the motivation if you're the chief attacker and you want to make this exploit happen? Take me through what the other guy's thinking and what he did or she. Sure. So you mean from like the attacker's perspective why are they doing this? Yeah, what the exploit happened and what was it motivated by? Was it a mistake? Was it intentional? Yeah, ultimately like I don't think any vendor purposefully creates vulnerabilities but as you create a system and it builds and builds it gets more complex and naturally logic bugs happen. And this was a logic bug. So there's no blame on 40 net for like having this vulnerability and like saying it's like a backdoor it just happens. You saw throughout this last year, F5 had a very similar vulnerability. VMware had a very similar vulnerability all introducing authentication bypasses. So from the attacker's mindset why they're actually going after this is a lot of these devices that 40 net has are on the edge of corporate networks and ransomware and whatever else if you're an APT you want to get into organizations. You want to get from the outside to the inside. So these edge devices are super important and they're going to get a lot of eyes from attackers trying to figure out different ways to get into the system. And as you saw, this was in the wild exploited and that's how 40 net became aware of it. So obviously there are some attackers out there doing this right now. Well, this highlights your guys business model. I love what you guys do. I think it's a unique and needed approach. You take on the role of I guess white hacker as a white hat hacker as a service. I don't know what to call it. You guys are constantly penetrating, testing, creating value for the customers to avoid, in this case, a product that's popular that just had the situation and needed to be resolved. And the hard part is how do you do it? So again, there's all these things are going on. This is the future of security where you need to have these, I won't say simulations, but constant kind of testing at scale. I mean, you've got the edge. It takes one little entry point to get into the network. It could be anywhere. Yeah. It's definitely security. It has to be continuous these days because if you're only doing a pen test once a year or twice a year, you have a year to six months of risk just building and building. And there's countless vulnerabilities and countless misconfigurations that can be introduced into your network as the time goes on. Well, autonomous pen testing is great. That's awesome stuff. I think it's just freeze up the talent in the organization to do other things and again, get on the real important stuff. Just because your network was secure yesterday doesn't mean it's going to be secured a day. So in addition to your defencing depth and making sure that you have all the right configurations, you want to be continuously testing the security of your network to make sure that no new vulnerabilities have been introduced. And with the cloud native app modern application environment we have now, hardware's got to keep up. More logic potential vulnerabilities could emerge. You just never know when that one end vulnerability is going to be there. And so constantly looking out for it's a really big deal. Definitely. Yeah. The switch to cloud and moving into hybrid cloud has introduced a lot more complexity in environments. And it's definitely another whole attack is it going after. All right. Well, I got you guys here. Great to commentary on this vulnerability and this exploit opportunity that Fortinet had to move fast on you guys help them and the customers. In general, as you guys see the security business now and the practitioners out there, there's a lot of pain points. One of the most powerful acute pain points that the security ops guys are dealing with right now is it just the constant barrage of attacks? What's the real pain right now? I think it really matters on the organization. I think if you're looking at it from in the news level where you're constantly seeing all these security products being offered, the reality is that the majority of companies in the US actually don't have a security staff. They maybe have an IT guy, just one and he's not a security guy. So he's having to manage helping his company have the resources he needs, but also then he's overwhelmed with all the security things that are happening in the world. So I think really time and resources are the pain points right now. Constant change. Any comment? Yeah, just to add to what Zach said, these IT guys they're put under pressure. These Fortinet devices they could be used in a company that just recently transitioned to a lot of work from home because of COVID and whatnot. And they put these devices online and now they're under pressure to keep them up to date, keep them configured and keep them patched. But anytime you make a change to a system there's a risk that it goes down. And if the employees can't VPN or log in from home anymore then they can't work, the company can't make money. So it's really a balancing act for that IT guy to make sure that his environment is up to date while also making sure it's not taken down for any reason. So it's a challenging position to be in and prioritizing what you need to fix and when is definitely a difficult problem. Well, this is the great example. This news article and this Fortinet news highlights the horizon three AI advantage. And what you guys do, I think this is going to be the table stakes for security in the industry as people have to build their own. I call it the militia. You got to have your own test and you got to have your own way to help protect yourself. And one of them is to know what's going on all the time, every day, today and tomorrow. So congratulations and thanks for sharing the exploit here on this zero day flaw that was exposed. Thanks for coming on. Yeah, thanks for having us. Okay, this is theCUBE here in Palo Alto, California. I'm John Furrier, you're watching security update, security news breaking down the exploit, the zero day flaw that was exploited in at least one attack that was documented at Fortinet devices now identified and patched. This is theCUBE. Thanks for watching.