 We're back with Keith Bradley, who's the VP of IT at Nature Fresh Farm. Earlier this year, we had Keith on theCUBE and we took you inside the ransomware attack at Nature Fresh. We're pleased to have Keith back to help the good guys get more prepared. Keith, always a pleasure to see you. Dave is always great to see you too. My tomatoes are almost there, you know, but not quite as beautiful as the ones behind you. Okay, let's review for the audience. You got attacked. What were the initial signs or alerts that made you realize that you were under attack? The first set of alerts that came in were fairly easy to see. We got alerts that a couple of servers came down, a couple of switches went offline. And when I first seen the alerts, I actually thought we lost one of our core switches. It just felt like that to me. So I basically popped in and ran to the office and evaluate and see what was going on. I figured it'd be a quick switch out the switch, put a new one in place, program it, this, that, and the other thing. And when I got in the office, I started to look and I'm like, nope, everything's running. And then I tried to log in my computer and it didn't work. And then I walked by one of the other techs computers and I could see that dreaded ransomware screen up on the computer and I'm like, oh, that's not good. And as things started to happen, it continued to happen. And I just ran and I just pulled the plug on the internet and stopped the feed and started to evaluate where we were. I bet your heart rate ticked up a bit. How did the attackers initially gain access? I think it was through an open port, wasn't it? Take us through what happened. Yeah, so what happened is we had a vendor set up a computer and they needed access to program it. So they needed to gain a port access to remote into it to control the computer and program it up. I had been on site in actually Delta Ohio or other facility for them to do this. And I opened up the port. Port was opened on Wednesday by the looks of the evaluation they got in on Friday and by Saturday they were executing their attack on our network. Okay, so was there anything else that you discovered? I think you brought in an outside consultant in terms of like, so they get into the port and then what happened? Did they start traversing horizontally? Can you give us any more detail up to the point at which you got that ransom notice? So yeah, they started traversing the network horizontally. They kind of went all over. They spread out. They gathered more credentials from the system. And at some point in time, they gathered an administrator credential that got them further. And then they kind of had the keys to the kingdom at that point in time. Once that happened, they traversed not only the Ohio network, but they also went through our VPN connection to the Canadian office, to the Mexico office, to the Laredo office and kind of took over everything. Once that happened, there was no stopping them. Okay, so you get a ransom note and you didn't pay the ransom. Take us through that discussion. Why didn't you pay? Did you ever think about paying? What was that like? When we first happened, we really didn't know what to do. We didn't know how to look at it and what to do. And that's when we started gathering our consultants both from our cyber insurance that we had and even Dell technologies that came on site to help see things. Well, they didn't go on site, but they did things remotely for us to help us gather what do we do next. There was a lot of conversations that the investigation firm had with the hackers to try to gain insight into what they got and to see what they got. But we decided that it wasn't in the best interest to pursue it. And that was about suggested by Dow insurance, but most cyber attackers just don't pay it. It's not worth it. And that we were blessed enough to say that we were able to recover all the data that we were missing. So we didn't really need to pay them to get it in back. We just had to rebuild our entire infrastructure from the ground up and start going. So how did you manage to regain control? How did you recover? How long did that take? So the initial recovery to get us back to their minimum was, so the attack started about 6 a.m., 7 a.m., so about 6 a.m., Saturday morning, we lost basically everything in its entirety. By about noon, one o'clock on Saturday, we were able to pick and shift product out of our distribution centers again by our backup and recovery software. We had nothing more than that. We just had their minimum to run and see where things are. Then we started to actually evaluate, okay, where is damage? What do we recover? How do we recover? What is the next steps? And at that point in time, we'd engaged with Dell to say, hey, what do we need to do this and how do we do these steps? So it was a double dip. So they expedited our deployment of Carbon Black. So our Carbon Black was in the middle of being deployed. Now we just deployed it everywhere, found out what was the hash for the attack and the other symptoms they were seeing and we just blocked entirely everything. Then we continued to just kind of rebuild things. And we continued to rebuild every moment of every second and it basically took us to about Monday, Tuesday or the next week to get back to a point where for the most part the end users didn't know what even happened. The longest part for my team was rebuilding probably about 300 laptops from scratch and some of them doing it remotely. So it was a busy time. Wow, okay. So all this was a catalyst as we talked about in May to restart your journey to cyber resilience. What else have you changed as a result of this attack? So we not only changed how we look at things but how we do things. We now make sure that the vendors are much more compliant with our cybersecurity policy. We are much more adamant about what's open on our firewalls and how things are. We reevaluate and even though our backup solution worked to recover for us, we found that that six to seven hour window wasn't fast enough for us for recovery. So we actually went out and rebuilt our entire backup policy and how we backed things up and how we'd recover. Then we institute what we call a cyber vault which is basically a virtual air gap solution that not only protects your data but analyzes the data to see if there's any hackers laying in the data that is going into this vault. So we really, really started to look at how do we back things up every day and actively saying, we don't wanna do this. We don't wanna change the way we do it. And this is how we are gonna secure a network. So you compressed your, if I understand it, your recovery time objective, if I got that right. Did you change the RPO or were you comfortable with the amount of data that you were able to get back? We actually did increase it. We made sure we were backing up every single thing now. So every single thing that we did was totally backed up and recovered now. There were a few servers that we were kind of using but not needing to have recovered that we just rebuilt because we didn't do it. But now we cover everything from a simple user's text file to an AI algorithm that we have on one of our servers. And your cyber recovery, your cyber vault is, you said virtual. So are you thinking about an offsite air gap? Is that something that's in the future? It's possible right now. Again, the power of tech that I'm managing that we're using right now, that's something we like always. We can keep expanding it to what we need. The next step I think that we would do is we would have an air gap to one of our other locations. So we would actually air gap to two different locations. Probably before going on the cloud, more because we have such a diverse area. I kind of like to say that we've built our own on-prem cloud to make things work. What would you tell somebody who, let's say was in the position you were prior to the attack, they see this discussion, they maybe see you at their Dell Tech World interview. What would you tell them? What advice would you give them? I would say to them is whatever you think you're doing today, it's never going to be enough. So you have to look at both sides of the coin. How do I protect my network? And yet, how do I recover when it happens? Because there is no if, it's always a matter of when. Like one of the things that I love from the intro was a ransomware attack happens all the time and by 2030 it's going to be every two seconds. So eventually one of those two seconds is going to catch up to you. So be prepared on both sides. Protect your network from the attack and be prepared to recover from the attack. Yeah, the statistics and the probability are not in your favor. Keith, you've always been a great friend of the Cube and I know you've been loyal Dell customer that sounds like they were there for you and your time of need and you're repaying that with your customer loyalty. So really appreciate you coming back on and telling your story. Yeah, I always appreciate to tell you and it's a good story to share as I feel that people don't want to talk about. They don't want to acknowledge that's happened and if we work together, it's going to make it feel better for everybody. Fantastic. All right, keep it right there. We're going to have more content on navigating the road to cyber resiliency. This is episode two.