 Hey, welcome back to theCUBE. It's Lisa Martin and Dave Vellante live, wrapping up our third day of wall-to-wall coverage on theCUBE of Dell Technologies World 2023. As you know, we've had some incredible conversations. We're about to share with you an amazing, very impactful case study on cyber resiliency you're not going to want to miss. One of our alumni is back with us, Keith Bradley, the VP of IT and security at Nature Fresh Farms. It's great to see you again. It's great to be back. I always enjoy being a part of your show. Yeah. So you guys, Nature Fresh Farms goes a lot of tomatoes, varieties of tomatoes and bell peppers and cucumbers. I'm hungry, just thinking about it. But talk to the audience a little bit about the organization and what you guys are doing, and you're a great edge use case. Yeah, we do a large number of edge. I like to say we're actually started doing the edge before there was a concept of the edge. We have hundreds of thousands of sensors looking at our data all the time. And we constantly are innovating what we do and collecting that data to help us grow a better pepper, a better tomato, and coming this July, a better strawberry for you guys to grow. So that's what we're trying to do. Oh, can't wait for that. Yeah. Fantastic. I mean, I've had, you've sent me some of your vegetables. They're amazing. When I first tasted them, I'm like, come on. You guys injecting sugar into these tomatoes? Well, maybe yours. Maybe yours. Yeah. Yeah. Sweet. Yeah. Okay, so we want to dig into your journey to cyber resilience and want to go back to the catalyst, which you had a ransomware attack. Take us through that. What was the anatomy of the attack? It must have been a terrible time for you, but let's start at the beginning. So we had an attack, you know, as well as things that we learned to happen. And it started off as something simple as we had an open port. And they got in and they attacked us. And it's funny how I went home that day. I actually did it on a computer. I thought everything's going good. I went home Thursday night. I went home Saturday morning. I got an alert that everything's going down. So my first initial thoughts, because you don't think of a hack. Nobody thinks that you got hacked was I lost a core switch or something weird happened. So I packed up, ran into the office on a Saturday morning. Look, everything's lit up. Start logging in. Yep, we were hit. Hit hard. It hit every server top to bottom and most of our workstations all at once within like about an hour time. Everything was shut down between all of our facilities. And okay, so where did they attack? So they got in through a port, open port. And then what, did they hit the backup corpus? Did they... So they hit a PC that was one of our IoT PCs that was basically controlling our packing line. From there it looks like they sat for about 24 hours and just gained credentials and started to figure out that. They harvest a few things that kept touching upon different areas. And then all of a sudden, they got another password and another one. And then somehow they ended up figuring out, you know, our main password or root password. And then all of a sudden, bang, they started doing it. We were lucky they came in quick and left quick because they didn't stick around long. So that was our saving grace that kind of allowed us to restore from our backups, get back going on our feet and get going. But it was that one port that was open, that one PC that we'd just gotten working that allowed them in because it wasn't, I guess I always looked at it wasn't a PC that we were worried about. We didn't think about things that way. We didn't think of security first. We didn't think it needs to be protected. And we didn't worry about shutting the port down once that our activities were done. We had to change that. What do you know or what do you think was the hackers objective? You know what? I don't even want to know what they had to think. It's hard to say what they were thinking but I think they thought they're just going to take all our data, they're going to shut us down and we were going to yield to their will and just say okay, that's it. We're all yours and they extorted us for Bitcoin. They wanted this and that to restore our data. Everything's encrypted, the whole story that everybody hears and fears. That pop up on a Windows PC that says you've been hacked was there sitting there prevalent in front of my face and saying hey we want Bitcoin to give you all your data back. So you got a ransom letter? Well, yep, on the PC, yep. You did, yep. Okay, and did you pay the ransom? No, no. We were able to restore our data. We were able to actually recover from our backups and it was a long process but we were able to recover from them and kind of get back going because we looked at it as why are we paying them and do you trust them to do it? Here's somebody doing an illegal activity and you're going to have faith that we give them this money and they're not going to ask for more. So in talking to our consultants that we ended up securing to help us through the process they said no, don't go down that road and we really agreed with that. We didn't want to encourage that field. And the consultants unpacked the anatomy of the hack? Is that right? Do you know where it came from? Was it Russia, North Korea? Was it, don't? They were 99% sure it was from Europe somewhere. They didn't want to get into specifics and we weren't really worried about it. We kind of really turned our focus not to what happened and more okay we just have to get a path moving forward. A path that gets us back to our feet, gets us back to normal and in the long-term gets us back to a safe place. You know in the last 18 months, two years we have seen ransomware become a household word. We've seen massive, nobody wants to be the next headline but we've seen it affecting critical infrastructure. There's no industry that is immune from this but you were interesting that you were saying before we didn't think security first but there was that sort of, we're a farming organization. We're going to be fine. No, it was weird to change that. Like they'd say now everything's security oriented. What do we do? What aren't we doing? How do we look at changing that mindset? And the mindset came kind of in three cents. How do we secure our environment? Educator users and then stay up to date. And now all three are hard because the best analogy I heard is cybersecurity is like moving the CN tower for Canadians. You're going to move at five feet every couple weeks because that poles are moving all the time and so you have to be prepared to change how you're doing your resistance. So how was it you were able to recover? So we were lucky. They didn't hit our backups. They actually, it looks like they tried to hit the backups. They were the one of the few things that I did have multi-factor authentication on. I did have those few steps that we'd already done with that. So it was able to protect us to there. We also had just updated to a new server system. We actually switched over and we had carbon black workload protecting that which actually ended up clearing that up. So it protected our VX rails 100%. So that kind of was a few servers we had and when we rebuilt from there. We started to restore the data into the new cluster. You dodged a bullet. Yes, yep. And so you realized that you did. Yep. So that was that a catalyst to sort of restart your journey to cyber resilience? Yep. It was a double step to resilience. We not only wanted to look at resilience, but we wanted to look at eight, well I can say a three prong approach. We wanted to protect our outside. We wanted to make sure our backups were safe. But we also wanted to make sure that we could, when and if it ever does happen again, recover much quicker than we do. We learned that we were down for a good chunk of time. How do we get better at recovering very quickly and cleanly? So that's where we started that journey looking at it. Okay, so what did you do? You hardened the perimeter? Yep, yep. So we hardened the perimeter. We made sure to update all of our firewalls. We added log monitoring. We got outside people to look at our logs. Things like that. Then we decided to, we looked at our backup recovery process and our current backup system didn't have the speed we need. Didn't have the ability to convert a backup into there. So then we switched our backup profile. We switched how it does. We switched to our power protect data manager, which allows us to recover now instead of hours and hours within minutes now we can do it. And then we started also educating our end users. And we really, I like to say I really took a heart to that because when we did it, I said I'm not only educating them to protect us, but we're protecting our family of employees. Because now they know, because they're just not getting emails to their nature fresh account. They're getting it to their Gmail account. How do we teach them to be safe themselves? And that was the fun part of that. I'd like to say one of the few fun things out of this to happen. But then the next thing we started to do is we now looked at, now we have a new backup system. It works great. But we want to make sure there's nothing there. So the next thing we did is we said, we're now looking at adding a cyber vault to it. We want to system a platform that we could build and expand our cyber security presence. So now we can create an automated air gap system that allowed us to keep our data safe. And then we want to go to the next step with that and actually record it. And well, it's not record it, sorry, but actually analyze that data so that we know a hundred percent that there is nothing malicious in those backups that have made it through our cyber vault. And then the last thing that was kind of a neat trade off with what we do, because we do so much IoT, we have so many databases as we started to restore quicker now. With our new system we were able to restore. So when one of our database gets mucked up, we just started to restore. So what I liked about it was it taught my team how to use this on a daily basis. Because how many companies, and we were no different. Actually tested, tried, tested and did a restore from nothing that's especially in our space. We're small, medium business. We don't have infinite resources. We have a very small amount of resources to do things. And that just was never on the list. But now it's something we do all the time. So you were optimized for backup. You weren't optimized for recovery. You dealt with that problem. You created a cyber vault, was that an air gap or was it a sort of local logical kind of thing? So we're looking at getting the air gap in place. So it basically turns on at a time so that it's only open for a certain time. So the hackers attack your system, that communication's only open for it's not. Small window. Yeah, a small window to let them in. Okay, and then is the process by which you, is that part of the sort of normal workflow or is it a separate workflow that you set up? It's a separate workflow. It's a separate, so both backup units are the primary backup unit and the cyber vault, they will act independent of each other. So they know what's going on. They know that I'm here to hold the data and I'm going to put it to the vault. But they work independently from each other. And prior to this hack, did you test recovery on a regular basis? No, no, we hadn't. I did a few little things over time. Hey, I need to restore this or that. But again, it was a siloed knowledge I said for my team. I was the one that did it. I was the one that did this, that. And now I wanted to get it so that they're the ones doing it. Because when I siloed that information, that was my fear next. What happens if I was somewhere like Dellworld? Where somewhere doing all these things. Somewhere at another event. How do I help them do this? How do we make a way so we all know it? Were you afraid you were going to get fired? An honest question because it used to be something like this, failure equals fire. And then I think that dynamic change where people realize you're going to get hacked. Everybody, you talk about that all the time. It is, yeah. And so were you worried that you were going to get... I'd like to say it was at the back of my head, but I really knew our executive knew that it was nothing that I meant to do. And they knew that. That they looked at our IT team and even all our employees that we do have. It wasn't something that anybody would want to have happen. And we've dealt with diseases in our greenhouses. And our growers feel so personally challenged by them. And they feel like, oh, we're doing it wrong. So we've learned that we will have challenges. I think every company will. And I felt good at the end and felt happy that we're there. I had a lot of meetings with our president and he said, Keith, you can't take this personally. He sat me down. You cannot take this person. Did you take it personally? Oh, I did. It was like having your child ripped from your hands in the own way. Here's this network I built. It's my baby. I know everything about it. And it's just gone in the blink of an eye. And what do you do? Do you think you could have gotten this capability funded if you hadn't been hacked? I think it would have been a much longer journey. And I don't think we would ever look at it the same way. I don't think we would have been as secure and as quick to do it. Do you consider, did the term cyber resiliency mean as much as it does to you today 18 months ago? I'm assuming no. And would you classify the journey that you've been on? You do such a great job of articulating everything that you've done. It sounds like a challenging, really multifaceted journey. Would you consider nature fresh farms now to be cyber resilient? I think that we are stronger. I like to say that there is no perfect. And that's like I said, that's the approach we took. We're going to protect as much as we can, but we want to position ourselves so we can recover and can recover quickly. Because like we learned, it happened to us. It can happen again. I don't know what's the stats out there that happens every two seconds. Somebody's getting hacked if it's not even more than that now. So the reality is we can never be perfect. So we have to be prepared for that bad day again. Oh, sorry. I think I saw a stat in 2021 that a ransomware attack happens once every 11 seconds. And I think by 2030, that's supposed to go down to every two seconds. That's what I was thinking. And that's where I was like, so there's only so many companies out there. And eventually it's like going to the lottery table. You're going to win. It just might not be the win that you're expecting. So you're saying the journey doesn't end. Maybe never ends. Yeah, I say it's a journey that we're taking. Like, you know, everybody talks about zero trust. Everybody talks about there. And maybe zero trust isn't exactly what we want, but there's pieces of it that we're going to start taking out and moving towards there. And maybe it takes us three to five years to get there. But, and maybe by the five year mark, it's actually changed a little bit. So we're still adjusting every time we see something new. Every time we look at things, we think, how do I multi-factor that? Who am I letting see? Who am I letting do things? And is it the safe way? Like you said, you're a small mid-sized company. Did you change the regime for cyber? In other words, whose responsibility, how you collaborated across the organization? You don't have like endless sec ops people running around, I presume. But did any of that change organizationally or outside consultants that you brought in? No, we didn't really change that. What it more changed for me is I became more part of the business process too now. As people started to realize we have to involve IT. IT, information security has to be a part of what we're doing because we can't just go and do what we did. Oh, we got a new computer from this vendor. We turned it on, it turned the packing line on, it started working, that's great. No, it has to be more of a challenge for us to look at. Is it safe? Is the software they're putting on there okay on our network? Does it have anything happening that we don't like? And so we challenged ourselves to make that better and we continue to challenge every time and that's what's made it better. As Dave mentioned and you agreed, you dodged a bullet. But what I'm hearing is so much learning and opportunities that have come from it. So not only start the journey and stay on the journey of becoming cyber resilient, but what you've been able to transform from the culture of the organization. You talked about the awareness and the training of people which is so critical, especially where security is at play. But I can also commend you for sharing your story because there's probably and even happened at this event so many people who will listen to the story that you've been on, take advice and recommendations from you and realize, okay, we're not probably as secure as we should be or we're not putting it first and we have to because this is going to happen to everyone. It's a matter of time. It is, it is just a reality. It's not a matter of if it's going to happen, it's a matter of when now. And I say just because we were hit doesn't mean that we are now immune. It's not an immunity that happens. But you do feel more prepared. I mean, you put it this way, I think you've, my inference is you're doing the things that you have to do and there's maybe some other things that you want to do, your journey continues, but you're much more aware of it. The organization's much more aware of it. Your management is quite supportive actually of that journey and hardening that journey. And there's a lot of uncertainty in the future and you just have to keep innovating. Yep, it's something that we look at every time we do something now. Every time we do and make a system change, we have to stop for a second and say, how is this secure? How will an outside attacker look at this? Is this their way in or not? Kind of flipping your mindset outside in thinking. It was great, Keith. Thanks so much for sharing with us. I really appreciate your time. Yeah, thank you for being willing to share the story. I think there's tremendous value in it for you, for your organization, and- And your peers. And your peers for sure. And of course, you know, as we've talked about before, your growers are data scientists. You got to keep them happy. Yep, they are the key to everything for us. They're the ones that makes the plants grow, live, thrive, and like you said, tasty as you can ever believe. Got to get my hands on some of those strawberries. Yeah. That sounds good. Keith, it's been a pleasure talking to you. Thank you for sharing the story on your journey to cyber resiliency. We really appreciate it. Thank you. All right, our pleasure. For Keith Bradley and Dave Vellante, I'm Lisa Martin. You've been watching theCUBE live for the last three days at Dell Technologies World. You can find all of our editorial content on siliconangle.com, and of course you can watch everything on demand on thecube.net and our YouTube channel. Stick around. We've got great content coming up on theCUBE all year long. You won't want to miss it. We'll see you next time, guys.