 Hi, this is your host, Sapin Bhartiya, and welcome to another episode of TFI Let's Talk. And today we have with us, once again, Rob Hershfield, CEO and co-founder of Rack and Rob. It's great to have you on the show. Swap, it's great to be here. I'm excited for the discussion. You and I talk a lot about infrastructure. How are infrastructure pipelines different or similar to CI CD pipelines? The idea of an infrastructure pipeline is really to encapsulate this idea of workflow, very similar to what a CI CD pipeline would be. And in a CI CD pipeline, one of the benefits is that you can add in additional steps. You don't have to have everything set up front. You can, as your pipeline grows, continue to add in security concerns, configuration concerns, additional teams, work, linting, right? That's the idea of this pipeline is that it's a start and an end, and you can add in a whole bunch of steps. And that way infrastructure pipelines are exactly the same, where you can build an end-to-end workflow with good injection points where teams can collaborate, where you can add security steps, where you can actually do additional checks and tests without breaking the function of the pipeline. So very similar from that perspective. But of course, an infrastructure pipeline is fundamentally about infrastructure. And so the thing that you're manipulating and working with here is infrastructure, a server, a network, a storage device, something that needs a workflow on it, as opposed to a CI CD pipeline that is typically around building code artifacts. And then there are so many types of infrastructure. How do teams kind of keep up with that? It is. It's a really exploding complexity challenge. At the end of the day, this is what Racken does fundamentally is we create abstractions that allow you to deal with different types of infrastructure using the same APIs and workflows and consistency. And it's hard. It takes a degree of flexibility within those pipelines to be able to detect and respond and change what you do, depending on what the pipelines are. But and people should be very relieved to hear this. At the end of the day, most infrastructure that you're running is much more alike than different. So the differences are important, but the operating systems, CPU, memory storage, networking, TCP, IP, naming, a lot of that stuff is identical system to system. And so when you do it for one system, you can apply it universally across different platforms. And then the difference, you still have to accommodate the differences. But at the end of the day, the differences are very small in what most people have between their systems. Now, let's talk about security for bare metal. We do talk a lot about security of cloud, cloud and you, but talk a bit about bare metal security. Bare metal does add some additional complexity in how those how the systems are configured, managed. We keep seeing and this seems to be an ongoing drumbeat, BIOS and RAID vulnerabilities. It was a UFI, UEFI, BIOS boot vulnerability exposed the other day, you know, the simple reality is when you're dealing with any infrastructure, you have to think through the entire environment that it's part of. If you're dealing with bare metal infrastructure and self-managed infrastructure, then you do have to be aware of the whole environment, make sure it's patched, updated and where things are going. Interestingly enough, some of those challenges are exactly the same as what you would see if you were doing a service provider. So if you were getting a operating system from a cloud provider and you let that infrastructure sit around without patching it, checking it, making sure it was valid, it will, you know, over time, you have more and more vulnerabilities known against that infrastructure. So in a lot of ways, you know, there's additional things to watch and check as you do more and more controlled infrastructure. But at the end of the day, it's automation and good discipline and hygiene solve a lot of those problems just as easily and you don't escape doing that discipline and hygiene, regardless of where you host. And is it like when we do talk about services or clouds, centric things, you know, actually it's not that bare metal is an exception. There can be remote execution as well as you have to have, you know, physical access to when you talk about a lot of vulnerabilities. So when it comes to security concerns for bare metal, are they different? You did say, you know, it's something similar to service where people are like, you know, we have to actually do more or we have to do less. You definitely have to do more from a bare metal perspective, right? Making sure that you have a way to patch an update rate and bios, that you do it on a regular, on the regular and check and test those processes. Right. That that's absolutely critical. You also have out of band management, you have certificates, you have the whole networking pieces around additional network interfaces and how you how you get out and provide access to that infrastructure. So without a doubt, I mean, there are additional ways in which bare metal infrastructure can be accessed. It just in some ways is very similar to what would happen if you allow somebody to have access to your cloud credentials and they were able to log in with your API tokens and and access the systems behind the scenes. Bare metal infrastructure does have additional control pass in it. Those are absolutely vital and, you know, the best practice and what we see our customers doing is they put certificates in. They have unique passwords per system. They have, you know, they actually make sure they rotate and control those passwords and certificates. There's a lot of just basic, you know, good hygiene that you need to do to make all these systems work. And once they're automated, a lot of our automation out of the box provides these types of features. That's a DevSecOps type of thinking. Then in those cases, it really isn't a lot of overhead to make sure those systems are controlled. I think the danger that people get into is not thinking through the management of that infrastructure from day one and also day two. And then they're going to be running around wondering if if there's a problem or if there's a patch or a vulnerability. That becomes a hair on fire moment. Realistically, what you want to be able to do, and we're helping one of the top banks do exactly this, they're going to be able to audit every server in their entire estate on a regular basis. I believe they're shooting for monthly where they can test, check, validate that their bias is up to date, that it's secure, the passwords are corrected. They know what the inventory is. It's been managed correctly. You know, that's the type of hygiene you need to make sure that things are running. Once again, none of that is specific to bare metal. If you're running any type of infrastructure and don't have an accurate inventory where you routinely sweep and check and test those systems, you are really living in an unsafe environment. And worse, you don't have a path to get back to it being secure. Right. There's two pieces with this swap that are really important for people to think about. The first one is, you know, are you secure, right? How do you know that you're secure? When was the last time you checked that you're secure? That's a KPI, right? It's a critical thing to say how, when was the last time I actually validated every system in my environment. But then equally important is how long it would take you to do the work if you weren't up to date. So how fast can you scan all your systems? How fast can you remediate them? Those three metrics are core security KPIs that everybody should be thinking through. And, you know, if you don't have that information at hand, that really becomes a first priority. You're talking about banking. There are a lot of industries which are very well regulated. There are a lot of, you know, compliance that they have to, of course, financial is one, healthcare equally one. Talk a bit about what does that mean in the bare metal infrastructure world? From a bare metal infrastructure work, you know, compliance, validation in any secure environment, the things that you really need to be aware of is who's accessing your system, right? Who's able to, how often was it patched? What does it take to get back to patching? The best practice in cases like this, regardless of industry, is actually looking through and being able to re-image and reset the environment. So what we found is that while it is incredibly important hygiene to be able to quickly go onto a system, patch it, you know, and do the work to mitigate any issues and validate that it's working, the best practice is to be able to do a full system reset, what I would call immutable infrastructure on a regular basis. So the idea is rather than taking a system that's been in the field for five years and has been patched every month, that's good practice. Best practice is actually to be able to reset that system back to an immutable state every month in whether you think it needs it or not. If you're continually rolling an update and a refresh of the operating system back to its pristine state, that gives you a chance to apply all your patches, check everything, validate that your build system works, validate that all of the componentry and automation around building systems is 100% up to speed. And if it is, then it's actually very low risk and creates a lot of peace of mind for our customers when they're able to take systems and just continually refresh the whole system. They're not worried that a patch didn't work. They're not worried that something's out of sync. They actually have a lot more confidence in how the whole system works. And fundamentally that's what security ends up being. The security of the system ends up, there's an element of, did I lock the doors? Did I apply the patches? But at the end of the day, it really is, is the system I have match what I expect it to be? And does it, can I get it back into the state where I expect it? The security vulnerability sneak in when you have a gap between what you think you have in that system and what actually is there. And that allows our customers to do things like be much more confident about, say a ransomware attack. If you can go in and say, you know what? Somebody's attacked me with ransomware. I'm just gonna reset back to my pristine data center and rebuild it to where it should be and then put the data back in. They're most vulnerable to a ransomware attack because you have a way back to working infrastructure. Talk a bit about what are the security risks if any of bring your own telephone? Oh, that's an excellent question. You know, what we see in infrastructure is code. And this is a really sad state of the industry at the moment is a lot of the tools that we have in use today, Ansible's One Terraform is certainly another. They're really not designed to be centrally managed or controlled. And so what we see happening is people will take a working, secure and validated Terraform plan or Ansible playbook and then they will make a copy of it and change it in order to meet their needs. And that doesn't mean they've made it insecure but it does mean that they now, instead of having the template that they started from exactly as is, they've actually modified it and now they have a copy of that template that is not gonna be able to be automatically patched and updated. And what that means is that a lot of times when people are using this automation, they don't have any way to easily go back and check to see if it's the latest version, if it's been updated patch secured or changed. And if the authors of the original, say the platform engineering team at a company gives you a blessed Terraform plan, if you make a copy of that and start changing that plan, as opposed to just injecting variables in it to change it, then if they patch or fix it, you don't get the benefit of that patch or fix. And even worse, if you made a copy, they don't know that their stuff has changed. They don't know what you've done with that system. And so this drift, we talk about drift all the time in Terraform, but usually we mean that the infrastructure is drifted. This idea that we have plans drifting from the approved version is actually even more serious of a challenge. And that becomes a very big deal. And then as most people know, if you take your plan and then overlay a different plan on top of it, you're likely to find a lot of infrastructure changes as Terraform tries to reconcile the two different versions of those plans. So definitely not the way to resolve security and drift issues just by slapping in a new plan. But that lack of knowledge and that inability to patch and change and migrate the infrastructure as people learn is a very, very serious security issue. What kind of advice do you have, to improve security posture? Also talk about consolidating operations, hell with security, how? Then also, once again, in general, what is your advice? Obviously, this is a place where Racken really helps our customers and provides a lot of guidance for how you can create infrastructure that is more secure. In a lot of cases, the answer for us is simply to have resilient, robust automation practices so that you can apply your security with confidence, that you can keep things updated and run through the system. And that ends up being a key point in how any of these systems get built. So my recommendation for security is always make sure that you can restore systems quickly, that you can get them back to known state, that you can scan them quickly, that you can apply patches. Your agility in managing the infrastructure is actually really important for security. And then it is always useful to have good hygiene, being able to control who has access to systems, limit that access, provide very clear guidance on when systems can be changed and attached to. One of the things that we find that's really useful is having a very good dev test prod process. And we really encourage this from an infrastructure as code perspective. If you have high fidelity between your development, your test and operations, production operations environments, that fidelity becomes a way to keep fewer people into the production systems. And at the end of the day, fewer people in production systems actually translates into a much better security posture. The same is true with automated development systems. The more automated your development builds are, the less likely you are to have a developer who needs to open up access to something and leave a development system insecure. And that is a potential vector into your environments, is having too much hand-built anything provides vectors for intrusion. Thank you so much for sitting down with me and talk about this topic today. And as well, I'd love to have you back on the show. Thank you. Thank you. Thank you to all the people and some improved their security posture.