 Welcome, welcome, welcome. Welcome to Cloud Native Live, where we dive into the code behind Cloud Native. I'm so excited for today's session. I'm your host. My name is Whitney Lee. I'm a CNTF ambassador, and I'm a developer advocate at Tanzu. So here at Cloud Native Live, every week we bring new presenters to showcase how to work with Cloud Native technologies. We'll build things, we'll break things, and we'll answer your questions. Today, as you see, we have Baruna Charya here with us to talk about securing the secrets manager with cube armor. So this is an official live stream of the CNTF, and as such, it's subject to the CNTF code of conduct. So all that means is please be nice. Be nice to the other folks in chat. Be nice to Baruna. Be nice to me, please. You don't have to be that nice to me. Just everyone else is important. Friends who are joining us live, will you please put your name in the chat and where you're joining us from. I love seeing that this is a global community. I think it's the coolest thing. And I think that's all. Oh, here's what I want to say. There's questions. If you have questions during the presentation, please do ask those questions in chat. We're going to make this more of a conversation. Baruna's going to share his screen the whole time and code the whole time. No slides. So we're going to have a nice, a great conversation, and we can't wait for you to be a part of it. So with that, I'm going to hand it over to Baruna. Baruna, introduce yourself, please. Hey, folks, I'm Baruna Charya. I am a fellow CNCF ambassador as well. And I am a maintainer and leading the development efforts of Cube Armor, a CNCF Sandbox project, which we will be discussing, diving very deep into today. So look forward to that. I work as a software engineer at AcuNox. AcuNox is a cloud security solutions company where Cube Armor as a project originated. So that's all. Awesome. That's amazing. I'm so psyched to have you here. You and I have streamed together before, so this is like a chat with old friends. Are you ready for me to share your screen? Yeah, give me a second. All right. Yeah, I'm ready. All right. Here we go. So I think it's lovely. While you're didn't do that, I'm going to say quick hello to all the folks in the chat. We have friends here from Palestine. We have Dallas, Texas. We have Vancouver, Amsterdam. How cool is this? We have Germany and I don't know where Sherry Povitz is, but hello. Greetings. Thanks for being here. Awesome. So, hey folks from all over the world. So I'll start with introducing Cube Armor to you. So Cube Armor is a cloud native runtime security enforcement system that helps you restrict behaviors such as processes, execution, file accesses, networking operation. And this happens inside containers or your virtual machines themselves. It could be either on Kubernetes or your bare material virtual machines at will. So Cube Armor is not tightly coupled with Kubernetes, but it can help you enforce things on containers running on directly on your virtual machines or virtual machines themselves. But you might ask what kind of enforcement are we talking about? So for that, what I'll do is we will dive directly into the Getting Started Guide here. Before we actually try to secure the secrets manager, I just wanted to have a brief introduction that what exactly is enforcement and how does Cube Armor go about that. Because runtime is something very new in the ecosystem and it's crucial that we understand what's that before we actually try to use like protect our secrets manager at runtime. Excellent. Will you zoom in a little bit on the webpage please? Yep. Thank you. Excellent. So this is the Cube Armor Getting Started Guide. And as any other tool you can use Helm to install Cube Armor. And it's already shows that okay, Cube Armor already exists. It's going to fetch all the repositories and do the installation as needed. And we can see that Cube Armor is, yep, Cube Armor is running in our Cube Armor namespace. And if I run Q Armor Pro. So Cube Armor in general, we are using Helm to install it, but we have an helper utility called KR Merch CLI, which helps you dive like get into like it's a user friendly way to interact with Cube Armor. In general, Cube Armor has many like it is Kubernetes native. So you can use CRDs. You can integrate it with Loki, Grafana, Elastic, any stack you have to interact and retrieve information from it. But for today's session, we are going to restrict ourselves to KR Merch CLI, such that which is one of the easier ways to interact with Cube Armor. So we see that, okay, Cube Armor is armoring up our clusters. And my cluster is a three node cluster on Google Cloud Engine, which has container security set to true. And all of these metadata, which we are going to dive into today. Awesome. So here we have a sample engine X application and I record and deploy it. Engine X already exists. So when did all that metadata get set? Did you do that? Did that happen upon installation? Yep, these are the default configuration of Cube Armor. So these information are node level information that, okay, Kubernetes has these nodes and this is the configuration that Cube Armor sees that. Okay, I'm running on a Cubelet version of 1.2.27 and the LSM that Cube Armor is using is, what's LSM? We are going, we will dive deeper into it as well. But that's the secret source that Cube Armor uses spoiler alert. So that's the secret source that Cube Armor uses to help you protect your containers and host. And all these are default configuration of Cube Armor, which we are able to tweak around, which we will be tweaking around today as well. So since engine X is already there, we are going to apply a sample Cube Armor policy. This is a sample Cube Armor policy. It's a Kubernetes resource called Cube Armor policy. You have a name, the name space is default because engine X is deployed in default. And we have a match labels here that, okay, let's set, let's enforce this policy on any application that has the label engine X. And at the end, we are going to call it block the execution of user bin app and app. This is a very interesting policy because you know how package management, you should not be executing any package managers in your containers, right? Because your containers always, always come already come with all the p package utilities that you want at the build time. So you know that if someone is executing a package manager in your container at runtime, something malicious is happening. And you should be very effective. But we will have a policy here, which will help us with protecting that malicious activity from happening itself. So while that policy applies, and the policy is there, I'll do is I'll try to execute app deploy slash engine X. Oh, that's being hidden when it's on the bottom of the screen. It's behind your name. Can you, I can remove your name. Basically, I can't see your commands when you're typing on the bottom line of your terminal. Okay, cool. Yeah. So we have, we are executing in the, to the engine X container and running app update and app install mass scan, mass scan is used to do port scanning and all that. And it's generally something that malicious actors do use for to look for open ports to perform their further attack operations. And you see we got a permission to write the process itself did not happen. And you might ask, how do I come to know about that? This was done in my cluster or not. So we have log logs for that. So KMR sends out telemetry information that something like this happened. And I will boil it again for you. I'm on the edge of my seat. I'm so excited. Yeah. So the, we have the alert here. Kuberma says that, okay, something happened in the host. This board name was engine X. There was a container. You have all these metadata that helps you identify where, which was the container where it was, it happened. And finally, it was that, okay, after night, to certain reasons, you may or may not need this information, but Kuberma has that entire context so that any like security operator realizes that, okay, this has happened. And I can take further actions to do that. Okay, probably this container is compromised. And I need to take further actions. But the action itself was denied then and there. So I have a piece of mind that, okay, nothing unsafe has happened yet. But I might need to be cautious about, okay, let's see what's wrong. So that's what Kuberma is for you. And I mentioned that Kuberma uses LSMs under the hood to achieve this. So we have this enforcer mentioned here as BPF LSM. So LSMs are Linux security models. They have been in Linux kernel for more than two decades now. So they are a trusted way to enforce policies on your Linux machine. But so you might ask, why do I need Kuberma then? So Kuberma comes into the picture because these LSMs were designed for Linux kernel back then. And the landscape has changed today a lot. We have Kubernetes ports, multiple machines running as part of a single cluster, thousands or maybe tens of thousands containers running inside our clusters. So it's difficult to manage those entities out there. And the LSMs in general, whatever mechanism they have, they won't have the information that, they would have these information that, okay, some PID 49696 was denied. But what is 49696 for me? It's just a number for me, right? I need that entire context that where it actually happened. And that's where Kuberma comes into the picture that I have this alert with the entire context. Another question that people ask is, do I need K-Armor utility to get these telemetry? And that the answer is not, that's false. K-Armor is just a utility so that you quickly gain this information. But you can integrate any of your SIEM tools. And we mentioned it in our docs as well that we have a guide entirely for the same that it's a generic way that it's in some ports STT out. So you can integrate with your Fluentee or Logstash to fetch those logs and push it to Elasticsearch and have that entire ELK, EFK, whatever stack you follow, you can integrate it with that or Kuberma exposes it on a GRPC endpoint. You can manually write your adapter to integrate it with, integrate the logs as well as we support OpenTelemetry, which is a standardized way of exporting logs and traces. So if you are operating on top of OpenTelemetry, any OpenTelemetry related tools can integrate with Kuberma. So that's... We have a question from chat. Can you discuss the overlaps between the commercial offering of Sistig to Kuberma? So Sistig is a commercial version of Falco. Yes. That's it. So Falco and Kuberma are like, they target different things all together. And Falco and Kuberma both are runtime-secured engines. But where Falco shines is that Falco is a threat detection, a runtime threat detection. It only monitors for what's happening and it helps you with identifying that something malicious happened and you want to take actions on it. There are automated ways to take action on it as well. Like you can go ahead and kill the container and all the liberated actions can happen. But that's still post-attack mitigation. So we talk about that in our Getting Started Guide as well, that differentiation. So essentially, if what happens is, okay, this is a pond pod that someone did attack your pod and there was a malicious execution. There was an EBPF event from Falco. There's an event handler and then the event handler goes and takes action on it. But how Kuberma does this is that it denies then and there that, okay, this execution itself at level one would have been denied. And you would receive all the telemetry information relevant. So that's the power of inline mitigation that any further containing nothing like the attack itself is contained then and there itself and all the other actions. But where Falco shines is that it has a rich ecosystem of integrations and it helps you with all the threat detection mechanisms. So that's the differentiation overall. I am not sure about what other offering that SISD conversion parts include. But that's how Kuberma is differentiated in general that we go for inline mitigation and help you contain the attacks that way. So inline mitigation is the terminology for when an attack is stopped as it's before it happens. Is that what I'm understanding? Yes, yes. So as it happens, it is denied. So that's an introduction to Kuberma. We have, we already saw that what's the capabilities of Kuberma. But what I'm now going to do is let's get on with the topic that we are here for. We are going to secure the secrets manager itself so that you keep your secrets safe and secure inside secrets manager. But what if something, some attack happens to your secrets manager itself? And that's where we are going to talk about the problem statements and how we are going to prevent that. For today's discussion, we are going to use hashikov vault. Hashikov vault is a secrets manager, very well known secrets manager, and we are going to go and secure it. So let's install hashikov vault. I think I already installed it. Check it using help. There might be audio. I think we're just quiet. We have someone in chat pointing out that there's no audio, which I appreciate. I would like to know that. But I think we're just quiet and watching Baroon work right now. Baroon is installing hashikov vault and then going to show us how we can use Kuberma to secure the vault. You do hear me, right? Just check. I can hear you. Nice. Okay. We deployed hashikov vault here and let's see if we can access it. Okay. It's still container creating. Let's wait for it to start. While we do that, let me talk about why protect secrets manager. So you might say that anything inside a secrets manager is already encrypted. Why do you need it? Even if some attacker comes into my container and tries to access the secrets, what would they get? It's already secure and encrypted and all that. So I don't have any losses, right? Even if they do able to access it. But now the issue is what if they go ahead and delete your secrets? What if they go ahead and add an encryption layer on top of that? Now you yourself are logged out from accessing your own secrets. And now that's how basically ransomware attacks work that now you need to pay for ransom, pay the ransom to get back that access. So where Cube Armor will come into the picture is that it will not let, even if an attacker sneaks into your secrets manager container, it will not let them delete, add an encryption layer. So all that, it's all completely safe and secure. And I hope the world is running now. It is running. So let's try accessing it. So to do that, Vault's looking at the system calls at the kernel level of the machine that the HashiCorp Vault is running on. Is that correct? Can you repeat it? To do this, to protect the data in the Vault, what it's doing is looking at the system calls on the machine that the Vault is running on. Is that correct? Yeah, that's right. So that's a nice question that how, where is Cube Armor running? Is it just a part or is it, how does it know that where, on which node Vault is running and how is it protecting it? So essentially, if you look at the Cube Armor manifest, you see three Cube Armors here. Essentially, this is the main engine that protects your containers. And there's an operator so Cube Armor uses operator-based installation. And there's a relay server, which helps you with all the SIEM integrations that we talked about. But the reason you are seeing three Cube Armors here is that Cube Armor installs itself as a daemon set. It's monitoring all the processes on all the actual nodes. So daemon, it is running as a daemon in all the nodes itself. And that's how it knows that, and it has all the metadata to actually go ahead and then construct the telemetry that it needs. I have a question around that part of it. Looking at all of the system calls on all of your machines seems like it might take a lot of compute. How does Cube Armor, is Cube Armor like a heavy technology? If you see right now, Cube C, it's 17 mCPU and 100 mM memory. So it's using not much. It's very decent amount of usage for the kind of things that Cube Armor is doing. And because how does it do that is... Yeah, but you're also not making a lot of system calls right now, right? Yeah, if you show you... Involved itself has started already, right? There are things running. If I show you the cluster itself... Okay. Okay. A, you will see there are a lot of things. I hope I don't seem... No, no, no. That's completely valid. Okay, cool. So if you see, this is traveling, like a lot of system calls are already happening. And why is it efficient like that? Because Cube Armor uses EVPF and all that, right? So everything is happening inside the kernel itself. And because we are intercepting the information right at the kernel level, nothing... So the overhead is pretty less from what we benchmarked with... Like it's an opinionated benchmark, but the overhead that Cube Armor adds is less than 3%. And we have published that data as part of Vicky. And that's decent considering what... How Cube Armor is helping you enforce things here. And monitoring all this is called... Like as you said, a lot of system calls are happening right now itself. All the network calls are being changed. But you might ask that... We will dive deep into... Even this is a lot of information, right? And you can choose to configure Cube Armor in a way that, okay, I don't need network information altogether. Let's disable it together. Disable it altogether. So all that configuration options are all there. So it doesn't have to monitor every single system calls. You could say, I only want it to monitor when a file is opened or something like that. And then it'll do that. Yeah, okay. Interesting. We have another question from chat. I just totally railed your... You're trying to talk about Vault and we're getting into other things. But the question is, what level of kernel exposure slash coding is required to make the most out of EVPF offering part? So Cube Armor itself, as you saw, right? These information that were coming right now, I mean, this was not formatted. But whatever you see here is all structured data. And you don't... You need to... Is the EVPF experience to get these data altogether? So it's purely how you manage any Kubernetes resource, you are going to manage Cube Armor as well. It's all YAMLs, annotations, labels, that's all the concepts you need. You don't need to write actually EVPF code to do anything. That's the complexity that Cube Armor handles itself. You don't need to worry about that. And that's why you need... You don't need to understand EVPF at all. There are some parts like you might need to understand... If you make a lot out of it, you might need to understand what's process IDs and how to go around. Actual protection will be handled by Cube Armor, but do the day to operations that, okay, this attack has already happened. What are the next steps to debug? Where in my entire infrastructure actually caused this attack to sneak in? That's where you might require further to understand some level of process ID, what's process ID and all that information to go ahead and debug that. We have a nice statement from Jay that he appreciates our conversation and digging in about... There probably aren't many system calls happening. And then we also have a question from Kuldeep. So we're talking about securing HashiCorp bolts right now, but does it make sense to use Cube Armor when we're using a CSI driver to mount secrets? So you could still use Cube Armor to secure Azure Key Vault or to secure AWS secrets manager, those two technologies themselves, right? Because Cube Armor is not very specific. But I'm not sure how AWS secrets manager will be posted in their own private clusters, right? So if AWS chooses to do that, I'm not sure if there's a self-host option on these two management solutions. But what you asked is that, like what Kuldeep wants to know is the CSI drivers goes ahead and mounts those secrets inside certain ports so that they can leverage those secrets. So you can go ahead, you can definitely use Cube Armor to secure those volume mounts themselves. So the secrets that are mounted inside, so that only certain binaries inside a particular like pod are able to access those secrets. You cannot go ahead and cat that secret or RM that secret. So that's there, yeah. So it totally makes sense. And that's a valid use case. We are actually going to secure volume mounts in Vault as well. But for some reason, Vault is not letting me initiate it. So I'm checking that out. Some live debugging is going to happen there. We promised up top that we'll break things. So we're delivering on that promise. Kuldeep has a comment. I just want to say you're talking about Sistig and Falco. So Kuldeep, you said you joined late. We already talked about that already. So I recommend you catch that as part of the recording. But the short answer is that Cube Armor, we'll see how well I'm listening. The Cube Armor does inline enforcement, which means as the attack is happening, it's stopped before it even happens. And Falco does asynchronous enforcement where it's the attack happens, the bad thing happens, and then an alert alert happens and then some sort of effect happens. But the bad, it doesn't stop the bad thing from happening. Falco is still amazing because it's an amazing community with a ton of integrations. It does. There's a lot you can do with it. But that's the differentiator between those technologies. How we do it? Give us an insight into your world, into your debugging. What I'm going to do is what's famously done by anyone, I'm going to uninstall and install. Very sophisticated work happening here. It's a complex debugging process as you all know. We can all relate. Do you all have any questions to chat about while we're waiting on this part? Or, Brutus, is there anything you want to highlight while we wait? Something. One of the things was that probably these all information are not easily consumable for you. Why are so many JSONs flooding my screen or for letting my databases? I need something to aggregate for me. These are network calls that are happening. Walt is going to make a lot of any application with a liveness probe is going to make a lot of network calls. So is there a way that something is aggregated here for me? So K-Armor has a nice utility called profile. And if I go ahead and run that, I think, okay, I uninstall Walt so nothing will happen now. But what it will do is it will start aggregating counts for the same process that's happening for you. So if network calls are getting executed frequently, it's going to do that and aggregate it. We are going to see it once the world itself is starting to run. So when your active line is on the bottom, we can't see it from behind your name. Okay. I will have to keep that in mind. Yes. Yeah. Cool. But I will show you that profile using default namespace maybe mean while Walt is installing. So we did Walt. And we did apply a policy on an engine. So I'm going to do apt again. So, okay, apt was permission denied two times for me. The result was permission tonight. And the pin bash process was executed once. And I have network calls were made by Prometheus, sidecar injector. All these are getting aggregated for me. Okay. This has happened 18 times. This has happened six times. So all that metadata is aggregated for you here. Cool. And then all that's also getting sent to like via open telemetry to whatever your stack is. That's awesome. We have a question. We have from Jay. Oops. So this is a tangent if we need questions to kill times as Jay, you all mentioned BPF, LSM enforcer is app armor, LSM also an available enforcer. Yep. So cube armor uses and like as I said, LSM is under the hood and the project actually started out with cube armor using app armor as an enforcer. And we pretty much have first class serve for app armor. Because if you see today, app armor is being used quite heavily because app armor also integrates with port security context and Kubernetes. So people do use app armor to protect their applications. And they do it at some level. But writing app armor profiles themselves is hard. So that's where cube armor comes in that entire narrative that why cube armor is needed there. But why we are like betting more on BPF LSM is that because with BPF LSM, we are writing our own code to decide what is being enforced instead of translating our policies to app armor based profiles. And there will always be limitations in app armor provided like because we are it's another tooling that we can do only so much there. So we are we were being limited by what app armor has to offer. And that's where we are today. Like if we say BPF LSM and app armor have achieved feature parity. So any of the both of them you use it's equal for you. But as as cube armor progresses, we are going to add additional features which can leverage the powers that BPF LSM provides us. So that's I hope that answers your question. I can't see your command line over there. It's behind your name. Yeah, thank you. I tried to look into just taking your name off the screen. And I was having trouble with that. I can try to make it into an X. See if that works. No, it doesn't on both. Not just you. Okay, I took your name away. Off out of that. Oops. What's happening is that for some reason I'm not able to initialize what but we can still try and secure go ahead and secure world right. Based on what's happening, if and this is a good time to launch the armor profile that the policy wrote only applied to like engine next things that wouldn't be the thing that's causing Walter fail. Would it? No, no, no, what's happening is what is saying that I'm already initialized. But then the UI shows that you need to unseal me. I'm not an inch and inch initialized. So I'm not sure why the command line thinks that I'm already initialized. Got you. You see, so as I tried accessing the UI itself, I'm seeing that world has certain processes getting executed here, right? Been busy box is executing wall status been bought responding wall statistics vault itself is performing some statistics. So all that execution is happening around vault as it runs. So and that's getting aggregated here. And file visibility itself is on as well. That inside my wall directory, okay, home vault is getting access ETC password. These some of these are security sensitive application files that are getting access frequently. And then itself vault is having some network calls all together. But that's how a cube armor is aggregating it all for you. But you might say that this is a lot of information, right? And you see file accesses are already happening at much faster rate there as 2316. It's increasing very fastly. And I wouldn't want that even network, right? This network is already reaching 86 on my aggregated while it only just started. And that's not like you might say that that's causing some performance overhead as well. What I'll do is now we can modify what you what all you want to monitor inside a particular namespace. So if I go ahead and describe vault NS vault, everything is installed install inside vault namespace. You see cube armor, I have an annotation here saying cube armor visibility process file and network. So these are the three entities that we are monitoring using cube armor. By default, it's just cube armor process and network. I had added file explicitly so that we show that like, I could show you this nice view that okay, what's happening inside. But if I go ahead and overwrite this annotation, annotate NS, and go ahead and set it to none, it suddenly it will just stop increasing now. Nothing inside the vault names, it will get stuck at 151. It was increasing for you a lot of time, but now it's not going to happen. And if I go ahead and care my logs log filter all inside this vault namespace, even if I now execute anything, I suppose I exit and keep this in and ID and SH. I can't see what you're typing down there. Thank you. It's okay. We need a secret word, jelly beans. So all these accesses I'm making, but I don't have any logs yet, but I will go ahead and now annotate again. And I'll just now have process annotation. And I only want to get alerts for processes. And now if I go ahead and do an LS, voila, I am only getting process based executions now. So you can fine tune what you want to receive inside. I know that okay, I know I already know what all processes are going to happen inside. But the nice part still is that even if I now apply policies and visibility is off, QRM is still going to enforce and alert me when something is violated. We will try that out. What I'll do is we already tried out, you might say, we did a getting started guide, right? And we saw a sample application that I had, a sample policy that I had in getting started guide. But I want to, as a user, I don't know how to, what all policies do I need? How do I know what are the, how do I know that package management execution is a bad idea? I do I need to educate myself with all that and all that things. But you actually don't need to, we have a community like community maintained repository of policy. It's called policy templates. So community maintained policies for all of you such that you can explore here. But going ahead modifying each selector for individual application here is is going to be happy. So we that's where key armor shines again for us. And we are going to run key armor, recommend me policies for my name says Walter. So I'll just go ahead and run this. So it's looking for at that, for the community policies and seeing which ones might be applicable for your particular namespace. Yeah, and also updates it like so like I had last time I guess I asked it for recommendation was maybe six months back. So the current version was 0.2.3. It said me that okay, new version is available. The community has has much more policies now, I will update it for you. And then I'll generate it all for you. And here's, I have a report for that. I, I will reduce a font size a bit so that you can see the full report and zoom in according. But here, here are all the policies that the like that we have in policy templates, but it's tailor made for your application now. So I had two pods in one was the agent injector, and one was the vault itself. And I have all these policy that have CIS command line warming and impair defense. These are based on certain frameworks out there like CIS, MITRE, and NIS. These are all the compliance frameworks out there with security automation help use to like have compliant products. And QBAR can help you achieve compliance using these policies. So we will try one of these policy out, say that I have, I have a question. I have a question. So say I want to apply one of these policies, but I already have my, my application is already running in production, and I don't want to break everything. Can I, can I do this in like an audit mode or like some sort of monitoring mode to see whether whether the policy is going to break my application or how I might need to change my application, and then apply the policy for real? Definitely, definitely. So what I'll do is so there are two, so like essentially QBAR supports three operations that you can provide in our policy. One is allow, one is audit, and one is block. So what we are going to do was block access to certain things. But what I can do is I can run it in, keep it as audit policy and monitor if something is happening. So let's do that right now. So this is not allowing rights inside. We already see involved that ETC password was accessed by world, but that's a security sensitive file system. No one should be able to modify it. And so what we are going to do is we are going to just allow anyone to modify anything inside ETC directory. What, what's the use case for allow? Why would you do allow? Allow? Yeah, I, I, I try to sneak that, I try to sneak out of that. And no, that's fine. But because that's the next part that I would have done after we would have tried that out. Okay. You can get to it when you get to it. You, we can have some suspense. You don't have to answer me right now. Okay. I like that. Yeah. So no spoilers yet. Okay. Let's say that. So allow is very powerful. And it's difficult to tame as a, but I'll keep all the spoilers once we try out the, these basic recommendation policies. Okay. So that's good. So my policy was right in ETC. Right. And I have this policy and I will edit it for you. We had block set here. What I will do is I'll audit it first so that we know you, as you said, it might break my system. So I will just set it to audit here. And QPC till apply the policy itself is live and exit. Just a quick point because we say, so as an enforcement works, even if you disable the visibility, right? We talked about disabling visibility just a while back. But because we are going into audit mode now, audit us to enable visibility. We tried to audit because audit is something, which is not like that's happening. That's a Q bar more feature. It's not something which is happening in line. So we need the visibility set to prove when we want audit to work. So I'll enable file and network back again so that the audit policy itself starts working. And CD at C LS. And what I will do is I will create a file now here. Okay. That's weird. I guess I did not change it to block change it to audit. So that permission was just denied because you have a policy in place that a file can't be created in that namespace. Yeah. Okay. We'll let's check the policy. We do have it in audit mode is a vault maybe that while you figure that out, I'm going to answer Luca's question. Maybe you can back me up. What are the best practices for managing managing sensitive application information in a containerized environment in a standard way using vault and cube armor correctly. For example, if you had to deploy a spring boot application via CI CD pipeline on Kubernetes, how we have it, it's longer. How would you manage his application properties with vault? For example, pass them the necessary information such as user password database connection, etc. Would you use config maps environment variables or anything else? I think that's a problem that's handled by vault itself. Vault provides an an ideal way both by vault as well as Kubernetes has a secrets operator which handles all this complexity for you that secrets are exposed in the correct way inside a particular so that your spring boot application here can access it out. What additionally cube armor can help you do is that wherever secrets are mounted, whether it be environment variables, config maps or volume mounts themselves, cube armor can go ahead and secure them all together. So they work together to secure things. So cube armor is securing at the system call level. And to be specific about a vault integration into Kubernetes, it can be done directly, but there are a couple of Kubernetes projects that help you do it. One is called external secrets operator. And that one will reach out to your vault API and then store the secrets you need as Kubernetes secrets. The other one is called secret store CSI driver. And that one reaches out to your vault and gets the secrets that's needed for a particular application and mounts it to that application pod as a temporary file system volume. And in both cases, cube armor can help secure at the kernel level. What's happening here is I guess something inside the vault configuration is blocking it here. But as we can see here, accesses are being made to vault. So probably a bad and it is working correctly because it's an audit mode. So as you said, if I would have applied this as a block policy, my vault application could have broken down. So using audit mode to monitor that I have zero alerts first is a good way to understand that if my application is going to break or not. My application is still safe because it's an audit mode. So that's where audit and block mode shines. What we will do is since we, so that that was the community part of it. Now we let's dive into how we are, what's the allow thing that I kept secret for so long. I was going to ask you to zoom in on this, on this terminal if you're going to use it. Yeah, I'm just going to do that. Cool. You see, because so many of it was getting executed, it said that, okay, just go, the bindings are now, a lot of binary inside bin is getting executed. So they are aggregated already. But that's something that profile is doing for you. The logs themselves are all as they should be. And you get both kinds of configuration. Thank you. So this is still a full information. And if you want to summarize information, you have it here. So that's where, but let's do one thing. I would want the full information just for some context. So while this loads up, what was allow policy? Allow is something, zero test is something that we have, we have like a lot, it is being talked a lot about in security, especially in cloud native security. Even if I go ahead and apply this policy templates, these are only helping me prevent against known attacks that these frameworks have like analyzed and created policies for me. But what about the unknown attacks? What if someone, something happens, which is not covered by these policies, there are only so many things you can go ahead and keep blocking, auditing, blocking, auditing that life cycle is going to be tiring for you. So that's where the allow policy comes in the picture. You can have zero trust policies created and enforced by Q bar one. So what's a zero trust policy is that I don't trust anything inside my container. What I want to trust is that, okay, I know that, okay, this is getting executed, bin vault, bin SS, bin busybox is getting executed. I'm just going to trust that and anything else that gets executed, no, I don't want it. I'll deny it. Makes sense. So that's, that's allow policy. But as I said, it's hard, right? You never know, okay, bin vault execute, I only saw bin vault earlier. And I created a policy saying that only allow process bin vault and deny everything else. Now, as soon as bin busybox get executed, my application is going to break. And that's where audit and again, audit mode is important. We'll see how we do that. And let's get this policy and I will write I'll, what I'll do is I'll just see into zp.yaml and let's so I already I'm taking this as a base policy so that I have all the required things. Vault zero trust. Action is set to block. I'll change it to allow now. And as I said, let's say that, okay, bin only bin vault is allowed to access nothing else. So match path and I'll say bin, bin vault. This was a point, bin vault was a process that we saw here. So I'll add this and alert untrusted execution detected. So that's my final policy saying that I only trust bin vault. Anything else go ahead and deny it. So cp and so we have this policy I'm going to apply it. And right now it should have broken it, but it will not break because by default, jubarmer keeps them in audit mode. Okay, I, okay, I missed something out. So, okay, read only something specific to file construct processes. So I have the updated policy. So zero trust policy created. And if I go ahead and even executing SS is already something which happened, right? And q, I guess qc, I'll delete the earlier policy. And if I execute LS here, okay, I have audit alerts saying that bin busybox got executed. LS is part of the busybox binary. Bin busybox is being executed. It's something that you should look into that probably your application still the zero trust policy is still not complete because you are receiving alerts, right? So take this and probably go ahead and add it. So I'll go ahead and add bin busybox now into the policy. So is this a normal framework for for profiling an application and monitoring it and then figuring out how to implement zero trust? Yeah, like this is what's the core core process of it. What I'm doing is in a very primitive way. There are tool, there are commercial toolings out there which do it in an automated way. Or what you can do is you can dump all the logs into some database or elastic search lookie and all that and have that entire profiling done and done from there. So a lot of things there. This is because I'm doing it in much more different way because I just wanted to show how at core level this works out. Love it. So now I've added bin busybox and I executed LS and I did not apply it. Configured, yeah. So I did not apply it. Never. So excited. LS and okay, no alerts yet. So LS is an expected outcome now. But let's check if there's anything else I can use which is not busybox here. Everything is busybox. Everything's busybox. But since file operations are allowed, I will be an attacker and copy vault into and rename vault into something else. Let's say test. Okay, I'm permissioned in that to do that already by QBAR. Okay, no, you cannot do that. So I guess I'll just ask you to trust me on this and say that if any other process is executed, it's going to be denied. You're ironically asking us to trust you about zero trust. Okay, no, I have some processes. Yeah. Okay, great. Yeah, show us. LDD, right? LDD got executed and I got the alert saying LDD was executed. But it is something that should not have been executed. So I will go ahead and annotate my name space saying let's set file posture equal to block. Now anything related to processes and files are going to be blocked inside this. And you see LDD is permissioned in it, but all my processes like been busybox and vault are working as expected and my UI would be working as soon. So that's how you have that entire zero trust policy lifecycle management for you that you can set it in audit mode per name space and change it to block mode when you see be confident that this is going to work. Awesome. Can you believe an hour? I was going to say an hour is passed already. It's flown by. I feel like I could do this all day with you, but we only have this hour. And I think maybe you covered a small fraction of what you hope to cover, but we just asked you a lot of questions. So I hope that you come back on soon and do some more with us because this was super, super fun. Is there anything you want to say in closing before I read the goodbye script? So, but like I already covered most of it and now I'll just add one pointer that I only did it for processes, but you can do this for file systems like how we were talking about entire securing the volume mounts and all that you can define which you can find great it at level that I can say which mount is going to be used by which. And I'll just show the final policy there so that you have an idea what it would look like. I'll say that okay, vault is going to be, vault is a volume mount inside vault. Lot of vaults inside. So inside vault, I'm going to block accesses to the vault itself and allow this vault access to only bin vault binary. We already see bin vault binary. And similarly, and this is something which we already discovered like bin busybox and bin vault. So this would have been the next part where we would have secured the volume mount itself. But now we already know the core concept of it that how to go ahead and generate this policy. And you have that entire lifecycle management of audit and blocks. So to help you out. So yep, that's all. We covered a lot of it and hope you all use cube armor. You start using cube armor to have armor up your clusters now. Where should folks go if they have questions after watching this video? Sure. So cube armor has its own slack. You can join us. This is cube armor.io. We have a nice go for it with a shield to help you project it. And you can join us, go ahead, use on GitHub. We have discussions. You can go ahead and ask questions there, create issues. That's one way of doing it. Or you can join our slack and ask us questions there. We have an active community there helping out everyone who is wanting to know how to how to start things here. So yeah, awesome. I appreciate you so much. And I appreciate all y'all in chat for asking great questions. Thank you for that. So thanks for teaching us about cube armor. And here at cloud native live, we bring you the latest in cloud native code on Tuesdays and Wednesdays. So at the same time of day. So thanks to you for being here. Thanks to Brune for presenting. Thanks to everyone who watched the recording. And please join us again next week.