 Hello, everyone. Thank you for joining me on my session on our marine cloud native workloads with LSM superpowers. I'm Baron Acharya. I am a software engineering intern at Aquinox, working on open source cloud security solutions with the various advantages that containers provide organizations worldwide are rapidly adopting containers in one form or another. We have applications running in containers across nodes across clusters with the rise of adoption of modern cloud native infrastructure. So has risen the cyber attacks on the same containers are not really protected by default as the various tools for security into place provides perimeter security at the host or the network and not necessarily the workload itself. We have security focused node images, hardened kernels, strict configurations, but there's a need to enforce security at container level. Hence the need for container security. We usually have some static analyzers in place, but they only let us know about what known vulnerabilities are containers are vulnerable to. But recent vulnerabilities like log4j and pwnkit could not have been detected statically. That's like zero days malware meant to evade analyzers and privileges escalation attempts manifest at runtime and cannot be detected statically. So how do we deal with vulnerabilities that manifest at runtime? We deal with them at runtime. So container runtime security. We actively monitor what's happening with our containers at runtime and try to detect for the malicious intent. But how do we go about this? To better understand, let's go through an analogy. Let's imagine containers as a private apartment. A private apartment with John and Jane each having their own rooms. They have a kitchen and some maintenance staff around John and Jane value their privacy. Jane doesn't like if John pokes into her room. Neither does John if Jane does the same. But they both do like to hang around in the kitchen for functioning. They generally don't want the maintenance staff to roam around in the apartment. But in case something breaks, like the shower is broken, the gas is not working or some electrical appliances malfunctioning, they need the maintenance staff around. Let's take this analogy back to containers. We have now a WordPress app. MySQL app. They each have their own configurations and directory. They need shared libraries to function. And then we have cat and PS and other maintenance binaries that help us around in case something is broken. But like I mentioned, Apache WordPress should only be able to access the WordPress config and MySQL app should only be able to access the MySQL directory. But what if they try to access something else themselves or the files are being accessed by someone else specific other than the binaries? We know that something malicious is happening. We cannot really restrict John and Jane to tell them what to do, violating their human rights. But we can restrict our containers to tell them what to do and prevent them from performing actions like these. We have established the fact that containers are not a black box. We need to profile and understand the entities inside the containers as well. Even if we imagine whatever is happening inside containers won't have anything outside, our containers have access to the outside configurations as well. Containers have to network with the world and the other peer network containers. They access capabilities from the kernel. They have access to host defined mount points as well. Now, what if a malicious binary somehow enters the container? It would gain access to all these mount points, network and capabilities. Hence, there is a need to understand the entities inside our container. We need to profile our containers beforehand and have fine grained access control such that even if a malicious binary sneaks in somehow, it won't be able to do anything. So let's see what we need to enforce this fine grained access control at runtime. There are two ways to go about it. Once we detect the malicious activity, we alert the activity and take action on the activity after we receive the alert. Or as soon as the malicious activity is detected, the action is taken by a security agent already and then we receive the alert for further investigation. The prior method is called async enforcement. Essentially, we restart detection engine, receives the kernel event, it alerts us and we then decide to terminate the activity. But once this entire cycle happens, the malicious activity would have already sneaked into deeper areas of our container. So the other method is inline enforcement. As soon as we detect that there's a malicious activity, we deny it then and there itself. And then we receive the alert for further investigation that what might have triggered this. So what kernel primitives do we have to enforce this inline, inline enforcement of systems? We have LDP load, we have Seccom that is secure computing and we have Linux security modules. Let's take a look at LDP load first. LDP load is a user space control. It overrides dynamic libraries and applications go through these libraries. So we have fine grained access control over them. But attackers can invoke syscalls without going through the dynamic libraries at all. Suppose go application statically embed their binaries with the library files. So they never need to go through the modified library at all. Next is Seccom. Seccom lets us sandbox our containers with various system calls. What system calls is accessible to them. It is natively integrated with Kubernetes, Docker and other container types. But the issue here is that dynamic enforcement is not possible and there are limited filtering options. We cannot decide inside a container what application needs to access what system syscalls. It treats container as a single entity. Finally, we have Linux security modules. Linux security modules include App Armor, SL Linux, BPF LSM. These are mature ecosystem of books with decades of development ongoing on them. These do not suffer from a kernel race condition called time of check and time of use. But they're not integrated with the modern context and there's a steep learning curve associated with the policy language of these LSM. So how do we integrate these security primitives with our modern cloud native workloads? We have port security context. Port security context defines the privilege and access control settings for a port or a container. Security context settings include but are not limited to Seccom, capabilities, App Armor and SL Linux. Let's take a look at Seccom, capabilities and SL Linux first. The issue with them is that they treat container as a single entity and apply the security posture across the container. But as we discussed earlier, we needed that fine-grained control over the entities inside our container. Then we have App Armor. App Armor provides us with the fine-grained access control. We need it and it is well integrated with the modern ecosystem. We can apply annotations to link it to our App Armor profile. But then we have a learning curve associated with the App Armor policy language. Also, App Armor is not readily available on all the systems. A bigger issue is here that App Armor uses ordered logs to generate the telemetry events. And these telemetry events do not have enough context. We never know which container triggered this alert and there's a lot of overhead required to realize that context. So defining our problem statement, we need access control on container entities. We need a declarative way to manage policies for these access control. We need inline policy enforcement and, importantly, we need the telemetry events to have the context. Introducing Q-Barmour. Q-Barmour is a cloud-native runtime security enforcement system that restricts the behavior such as process execution, file accesses, and network operations of containers and nodes at the system level. Let's see how Q-Barmour does this under the hood. We have Q-Barmour listening to various probes in the kernel, listening to the events and acting as a detection engine. It sends out the log and telemetry to the users for them to analyze. Q-Barmour then listens to the security policies that user sends and converts them into the kernel space rules to have inline access control inside containers. Q-Barmour utilizes EVPF for observability. EVPF lets us extend kernel in a safe and performant way. It has established itself in the cloud-native space and is a go-to solution for observability these days. And then Q-Barmour utilizes LSMs for enforcement. When Q-Barmour receives the policies, it converts them into app-armour profiles, BPF LSM bytecode, or SLNX labels, which are enforced and are responsible for the decision-making inside kernel. As we established earlier, LSMs are a proven technology for access control and paired with EVPF and leveraging information from container APIs, we have a holistic solution for container runtime security system. Let's check out Q-Barmour in action. Some context about what we are going to see in the demo. We will have an Apache WordPress MySQL application, in which we will showcase how we can allow specific entities and deny the rest, how we can restrict our application access to certain kernel calls. So also, Kubernetes auto mount service account tokens, even if we like set it or set auto mount to false, which will might need access to those tokens. So we will showcase how we can lock down access to a particular application inside our container to that service account token. And finally, we will showcase how we can secure our port data, port data including MySQL directory files or configuration files. So let's get on to the demo. We already have a Kubernetes cluster running with Q-Barmour installed and our MySQL WordPress application installed as well. Let's see how Q-Barmour is configured to run here. We have a two node setup with container optimized OS and Q-Barmour is configured to run VPF LSM. We are armoring up two of our workloads here, MySQL and WordPress and currently they have no policies applied to them. Let's start with applying some policies. So this is how a sample Q-Barmour policy looks like. We define it as Q-Barmour policy. We can specify which selector labeled it should match up onto. We have file network processes and other entities that we can specify and finally our action which can be allowed to specify by listing policies, blocked to specify blacklisting and we can audit things as well. In this policy, we are trying to block access to anything inside the MySQL directory. Let's try applying this policy. At the same time, let's try to monitor our alerts and telemetries as well. Now that we have applied our policy, let's execute into our MySQL port. As soon as we try to write a file to this directory, the batch told us that permission is denied and a telemetry event was generated for us. The telemetry event included what was the host, what namespace, what pod, what was the container ID, container image. All the relevant information that would not have been available in our audit logs is available here and it also includes which was the policy that caused it to be denied and what was the process name, file name. All the information is available to us which can be useful for debugging and tracking down how and where this malicious intent happened. So similarly, let's try to secure our service account tokens. We have this policy where we deny access to service account tokens in general but allow access to this specific service account token by cat. So now if we move into our WordPress application, if I try to access my service account token, I can access the service account token since I mentioned that I should allow access to the service account token but what if some other binary in this specific container tries to access the service account token? We never applied the policy. So now cat can still access the service account token but what if more tries to access the service account token? We see the permission was denied and a telemetry event was generated to us that more tried to access the service account token. Please look into it. One more thing I mentioned in our analogies that maintenance tools ideally maintenance tools shouldn't be allowed to execute in the production containers but it still might need in a developer environment to debug things. So let's try to restrict access to the so now this policy includes that we should deny access to apt and apt-get to be executed. Now if we try to execute either apt or apt-get, they are denied and we have the telemetry data with us for that. A final use case was that how we can restrict network accesses down to the process level. So if we try to apply this is an allow policy where we have mentioned what what processes can draw be accessed by what processes can TCP be accessed by and what processes can be accessed can access UDP. This is generated after profiling our WordPress application. Now that we have applied this we see that curl is not part of this policy. So if we try to access anything using curl, any network using curl, we will see that the permission will be denied. We can see that we could not resolve the host for curl and we had a relevant denial to this event. This might have been just due to UDP. So let's try to access TCP as well. We couldn't connect to the server. So we have successfully like limited down, which what all can access TCP and UDP. So to summarize, Q-barmer provides you with access control on access control on container entities. We provide you with a declarative way to manage policies for access control. We have inline policy enforcement and we have telemetry data with context. Thank you. You can reach out to us on Q-barmer Slack. You can check out more details about Q-barmer on Q-barmer.io. I'm available on Twitter as demon1024 and you can know more about me at baron.cc.