 Welcome to uncovering the sophisticated Kubernetes attack in real time. Kubernetes has become the de facto cloud operating system, and every day, more and more critical applications are containerized and moved to Kubernetes. This means Kubernetes is quickly becoming a crucial component to secure. Today, we as defenders are in the dark because we lack the observability needed to detect or respond to attacks on our workloads. Today, we'll show you how we can build out a detection program to uncover a sophisticated attack against a Kubernetes cluster. Hi, I'm Jed Salazar, Platform Security at Tesla. I like mountains and distributed systems. Hey, my name is Natalia, and I'm a security engineer at Isabel and I really like traveling and flying on planes. Observability in distributed systems is a crucial element in detecting issues and making changes to improve the stability and security of the system. So for SRE, we use things like Prometheus to actually come up with metrics to define how we can actually improve the reliability of a service. What we're advocating for is that security takes the same approach, being data driven and using data to drive forward security improvements within our system. So a lot of security today practices, they really focus on the pre-fail and post-fail approach to security. Pre-fail tries to predict the necessary protections that a system is going to require based on threat modeling or an assumption of risk. Post-fail ensures that you have the necessary logs for intrusion detection to be notified if you get compromised despite those protections. What we're advocating for is that we actually change that paradigm and we move from a pre-fail and post-fail to a pre-data and post-data paradigm. Data allows us to continually measure that our hardening and security configurations are suitable to protect against real-world threats that are detected by observability within our environment. With observability, we will open your eyes to the reality of the threats in your environment. So how do we secure this? This is the CNCF landscape, basically the set of tools and systems that make up the Kubernetes ecosystem. I like to call this the CNCF threat landscape. If you're tasked with securing Kubernetes, you might look at this landscape and become overwhelmed. It takes a lot of resources and time to successfully build an operated cloud-native environment, let alone secure it. However, different components of your stack have different levels of risk and prioritizing your focus where there's the least trust and the most risk is a good way to get started. For example, it's reasonable to assume that the Kubernetes scheduler is less of an attack vector for your cluster than an internet-exposed pod that handles untrusted user input. But what if I'm following CIS benchmarks and scanning my container images? Is that enough? Don't do this. Not to say that vulnerability scanning isn't important, you should definitely not deploy vulnerabilities in the problem. There are a number of different protections that we can implement within a system. Admission controls like OPA can define security standards and enforce policy for objects before they're persisted to LCD. You can protect your containers by removing dangerous debugging tools like interactive shells, package managers, compilers, and unneeded runtimes with distro-less base images. Vulnerability scanning should be integrated to a container image build stage and runtime sandboxing such as Setcomp, AppArmor, GVisor, and Firecracker can limit the potential of privilege escalation or container escapes. Still, each of these tools offer varying degrees of locking things down. So how can we verify that these protections give us sufficient protections from the real world attacks that we're actually going to see within our Kubernetes environment? So the way that we can enable that is by making security look more like SRE. So if I were to ask, how secure are you today? Could you provide a quantifiable answer to that question? Are you measuring your assumptions on how secure you are? We need to make security more like SRE by sharing a lot of the same SRE principles, like being data-driven and measuring the data that we actually receive within an environment. In this new paradigm, where security shares SRE practices, security collects data, observes the feedback, and then makes continuous data-driven decisions to improve the security of our systems. Are you scanning your images? Do you apply network policy? Are you encrypting your data at rest? How are you measuring the efficacy of these protections? And then can we detect if the service running in our environment is outside of our security policy? The way that we do that is we trust but verify. This is an old security adage. So you place a lot of trust in your systems at deployment time. For example, if you build up a container image, you should have this container image represented as infrastructure as code. So the configs that we deploy are pretty much well understood. For example, when the Kubelet launches our workload, we can assert that the configs have been statically analyzed, hardening measures have been applied, and we can basically account for every bit in the deployment. However, over time, entropy reduces that trust. For example, an internet-exposed workload gets attacked. An SRE runs kubectl exec and installs some debugging tool and fails to clean up after themselves. Over time, this entropy becomes a problem and observability verifies the trust that we place in a system throughout its entire lifecycle until that trust is broken by some type of compromise. That's what we call detection. So we wanna test our security assumptions and measure whether the protections that we're applying are suitable for the threats that we actually detect within our environment. So DevSecOps integrates security into DevOps and DevOps writes tests for code. For example, unit testing, which is ensuring that the behavior of a system is understood. Observability continually validates this behavior. If you have security controls applied, for example, a network policy, do you have the observability, for example, via network flow logs or DNS queries to verify that that network policy is suitable for the type of protections that are needed? Capturing system calls and file system access from a container can validate the efficacy of our SecComp and app armor profiles. On the left, we actually have unit testing and on the right, we have the observability that can test and verify that unit test. These measurements can continually inform us whether the protections that we're utilizing are sufficient based on the attacks that we've identified in our environment or if we need better protections. So why BPF? BPF is really the optimal tool to provide this observability in Kubernetes. Because a pod is just a set of Linux processes running in the context of a kernel namespace, the pod makes system calls and implements CNI network functions within the kernel where eBPF resides. eBPF acts as a virtual machine in the kernel with a generic set of 64-bit registers and eBPF programs that are attached to kernel code paths. This means eBPF can intercept system call events and react in real time with fully programmable logic. Using eBPF, we can build out a detection program and the best way to get started is to work with your red team or just attack your systems yourself. Even as a beginner, you can start to collect the knowledge of tools and techniques and use eBPF to detect these attacks and configure alerting options to detect attacks in real time. But the best way to detect is first knowing how to attack. So in this talk for detection, we are going to use Celium and show how it uses eBPF to very efficiently monitor our malicious network and process behavior inside of a Kubernetes workload and give you identity of our visibility into those malicious behaviors. So a little bit of background, Celium deploys as a demon set inside of a Kubernetes environment. This means that there is a Celium agent running on each one of your Kubernetes worker nodes. These agents are communicating with the Kubernetes API server to understand pod identities, network policies, services, and so on. And based on those identities of each one of the workloads deployed inside the Kubernetes environment, Celium generates a customized and highly efficient eBPF program to do connectivity, observability, and security for those workloads. Celium is able to both enforce what behavior happened inside of a Linux system as well as very efficiently observed that behavior. It can collect and filter out observability data directly in the kernel and export it to user space as JSON events where it can be sent to many external system, for example, a CM to do investigation and alerting. In this talk, we are focusing on the security use cases and security teams and how they would like to consume those data. So this is how our JSON event looks like, which was exported to user space by an eBPF program. You can see the Kubernetes identity of our information like pods, containers, labels, and the OS level process visibility information like binary arguments, Docker ID, and the PID. For simplicity, we will just use some Python scripts directly consuming the JSON events that would be otherwise sent to an external system like CM system, like Elasticsearch or Splunk. In this talk, we are going to see two attacks both happening on a live GKE cluster. In the first one, we are going to reach the host file system and break out from the container. And on the second one, we are going to download some sensitive data from an external S3 packet. And for detection, we are going to use Selim and the power of eBPF. So from an environment perspective, let's assume that chat has already accessed a GKE cluster. That's what we can see on the right terminal. And on the same cluster, I have Selim setup and running, which monitors all the Kubernetes workloads and provide observability, connectivity, and security for them. For simplicity, I'm going to use a Python script which would parse all the JSON events coming from the kernel and export it to user space via an eBPF program, and then which would look for malicious behaviors. Now the question is, can JED get access to the host file system? So let's find it out. So the easiest thing to do would be to spin up a pod with the privilege equals true flag. Note that Kubernetes allows us by default and this disables all of the sandboxing mechanisms inside of a pod. Okay, let's apply to the privilege pod spec. Now we have a privilege pod which gives us the same permissions as root on the node. Why is it so powerful? Privilege enables the capabilities, all of the capabilities which the pod, as you can see has started. This includes Capsis Admin, which is basically the new root in Linux. And on the left terminal, we can see that the generated eBPF program was able to pick up the new container that has started. We can also see the privileges that include Capsis Admin. So now let's try to queue Cuddle exec into that pod. Let's see if we have access to the host's file system. I'm gonna run cat, etsy, or sorry, cat proc partitions and see if we have access to the resources on that host. Here we see that the partitions listed are actually partitions from the host. And on the left terminal, the generated eBPF program was able to pick up the sensitive command detected. So now let's mount the host's file system inside the container, breaking the boundary between the container and the host. This is also known as a container escape. And on the left side, we can actually see the namespace, the source, but then the container. So now that we have access to the host's file system, we can just directly CD into that mount. This gives us direct root privileges over that file system, which is effectively game over. And we have full root access to the file system, which means that we can mutate things like, the roots, bash, trc, we can change, we can add users, we can really do whatever we want at this point. So the container escape is not the only thing that you want to monitor for. Sadly, most of the people are only looking for this. In the second demo, we will detect an attack where it is able to download sensitive data from an external S3 bucket. From an environment perspective, the setup is the same. So let's assume that Jet has access to a GKE cluster, and from a workload perspective, he's currently seeing running application with multiple services like Joe posting and Recreater, and with multiple data databases like Elasticsearch and ZooKeeper. So let's assume that he has already compromised the core API service and got into the core API pod. That's what we can see on the right arm now. So now he can run arbitrary commands to discover the environment, do any lateral movements, connect to external services, and get access to any sensitive data. And on the left terminal, I'm going to run a custom Python script which processes the JSON events exported by the ABPF program directed from the kernel, and which monitors all the malicious behavior that he will do. So now that I've compromised the pod, I'm gonna run the end command to see if there are any credentials or secrets mounted in the pod that I can take advantage of. Excellent. We can see that there's an AWS access key and secret key as well as an S3 bucket name that I can use. So ideally, secrets are mounted in a volume inside the container to avoid having to restart the container if a secret is rotated. But in this case, I'm able to take advantage of the fact that they're environment variables. And on the left terminal, I can see that the sensitive command got picked up by the generated ABPF program. I can see the namespace, the source, but then the container. So let's access the S3 bucket and list the contents. So here we see three folders. What's this super secret PDF? And on the left side, I can see that the sensitive command get picked up by the generated ABPF program. So let's download this super secret file. And we can see this was successful. Excellent. And on the left terminal, I can see that the sensitive command got picked up by the ABPF program with all the namespace, the source, but then the container. And then the network socket that was initiated was identified as well. So now that we've downloaded the super secret file, let's inspect the contents and see what it contains. Ah, okay. I don't know what to say there. I'm just like, yeah. All right. But how do we translate system calls and network flows into an attack? Not everyone has the detection skills to identify an attack based on these signals alone. The image on the right comes from a blog post called the Pyramid of Pain. And it identifies the value of detection data progressively from trivial to collect and the relative value that that data provides. In a regular environment, at the bottom you have a hash of a file. If you flip a single bit in a file, its hash value completely changes. So its value is minimal. However, in the environment of Kubernetes, we can start to deploy the concept of immutability. Immutability enables us to declare patterns to understand the behavior of an object. If you need to change an object, you don't change it in place or patch it, you replace it all together. So we can use this immutability to unlock the detection of tactics, techniques and procedures or TTPs. For example, if there's an execution of a binary that's not part of the container image, we immediately know it's suspicious. Immutability really brings a lot of benefits to security. All the binaries in a container should be packaged into a container image. Using a hardened base image like Distralis, it makes it easy to identify all the binaries in the container and those that should be running. Downloading binaries in a production container for purposes other than debugging is an anti-pattern. So if there's a execution of a binary inside of that container image that isn't part of the container image, you immediately know that it's suspect. What this further allows is those hash values that are trivial in value actually become very critical inside of an immutable environment because we don't make any changes to any of the files inside of that environment. So a change to a hash in the case of immutability might actually become a tactic, technique or procedure. So wrapping up, just wanted to kind of go over some of the things that we talked about through this talk. So first is that to measure the efficacy of the security in your environment, that requires observability. So the key thing here is starting to collect the right data so that you're actually gonna be positioned to be able to detect the sophisticated attack that occurs within your environment. Today we've talked about a couple of kind of simple attacks, but the key here is that we're actually measuring that data to be able to detect those things. Oh, and by the way, Cilium is going to open source the EBPF observability tool that was used today for the demos. Awesome. So when we actually start with this observability, we can detect the types of behavior that we expect in a system. When things go wrong, for example, whether we have a compromise in the environment, that's called detection. And really, if you don't collect this data, if you're actually lacking this data, hope is your best strategy. And if I can take one thing away from the SRE book, it's that hope is not a good strategy. And lastly, we just wanted to give a huge shout out to everyone on the slide. Everyone here has given us either feedback directly or helped us somehow throughout our Kubernetes security learning journey. This talk wouldn't be what it is without these people. So thank you very much. Thank you.