 Hi and welcome to this CNCF Recorded Webinar. You know everybody's been talking about shifting security left. I'm not here to dispute that trend and its importance. Like any security guy, I'm here today to remind you that there are threats on the right that needs to be acknowledged. First, let me introduce myself. My name is Amit Chandel. I am a security researcher here at Armo. What I'll be showing you here is a proof of concept we did at Armo for runtime threat detection in Kubernetes. Code that Armo has recently donated to the Cubescape project with the intention that the community will integrate it into Cubescape. I am a Cubescape contributor myself. So the agenda for today is to introduce you to the idea of KDR, as well as an overview of Cubescape KDR, which we developed here at Armo. And finally, we will see the KDR in action in a live demo. So let's talk a bit about the gap in cloud-native security. Securing cloud-native workloads has become synonymous with passbo and vulnerability scanning, which I'm sure all of you have heard of like a million time before. Everything we do in our day-to-day scanning, patching, et cetera, is necessary and important. However, history has proven that regardless of scanning, glomerating, and patching, it only takes one vulnerability or misconfiguration slipping through, which will lead you open to attack and its aftermath. To bridge the gap, we need a good understanding of what is going on with our workloads. The EVPX stream enables a baseline recording of normal application behavior, as well as continuous inspection of it. Essentially, we identify anomalous behavior by comparing the live data stream with the baseline. In parallel, the behavioral inspection enables real-time detection of threats using an advanced rule engine that analyzes the EVPX data stream. Lastly, signature-based scans search for the distinguishing signatures of malware based on a database of non-malicious programs. Please note that the use of signature here is not an actual signature you would find in an S-bomb. It's a fingerprint associated with a malicious program. Before we jump into the demo, I'll walk you through how we put this all together. Here is an overview of the architecture of Cubescape KDR. We have our recording system, which streams events to our detection system, as well as creating application profiles to have a baseline recording of the workloads. This can later be used in order to find anomalous in the behavior of the application. Then, we have our detection system, which is packed with a rule engine that inspects and validates the EVPX data stream coming from the recording system. You can bind rules to workloads, and by that, letting the engine know what to enforce. Finally, we export all the alerts into a unified view. Kubernetes-detecting of the response is functionality developed by Armo, which will be contributed to the Cubescape project in the near future. Let me walk you through the repo currently on Armo to familiarize you with the functionality. Here is the KDR repository under ArmoSec. Again, the near future is going to be under Cubescape, but for now, it's over here, and I'm more than inviting you to come to this repo and met around with the KDR yourself. In this demo, we're going to follow our demo walkthrough, which is under the demo folder in the Cubescape repository. And in this demo, we're going to see the KDR capabilities, most of them, not all of them, and let me just hop right in. So, I have a Kubernetes cluster installed, and I also installed Cubescape inside of it and configured the alert manager, where we can see the Cubescape alerts, and in the alert manager, we can see the alerts is aggregated by the namespace in which they occurred. And in this demo, we're going to basically attack a web application, which is the IP ping tool, and this web application basically allows you to ping an IP address of your choice. So, let's see if we can ping your IP. Okay, great, we can see the output, and it works as expected. Following this read me, you can see how to install everything. It's very simple, simple installation, but overall the installs for you. So, in order to start the demo, first, we need code execution. Okay, so let's try to attack this web application, eventually get the code execution that we want and execute a Kubernetes client inside a pod in order to see what the KGL detects. So, looking at our web application, like every good attacker, the first thing that comes in mind is a command injection attack, right? So, if the web application behind the scene doesn't validate the input, an attacker can abuse it in order to execute code. So, we assume there is like a ping dash C behind the scene, and then the input of the IP address. So, let's see if we can inject a malicious command and execute code. Let's try to do an Alice command over here and see if it works. And great, we see that we have successfully executed the command injection attack, and we have two files. Awesome, let's continue with the attack. So, what every attacker will do when attacking a Kubernetes cluster is once I'm inside a pod, if I want to move a lot laterally and talk to the API server, I need to use the service account token, which is mounted by default in all the Kubernetes pods. So, here I have the command, which basically reads the token. Let's try to execute it. And hit the ping button and see what we get. Okay, we can see we have successfully read the service account token. So, stopping with the attack for now, we have code execution, and we have the service account token, which is great from an attacker perspective. Let's see what the KDR detected. So, looking in the update manager, we can hit the refresh button and we can see we have four new alerts. Let's look what we got. So, we can see QCOP rule violated. We can see the rule name. For example, unexpected service account token access. Okay, QCOP, the KDR, which is QCOP for now, detected that this process access the service account token access, and it is not in the baseline behavior of the application. So, we also have a recorded baseline of the web application, and we know that the application by default doesn't touch the service account token. So, if suddenly the application touches the service account token, it means something bad is going on. We can also see some metadata such as the pod name, the host, the node, and all the needed information in order for us to further analyze this event. If I click on the info, we can see that we can apply a patch to the application profile and if this is a false positive, you can add it into the application profile in order for it to disappear. So, let's see what else do we have. We also have unexpected process launch, which is a process that usually it's not in the baseline again and shouldn't be launching. And we can see unexpected file access and another unexpected process launch. Okay, so we can see already that the KDR pretty much is on to us, right? But let's continue further on and see what else we get. So, the next step in the attack will be to download the Kubernetes client. Okay, we'll use kubectl for this demo and execute the kubectl inside the pod in order to talk to the API server and get the pods. So, the first thing we need to do is to understand what is the architecture of the underlying node in order to download the correct kubectl into the machine. So, I'm going to execute this command, which basically runs a uname and tell me the underlying architecture. So, we can see it's AMD64, okay. So, we'll take this information and now we can download the kubectl. So, I'm going to take the command from the demo with me and I'm just going to modify this part to AMD64 because this is the architecture that we want. Let's hit the ping button and this may take some time because it's downloading the file and it can be quite heavy but we can see the return status is zero and we've successfully downloaded kubectl into the machine. We can even verify it by again doing an LS and hit the ping and see what we get. And we can see we have kubectl in the current working directory. Awesome. So, next we need to give it execute permissions, right. So, let's quickly give it an execute permissions Awesome. And the last thing we want to do is to get the code in the console. So, for that, we're going to execute kubectl. We're going to give it the API server. We're going to tell you to ignore TLS and we're going to supply the service account token which is again mounted inside the pod and we're going to specify get pods which is what we want. So, let's take this command. Go back to the web application and try to get the pods great. We can see we have successfully executed code and executed the Kubernetes client and got the pods. This is an awesome starting point from an attack perspective that from now on I can continue to move on inside the cluster and maybe eventually even reach out to the node. So, let's stop right here, look in the alert manager to see what we get. So, we already see quite a bunch of alert which is obvious because we are doing a lot of stuff that aren't supposed to be happening inside that application. And most of the alerts we've already seen like the unexpected service account token access, the unexpected process launched and again all this metadata that is important to further analyze this incident. And I want to show you a new alert that we got which is Kubernetes client executed. So, QtScadeKDR also detects when a Kubernetes client is executed inside the pod which again is not in the baseline behavior of the application. Okay, and this is a very severe incident because most of the time if it's not in the baseline application behavior, it means that an attacker is inside the pod trying to talk to the API server. Again, we can see further information, what happened, which binary was executed and all the needed data in order to further investigate it. So, this was the first part of the demo where we saw how we can execute code on the web application, how the KDR detects every single anomaly that happened and have also behavioral inspection rules to detect malicious stuff. And the next part of the demo will be to show you how the KDR detects follow smaller attacks. So, those attacks are quite common within threat actors. And for this demo, what we're going to do is use the Azure descriptor, which you can find in this repository. And it's a simple Linux L front end descriptor. And behind the scene, how does file a smaller attack work? So, using the memathlete create syscode, an attacker can actually memory map and malicious program into memory. And that way it doesn't reside in the disk. And then basically called exit on that file descriptor that holds the malware in the memory. So, this attack is very common within threat actors because it's quite a step. So, let me quickly install this manifest and get back to you. Okay, so I've installed this manifest which contains a pod with a malicious application, which is basically a file a smaller. And if we take a look in the alert manager again, you can see that now we have 22 alerts. And if I leave the refresh button, we can see we now have 23 alerts. And let's try to find the other. And we can see we have exec from malicious source. Okay, we can see the comis3, which is already a bit weird. And if I'll press on the info button to see further description, we can see an exec code accrued on slash procs slash self fd3, which is obviously a file a smaller attack because no normal application does this behavior. So, we can see the KDR also detects those types of attacks. Great, let's move on to the next part. We also have in the demo a one more capability which is a malicious image, which is packed with malware and crypto miners and web shells. And we integrated with Clamav, which is an open source antivirus developed by Cisco in order to also bring in the antivirus world into the world of KDR. So, we can also scan the node with Clamav, which was a special signature database that we created here. And eventually enrich the signatures with the Kubernetes content. So for example, if we'll take a look here in the image, we saw that Clamav detected a coin miner, so a crypto miner basically and we enrich that data with the size hash and the workload and everything that needs to be further analyzed. We also added the feature of is part of the image. So, if this is part of the base image or it is a new drifted binary. So, this is an other capability of the KDR and there are more capabilities that are not inside the demo, but they exist. So, I more than invite you to play with it yourself. So, let's hop back to the presentation. This was the demo and I hope you enjoyed it. As I mentioned earlier, shifting security left by scanning early and often in the software development lifecycle is important as ever. However, runtime security will help you identify the thing that got through the scanners and still causes a threat. In addition, adding anomaly-based scans behavioral inspection and signature-based scans to your toolbox with Kubernetes detection response is a necessary thing. The code to Cube State KDR is available on GitHub. Give it a spin. If you're interested in open source Kubernetes security and would like to follow or join the Cube State community, you can check it out on Twitter or GitHub. There are also Cube State channels on the CNCF Slack. Thank you for watching this webinar and I hope you learned something new.