 Hey, all. Good morning, good afternoon, good evening. Hopefully, you're going to all see my screen right now. So welcome to our talk today about acting misconfigured Kubernetes. My name is Oka Mara. I'm a senior development team lead at Ericsnick. For those of you who are not familiar with Snick, Snick is a developer-first security company that helps developers use open source. And as part of it, we offer security solutions for cloud-native applications, including open source dependency scanning, container scanning. And lately, we also released a new official product for Inter-State risk code scanning. Before Snick, I worked in the government of Israel and that's it. So what exactly are we going to talk about today? So we're going to do a really short kind of intro, but a really short one about the potential problems and the cloud attack vectors just as a background. Then we will continue to talk about three different issues as part of the Kubernetes configuration. We will not do a huge grill down into each one of them, but we will just show how easy it is to fix different potential problems that you can have on production. So we'll start by talking about security context, and then we'll talk about resources limitation. And yeah, let's start. So again, like we have the clouded, like multiple attack vectors in the cloud environment. So why is it so complicated? And basically I wanted to start with an example. Probably you're all familiar with this one, the attack on capital one. I think it was almost like one and a half year ago. And it was a really, really big one, expected one million users only in the US and around like one million dollars in costs. And I think it's a really, really good example to start with because it's kind of an example for misconfiguration that was part of the attack, but also a good example for the amount of potential as security pitfalls that we all should be aware of. And for those of you who are not too familiar with this attack, so there were like multiple steps that helped the attacker. Part of them were like misconfigured load balancer, and we also add like a server side request for jerry vulnerability like SSRA vulnerability in the application. And in addition, we also add like an over permissive S3 bucket on AWS. So all of those together basically helped the attacker while doing the attack. So now let's talk a little bit, just a little bit about the ownership of developers those days. And basically what does my service contain? So obviously we, as we all know, we have the source code in my application. So you can see right now an example for my Python app. And of course I need to make sure that all of my code is secured and I'm writing like good code. And then we have third-party dependency. So in this example, I just installed a few packages as part of my requirements TXT. And of course that I need to make sure that each one of those dependencies is secured as well. And then we're taking the application and we wrap it with our Docker file and in order to build Docker image, in this case, we just took the Python 3 base image and we run some update and install commands and then we have our own Docker image for the application. And of course that as part of this Docker file and the Docker image, we need to make sure that all the OS dependencies are secured as well. Exactly like we need to make sure that the dependencies of the application are secured. And then we have the platform, right? We need to make sure that all the infrastructure as code files are secured. In this case, we have just an example for a telephone file and again, just another thing to make sure and to take care of. And last but not least, what we're gonna talk about today, Kubernetes file. In this example, you can see just a deployment Tiamo for my Kubernetes configuration, but basically there are lots of things to take care and like lots of knowledge that are part of the Kubernetes configuration. Yeah, basically lots of things to handle and good luck Mr. developer. So now let's start talking about the security context, which is part of Kubernetes configuration. So let's start with simple, very simple definition for it. So what security context is, is basically defines privilege and access control setting for the pod or the container. So raise my like pod configuration. And as you can see right now, I also have the security context is a special section for the security context. And under that, you can see several options like privilege or capabilities. And we're gonna talk about each one of those today. So let's start with privilege pods. So when exactly do we need privilege pods? Think about scenarios where your application need to access the host resources, cases like accessing the GPU, the graphic card or manipulating the network stack. For all of those, you actually need to access the resources that are part of the host. And I think that the security risk is clear here. Basically each one of the processes inside the privilege pods, the privilege container is basically exactly the same like a root process on the host. And in other words, it means that an attacker can basically do anything they want. So the solution is simple as well. So if you don't need it, just don't use it. So don't turn it into truth. And of course, again, there are some cases where you can use this option, of course, but you need to be aware of the consequences. So now let's start with the demo, with a nice demo. In this demo, we have two different applications. The first one is supposed to be supposed to be a secure payment application. And again, it's supposed to be secured because it's supposed to be isolated as well. And the only thing this application does is just to write files to the disk. The name of the file in this case is CardsJSON. And basically, whenever a user pays something using the payment application, we just write the details to the CardsJSON file. Again, very dummy application, but it's like our example. And the second application is our vulnerable application. So as part of this application, we have two different problems. The first one is that we have an RCE vulnerability in this application. RCE stands for the mode code execution. So basically think about an option for the attacker to run code from the outside on the pod itself. So this is like the RCE vulnerability, but also the main problem that we're gonna talk about that this application is running inside a privileged pod. So again, we have our node and inside this over node, we have those two applications, basically because both of them run on the same node, they basically use the same Docker engine, which means that the Docker layers are stored in the same local storage. So in our case, if an attacker has an access to the vulnerable pod, they can basically access all the files of the others supposed to be secured application. And now let's see that. So let's just see the application itself. So this is like the vulnerable application. And as part of this application, I can basically upload pictures into this page. And now I want to use the payment application. So I'm gonna donate $1 and I'm gonna show you my secret credit card. And that's it. We basically donated $1. Let's see what's going on in our environment, in our Kubernetes. So I'm gonna run to kubectl get pods. And this is our two pods. We have like the one for the deployment and the one for the payment. And now let's take a look on the configuration themselves. So this is the configuration of the payment application, nothing too spatial here. But this is the configuration of our regular vulnerable application. And you can see that the pod is privileged. And now let's try to add this application. So again, we're gonna assume that there is an RCA application, RCA vulnerability as part of this application. And basically the RCA that we're gonna use is kind of an option to upload a PHP script into the application and then access this PHP script. And then we can use this PHP script in order to run commands on the pod. So first of all, we're gonna upload the PHP script which we're gonna see in a second. Then we're gonna create, we're gonna access this PHP script using Carol command and we're gonna run mkdir in order to create a directory. We're gonna mount the host file system. And then we're gonna look for the file that we're interesting in. So this is like our PHP script. And as you can see, I can get an argument and run it as a system. And now let's try to upload this file. So I'm gonna upload again the PHP script. And now let's inspect this page and take the name of this file. So now the only thing I need to do is to run the Carol command and to access this file from the outside. So now let's copy this Carol command. And the first thing we're gonna do again is to create a new directory on the host. On the pod, I'm sorry. So this is, so basically the upper terminal is the kubesidil exec into our pod just so we understand what exactly is going on. As you can see, I have nothing in the time directory. And on the lower terminal, this is the attacker environment. So as you can see, we run the mkdir command and we now have a new directory named host. Now what we're gonna do is to run a mount command and we can do it only because this pod is privileged. So before we run the command, we had nothing under temp host and now we have multiple directory. So those are the directories from the host itself. Next thing you wanna do is to look for all the files named card JSON. Again, let's just assume that I already know that the name of the file with a secret script card is a card JSON. And the last command we're gonna run is just the cat command in order to display the content of this file. And as you can see, this is the credit card we just entered in the payment application. And now let's see how we can fix it. So simple as that, you see just turn the true into false. I have a clean up script. So we basically deleted all the pods and now we're gonna build everything again. I'm gonna refresh my application so you can see that we'll start from scratch. And let's start, let's try to do exactly the same. Let's start by uploading the PHP script into the platform. And then we're gonna run the same commands one by one. So again, let's look for all the new pods. You can see that they started 20, 28 seconds ago. Let's execute into the deployment pod. And let's see that we have nothing under that time directory. And now we're gonna run the makedir command. Second, yeah, sort of with that. And as you can see, right now we have nothing under the temp host directory. And now this is the interesting part. So now we're gonna run the mount command and we still have nothing under the temp host directory. Let's try to understand why. So we're gonna run the mount command, notice an attacker this time, but just inside the pod itself. And you can see that we got permission denied error. And basically we got this error only because the pods are not privileged anymore. So that was it about privileged pods. And now let's continue with our second topic. And the second topic is root containers. So when exactly do we need to run containers with root users? So basically almost each and every scenario, think about cases when you need to install system packages or when you need to change part of the configuration and even simple commands like ping, all of those, you actually need root privileges. And the security risk is kind of similar to what we had with the privileged pods, maybe a little bit like lower, like it's like privileged pod is basically, okay, and you can do anything, but root service, root containers, it's still very risky because the attacker can, for example, access the files or if they wanna explore the network and check other devices in the same network, so they have those option. And I think that one of the problems here with root containers is that we can easily forget or miss the fact that our images can run as root. So even in this Docker file, in this example, you see that we have a base image of PHP, and I did nothing special. I mentioned nothing related to the user itself, but even then when I will run the UMI command, I will see that I'm running as a root. So again, this is like something that we only to be aware of, because there are lots of images that we probably can use, like base images that we can use as part of building our images. And in part of them, we can add root users. And now let's see two different solutions, okay? Those are not like the only solution, but two different solutions that we have for root containers. The first one is kind of like an old legacy Linux solution, which is like the Linux capabilities. And basically for those of you who are not familiar with it, like the kernel, Linux kernel added lots of capabilities in order to allow the process to start as root and then to drop the privileges as quickly as possible. So this is like the original solution. In our case, in Kubernetes, it's basically, we can basically use Linux capabilities as an option to grant a very specific permissions, list of permissions to our application. So basically this is how we can prevent granting a full root access for the application. So my recommendation here will be, just start by dropping all the capabilities and then gradually, gradually one by one, you can add those that application actually needs. And let's try it out. So in this example, we're gonna use the same Docker file that we just, the same image that we just saw. It's like a PHP image. And as you can see, I'm not running as a privilege pod. And now, let's say that the attacker got an access to my pod. So maybe one of the things that I wanna do is to export the network. So one of the common tools that can be used is, and that is basically a free network scanner that can help the attacker to export the network. And here you can see that we run NMAP and we got the IP of the payment service. And of course that we can ping this service as well. And just assume that maybe, you know, maybe we have a vulnerability and this is how we can access somehow or exploit into the payment application. So now let's see how we can drop all the capabilities of this image and how it can help us. So again, let's start by dropping all the capabilities and then I wanna clean up all my environment and rebuild again. So as you can see in the lower terminal, we now have new pods that are being created. And let's try to do exactly the same. Let's execute inside into the pod. We're gonna run this config in order to get our IP and then we're gonna run again the NMAP tool. This time we got a fail to open the device and even if we will try to ping, we'll get operation not, we just got operation not permitted. And the reason for that is basically that NMAP sends and receive raw packets. And this is why when we drop the capabilities, all the capabilities will also drop the capability name cap net raw. And so we kind of disable for NMAP the option to receive those packets. And this is why we cannot use it anymore. And now let's see our second option here. And the second option, our second, maybe the second solution is to try the option for run is non-root. So run is non-root is basically an option that might help you to determine whenever the container should run as non-root user. And basically behind the scenes, Kubernetes will not let you to start the container if the container itself asks to run as root. And the recommendation, if you know that the pod or the application should not run as a root, just turn it on. And let's see how exactly it works. So again, we're gonna use exactly the same environment. This time we gonna disable the capabilities feature and turn the run is non-root option. I'm gonna clean up the environment again and rebuild everything. And this time in the lower terminal, you can see that I have a problem in the statuses create container config error. Let's try to understand what exactly happened. So I'm gonna run a kubectl describe command and you can see that I have an error that the container tried to use root container. So basically because the container was root, like the user of the container was root, Kubernetes helped us by blocking this container from running. Okay, that was it about privilege pods and also about root containers. And next thing we're gonna talk about are resources limitation. So it's kind of a different type of security issue because I think that resource limitation is a good example for the default behavior in Kubernetes configuration that we all should be aware of. And let's start talking about that. So we have two type of resources. We have CPU and memory, of course. And again, the problem is that by default, all the pods run with unbounded limits. And it basically means that a single pod can consume as much as CPU and memory that are available on a specific node. And of course that Kubernetes might kill application or even nearby application inside the same node. So of course, as you all know, defaults are never good. And this is why you need to manually make sure that we have proper resources, both with their requests, but even more important than the limits. And now let's talk about the CPU and the memory. So I think that the CPU is maybe the easy case because of the throttling mechanism as part of Kubernetes that occurs each and every time the application eats the CPU limit. So Kubernetes will not terminate our applications if it will eat the CPU limit. The worst case scenario in the CPU with the CPU is that it will affect only the performance. So everything might run slower, right? With memory, it's a bit different. So we cannot compress memory. And this is why pods will be terminated once we reach the memory limit. So think about scenarios where someone, the attacker can run a dust attack on your application. In this case, if they will use all the memory as part of this application, they might block legitimate users from using our application. But think about like even a much worse case where you might have different application running on the same node and an attacker tried to attack one of the applications, but at the end, they also blocked the second application that is running on the same node because at the end, both of them are using the same memory, the same memory sources. And now let's see an example for that. So in this demo, we're gonna have again, like two different applications, again running on the same node. We'll have the innocent application and this application should not be affected by any other applications. Again, this is only the trivial assumption. And we also have on the same node, we also have our vulnerable application. And when we'll start the demo, we will run this application without any resource limitation, without any limitation of the memory. And we also gonna assume that this application contains a vulnerability that will help the attacker as part of the dust attack. So the attacker can use this vulnerability in order to take more resources from the pod. And again, the problem is that the attacker can use the vulnerability in order to take those resources and then they will affect the innocent app as well. So let's see. So this is our two deployment. So we'll start without any resource limitation. This is like our innocent app. So as you can see for each one of the app, we can see how much available memory we have. And now let's go to the vulnerable app and let's assume that we can run the dust attack and we can consume, let's start with 100 megabytes. And you can see that there is a drop in the amount of available memory. Again, we ran it for five seconds. So now let's do even a drastic step and you will now see even a faster drop in the amount of available memory. So again, that was our problem. We had like two different application and we saw how one of them without any resource limitation can affect the other application. And now let's see how we can solve it. So I'm gonna turn on the resource limitation. I limited the number of memory and again, I'm gonna clean up my environment and rebuild it again. Let's wait for the new pods. Yeah. Okay, so now let's try to see those two application again and let's start with a smaller size this time. So you see that I managed to allocate 10 megabytes but when I tried to allocate more than that, like 100 megabytes, which is more than the limit, I just failed to allocate this amount of memory. So again, a very, very simple solution but it's all very important. It's part of taking care for all of the applications that are running on your environment and your production environment. Okay, so now let's go to some conclusions and maybe, you know, what are the things that you can take from this talk? And so what did it talk about? We talked a little bit about the ownership of the developers those days and our complex it is to make sure that everything is secure as part of our pipeline. And then we started to talk about three different issues that are part of Kubernetes configuration, three different security issues that are part of the Kubernetes configuration. The first one was privilege pod and then we continue with good containers. We saw two different solutions for good containers and then we continue with kind of a different problem of security, of first of all, having no limitations but also using the default values of Kubernetes. So Kubernetes security is hard but definitely doable. And I think that, you know, that developers should be, they should start getting familiar with those issues. They should start getting familiar with those risks because it's, as we all understand now, it's kind of an inseparable part of the application, right? You can just say, okay, this is my code and this is my dependencies and I have no idea what's going on on my Kubernetes environment. It's all part of the same application. And I guess that, you know, one of the things we need to make sure is that there is a good education and we all will all be familiar with those risks as part of the Kubernetes. And maybe another kind of a takeaway from this talk is the fact that we need to help the developers and it's kind of related to the previous slide. We need to help the developers by adding more tools to the pipeline. And more specifically, I'm talking about the fact that we need to catch those issues as soon as possible. I'm not waiting for the last moment and see those issues on production but to find them as soon as possible even during the development and to fix them. And what did, and we talked about today, so lots of other staff that we probably missed. Again, we don't have too much time today so I will not talk about each one of those. But there is a great list of other issues that you should all be familiar with. Things like like RBOT, like role-based access control and but also networks and network policies. All of those have a specific context, a specific value that you can use in order to improve the security as part of your, the security as part of your Kubernetes environment. So last but not least, I would like to show you one of the solutions that we released lately as part of SNCC, which is named SNCC in price code. And basically, SNCC in price code helps you to find and even fix security issues as part of your configuration files and right now we support Elm, Kubernetes and Terraform files. And we're basically making this list even bigger and bigger. And you can scan everything from your Git environment, you can scan everything locally using our CLI and you even have an option to fill their out policies that you don't agree with. Like if you don't agree with the severity if you think that one of the policies that we think that is very low, you think it's very important and you wanna make sure that all the developers and the organization are taking care of so we can change it to be an eyes very deep policy. So I don't like those two slides, I'm just gonna jump directly into our SNCC project page. So this is like SNCC environment, this is like SNCC platform, and I already imported, like using GitHub, I already imported my cloud native config repo. And now let's go to one of the files inside, the Kubernetes file inside this repository. And as you can see, I have a view of this file I have a view for what exactly is going on. So yeah, we just talked about like education and I think this is part of it, right? Cause if for example, we have a privilege pod, it's not enough to say, okay, it's just a privilege pod, but it's very nice if we can get a context to the problem, okay, like what's the issue is, what's the impact of the issue? And we already talked about this in this talk, but and also how you can fix it. And so this is like our integration with Git. And what we also have is an option to scan everything locally from your environment. So I'm gonna run SNCC infrastructure test and I also can see the results of my local files. So again, both the CLI and the regular input flow of SNCC. So thank you all for listening. I really enjoyed and I hope that you enjoyed as well. And I think that now we have time for some questions. If there are any questions, yay. Okay. There's the second, first question. Will we talk about the open shift? And the answer is no. Sorry for that. Another question. So we have a question about does the attack vector will work on open shift or only in vanilla Kubernetes? And I think, you know, like the point for second, the point for this presentation was to talk only about the core concept of Kubernetes. Maybe part of them will not work. I think that, you know, the privilege example, the privilege part example will be different in open shift but the recent limitation problem will definitely be as part of open shift as well. So we have another question. Private Docker images from Docker app is mostly privileged. Kubernetes security context can fix it in the container level using run as user. I'm not sure I follow the question. So if you can make, if you can clarify that on the chat as well. Thanks. So another question. You can get the resources related to the demo. I will be happy to share the YAMLs and the Docker files after this talk as well. I'll make sure it happens. And whoa, too many question. How to run untrusted application with special third party dependencies from internet inside pod running in on-prem Kubernetes cluster Docker runtime. Can you clarify what exactly the question is? Thanks. Thanks all for the good feedback. And I'm not sure if I can provide exactly the same GitHub repo that I just saw, but again, I'm gonna share the examples as well for sure. Yeah, good question. The sneak infrastructure is code scan. You can try it for free. And basically old sneak product as a free version. So you can try sneak infrastructure as code for free as well. And I think that right now we have 300 tests per month free for infrastructure as code. So yeah, definitely gonna try it out. A question if the tool that I just demoed will find errors and provide suggestion to fix the error. So yeah, as I just demonstrated, the purpose is to have a list of all the issues, but not only that also to give you some explanation how we can fix it and what's the problems part of it. Interesting question from anonymous person. If someone has access to the cluster, can they just change the almost files and privileges? I'm not sure in the example you showed us if it's a separate terminal to execute the attack. So again, like add only a separate terminal just to show what exactly is going on inside the pod, not as the attacker, but as for the first question. So it depends what does it mean that the attacker is an access to the cluster. If they have an access only to the pod itself or one of the pods, so it depends what's the configuration of your cluster. So of course that maybe it's possible to somehow get an access to the YAML files, for example. But if not, and in most of the cases, I guess it's very easy because like the deployment environment and the source code environment is kind of a different environment from your Kubernetes cluster. So in most of the cases it's not doable or at least it's very hard, but I think that the core concept is even if they have an access, even if the attacker had an access to the cluster, you need to make sure that they can do the list as possible as part of this access. And I hope I answered the question. So right now we don't have any automatic fix for those issues, but we are working on that at the moment. So stay tuned. And again, same question. I'm gonna share a part of the configuration I just talked about in this, like very soon after this talk. Nice question of where I can learn more about Kubernetes. Very good question, I like it. I can say about myself that I learned like most of my experience is just by playing with Kubernetes. And I think that lots of good podcasts about a bit Kubernetes and how you can use it, but also like very good courses that you can take as well if you don't have any context at all. But again, my recommendation just start playing with that. I think that if I will share some resources after this conversation after this talk, so I think it will be a very, very good start. Any other question? Another anonymous question. How to do pod sandboxing and Kubernetes cluster? So I'm not sure like, I'm not sure there is any similar like concept for sandboxing not that I'm familiar with, but again, I think that in most of the cases, at least what you can do is try to make sure that like the pods that you have inside your cluster are like don't have all the privilege, all the privileges, all the capabilities and then like if they don't have any options for that. So even if the attacker has an access to your cluster, they can basically do nothing. They maybe they can ruin your pod, maybe they can keep the pod, but they can do nothing outside of the pod. Question about the vulnerable application with the recent limitation. Is there any other way to handle that vulnerable or just limit the memory? You know, probably, you know, as part of the demo I saw I explained it are two different problems here. The first one is what was kind of a given just for the demo of the like the vulnerability for the attacker to use more resources. So for this one, you know, just fix the vulnerability, just to make sure you don't have those kind of vulnerabilities as part of your application. And for the second part, even if you have those kind of vulnerabilities, so yeah, you can, I think it will be enough for you to start with a memory limitation. Is there any way to inspect the pod for anomalous behavior? I think there are like multiple products that might help you, I don't wanna enter this as part of this application, sorry. And another question about SNCC, is it open source? So we do have several repositories, for example, our CLI is open source, but not everything as part of our company is open source. And I think this is it. I hope that I answered all the questions. And thank you everyone. Thank you Orr, that was awesome. Thanks so much for everyone participating in all the great questions, that was great. As a reminder, the presentation will be posted on the Linux Foundation YouTube page in the next 24 hours. So we look forward to reviewing with you there. Hope everyone has a great day. Thank you.