 Thank you very much for the invitation. So second talk tomorrow. Thank you Today, so this is my background More or less I do Kubernetes security in critical infrastructure founded several companies and I know my main focus is in Kubernetes security and this is a rabbit hole of Kubernetes if you look deeper into it then you should Be a little bit scared and I want to explain why The first approach is the application and then the application normally is run in containers and The containers What is not really well known have access to service accounts in Kubernetes? Which is one of the major flaws in Kubernetes at the moment if you take over the service account It might be that you can take over a cluster and if you can take over a note If you can take over a cluster you might take over a note and then your entire cloud service account which is work of Somebody else I will mention later on this slide. So let's look what happens so the target is I have an application exposed to the internet and I Want to own the entire cluster from outside Application might be vulnerable examples Yeah, lots of them one example. I want to present is image-tragic You normally should not do evil or exec statements in any framework PHP Node.js or any other framework and execute commands in The context of your application because something can go wrong Developers are responsible for this. Let's see how it looks like. This is the attack model based on an attack, I thought it was old and has been fixed 2016 but now There was a new overview overview by Amy Lerner who again showed yes, you can in current versions of image magic Exploits this attack. So it works you image magic is for uploading Images so you convert the image in a different format scales The size and then if you do something wrong in this image, you can own the entire Container this also works for non containerized applications. If you have a server running something with image magic on it Please be careful Okay if we have Mastered this step the next step is Yes, we want access to the service account and this is by default enabled in Kubernetes. So you have a Kubernetes design flaw because your service account is Exposed to the container Where the application runs in the next step of an attacker is Installation of additional software. So you want to take over you need a curl or a cube CDL a change mod and then you are Owner of the service account and can actually do commands by uploading pictures in image tragic So responsible for this flaw is the image creator Let's see what else can happen To get total control you also need roll binding to cluster admin role. This is not enabled by default But the internet is always good for bad advice. So if you copy installation Requirements or recommendations from the internet you might take over Somebody else might take over the entire cluster. Let's look deeper into it Worst practice here is what you can see in the elastic Installation recommendation They just mentioned they have a newer version, but they use the cluster admin permissions here To install Elastic search in your Kubernetes cluster. So they recommend it and a lot of other Lot of other Applications also have this which is a little bit outdated, but it's quite common in the installation requirements Never ever do this, please It also can come with Helm charts. So you have Helm charts where the cluster admin role is Included Here you see it. It was in Apache Huron, which is an Apache project and It uses a cluster admin role. So by in Helm install you might be affected by this floor too So with these four steps, which effectively are three steps you have an explic you have a cluster application exposed and Through that path You can take over the entire cluster from the outside and Do anything what the cluster admin role can do? Effectively this cluster admin role binding is like a Doormat attack. So you have the best cryptography the most expensive locks on the one side and then you put the lock under the doormat or under The flower at the door something like that. This is something which is not really What you want You can do an example walkthrough Which shows how it goes. So I've published all my Trainings notebooks on github Here's the way you can build this out dated image tragic version In open shift. So I use CRC, which is the code ready container version it's based on the image tragic proof of concept by Mike Williams and Here you run and create a vulnerable image a Little bit lengthy. It's compiled inside and so on. So you don't get a Full version Reason why I don't show it here, but effectively at the end you have a vulnerable Application in a container internal in open shift and that's what exactly what we need to run the application Here's the exploit and the exploits starts with the deployment of This container, which is standard Kubernetes here. OC is like Qubes CTL. So you get an overview Additionally in open shift you have a very simple version of creating a route Which is connected to a host name and then you can upload it by using that hope host name You expose the deployment you expose The service which is created you expose The route finally and then you have access The next step is you get these route and then here you have in URL Which you can use and in a full demo. I would just simply Call this URL and then I can upload images here. I've created these files, which are valid Postscript files, but you see at the end of the there is a full Command and here because there's a curl in the container. I can download a version of Qubes CTL Effectively the containers especially the redhead containers are not so vulnerable vulnerable as others but You have always Rightable temp, which is enough to deploy some software. So we curl Tube CTL from the internet put it into temp and then we use simple change mod command To activate Qubes CTL. So now we can call Qubes CTL commands, Kubernetes commands from inside an image this is a It's a death belt more or less. So exactly at the right place we have a working exploit now and Warning it might also already work in older versions of Kubernetes Because in newer versions we need some Pill of poison additionally and this is exactly this cluster all binding to the cluster admin which Needs to be done that we have full access from the outside and if we do this And expose our cluster admin account To the same account which is already exposed inside the container we can execute commands with this Qubes CTL So we can create deployments by uploading pictures Which is exactly what you never want but an attacker now has full access to your cluster by simply uploading Prepared malicious pictures can do this. So this is an example here and just Create and delete containers and deployments This way you can effectively do everything and Again, this is the problem here from the application side You have if you have a vulnerable version of image magic you can include Commands and you can definitely install software on the community server side There are several trials to fix this For example, you can use better images like redhead does so this is a redhead health index Which is quite good, but effectively these images have the advantage only that you not run anything as route But you run the same as Another user ID and if the same user is allowed to write to the temp directory effectively Yeah, you don't need root for installing software. So the container also Was good practice no root inside it has in immutable root file system, but At the curl which is completely unnecessary was also deployed. We had write access to a temp. We had a change mod And the first thing you prevent all the stuff I'm doing here is and if you Don't do and don't learn anything from this talk. Please go Look into your service account and try to disable the auto mount service account token Peaches so all the service accounts which are not running operators don't need this service account token If you have an operator it might be broken now and it can be overwritten by the pot definition, but effectively this Entire example would not work without this service account token So we have fixed that we cannot fix the application because this is something Somebody else is creating for us and we might even have a floor which is not affected. So there might be zero day The next thing we must prevent is the installation of software Fix the images so use really immutable images Temp only if you need it PID is and is one anyway Okay, you might have some variable data, but you should use containers from scratch no curl no w get and also affects redhead ubi's and And most of the standard images have this flaw So you have a full operating system inside with all the tools you like, but this is not Your Territor It's just Yeah, it's it's a tool for the attacker. So please run only Trusted images build your own images and build them from scratch This is my example. I also have Uploaded To get up how to harden a container which is based on engine engine x alpine engine x alpine Normally is a very small container, but you can do more you can use the script Which is in this repository Just to get only the tools you need so this is not Statically linked because the original engine x is not statically linked But it's very close this means you only positively install the software you need And this is dynamically linked there for the minus these so we use LED to Extract all the dynamically in libraries and then all the configuration files Which are necessary atc password atc group. Okay Some licenses in in share And needs some directories for logging and then you can install it from scratch because this script installs it in a directory temp harden and you can with this Multi-stage build you can install All what you need from temp harden and then the next And then the next container is based on scratch and you can use engine x the same way you would use a more or less an Application which is statically linked So now we have Created a hardened image with outcube CDL curl inside. So we are much closer to a secure application The next thing is yeah, roll binding to cluster admin roll don't do this If something in your application goes wrong, you have additional measures which you can take just to prevent the application from Break out of the container so you can separate the internet exposure of services or ingresses and co-ordinators from privileged as Operations, so you have node settings elastic search is doing a lot of these things so a lot is Really not true. So it's doing is this CDL some applications Have host passes on have connection to the host inter-process Communication which is not necessary if you have exposed it And then separates the applications who needs this from the applications Which don't need this so cluster admin should be more or less restricted to very privileged Operators and by the way, Argo is also a very privileged operator Don't run an Argo on a Kubernetes cluster any security critical environment because I've seen Argo Also is binding to cluster admin it Doesn't mean that Argo by default is unsafe, but it's a very complex application and I would Definitely run it in a separate cluster not in the critical cluster and What doesn't architecture fix look like here you have the Lifecycle of a pot so the time is going to from left to right and here you see if the Container is ready. It can be accessed from the internet and If you do something from the internet system also like assist CDL Please do it inside a container which is not connected to the internet Just use a pause a container the pause container to limit it and restrict it That it is not really Connected to the network. So this is something which covers the architecture Additionally, I already mentioned here's a network policy Which will come later. So this is our threat matrix. You have exposed and not exposed Services you have unprivileged and privileged things the dangerous ones are the privilege ones which are up which are Exposed but normally you only have an exposed privilege application If you have an IDE running in Kubernetes, which is not what I would like to see in critical infrastructure something like our studio Or have a web UI to a GitOps framework and Normally, you only have a web application and What should not be exposed? Under normal conditions is an operator CCDL build systems host operators and so on If you do this, it's literally not possible to own the clusters. You should do all the three because if you Have security in-depth you can make a Mistake on one of these levels and the other miss other levels keep you from being exploited You can even do more isolation on the network side. You have network policies for egress On the node side, you can activate second JV so and the common common frameworks as the Linux app armor You can use pot security policies or in the future the open policy agent to prevent the node from being hacked For the identity and access management, you should use individual service accounts for all your tasks So you have enough have a lot of roles You should use role-based access control to check this Okay, but I promise yes, we can go even deeper and This needs a little help from your cloud Administrator and he is the example from Nico Meisenzahl who does a very similar Example on hijacking Kubernetes and he's doing it obviously in one of the clouds and What he has found out is you can get access to the asia.json file Which has a user assigned identity. This is not the Kubernetes identity. This is the Azure identity You can get a token. You can get a subscription You can get a resource group and then you can use a curl command with this token To change things on the API version of this resource group with this subscription So you might be able to hack your node with a privilege container and then Take over your cloud account and he told me that this is also the truth for the other clouds So it might even work something similar in AWS and GCP so please also protect your cloud account Understand your identity and access management in the cloud. So at least somebody someone in the team should understand it and limit also the underlying account To the bare minimum it might even be a good idea to block access addresses like 169 to 54 something And the other clouds as I already mentioned also might be affected And my call to the cloud providers is don't deliver account data in containers or nodes. This is not necessary It's yes, it's very comfortable as the service account token is very comfortable for running operators But it's a major security flaw and it might Be that you lose all your accounts and data so Conclusion we have a full attack chain from the application to the cloud account and It's your task to Prevent it and fix it This is called shared responsibility So the cloud providers effectively only care for the infrastructure, but not really for the security in your clouds This is your task Okay, thank you for your attention. I hope it was interesting. Please ask your questions and Now I am open of the Q&A Thank you for the talk. Is this working? Yeah So do we have any questions from the internet I Don't see any coming in so far, but we are think usually a bit ahead So I'll ask one and what do you think so who's in in the responsibility mainly to fix these insecurities? So do you think this can be fixed by better defaults in these? infrastructures and configuration files is this to be fixed for better tutorials and better education for the DevOps engineers or where's the main point of responsibility? I Definitely would prefer to have secure before installations But then you have this shared responsibility in the contracts that from a certain point you are responsible for the security of the account and We have seen this complexity Because these might be 20 steps every step is very simple and every Every step is looking very harmless, but All the steps together might Create a full exploit of clouds. So this must be overseen and it's very hard for developers who Are cloud native and are focusing on their application to Have an overview of the security developers now have 10 or 100 times more code on the hard disk than 10 years before and this means developers are not able Really to have a full judgment about what is going on in terms of security And this is something developers talk about security Either they are specialized on it or they have not seen Things like this. What I normally notice is developers are not aware of these Problems Okay, and what do you think what can we do about the complexities? So do you think we need better education for people to actually understand the systems or is there a way in cloud infrastructures to reduce the complexity? better education and do I think Do all the simple fixes These are five steps and the fixes are also very simple and You have to check them and then you need a tool because you might have 20 clusters like this Every cluster has 20 applications. So this might be quite complicated So you need tools for an overview and in the trainings material you see examples how you can check your Kubernetes clusters for exploits like this Okay, thank you very much. Thanks for being here. We will continue in About half an hour with the next talk then again in German Thanks Thank you very much Everything is licensed under CC by 4.0 and it is all for the community