 Okay, I think I'm going to start so I will gain five minutes more for me and Have you ever heard about at CD? Sure. You know what it's at CD. Yes Yes, yes, okay, perfect So, okay, first of all, good afternoon everyone I will like to thank the organizers for accepting this talk and today we're going to talk about About how to post exploit a compromise at CD But first let me introduce myself. I'm Luis Toro and I start on the offensive side of security in 2019 and for the past two years I've been working as a security consultant at NCC group and here you have some details about me and some budgets and And the thing is the core concept behind this technique is not completely shiny new Yep, and there have been other talks in past editions of Qcon that have already discussed the importance of at CD within the cluster and You have here just a couple of examples in the first talk Joe Betts release Tool that has been the center of focus for developing this technique and I will talk about this tool later on and For those you may not be familiar with cyber security terms It's important to define what post exploitation means and It refers to the techniques that an attacker employs after gaining an author right access to a system So consequently post exploitation techniques always come at the end of the of the workflow of the Security exercises because it implies that at this stage we have already compromised one asset and we want to establish maybe persistence or We try to jump to another asset in the same network But before that being in into details I would like to highlight these points as we already said this is a post-acquisition technique which means that we should already have a compromise asset and It should provide us the the following requirements. So first we need to have the certificates obviously to to get authenticated in at CD and The second requirement is the at CD service must be reachable it could be locally or or remotely and It's worth noting that this technique only works in self-managed environments and also it works in environments Where at CD is implemented as a cluster. We just need to compromise one of the at CD nodes and However, we won't be able to apply this technique in managed environments because you already know at CD is not reachable in control planes that are configured by the cloud providers and Here's a brief reminder about what is what is at CD and as you know, it's a key value store And it's usually located in in a control plane node or bound to the control plane node in another host and Kubernetes it uses at CD for baking up all the cluster data So we can consider at CD as a kind of living backup And it's constantly updating the state of the cluster components and talking about backups we What what should happen? What would happen if we have a legacy system on Which we have backups or dams that are not encrypted or are not encode or not serialized We have just backups in in Row data, so what would happen if someone have access to these backup files? Well, most likely an attacker could use these files to obtain of course sensitive information Let's say hard-coded patch words in configuration files, or maybe could retrieve private keys for SSH or even worse It could modify or inject malicious files waiting for this backup to be restored and gain a compromise Asset within the network so based on this principle. I was obsessed kind of into finding a way to Reproduce this the same principle in in at CD But first things first and how does actually work and as we mentioned is a key value store which is a simple concept you can add keys and values which are associated to to those keys and You can if you know a key path you can retrieve its value or delete the key and basically that's all it's quite simple Then what happens when for example we create a pot? Well, the API server will take our request and will instruct the hcd to store this value in in a specific key In this case it will use all the information of the pot manifest as a value and the key will be located usually at Slash registry slash ports slash default slash and jinx in this example and This is just an example. You know that not all resources in Kubernetes are namespace it So not all resources will follow this path a structure However, most of the namespaces objects like deployments or stateful sets or service accounts will be safe following this this structure But that was just a high-level overview of the entire process It's actually more complex and this workflow is much more detail. I want detail every single step here I just want to highlight that the API server is in constant communication with the hcd at every single step and When the API communicates with the scheduler, for example, it updates the state in hcd when the API server communicates with the cubelet it tried to updates the information in hcd even when the cubelet receives some updated information in Sorry, I lost my mouse okay, right here even when the QAPI server received information from the cubelet it also tries again to update this information in in hcd So as you can see hcd is in the middle of every single communication managed by the API server So we can agree that it's a key component in in the cluster, right? now we can retrieve this information about the newly created pod from hcd and It's quite simple with this command we specify where the service is is located in this case It's in exposures in in the local host But if we have the hcd in in a remote cluster, we should just change the endpoint and and We also have to specify the files that we use for for the authentication against hcd and finally we indicate with With the command get that we want to obtain the value of the following key in this case it's our pod call and jinx and We have this if we take all the information from hcd we get this string and It appears to be something like a JSON, but it also contains a special non readable characters. It's strange But this is actually the pod data serialized with a protobuf format And you may wonder what is protobuf and and how does it works? Well, it's just to put simple here is just an example on how to use protobuf how to serialize data from from a person in this case and First first of all We'll need a protofile that will define this structure of the data that we want to serialize in this case we want for example to include the name age and an email and Then with the command below you can create a module that can be imported into your Python script And there we can create the serialized object and here at the right you have the output in byte format But what happens if we want to serialize? Another thing not a person what if we want to serialize a book or a car then These objects do not have the same parameters as a person so we should create additional Protofiles and and more structures and this is the problem that we have in in Kubernetes we have several objects and all of them are defined in a different ways as You know a service account is not defined with the same parameters as an equal policy And a pod is not defined with the same parameters as a role binding. So what happens if we want to serialize these different objects? because we will need to know the the structure of everyone and fortunately for me there is already a tool available that allows us to Serialize and deserialize entries in it CD and it's the tool that I mentioned in in the beginning and was presented by Joves at KubeCon North America in 2018 and this tool basically enables us to pass a serialized object through a pipe and it converts into a yaml or or adjacent and Similarly, we can manually modify this yaml file and then we can try to serialize it again and put it back into it CD Quite simple Basically, this is what we start from at CD and the previous tool out here will return us this beautiful yaml crazy so I'll start playing with this tool testing some changes in those extracted yaml files and Then put it back into at CD just to see how the cluster reacts and I found some Interesting results. So I thought that it would be a good idea to create a wrapper for this tool that automates the the entire process and You can see here in the middle that say tricks and I'm gonna explain and detail. What are those tricks? So here I've detailed the entire workflow of this wrapper called Kube at CD along with some of the tricks involved and first we need to select our resource that is already in the cluster and we'll use this resource as a template this tool does not work with row yamls and At the moment, I've only implemented pots, but this could be done with any other resource in in the cluster Then the first we deserialize this pot template We take the yaml and the first trick is that we need to create a new UID which can be randomized and If we do not change this UID it may create some cross-reference between objects and may result in inconsistencies and malfunctions of the API logic, yeah API server logic and there is an interesting aspect here When the pots reach the running a state and neo-hash will be created and link it to this pot And also it serves as the container ID as well and we can simply remove this hash and everything will go fine perfect and Now we have modified the template we have changed the UID we have Removed the hash now we have the bare minimum the basis to create a new pot and make some some changes for example, let's say we could modify the creation timestamp could be or we can turn in a regular pot into a privileged one and We can also experiment with some creating Creating some inconsistencies, which I will show you shortly and After making these changes We can serialize this modified object and we try to put it back into hcd So here we fire so here we have the first example. It's nothing really harmful and We have our pot and jinx running and with the wrapper cubetcd we can simply modify the creation timestamp and Suddenly as you can see here the pot appears to be running for 23 years without a restart And it's a remarkable achievement because it has been running even before the Kubernetes was was created and Want to take a picture of this? Yeah A second scenario and I was thinking about what would happen if if the name in the pot In the pot manifest does not match with the name in the hcd path Look here. We create a new pot called and jinx persistent and Then I implement the dash p argument to create a custom path In this case the path will be called random entry, but could be any any name So if we list the the pots we can see the and jinx persistent pot running However, if we try to delete this pot We'll get an error telling that this pot is not found and That's normal because from the API Logics perspective this pot does not exist because the real name in hcd is random entry and It does not match with the name in the manifest So in this case, we won't be able to remove the pot using The API server through kubectl or any other client instead of we must delete the spot Deadly from hcd. It's the only way to remove this pot from from the cluster And I found these previews a scenario of funny So I thought hey what happened if I change the namespace So here I create a new pot call and jinx hidden and now you can guess what will happen Let's gonna try to find this pot So we have just created our pot in a non-existent namespace So if we check the default namespace, we cannot see our pot. It's not here It's not in the default namespace apparently and if we're trying to find our new invisible namespace that we defined in the previous slide right here and It doesn't exist either so where is our pot? Any guess Any guesses, you know, where is the pot? It's in the in the heaven of the pot No, it's in the default namespace Our pot has been always in the default namespace Because we didn't change this value in the in the manifest. It's just changed the hcd path So when we run kubectl get pots the API server iterates over the default namespace path in hcd and this pot is not under this path However, it will list all the pots in all namespaces. It will iterate Over the the pots path and that's how we can find our pot here Well as a security consultant when a customer seeks a Kubernetes assessment one of my main concerns is deploying secure workloads and you already know that kubernetes has some built-in features for deploying secure workloads such as you know security contacts and all its parameters and We can also apply second up armor selenux and furthermore we have also cool third-party solutions to protect our cluster But for now we'll focus on on on built-in features But you know nobody is perfect and We could miss setting these configurations in accordance with our security standards of our organization So we need to implement some policies to to ensure and to enforce that our design configuration Will be applied and this is where the admission controllers come into play You already know that About the deprecation of pot security policies now we have the pot security admissions which define three pot security standards and as in names indicates the most restrictive is the restricted standard and it requires many security parameters and as a validation and mission controller we must comply every single of its Directives if we want our world low to be to be deployed As you already know pot security admissions apply only to namespaces, so let's create the most restrictive namespace ever call restricted namespace restricted NS and Here you have the definition and What happens if we try to deploy just a single part? It's not even a privileged part. Well the policy admission will not validate this insecure configuration and This pot won't be deployed in this restricted namespace Here I just highlight some of the missing configurations It's it's we have to disallow the privilege escalation the capabilities and the run as non-root and Do you think it's possible to deploy a privileged pot in a restricted namespace Because it is if we deploy that early through at city in so here I implemented the privilege flag to turn any Template into a into a privileged one and as you can see here We managed to deploy a privileged bot in a restricted namespace by by passing completely the pot security admissions defined in the namespace and Time for a demo. I think I'm going so fast Let me show the internet with my because I need to pull some images Hold on and Demo time Okay, you have to see okay the same Screen as me and I should connect my iPhone. Yeah Please don't hack my iPhone And Okay, now we have three Three panels and here we have the administrator console. This is the cluster admin. It's legit. Nothing wrong Here then we have a compromise control plane. So the HDD is running here And we have another remote machine that belongs to the to the attacker So that's gonna see what is running here. We have just an an engines That's gonna see what we have We have many things running. We have key burn also implemented Nothing nothing really harmful everything in place Okay, so now here from the compromised Control plane we can run Oh, let me show you something. I'm gonna try to run Privilege and jinx of course it doesn't works because the cube CTL We have implemented a cuba in a policies to avoid privileged pots in all the the cluster so it doesn't matter the the namespace and Here with this fancy wrapper, I will cuba CD. I will create a pot That I will call it and jinx brief. Let's say I will use the the running and jinx Pot as a template and I will say a I wanted privilege That's gonna try this thing. I'm gonna use a Watch yes, you can see what it's happening on real time Okay And cross finger and here it's creating the pot pulling the image It's running and the big question is is this pot really privileged? Okay. Let's get Keep CTL get and jinx brief and And I will get this in demo. I Don't have the mouse here And jinx brief get oh get pot and here if we find the second contacts Thank you context should be run here Here we have it we have a privilege pot running in a Kibirno protected cluster and In addition, we have I've also had not only This flag but it's also sharing the these namespaces IPC network and PID Well, that's crazy, right? but We can do More fancy things That's gonna say a I want a Reverse shell In to an external machine. I'll try to get My IP I think it's check It's not this one Remember that this demo is running in in kind so I'm looking for the docker IP the docker zero interface IP Okay, please click more. Thank you Okay, anyway, I will be listening on this board and Here I should have in the history Okay, got it got it. Okay Here with this command, I will try to get reverse shell that early from from inside the The iPod and we'll try to create a fake namespace and also will try to use a Fake name for the at CD path. We'll try to watch again What's happening in the cluster and That's gonna do this what happened We have here a remote shell as you can see we have no news pods here. There is nothing Of course Ctd get pods slash a Let me check. Let me reduce this we have here the new pod remote shell in the default namespace, but as a Legit cluster. I mean, I will try to delete the spot call and jinx remote shell. I Cannot So Okay, let me check where I am. I cannot use this one Post name I'm barely in the worker. I've also in the parameters. I add the option to mount The root file system in in this file system in host and Despite this error we have All access SF in it. Oh, no, sorry. Oh I lost my Anyway, I have full access to the control I'll try again That's it. We have root access full access to the control to the node. Sorry and This could be also. There is another flag Telling a I want to move to another note. That's quite simple just using no name I'm not sure if this will be You will work Not sure not sure not sure not sure. No anyway the thing is I'm gonna Stop this and turning back to the presentation Nope, not this Okay, and here comes the big question. Why does this work? Why this happens and to be honest I have not analyzed the Kubernetes code But all the tests demonstrate that the implementation of HCD in the Kubernetes architecture provides a level of reliability regarding the API that Kubernetes clients do not have and as a users we are subject to authentication to authorization process based on roles and Finally to admission controllers that even cluster admins cannot bypass without disabling them however HCD only needs its certificates to get authenticated and then everything it it's In HCD will be trusted as as valid and The thing is how can we mitigate this threat? Is that possible because attackers here have an advantage and this technique affects you know a very specific environments and there are At least to my knowledge. There are no a wildly Adopted solution that serve as as the factor standard for checking the integrity of HCD and We have to respond a security ends incident that exploit this technique the the The events on Kubernetes. They are also storing HCD. So they are completely compromised Even we have root access to the node so if the logs are safe in any note of the cluster could be also compromised and However, we have runtime security solutions that could provide early hours. It's something unusual it's happening in the cluster as your tool and Especially, you know end point security runtime security solutions could detect easily Processes that are running on the node. For example, obtaining a remote shell. It's it's obvious and We have seen gaining root shell access on any cluster. No, it's quite easy Even using images that are not intended for offensive exercise Remember that this image use its jinx latest. There is no offensive image at all So finally let's rub up some conclusions here as we have seen this technique When all the requirements are met is it's very powerful and It can compromise the entire cluster in in a matter of seconds And if you check any book or blog post or anything related on how to exploit a compromise at CD You will find that in most of cases They treat at CD just as a database from which we can retrieve six secrets But now we have demonstrate that at CD is a key component of the Kubernetes architecture and perhaps it's the most important and the most critical one and Suddenly despite all this remember that this technique it won't work on managing environments And now I would like to pose a question to you and the community I won't answer it and the thing is should the trusted position that at CD currently holds in the Kubernetes architecture should be reconsider What do you think and I hope you catch me out there and share your thoughts about it and that's all. Thank you That was You have any question Yep, yep, okay Yeah, it is yes Well indeed it was not a screenshot. It was my calling indeed. Yeah A question Now I have Now I have a technique to exploit the atcd. So, how do I get access to the atcd? In other ways What's the common pitfalls in Current to KBS management to let the attacker have the access to the you say atcd database You sorry, you are asking if we have anything to to to be safe against this technique I'm asking is there any common pitfalls to let the attacker access the atcd database. I Don't know to be honest So how do I to exploit this is the atcd? The with what's the common ways to? Yeah, usually the common way is just to if you get the the credentials. I mean the certificates Usually if you check some blog posts and and books and All the information we have in the internet most of times they are treating atcd just a place where you can retrieve secrets and that's all Okay, so That's the it is it is certificates is stored in the worker node. Yeah. Okay. Yeah. Okay. I get I get the point Yes, for sure. Yeah, first you have to compromise the the control plane or the atcd node But this is not covered by this presentation. Yeah. Yeah. Okay. I got it. Okay. Thank you It wouldn't be it wouldn't be easy to compromise for sure, but Sometimes, you know, some system administrators will like to put some services in in the same node and you know and the thing is Security is is multilayer. I mean if let's it's an example If we have an image that it's it's configure to run To run a user that is not root it's image that it's properly Configured in the docker file It doesn't mean that we do not have to use a mission controllers to avoid running Pots or containers running as root. I mean it's okay. It's pretty difficult to to Compromise a control plane. It's pretty difficult to compromise an atcd node that only runs it city but the things works until It doesn't works. I mean Vulnerabilities are always a rise every day. We we have already seen the supply chain attacks I think this could be also secure in in any sense Atcd should be in my opinion Also Restricted by the mission controllers Yeah, that's the point Hi, you basically answered my question. So thanks Hi, I have a question In a demo we notice that we created a pod with a path that Invisible name space in the past. So I wonder why we can use kubu kado get a pause dash a to show the pod But we cannot delete of this part Sorry, I don't get it. Okay. Let me Reproduce my question. So you created a part Directly through atcd your kubu atcd with a path The name space name is invisible. Yeah. Yeah, so we cannot Get the part of without a dash a yes what's But but with dash a we can see that part. Oh, yes, because yeah, yeah, yeah. Okay. Got it The thing is when you run a slash a with getting all the information the the API Goes to atcd and go to slash registry slash pots and Iterates over this path and it gets everything on this path Okay, so when we try to delete this part Can we or because we don't have a name space named invisible so we cannot delete it It's in the atcd path. It's a slash registry slash pot slash invisible But the thing is when in kubectl, we try to remove this pot or to check this pot it's Taking the value of the name space from the manifest and it doesn't matter it creates an inconsistency and You cannot find with with other command. Okay. Thank you. Thank you. Yep. Ready Thank you