 Hello everyone. This is the art of IDing Yourself and I'm Lorenzo Fontana. I work as an open source software engineer at CSD, where I am mainly focused on the CLCF project, ALCO, which is the project for container and time security. I'm the author of Linux Observability with BPF with David, and today I want to, you know, put the attention on something that we always do and that we never think about, which is closing the doors, right? We continue closing doors, putting firewalls, doing all the prevention, all the putting our system in upstate where we feel like it's secure, because security is a feeling, right? You always put in place mechanisms, put in place, you know, redollify system, rootless containers, everything in your power, and that's very good to do. But you never know if tomorrow someone has a zero day on the kernel, on the software that you use, on your bootloader, on some hardware that you're running, right? So I think that you should be always prepared for what happens if someone breaks in. Let's do an analogy with your house. You're in your house, you have the thickest door in the market, you have always, you know, all the windows are gated, everything, but if someone breaks in and they find a way to break in without passing through, like they come from the roof or whatever, they are there. They might do a lot of mess or whatever. They might spend their time, you know, getting all your stuff and doing a big mess, but what if they don't? It gets worse, right? Because let's say that they just, that they are just in and they don't do anything. They just, you know, sit in a corner and wait for you to go to sleep. You know, it sounds scary, but that's what people usually want to do with your clusters. They usually don't go in there and start being very loud. They go in there and they want to be as hidden as possible and they want to be catched so that they can maximize their profit. Let me play this role now. I'm Lorento. I'm not an open source software engineer, I see this degree more for the next 10 minutes and I will be the person trying to hide. This is my new vest with my mask and everything, like not the face mask, but the eyes mask. And I want to be in a position where I have a cluster that I already compromised. I have, let's say, a zero-day Kubernetes. I am in the cluster and my duty is to minimize the risk. I don't want to be discovered. Why maximize the profit of, you know, getting access to other machines in the cluster and I don't want to specifically be discovered because getting access to the machine costed me time and time is money for the hackers. To do that, I have to keep in mind always two specific things, environmental awareness and persistency. I have to be very aware of the environment where I am in because I have to be in a situation where I can identify what are the best ways to hide myself. And environmental awareness is also important for persistency because with persistency, that means that you, once you're there, you want to do the best as you can to be still there at the next reboot, at the next, you know, reset of the system. And in a Linux machine, persistency will mean, you know, putting a system in the unit while on a Kubernetes cluster, persistency will mean, for example, putting a pod in the cluster, maybe as a deployment or as a demo set or in a way that once it stops, the cluster will put it up again. A Linux environmental awareness means being aware of three fundamental things. Process is now working as storage. You will want to create a process. No one should be able to see it. You want to add your network activity because no one should be able that while you're 15 bitcoins or any coin, you send out the tokens, for example, and you want to hide your files so that no one can see them. And specifically, you might want to hide the files you are exfiltrating for the machine if you are exfiltrating the different machine. There are some strong assumptions in this talk, which are basically that I'm not here to show you how to break into a cluster, how to, you know, have a root kit or a zero-day. I'm just here trying to create awareness on the fact that once you are in, once you have, let's say, access to the master, to HCD, to everything, you don't want to be caught. You want to be there as long as possible. The company will last 10 years. You want to be there in the machine 10 years without them even noticing. To understand how to hide yourself, you first have to understand how administrators of the company that you are hiding into look at the processes. One very common way of looking at the process is just QCTL exec in a container or SSH into a machine and just do a PS. PS is a very common command that everyone uses and in my demo later and in all the examples, my malicious code will be just a ping, right? I just ping the Cloudflare DNS and that's all the bad I do. To understand how I will be hiding myself and my processes in the cluster, I will be using a technique which is exploiting the fact that processes are read from the Procfi system. Usually commands like PS, top LSOF, all the standard Unix command don't access the Procfi system directly with a read syscall. They access the Procfi system using the readdir function from Glibc that then uses the syscalls underneath. That is also a dynamic library which is subject to the rules of the global dynamic linker. Using the global dynamic linker can replace the function that PS, top LSOF use and use my malicious function instead. I will also be showing once we have the process seen then outweigh the process in the Kubernetes cluster and outweigh the pod that transfers the process in the Kubernetes cluster because we are using Kubernetes and we want to achieve persistence via a pod. I also put a link to the sysdig blog where there's an article for 2014 that's about IDIN processes in Linux machines. So I'm building this talk on top of that by also talking on how to add Kubernetes pods for persistence and outweigh the pod itself. Once you have the slides, you'll be able to see the link. The tool that I'm using for IDIN processes is called LibProcessSider, which is from an ex-colleague called Gianluca Borrello. LibProcessSider has a very good track of being used by hackers to hide crypto miners in Linux machines. Just look for LibProcessSider crypto miners on your favorite search engine and you'll have a glance of that. IDIN network activity, as I said, since you just used the standard libraries again, is basically the same. Of course, there are more advanced techniques for IDIN processes in the network activity, like I am not talking about loading a kernel module to hide the process from the profile system or loading a program to hide connections directly. I am using a more lightweight approach but I'm here to show a concept, to show the concept of that, even with this lightweight approach, the actual end operator that manages the cluster, if they don't have anything special in place, if they just, you know, go and QCT LXAC or SSH in the machine and look for the processes, they will have no clue of this. IDIN files is another important topic in here. I don't have an example on out-wide files but I have some good ideas here. In a Kubernetes cluster, you could add files in HCD, you can add files in network interfaces, like, let's say, create a name and, you know, create many network interfaces with different names and then just hide them using the technique that I said before or hide stuff inside other binaries, hide stuff in files systems that are not mounted, just create a partition and hide files. I didn't want this talk to become three hours so I had to choose. Environmental awareness on Kubernetes is even more important than on Linux because you have many, many components. Kubernetes is, I think, in general a very secure environment but you always think, there's always people thinking on how to, you know, exploit its components. For example, how do I, you know, hide a pod? How do I hide containers? How do I hide different ways of scheduling, like deployments or with demon sets? Because if I have access to HCD, if I have access to the Kube API server, it becomes easy to hide whatever I want. What we will be looking at in the demo at the end is that I will be able to hide the process using LibProcessSider and I will be able to hide pods by tampering with HCD because, as I said, I have full access to HCD. I have full access to the API server. I will use a trick of deleting the pod entry from HCD to hide the entry from QCTL get pods because basically, if I hide my process and you enter the machine and you can see the process, like the ping process and then you QCTL get pods and you don't see my container, I'm done. I can start crypto mining or whatever I want. Of course, all the time saying I'm showing here as mitigations like if you enable secure boot, you're not being able to load a different Linux, load a different thing at boot phase in the machine. If you use rootless containers, you're not being able to replicate what I'm doing because I need root privileges. If you have a redollify system, you will not be able to go and inject the Kubernetes pod manifest where I'm going to inject it and in general, if you keep your system update, it's a lot more difficult that there are CVs in the system that there are things that can be exploited in general. Before we go forward and see the example that I prepared, I want to just do a recap of what we will be doing We will be creating a pod in Etsy Kubernetes Manifest. We will be in a master, in a master where we have access to Etsy Kubernetes Manifest. The master will also have the folder Etsy ldsopryload the file ldsopryload and we are able to connect to EtsyD for deleting pods. Of course, all those reactions that we do can be seen from a kernel perspective. I will also be showing how to use Falco to see those activities right after I do the the IDing. I will hide myself and then we see how Falco complains about myself hiding in that moment demo time. Our demo is starting from the evil pod. The evil pod is my hiding vector, my attack vector to stay hidden. In this evil pod the first thing that I have is an init container which has access to the host network access to the host process namespace. This pod contains a base 64 encoded version and zip the version of the library, LibreProcessSider modified to hide a specific process which is called ping. This is basically the code that library contains. I will be linking it in the slides. The library I just changed the process to filter with ping so that when the library goes and lists all the processes for the process command or for top it will not go directly to showing the process if the process is named ping. How the library works it basically just loads itself before gmc and wraps the readdir call in here if you see it and basically removes the readdir call if the process is named ping and instead wraps the readdir call for all the other processes. I have the library I uncompressed the library in userlocallib processSider as so and in particular I do it in a way that I copy it in the host using the proc by system. Every process has a process ID under proc every process ID under proc has a folder called root that folder contains the full file system of the process, the full snapshot of the file system of the process, the view. So I can just go and load this is one of the three parts. This pod does three things. First it loads lib processSider in the host so that when this is loaded you can see any more my evil action which is just a ping to the call for DNS as I said. The second thing it does after that is that it register itself and continuously removes the pod this self from hcd by removing the key from hcd which is registry pod default pod can control plane every five seconds and this is done by connected to hcd using the same technique so I just go and get the keys since I already exploited the API server I'm in there I just need to add myself just using the hcd key so I'm authenticated and everything and then the third thing it does is just starts a ping command and starts hiding itself and starts doing bad things here I'm in the container because I created the class with kind and I just have a container with everything so I'm in a container with the cluster and I go into the Kubernetes folder and go to the manifest folder and I decided to create everything under manifest instead of using instead of putting my pod in there with the cube API server because in this way I don't have to deal with the hiding of deployment or a demon set I just hide the pod itself and I also don't have to deal with the fact that the request of inserting the pod goes through the API server which might log stuff in the it is how this log or other places I just have a guest for simplicity where I download my explore from and then I can once I start this command I will need to be very fast at doing QCTL get pods because I will be able to see it for really for some seconds where is it and my evil pod started let's see what happens with it so it gets initialization so now it's actually installing the hiding processes library lib process sider it's initializing now the two containers are pending there's one that remove start from CD specifically remove the pod itself from CD and the other is actually there for doing the bad activity which is just a pin it looks like they went away if I now do QCTL get pods it's not there anymore and if I do in the container with the kind plaster a graph of the pin process it doesn't show it this means that my attack went through I can verify that by going to let's see so reload at it so it actually went through I left QCTL without you know hiding from QCTL for the purpose of actually showing that the thing is still there even if we don't see it so I will do QCTL PS collection is actually there is actually doing its stuff and also at CD cleaner so at this point I will then attack her be able to remove this the first thing that they might notice if they have some kind of software that you know let them know that someone's writing under the SOP reload is that they might want to remove the preload so at this point they are able to see ping again but unfortunately they don't know what is doing this yet because I left QCTL behind it could have masquered myself from there so at this point the only thing left is either to have a tool that you know look at things from a different angle for example Falco which is already in there and which is actually still up and looking at what happens Falco already found this is the Falco sidekick UI Falco sidekick is a tool that aggregates all the event from Falco and shows them in different outputs in this case it's using the web UI so that we can see through the browser and I filtered the errors because this is actually a critical log alert and this showing that someone opened up a writing under etsy folder and this is just a cool with the evil body ammo this looks suspicious so this was actually cool that Falco was able to detect this in reality it's not that you have a way like Falco to detect everything there's no magic in there it's just that you can see things from different angles the way the Explorer worked was to hide myself from PS that uses gdbc to the syscals if I do the syscall directly let's say that if PS did the syscall directly I wouldn't be able to hide myself from PS the kernel and change the code in the kernel for the PS syscall so since Falco looks at the syscals directly Falco was able to see it on the other hand Falco was also able to see the container Falco was connected to the Kubernetes API server and also to the audit log so in general more data sources you have different things more you want to see if something happens of course this doesn't limit the ability of the eventual hacker who want to try to hide to find other creative ways of hiding themselves but it should give you an idea of the fact that they might or not might be able to hide themselves depending on what you put in place right so if you just leave the door open if you want to be there you won't see them if you close the door it's more difficult for them to be there and if they get there you don't see them if you close the door and you also install cameras maybe you see them but what if they have an invisibility shield or something you won't be able to see them so you need like a detector for invisibility shields so it's just a game of continuously catching up and that's what we do that's it this was my demo I hope that I passed a message I hope that these can help you in thinking about your security and also in thinking that in general there's no such thing as good security you just have to play the game and thanks a lot for listening to my talk to see you around