 Hello everyone. Good morning. Good evening. Welcome to the webinar, defense strategies against Kubernetes attack TTPs. My name is Manoj and I'm a threat researcher with Tigera. I'm currently focusing on Kubernetes and cloud attack surfaces. My background has been vulnerability research malware reverse engineering and exploit development at a point. And my previous roles were at Juniper network and Intel technologies where I looked at their security posture. So today's agenda is straightforward where we will look at the attack surfaces available to external threat actors and how each external threat actor which is active right now in the wild is targeting them. And what are the mitigation strategies we can use right now to stop them. So current trends from the Kubernetes perspective is the external threat actors are interested in mining cryptocurrency and this mining activity kind of picks up when cryptocurrency price comes down. And to to mind this cryptocurrency, these attackers kind of try to find the misconfigured Docker API is Kubernetes API to access available compute on the Internet. And for that this can entire Internet to see if they they can find anything. And once they find such misconfigured API is they do. They do get that initial photo. They do privilege escalation and move laterally. And in some cases we found they do exploit and patch CVs so that they get that initial photo easily. So with that this is the third metrics for Kubernetes from Microsoft and as you can see there are almost 45 techniques listed here and each technique can be used to compromise your container attacker can do privilege escalations can can can move laterally within your containers and do the data filtration ultimately using this technique. So today we will be focusing on this initial access techniques from Kubernetes perspective. So on the right you can see this is your typical Kubernetes cluster where you have host or node which is running on your cloud hardware. And this is particularly instance and inside that post you have your container runtime running and you have Kubernetes vertical, which is kind of managing all these containers that you deploy. And by configuration Kubernetes API, Kubernetes API, Docker API can be exposed to Internet and attacker can really find out these APIs and try to get that initial put on if you haven't properly secured it. Your services are also exposed out of these system. So Kubernetes can really expose your services. And also there are services like SSH, MySQL, which can run on host itself. And last but not least attacker can really make use of the malicious images that are hosted on public infrastructure or public repositories like Docker Hub where you can end up pulling one of the compromised or image uploaded by attacker and your container can get compromised because of that and ultimately your cluster. So this is the attack surface usually is targeted by threat actors currently to get that initial put on. And this is just a point in time just before this webinar I tried to see how many Docker APIs are available and how many Kubelet instances which is Kubernetes process running on node is exposed on Internet and I almost found 500 entries for each. And Palo Alto data study a few months back where they almost found 50,000 containers publicly accessible. And we do agree with that number as each Kubernetes cluster can run multiple containers, even the Docker API can run multiple containers and numbers can really add up really fast. So let's begin by looking at our first external threat vector which is Team TNT. And Team TNT came up with its new malware which is Heldegard and this particular malware is discovered by Palo Alto. And this is targeting the Kubelet API to be specific to get that initial photo into the Kubernetes. And this API has to be unsecured in order to get that initial photo and they did the same thing for Docker interfaces in the past. And once they get such APIs, they usually drop a bunch of shell scripts and do the initial installation. And the intent and focus for this particular attack vector is Monero coin mining and they have a bunch of scripts which can look at your cloud I am AWS or SSH keys so that if there is a chance to expand their presence, they would be able to do that. And they have typical CNC capabilities, interesting evasion and DDoS capabilities. So as we started looking at this threat actor closely, the first thing we found was their rootkit and this rootkit was compiled dynamically so that they wouldn't be targeting specific Linux architecture. Rather, the binary should be able to run on as many Linux architecture or flavors as possible and they kept the dependencies minimum. As you can see in the picture only libc and libdl. So with that, the interesting functionality abuse is LD preload LD preload is a Linux ability or Linux functionality where if, if you are running any process inside a Linux and you enable this LD preload functionality. And the binary defined by LD preload will be loaded into that process memory. And, and this will be enabled globally so any process you run, it will be adding the library or binary. This is kind of showed by this LD preload parameter and attacker is exactly doing that. He is loading his, as soon as he compromises or they compromise this particular container. So you get this LD preload parameter. And after that any process that anyone runs, it will load their malicious library into it and what this malicious library does is hooks into read the air and read the air 64 system calls. What this system calls do is if you're going to do PS or LS, it's going to give you a list of processes or list of files on file system right. Attacker will be able to always hide the files or processes it wants. So if administrator goes into a container and does what are the processes running here, he won't be able to see healthy guard processes running or crypto mining, mining processes run by this particular malware. And, and as we looked into this closely. So they kind of have all the symbols already available inside the binary. They didn't strip any. So we were able to look at the code pretty easily and here in the red box you can see that they are directly making a call to a particular symbol to look at the read the air 64 and and they are able to override the read the air 64 after that just to show you how it looks like. So at initially in this, this is a particular container where we are just running a PS command and you can see there is a net cat process which is running here. And after that we just used LDP load and loaded our malicious binary which kind of hides this net cat process. And after that we did this PS command. And as you can see in the output below there is no net cat binary. So that's how this particular technique works. And also team TNT used already available. But the right teaming tool. And what this tool does is it can abuse C groups, if you have a privileged container. So, if you TNT ends up breaking into a privileged container. They can basically abuse C group and run the commands on the host. Here we use this body framework and just did the host name command and as you can see in the red. This is the name of the host rather than name of my container. And after that we in next red box you can see, we were able to read the files on the host, skipping the containers. So this threat actors kind of here what we saw is they used ready to use third party tools and integrated into their malware scripts. So for command and control. They used teammate and IRC as a Viki startups. And as we talked they use LDP load technique using the process header and body and periods for container breakout and credential harvesting. And for DNS monitoring bypass they simply override the host file, and they have a lot of scripts to scan the internet so that they could expand their presence once they compromise your infrastructure. And the intent seem to be the same as we discussed earlier a Monero coin mining. So, this is the, these are the basic capabilities of this particular threat actors. So let's move on to next one. This is kinsing. So, kinsing targeted the Docker API, and we didn't see that kinsing upgraded and targeted Kubernetes kiblet API, but it used a another interesting technique where it kind of uploaded its images to public infrastructure that is Docker repository. And, and users, I think at some point there were millions of users which were pulling down images uploaded by them. And this, this is in our view, the kinsing was kind of advanced compared to team TNTs. And, but the methods were almost similar. And what we found that this kinsing malware just use the bulk project which is already available on the get up. And focus seems to be similar, the coin mining command and control and encryption and the evasion. So, when we looked at this particular malware's root kit closely, it was similar to team TNTs, they use dynamically linked binary to target as many flavors of Linux and our pictures. And this is comparatively advanced as you can see on the right there are a number of functions this particular malware embedded inside their root kit, and all these functions, they could do. They could hide process named TCP ports, directory names and file names. So they were not limited to process names, as we've seen earlier. And the techniques were same that we discussed and the preload to hook this particular system calls. And additionally, these guys used encryption to hide their hide their modules, and this is a simple XR encryption. So, this is one of the image which was uploaded by kinsing operators. And this is, this is the, this is this picture is shown of a two chord die, which can read the container images and each layers and show what were the changes in each layer. And as you can see, it has a Monero coin miner, which is being downloaded here in this particular layer. And this particular image had more million hit when we kind of got this particular image. So, almost a million compromises, wherever it ran. And our another threat vector from Kubernetes perspective is doki and doki is a backdoor and this is a particularly a binary which was previously undetected with n growth partner. The reason to mention this because it uses a domain generation algorithm to contact, though this is not a new technique, but they use the really novel DGS seed in their operation. And they use the dodge point API and dodge point is you may be familiar with the popular cryptocurrency cryptocurrency, which is available on the internet. And if you are not familiar with the main generation algorithm so this is really simple where attackers malware, which is running inside your compromise cluster, and attacker which is sitting somewhere on the internet has access to same seed, which is dead and time or currency or temperature of particular city or even a pending topics on Twitter, and they will use that seed and feed it to a pseudo random string generator and get a random string and append a particular tld to it like calm, depending on a day of the week or month of the year and query that particular domain. And interesting thing is malware which is running inside your compromise container and attacker will be able to create same this and what attacker needs to do is just go ahead and register one of the domains from there and malware eventually will be able to query and contact to that particular domain. And here attackers use the dodge coin wallet. So they pre built a wallet address inside inside this doki backdoor. And what they used as a seed was last money spent by attacker. So whatever the last transaction amount was that was used as a seed for this particular algorithm. And as you can see on the right, this is the dodge coin function that they implemented where they queried and called the last money sent by attacker. So with that let's move towards mitigation and how we can mitigate all these actors that we discussed right now. First suggestion is use scratch images, because if you use a scratch image, it will contain only dependencies, which are required by your application and won't have any API to execute or invoke any other binary which is, let's say, a Rob attacker, and it may not even have the dependencies required for that particular executable. So this really helps where attacker have to be very crafty to execute any executable. If you can go a step further you can compile your binary for a specific architecture and link it statically so that there won't be, there will be only one file on your container which will be running. So there is no way attacker can drop another executable and invoke it. If, if you use this particular strategy, and this really mitigates any attack that we have seen using installation using shell states or root kids that we discussed. So when you use use zero trust policies to block access. It is easier to block access, increased access to template and Docker API. You can just implement the policy for that, but we will want you to go further and implement zero trust for your north south traffic because that is possible with respect to your communities, communities clusters, and only white least necessary flows when are coming inside. You can implement that same zero trust for your East West traffic so that if attacker ends up compromising your container or particular service, he won't be able to move that easily. And third thing is blocked access to external DNS. If you are running a Kubernetes cluster, the only DNS server you should be talking to is a full DNS. And with that, on the, on the right, you can see that we have particular suggestions in new that is, if possible, log on your L seven traffic for our L seven data for your service. So any processes that is being spawn inside a container and audit locks for your Kubernetes API and have a machine learning algorithms to monitor all these so that if there is any suspicious activity machine learning algorithm can notify you of that. But not least implement threat feeds. They can tell you if attack the attacker attacks seen by other members of the community, and you can kind of, if you get compromised in the same way that somebody else has, you would, you should be able to Next thing is, you should be enabling Docker contain press on your Docker API so that only images signed by you are running on your Docker. So if you won't be able to run any publicly available Docker images, not just you attack or won't be able to run any publicly available Docker images from unknown sources because of it, and last suggestion is have your container isolation policy ready. So when you suspect a particular workload behaving suspiciously, or there is an issue with that you can add that particular label of a policy to that container and isolated right away so that you contain the blast radius for the issue. And the useful projects are here given by us is project Calico of course to implement all this zero trust that we talked about. And that is a really cool project to gain tell if you are logging DNS domains from your core DNS, or within the Kubernetes cluster, you can check those domains against this LST and this deep learning model to know if it was a DTA or not. And, and that's about mitigation with respect to external threat actors which are currently targeting Kubernetes. So, if you have any questions related to one of this threat actors or if you have any particular use case. Do visit us at tiger.io and let us know. So thank you, and have a good day. Good night.