 Hi everyone, I'm Mour Weinberger and I'm here to talk about from illuminating to eliminating cryptojacking techniques in cloud native. On today's agenda, I'm going to present myself, then we will talk about the crypto mining process, then we will go over the berth of cryptojacking and why it's so appealing to bad actors, then we will go deep dive into some of the cryptojacking evolutions and the trend and we will end up with detection and mitigation. Okay, so quick intro, so hi everyone again, I'm Mour Weinberger, I'm a staff software engineer in aqua security. I joined aqua a year ago as part of an acquisition of Argon security. Before that I worked for Microsoft for about four years around the cloud security products and I really like to mix the engineering work with threat hunting and security researching stuff and this is basically why I'm here and this is basically why I'm talking about this topic. Okay, so let's start. So let's talking a little bit about what is the actually crypto mining process. So in short, it's basically the process of verifying transaction on the blockchain. This verification process is actually solving cryptographic puzzles and this requires compute power and because this activity of verifying transaction is super important for the security of the crypto network, those miners are getting reward with fee, with transaction fee in a type of, you know, new coins. Okay, so let's see how easy is that to become a crypto miner. So all you need is basically four things. So the first thing you need to choose your cryptocurrency coin. There are like multiple coins and each one has its own capabilities and its own features. Then you will need to buy our equipment and as far as it will be a strong equipment, you could mine efficiency, efficiently, sorry. But if you're a bad actor, you can just hijack systems and run your miner. You will also need to create a crypto wallet. It's very simple and totally open source and free and you need to choose the mining software and configure it on your device. So it's super easy. You can do all those steps in just a few minutes. Okay, so let's quickly review the birth of actually cryptojacking activity. So it kind of start on 2017 when Kornhive offered a web client miner code, which basically means if you are owner of a popular website, you can take their code, put it in your websites, and then the visitors will start mining crypto coins for you, for helping with the funds of the site. Then we start seeing that there were some website, popular website, especially on the video sites that adopt this approach and add Kornhive code into their website. But I think most of them added it secretly, which means they didn't notify the visitors. And of course, the visitors, the users didn't really like it. It basically utilized their CPU till the maximum. Then we start seeing that bad actors that breach websites, focused on government-based and university, affected them with Kornhive code. And we also saw that those bad actors targeting unpatched servers with some known vulnerability. And again, they just affect them with crypto mining. Then we start seeing that they also targeting unpatched routers and actually affect them with Kornhive code, which basically means that on every HTTP request that's sent into the router, the router injecting response add a Kornhive code. And at that point, we start seeing like repo mining everywhere. This is basically an advertising screen in some parking lot. And did you spot a miner here? It's on the right. Here it is. This is a nice hash miner client that run on this IoT device. Okay. So basically, but why so appealing to bad actors? So there are multiple reasons for that. So the first reason is about the anonymization. So basically, when you create a crypto wallet, you don't need to provide any identification. And to do it, to do in the correlation is impossible. And there are like additional coins like Monero that add another layer of anonymity. And they basically not allows to see their blockchain. So for example, if I have a crypto wallet, I'm a bad actors. I steal crypto coins. I am a hijacking systems. And now everyone detects my crypto wallet as suspicious. So once I just transfer the funds into another wallet, it couldn't be traceable because the transaction on the blockchain are not public. It's also very easy to fast cash out. Once I have the funds on my crypto wallet, I'm good to convert it to other coin, use exchange to cash out. And we will see that it's super easy to do it in scale. It's also suitable for newbies and script kiddies. There is no like deep knowledge that you need to have on the cryptocurrency network. You can just run one line of code and you are good to start in the crypto mining. And unfortunately, organization consider this risk as a nuisance. Okay. So enough with that. Let's start seeing some crypto jacking techniques that we saw in our research team called Nautilus in Aqua Security. And also when connecting with organization. So the first thing we will focus on is Kubernetes environment. So on the left, you could see the master node which contains the API server and other stuff. On the right is yet another node. Okay. And we will go over on the popular attack vectors that bad actors, specifically crypto jackers are targeting and looking for. So the first attack vector is basically looking for a vulnerable application which means that in this scenario, we have engine innings that running on a pod. It exposed to the internet and this engine innings has some RC vulnerability. And the attacker doesn't, at that point, doesn't really care if it's run on Kubernetes or not. Okay. Just scan the worldwide APs looking for this non vulnerability since it exploitable. So it's very easy to retrieve the exploit code and just execute it. And now we have a foothold in the pod. So at that point, we starting in the last few years that those actors are start being familiar with Kubernetes environment and they are trying to look for a lateral movement for a privilege elevation. They are checking if this pod run in any privileged container or any configuration like mount to us that allows them to control the whole node. And this is basically the first attack vector that I review here. Second one is misconfiguration on the Docker demon. So Dockers was around a little bit before Kubernetes became so popular and we still see customers and organizations that are using Dockers. And this one is about misconfiguration of the Docker demons which basically means that the Docker demon pod is exposed to the internet and there is no any authentication or access level control that is required. And again, this bad actors basically scan the worldwide looking for this specific port and they are just starting to communicate with the API. What they are actually able to do is to list all the images, list the running containers, getting shell on those containers and deploy new containers with crypto mining. Third attack vector is about poisoning the public registries. For example, Docker hub is the most famous registry and we see that actors doing something that call this technique, sorry, called typosquadding. And basically what they are trying to do is to push malicious container. For example, this on the right we have the valid and popular tensorflow Docker image is very popular machine learning framework. And this actor published Docker image called tensorflow, which as you guess, contains crypto miner. And they are basically depends on the fact that developer will misspell the container image. As you can see, this image specifically has more than 1K pools. So this technique is quite work. Okay. The fourth attack vector is about misconfiguration of Kubernetes dashboard. So Kubernetes dashboard is an easy way to interface on your Kubernetes cluster. And what you actually capable of is to list all the running pods, list secret and even get a shell on those container, all just from your browser. Okay. This is actually an exposed Docker dashboard in the wild. And once you and this basically allows you to have full control on your cluster. Another attack vector is about accessing the Kubernetes API when it misconfigured. Again, the API, the community API port is exposed, and it doesn't require any authentication or access level control. And the Kubernetes API is actually responsible to communicate it with the node. And once you able to control it, you basically have a cluster level control. And the last one is about misconfiguration of the Kublet. So the Kublet is stored on every one of the node and irresponsible to communicate it with the Kubernetes API. And once you misconfigure it, so the bad actor has a node level control here. And basically he allows to execute command on the running containers and deploy new containers. And at that point, we also saw that those bad actors are familiar with the Kubernetes features itself, and they are leveraging them in order to deploy the mining containers in a scale. It means they are using demon set deployment and replica set in order to deploy their stuff on every node and on every pod. Okay, just a second. Okay, so basically those attack vectors are being around for like many years. But we still see that cryptojackers are upgrading their techniques and doing the optimization. One of those optimization is about changing the huge pages, which basically allows the actors to increase the memory page. And basically those actors, once they have a foothold on the containers, if they will just run the miner as is, it will not be that efficient for them. So they are basically looking for how many cores and basically the architecture of the environment. And they set the huge pages to be optimal. And by the documentation of a few of those open source miners, this could help with increased efficiency by 50%. We also saw that those bad actors are aware that there are more bad actors on the environment. We also see it on our honeypots that are like multiple actors that basically battling each other because no one of them wants to share the resources, right? So we see them trying to disable and kill each one of those. And we also see that they are adopting techniques of wood keys and filers, and they are also trying to disable and remove the cloud security agent. Those two examples are one of Ali Yun, which is the Alibaba cloud agent, and bottom one is GCloud agent. But what do you think? Okay, we saw some techniques on the Kubernetes side, on the runtime side. What do you think? Are those cryptojackers shifting left with the ecosystem? Are they aware of the new technology that the organization are adopting around the left side? So of course, yes, this is basically a campaign that we discovered a year ago of bad actors that are trying to abuse the CI-CD free tier. So basically, they are not needed to find any unpaid servers or misconfiguration. They are only need to craft some techniques that they can do it in scale. And basically, this method is unlimited. So what those bad actors are basically done here is creating a GitHub repo, connect it into CI-CD platform, okay, use the free tier, of course. And on every change of this repo, it's basically trigger a build job, a build process, all right? And basically, those actors understand that the build step could be hijacked, the resource of it could be hijacked in the favor of crypto mining. Okay, so what basically happened here? How they made it in scale? So on the build step, they set that first they will clone the repo, then they will choose some random file and made some random change, then they will push this change inside the repository. And what will happen? Another CI job will trigger, which basically means at that point that they have an infinite loop, because every build will trigger another build. And after they trigger another build, they are pulling a custom-made miner from another repository, and they run this miner until the maximum time of the CI offering. This table we found on one of those bed actor repositories is basically elaborate what are the free tiers on those CI-CD and SaaS services. And we also saw that those bed actors are learning and they are aware of every limitation of every CI-CD platform. This one targeted GitHub action. As you can see, it set the maximum parallel jobs to 20, which is the limit, which is the maximum allows. They also set that every job will run no matter if a previous job were failed. And they also use interesting evasion techniques here, which their miner is an ELF binary, which basically run on Linux, but they run it on a Windows box. And they use and they able to execute this ELF by the WSL feature of Windows, which is the Windows subsystem for Linux. And we believe they do so for running the ELF binary in kind of sandbox mode. Another evasion technique that we saw here is in order to make the network detection a little bit harder. So what they done here, they are using root keys techniques that are wrapped by NPM package. So they basically just pull an NPM package, which is very regular and popular thing to do as part of your build. But this one is wrapping root keys CLI that allows the bad actors to hide his miner from the process list. And as you can see on the right, they managed to run multiple GitHub action jobs successfully. And each jobs run into the maximum of the allow of GitHub action, until it exceeds the maximum time. Okay. And this one is kind of funny. Shortly after I published the write up about how bad actors abuse those CICD free tiers, I got a message on LinkedIn that says, hi, more, I'm a crypto miner. And I think we share the same interest regarding the field. And I think to myself, that shouldn't be the key point, right? Yeah. Okay. But what about supply chain techniques? Are those bad actors adopting some supply chain techniques? So as you guess, all the answers here are yes. So yeah, they are shifting left and they are aware of the risk on the software supply chain. And just to align here, so basically the software supply chain is start with developers that write code, manage it in source code management, then build it, package it into artifacts, and then deploy it into the runtime. And the thing about it that every one of those steps that I just mentioned, using open source third party dependencies. And those bad actors understand one single point of failure here. And they target exactly that. So the first attack vector that we saw is about techniques that call typosquadding. And this one was discovered by our friends on checkmarks. They saw that these bad actors push multiple open source package into PyPy, into the public PyPy. And it contains, as you guess, a miner and they try to mimic the popular request package on PyPy. And because PyPy just requests the GitHub repository that correlate to the package, so this bad actor just put the valid request one and he gets the exact amount of stars and forks and he was able to really make a good view of this package. Another attack vector is around something that call dependency confusion. It happened with PyTorch just like a few weeks ago. And basically what happened here is that as part of the PyTorch build processing, its load owns dependencies. And some of them were internal dependencies and the way that when you're trying to load dependencies on PyTorch, so you're first going to the public PyPy index. And if he can't find it, he will get into your internal repository. And because PyTorch didn't preserve this specific library on the PyPy public index, so this bad actor found it. He pushed a new package into the public PyPy with malicious code and PyTorch pulls it as part of the build. And it totally got compromised, this version. And if you think to yourself, okay, we saw a typosporting techniques. We also see dependency confusion techniques. It could be like some security solution around that that could help with defense. The thing about open source dependencies is that there are multiple popular packages that are maintained by just single maintainers, maybe few maintainers, and we are not really aware of who stand behind them. And what is the security measurement that he takes and how secure he is here. So we see here a bad example of UI parser that happens a year, maybe two years ago, which is a very popular UI library on NPM. And this one got compromised by a leaked token of the owner. And it got breached badly. This library was used by the most big tech. And this bad actor had crypto mining, but also a password stealing code. And he targeted both Windows and Linux. And he basically tried to target both the developers, the build system, and runtime. But you could think to yourself, okay, what is the actual risk here? So the obvious one is about financial loss and service degradation. Either it runs on your AWS account or like cloud account or on your build, it will increase your charge. You will also face service degradation. But the most important here is about those bad actors has the ability to run arbitrary code on your environment. If it's on your build, they can basically steal your secrets. And on your environment, they will try to elevate permission and lateral movement to other high assets like your DB and other sensitive information. So what we can actually do about it? Okay, so if you don't have any security measurement or if you have some, so my suggestion here is first, follow the compliance and best practices. There are like multiple organizations out there. This is just two of them with security experts that invest their time and their expertise in order to create standards and best practices specifically for each popular technology. And the second thing about it is you need to understand your architecture from your cloud into your, from your code into your cloud, understand what are the technologies that you are using? What are the integration parts? And basically follow those best practices and guidelines. And there are many like open source tools out there that could help you run those, run those guidelines and checks. This is just two of them. The first one is Trivi that basically can help you run on your Kubernetes cluster and even on your YAML before it gets deployed, security posture management to find those configuration that I just present you and others. And Chainbench which is focused on your software supply chain and running the same approaches but on your left side. And those two can also provide you compliance report. Here it's a CIS or other and then you could see your security posture management based on those compliance report. If you got to here so you can also scan your code, your infrastructure as code and dependencies for vulnerabilities, for misconfiguration and you can also do it with Trivi. And you can also run static code analysis. SEM group is a great tool for that with more than 1000 rules that supports most of the popular languages. And it's able to detect risky code even before it gets merged into your source code. Like for example, it could detect XSS and injection risk within your code. But this is not enough, right? Trying to reduce your risk and fix misconfiguration most of the time is not enough because basically you have some vulnerabilities and other misconfiguration that you didn't have the time to address. And there are always like new zero days and other vulnerabilities. So we need to get one layer deeper. We need to look for abnormal and suspicious activity again on your cloud account to look for any increase in your resource creation on your abnormal identity activity. And also you need to have some random protection. There are like great tools out there. I will just mention one of them which is Tracy is based on the EBPF technologies which means it runs on your kernel level and it allows to detect root keys and file s attacks. And if you don't trust your third party libraries, you can use Guard Dog by Datadog that has the capability to run static code analysis and detect malicious code like crypto mining on your PyPy libraries. So this is it basically. Thank you everyone. And I would appreciate if you could scan the QR code and provide a feedback. Thanks. I don't think I have enough time for question but feel free to connect me in person. Thanks.