 Hello, everyone. Welcome to Cloud Native Live, where we dive into the code behind Cloud Native. I'm Annie Talasto, and I'm CNCF ambassador, and I will be your host tonight. So every week, we bring a new set of presenters to showcase how to work with Cloud Native technologies. They will build things, they will break things, and they will answer all of your questions. So you can join us every Wednesday to watch live. This week, we have a really great session by Nigel here with us to talk about detecting cryptojacking in communities' workloads. As always, this is an official live stream of the CNCF, and as such, it is subject to the CNCF Code of Conduct. So please do not add anything to the chat or questions that would be in violation of that Code of Conduct. Basically, please be respectful of all of your fellow participants as well as presenters. But with that done, I'll hand it over to Nigel to kick off today's presentation. Cool. Just to confirm, you can see everything that's on-screen. There's no issues there with the screen share. I think now we see it perfectly. Yeah. Fantastic. So yeah, thank you for everyone who joined today's session. What we're going to do is we're going to work with Alco. So if I say QCTL, okay, I'll start today. We've got a simple two-node Kubernetes cluster right now, and both those are EC2 instances running on AWS. So with this, it's not a complex environment. You can see we have a networking plugin, we have Falco. We also have a Falco sidekick, which you see on the right side pane, which is for the purpose of the demonstration, it's a visual UI to demonstrate the alerts that are coming through in real time. So with that, I'm going to kick off what we're going to try to do today, which is detecting crypto mining in Kubernetes. Now, we're doing that with a tool called Falco. For those who aren't familiar with Falco, it's a runtime forensics tool, and it can show you intrusion detection, any kind of unusual sign of security compromise in real time. Now, how we're doing that in today's session is we're going to achieve that via system calls that are already happening in Linux. So that applies to both containers, because they're Linux and to the host, which are all open to instances in the case of my cluster. So with that, I'm just going to cover, first of all, what is happening here. So historically, what someone would have done is they would have pulled, say from a GitHub repo, they pulled the download package associated with Exim Rig, which is a mining binary for crypto mining. With that, they would have on this same Linux host, they would want to unpackage it. Once they've unpackaged the folder, they'd want a CD to that directory, which is the Exim Rig, whatever it is. And from there, they can then initiate cryptojacking. Now, this whole process is pretty straightforward. You know, I can, you can see here, this is one single command, and this is going to perform the cryptojacking, or the crypto mining, I should say, on the host, which I'm using the executable.exe, or .forward slash Exim Rig. In that case, I'm pointing to a mining pool. Now, for the context of this demonstration, all kind of crypto miners, in order to share resources or pool resources, they connect to different mining pool IPs or domains and associated ports. In this case, it's, the domain is XMR US East One location on bananapool.org, and the domain is 14433. Now, in order to authorize or authenticate to a mallet, you know, you can see you've got this hash value here. And also just for the purpose of my command, I'm silently using it using the background command. Now, once I run that, I'm crypto mining. So it's that simple. If you want to see what it looks like, you can say .exe-m-rig, and this is what crypto mining looks like in reality. Now, that's pretty straightforward. You can see there's no errors, so I'm able to connect to that Exe-m-rig, whatever location in the port. So I want to close that process out for the second, and I want to explain about where is it becoming such a problem now for the disease environment. So we can CD out of that directory. We can remove the directory. I'm even going to go ahead and remove the zip package that we just pulled. Now, the thing about crypto mining is it's so much more than just a host. You know, if I just kill that process there, you can run it as a container in Kubernetes. So I'm just going to kill that process. That should be fine. That should be gone then. So with that, you can run your crypto miner in this case as a Docker container. You can run it as a pod in Kubernetes. Since Kubernetes is highly scalable, it's an ideal target for cryptojacking, especially at the moment when we talk about the cost of cryptojacking in different environment, or crypto mining. The reward isn't really good for individuals. So utilizing someone else's resources, compromising that environment, especially the scale of Kubernetes or the scale of an EC2 instance in AWS, you know, it's a broad target there, and we can take advantage of that. So to show how simple it is or even easier it is to do crypto mining as a Docker image, you can see, I just run this single command and that's it. So I'm saying sudo run as the administrative user. I'm using the Docker run command, but I'm using dash dash rm. So essentially remove it after I run this interactive command. And the image in this case is metal3dxmrig. So it's the director of the folder where it's hosted is metal3d, but the binary in this case is xmrig. So if I hit enter, you know, same thing again, you can run the crypto miner. I might just give that a, oh yeah. And you see in this case the connection failed. So I went it to any, you know, authorization. I didn't even generate the wallet associated with it. It didn't authorize the connection. So we can just close that and that's the end of the crypto mining as Docker. Now with the fact that you can run it in different environments, it means, well, first of all, how are we going to detect it? That's the whole point of today's session. So you can see on this right side, pain, if I refresh the URL, you can see I got a bunch of detections. I actually forgot to turn off the refresh or enable the refresh. So at this moment it's going to update every 10 seconds. So I have a bunch of rules I configured in Falco to detect the indicators that compromise associated with crypto mining. Now the first one that's pretty obvious is to detect outbound connections to common miner pools. So if I said cube CTL get CM config map in the namespace, or yeah, it's going to be called Falco. It's named the config map and it's in the namespace Falco. Now, oh yeah, that shows that it's there. So if I want to edit it, I can just show you quickly what the rule looks like. If I run fordash and I just want to look for, let's say minor or pool, I think it's the best way. So you can see here, what Falco can do is we can figure a list. So a list is just like a list of things. So we have a list of ports, which we defined as the name minor ports. We've list of domains, which are all the domains associated with crypto mining. And also if I was to scroll down a bit, I can actually hit enter and go down through these. I've also defined, or you can do it as a IPs if you want to do it that way. You know, there's no limitation to how you define the different attributes that are going to find your rule. Now, I don't know if I have to find it below or I'm going to do it above. I think it's just above it. So we'll just scroll up a little higher. So yeah, you can create a macro that finds what is the conditions. The list is just a list of objects. And then we can create the rule, something like this, where you say create hard link oversensitive files. So if I wanted to say forward slash, let's say common, I guess. Yeah, there's the name of our rule. So it's detectable and connections to common minor pool ports. So in this case, we would say there's a rule name. The description is what is it going to do for us? And then the condition is defining those macros and lists. So I have a macro called netminer pool. And the conditions of the netminer pool is saying, it has to have these domains and it has these ports. If I define that condition, we've already proven that we can detect unusual network connection. Now, this second one here, we see that on the right side pane, I have this minor binary detected. This is not actually in the Falco rules. So if you want to actually see Falco rules today, you can go into the Falco repository. You can go to the rules view. And from here, you can click on Falco rules. Here you then see all the default rules that exist, like the one we talked about there a second ago, like pool, you would see all of the list of items that are in it, but there's no default rules similar to this where we see minor binary detected. So in that case, what I've done is I've actually created my own custom rule just before this session, where you can say, maybe if I type in binary, you can see, yeah, minor binary detected. All I'm saying in this case is I want to detect malicious scripts or binaries detected in a pod or host again, because it's all based on system calls. I can see it happen in the container. I can see it happen on the host. I can see it happen in Kubernetes because we're relying on this exec CVE system call. Now, as for what the output is, that's essentially what's going to show up in the alert context. So what you see as an administrator or an SRE is the output of the alert. So you can put whatever context is relevant. In my case, I want to see, for instance, user ID. I want to see what was the node that it happened on. All of this relevant supporting context, even container name. So I understand where the incident occurred, when it occurred, so like timestamp, so I can then take further action on it. Now, what defines this minor binary detected? I basically said, you can see at the bottom here, list malicious binaries. So I made a very small list. It's only like six, I think, defined binaries. You can modify this whatever way you want. But I said, if I see an example of XM rig or nano-minor, pawn-rig or astro-minor, whatever the minor is, you can constantly play around with this however you want. You define in the list, you can build a macro, which says in malicious binary. So I say, if the process name contains any of these listed items, that's how our macro is defined. And then when I create the rule, I can say, spawn process and it was listed as the macro in malicious binary. So there's this logic of rules embedded macros and in macros, there is the list items that support that. So now that we kind of understand the logic of how rules work, we can go a little further into triggering these different rules. So yeah, with that, I said, we can run it locally, crypto mining with XM rig. We could do it as a Docker image that we saw there a second ago where we just run Docker run. And it's super simple to do this. And you can see the XM rig that was run there would have triggered under the minor binary detection. Now, things like the first thing I think is important to notice, how can we prevent this from happening? So there's different initial access issues. So the first one is if you have a pod and it's given elevated permissions, let's say I run LS, I have the first pod here we're looking at, it's called security context YAML. In this case, in the pod definition file, you can say security context and this is like what kind of permissions we're gonna give them. So we've run user 2000 as the permission set, but also there's this allow privilege escalation that's disabled because I guess as an admin, I don't really want the pod escalate permissions from there. The alternative because you can say true is if I was to cap the privileged pod, you can see this one has security context set to privilege true immediately. So essentially root permissions. Now, if I wanted to show you what this looks like in real time, we just have to say cube CTL apply dash F on the security context pod. When I do that, the pod is now created. If I wanted to shell into the pod, you can see then it'll open up the session. Now, refreshing the URL, this in itself will trigger an alert. So I terminal shelled into a container, unless you have a justifiable reason why are end users shelling into the pods? Like preferably we don't wanna drift from the original design that we deploy in our environment. So yeah, again, there's legitimate reasons you might shell in for troubleshooting purposes. That would be the only real reason. But for an adversary, they would do it for other reasons. So in that case, we've shelled into the pod. If I was to go inside the pod, it works the same way as any other Linux host. You can run PSOX and you can see what are the commands or what permissions are set inside the environment. So if we tried to do what we did earlier to manually install the crypto miner in a non-privileged pod, it would say something like this, permission denied. I'm unable to install the crypto miner in this environment. So with that as the attacker, I'm kind of in a bit of trouble here. Now, if I was to say Qubes ETL delete-f the, I don't know what it was called, security context set to and create the other pod, which is Qubel apply-f privileged. So if I create a privileged pod, same logic applies here, the pod is now created. You can say Qubes ETL get pods to see the pods running. Hopefully it's running by now and I can just hit Qubes ETL exec with that pod. And now you can see the difference between the first example and the second one is rather than running as the standard user, I'm running as a root. So now that I'm running as a root, I can do the same command except this time it will work, which is to curl, I'm pulling the package. I'm gonna do exactly what I did on the main EC2 instance, which is unzip the package, which gives me the same directory. I can see the Xembrig to that directory. And once I'm in there, you can see the files, which is the conflict file, the Xembrig binary. So if I wanna do crypto mining within that environment, it's quite easy to, well, actually I can even elevate permissions on a pod. So say something like CHmod, if I wanna set a repeaser ID or set git bit, sorry, on this executable, if I was to refresh the URL, again, you can see these different behaviors. So things like running CHmod, any CHmod, so drift from changes from the original design, that will be flagged as changes in a file directory on a container the same way it would be on Kubernetes host. But all these other unusual behaviors, like if you looked at this holistically on the right side of the pane, there's someone's launched a privilege container, then they've terminal shelled into that container. If it was a real crypto jacking incident, they would probably just take an existing pod, executive pod, if it had the privilege set to true, and do the damage from there. But in the case of like, trying to do further escalation in environment, yeah, they would want to create that privilege pod. There's nothing in place to prevent them from creating that. And if there was no visibility, if there's no file code, you wouldn't know that some created a privilege pod, they were jumping into it, making all the file changes. They called the pod some legitimate name, not like my one, they might, yeah, they might call it like Redis database, something on those lines. So you want to be able to see examples of just unusual patterns of behaviors, like setting escalated permissions on the pod. So in this case, when they run the crypto mining, which is basically dot forward slash XM rig, it works the same as it did in any other environment. If I was to refresh the URL, we should see the binary detected where it was the use of XM rig. Immediately it tells us, you know, the command we just ran. So it gives us all that additional context. If I was to refresh the comment from the audience that if we could make the C and D prompt a little bit bigger because they're having some trouble seeing that. Oh, of course. No apologies. Does that look better? Or is that hopefully more visible? I think it's better, we could maybe make it a bit more bigger as well. Yeah, of course. Yeah, it's my, unfortunately the screen is quite big. So yeah, this should be okay. You should be able to see what's going on now. So in this case, you know, we just showed that crypto mining works the same on the container as it does, you know, on the host. So if I was to exit out of this container and we said, cube CTL delete dash F the privilege pod, you know, that has solved our problem for now. But it's that knowing what is the dangerous part? Like no one's gonna call their pod privilege part. Well, in this case, the name was actually test pod one, which is like, again, a bit sketchy from a visibility perspective. So what we wanna do is maybe look at that example again, something about creating the privilege pod. There's a few things you've noticed here is like, if I was to create a new privilege pod, this is something from scratch. Now, like people might, or sorry, adversaries might, yeah, hit enter, cool. So we created that new same definition file, actually, we're creating a brand new pod. Every time we do something like this, destroy a pod, create a new one, notice how like, delete or rename shell history, like you just deleted all the associated data. So even like leaving a trail behind the attacker, every time they delete the pod, they created five minutes ago so that they don't get detected, we get the detection, we see all of those events. So we stream them to sister, or through Falco, sorry. So in this case, we would see, delete or rename a shell history. So they don't hide detection. So in this case, if they wanted to shell into that same pod, which is the test pod one, they might want to, before they do the actual connecting to the minor ports, so on, they might wanna put in some kind of suspicious networking tool to actually perform those actions. So one that they might want to use is something like Telnet, or NMAP, or whatever. You can define these tools yourself. Now in my case, I tried installing Telnet. It told me, look, you can't, because some issue with mirror list. Now, if I was to try and bypass that, so I moved into the repo directory. And again, it's even this example of, why is the attacker, why would anyone in our employee, in our company, shell into a pod, start using the YUM package manager to download things. They might be trying to download a tool that they need to use, but again, we should treat those as kind of suspicious behaviors. Like, why are you bringing something, an insecure tool like Telnet, into your environment? So this whole, like, making changes on a pod, then trying to run, for instance, YUM again, they would be able to do it, but we want to be able to trigger those detections. So we can refresh our URL, we can see, like, launch package management process in a container. Again, unusual thing. Or even just the process of updating the package manager. So when I ran YUM update dash Y, that in itself is suspicious. There's obviously a spew of things happening from the package manager. So we see all those different alerts coming through. So again, it's just how you want to define these guardrails, so that, I mean, of course, detecting the connections going out to the IP port or domain. That's all great, but it's already happened. They've already compromised your environment. They're already wasting your resources. So what we want to do is be able to assign a compromise before they happen. So in this case, if I wanted to install Telnet or any of these other networking tools, you would just run YUM install Telnet on our side if we were to refresh this URL. It's just because they don't have the automatic refresh. And you see their launch package management, sorry. There should be, I think, the suspicious tools if I look at the suspicious. Oh yeah, there it is. So launching, I actually did this one from earlier. Sorry, I'm still waiting on it, but earlier I did the same thing. I installed Telnet on the container and yeah, it launched, it effectively created the alert telling us that a suspicious network tool was created in the container. So what we want to do is again, look at things like terminal shell into the container, any file changes, moving to the temp directory to evade detection, deleting historical changes they made, deleting shell history. These are all things that an adversary would want to do to evade detection so that you can't, as an incident or some problems in forensics team, see what's going on. But because we are streaming all the events somewhere else, so even if the container is killed, we still see what happened within that environment. So in that case, even if I was to exit and say QTTL delete dash F privilege pod is gone, we still see all of the actions as they happen. So we just have to refresh URL and we see these examples of all the, again, delete or renaming of shell history. Now, the other thing that's worth pointing out is creating a crypto miner as a Kubernetes deployment. So if I was to say LS, we can see the miner deployment. So if I say cat miner deployment, it's really straightforward. So you can, as an example, create a deployment definition file. You can call it miner. Again, if you don't want detection, you probably call it something different than miner. You would call it legitimate database and then you would put it in the namespace of, database name, namespace. So in this case, what we're trying to terrify is whether you create it via, your W gas, you pull it and you run CH mod to turn into an executable or if you create it via deployment we will still spot that image where it's pointing to, again, XM rig or to any of the other miner definition that we put in that list item earlier, which is really important. So in this case, if I want to say QTTL apply and create that pod, we can. Now, one other thing that's worth pointing out as well and I may not have covered this already but if I say QTTL get CM and if there's any questions at this point about anything you've seen throughout the demo so far just feel free to ask. There was actually a audience question right now from Oliver. Would the hacker not be able to knock out the Falco container running in the same pod? They're assuming that that's how Falco does it since it needs to be able to monitor process names and so forth. Yeah, absolutely. Like that would be one of the things that they would try to do. Important to say there is like, you know, when you create there's still the deployment file if you set the what's the term where you only administrative privileges to make those changes in the first place you know, only, you know an elevated user would be able to make the changes on it. So even if they try, okay, killing the pod it'll still get recreated as part of the deployment. That is true. That is one of the things that they would try to do but of course the pod is gonna keep recreating. So they'll only be like short outages, I guess and it'll make it nearly impossible for the attacker to continue that cycle of trying to evade detection as long as the events are still being streamed but it is a good question. Great. Cool. So one thing I would say there as well is oh yeah, we talked about this config map for Falco in the name of Falco is not every, oh yeah, I should edit not every rule is going to be enabled by default and there's good reason for that. So when we talk about the default rules I think there was the setGid bit. Yeah. So I have a rule here and if I hit enter you can see it where it's setUID setGid bit true. In order for that to work I did of course change the enabled to true from its default setting which was false. Now it's really important to clarify here that there are some really interesting rules. There's I think there's an example here for C2 servers. Yeah. So like you can create a rule as you can see here the text outbound connections to a C2 server. Although I believe yeah, it's enabled by default. When you look at what is the definition which is the list, the C2 server IP list we can actually then look that up. So C2 underscore server IP list and you can see that the item is actually set blank because at the end of the day it's for the end user to decide what is it I'm trying to block. Like if you wanted to say we don't want outbound connections to a service like Tor because again, an adversary might use Tor to keep the connections or they keep a bit of anonymity when it comes to speaking outbound from the environment they don't want you to track where those connections are going. So what you could do is create an item list of all of the IPs associated with the Tor network that doesn't change often. So again, it's not something that you need to be updating manually regularly. So you could certainly put that in the form of a list or create your own new list and call it something like Tor Relay it's something I've tested before. But ultimately the rules are really easy and customizable to say something on the lines of outbound connection to a C2 server. And from there you can again specify what is it we want to detect. More importantly, the output is what is it you're going to ship back home for further inspection. The other thing as well is when we talk about moving through the environment into different areas if I was to go quickly since we're looking at this purely on system calls but since we have a different we have a plugin architecture where you can plug into different tools or utilities. If I was to go to here, we have a GitHub repo. Sorry, we have rules feed for a GitHub plugin that we have for Falco. So you can see if you plug in to GitHub so for the logging source that comes from GitHub you can build your own rules, these default examples but an example of like for instance GitHub actions that are being performed on GitHub and that if they contain a miner so like we talked about XM rig or any of those minor examples you could absolutely have something say GitHub workflow has miners set to true and trigger as an alert so that you get a critical for instance alert to tell you this is happening. The other way of looking at this is if you have the different plugins from the different sources it's very easy to use Falco sidekick to filter through to understand okay, where is the big issue coming from? So for instance, we have our different sources right now only using system calls. So plugin sources include Kubernetes audit logs so if you wanna see for instance events for when a deployment is created a config map created or deployment deleted or a config map deleted for instance you would certainly get those events. You could have a plugin for GitHub or Octa so again you're now going from or yeah AWS for CloudTrail so being able to say here are my identity provider like Octa, here's my supply chain and form a GitHub and here's my cloud provider in the case of AWS being able to stream all those events handle them through the flexible plugin architecture it's important to then say I can send it all to a centralized view again I can forward it then over to automation tools for deciding what to actually do with those events or ship it off to for instance a notification tool like Slack but also from the search I can just search for an arbitrary string so we've been continuously talking about XM rig so if I was to type in XM rig even though the tool doesn't inherently know what XM rig is it's just looking at string matches in the case of things like the minor binary that was detected in the case of riding below the root directory or again escalating permissions any of this we will see that happen throughout not just in the case of the minor binary for instance like a pod could be called anything and it could talk to these outbound connections at the minor pools and ports but in our case if we scroll across we know it's happening for the yeah it was the command that was actually written in this case was XM rig which makes sense because that was the utility we just ran so in that case whatever the context is like again the command it could be the pod name you would get Kubernetes context so like if the pod or I don't know whatever the context is if you see that match string context it will show up in the rules alert context building your own rules is really easy there's a few different ways so the first way is we can actually just go into the config map like you would done here you can click in I and you can just paste in the rule list macro whatever as long as they match you can then close it and then kill the pod and recreate and you get the changes the alternative approach is if I was to go to I believe I have it here oh yeah I'll just put it on screen so I'll quit so if you were to go to cd forward slash ETC forward slash Falco is there a Falco? Yeah there should be a oh yeah I might have to create it there should be a rules.d directory in the Falco folder and from there oh yeah I don't know why I couldn't move to that directory but that's okay basically you should be able to modify the existing like custom rules file and from there you can put in your own files and then Falco will update it to respect those new rules that you've created which is a really nice thing to be able to do I think it should actually yeah you can do it via Helm as well so you could say Helm install Falco dash F specify the YAML file and then say whatever the repo was which is Falco security forward slash Falco regarding creating the environment that we looked at today I put it all into a GitHub repo what's also important to know as well and I think we might have put it in the chat in case we can confirm that but we have like we've created like Falco blogs on these topics so again crypto mining through GitHub actions or if you want to our official blog you can see the examples for like crypto mining detection using Falco we go through all of this in depth so that if you want to create the same rules if you want to understand how the rules are built what is the logic behind it but most importantly how it shows like we showed in the demonstration today the real-time alerting you know that's the ideal outcome is that you can go in here test out Falco since we're talking about system calls whether it's an AOT device again whether it's an EC2 instance running in cloud it really does not limitation anything Linux related it's going to be perfect on and that's really exciting because it means that as you're testing these other tools you can use Falco for intrusion detection again it's a bunch of different operating systems of sports so you know whether it's a Ubuntu in my case or if it's CentOS or whatever I have pods I was creating earlier that were CentOS it should work the same in each of those cases so yeah just to wrap up I guess we just had a few slides on this topic and I can just share on slideshow and again if anyone has any questions you know feel free to ask but the main thing is that like Falco could be used as a network intrusion and I should put on a little light in this room it got very dark very quick so yeah you can like detect outbound connections to those IPs, domains, ports so it can be used as a network intrusion detection too also for looking at ingress or egress traffic again from suspicious IPs or from whatever you can define what is even from logins or some other activity you could say some new login change was made from a user from this approved IP and that way anything that isn't the approved IP you could say is not equal to and therefore see the unusual activity but in the case of crypto mining it's pretty consistent the only exception to when you wouldn't get this detection is if for instance they hit the mining server behind a CDN service like Cloudflare that way and again in the Tesla incident they did something like that so their IDS tool didn't actually pick up on the connection so that was made to those IPs that's why it's important to look at all of the indicators of compromise not solely connections made to IPs so one thing we said as a clear guardrail for organizations was to set this allow privilege escalation defaults I mean there is so much more than you can do to that like you can use OPA like a gatekeeper do you know to define what can and can't be admitted do you know admission control in the environment that's an important tool as well to implement but also things like your network policy enforcement so you could use a calico or a psyllium or your service mesh to define what connections you allow so you can ultimately prevent the connections going into those mining pools but that only stops the you know the connections for the miner the miner still is using up your resources so you know good place to start is to look at privilege escalation because when we look at it in the context of the MITRE table you can see yes privilege container is a technique in the tactic privilege escalation but where else can they move to so things like node to cluster escalation control plane to cloud escalation compromise submission control there's so many other things they can move to and that also justifies more use case for why it's good to use the pluggable system of plugins our hither flexible plugin architecture of alco so that you can see okay someone's escalated to cloud these are the changes made in the cloud environment so they've escaped easy to instance but sorry we have two questions from the audience coming in right now already so Tyler says this is really powerful stuff when should I be implementing Falco directly into my cloud native ecosystem as opposed to paying for a tool like light spin that uses Falco to do this stuff for me yeah you can go whatever way you want to approach this I think when you're talking about when to bring it into my cloud native ecosystem I think a good place to start is when we talked about the documentation we have I'm pretty certain we have a quick start guide on this so if I was to look up a quick start the reason I'm pointing towards the quick start guide hmm I don't know getting started it must be and then you can say running yeah so that's cool so there's different ways you can go about it obviously you can just install it in your environment there's the Falco binary you can install as a docker container and then we also have like helm charts and so on for installing it but I think we also have the example for mini cube you know if you if you have a little test lab and you're running it in your you know your like a home lab environment or running it on your desktop and you you know you're not ready to test out Falco and all its capabilities like in your production environment this is a great place to start you can just you know you would get the tool mini cube it's just a little utility from that you just run mini cube start specify the virtual box driver and from here it's like a few simple commands to say look here's the helm repo command so I'm pulling from that helm chart I'm updating it naturally installing Falco it's like three commands once installed yeah you can say you know logs which is this is just using purely Falco by the way this way we can say here's the log output to prove that it's working and you know you can start modifying rules accordingly in our case we went a little bit further because we use extra utility called Falco sidekick we also have this document to this well and that's for streaming events out but also just streaming them to this graphical UI which I'm currently accessing it via port forwarding which has been running for throughout the whole session but yeah if you're talking about a place to get started I think mini cube is the best one get familiar with how the rule structure works how to modify your own rules and from there absolutely then start looking at bringing it into a staging environment perfect and then there was a question from ITSploit who asks is Falco able to do remediation or is it only alerting and actually Miquel already chipped in and said that Falco is only for threat detection tool yeah that's a good point so right now Falco is yeah Dick you can see it on their landing page it's purely about taking all those different sources and then being able to inspect what's happening rules of course make it easier to see what the unusual activity is and when we create some curated rules for you and when I say some quite a lot so that you know you already have them in your hand but you're right in saying it does not automatically take action to you know to do something we could kind of we could use Cisco or sorry Falco sidekick to hook to send webbook to the you know your third-party tool that can do that automation process but that is not handled within Falco so in here it is purely think of it like a security camera it's if it's always on it always sees what's happening perfect is there any other questions on that topic I just want to make sure not at the moment there was a person saying this is amazing thank you so much and so forth so but no questions at the moment awesome no problems so I might just go on to those last slides just to make sure you know we have as much information into your hands as possible so you know we talked about detecting privilege escalation if you can avoid creating overly permissive paths same logic lies to cloud environments where we talk about overly permissive IEM policies the user policy of what they can and can't do think of the same logic to pods you know it's easy to create a privilege pod for testing purposes but it's not good in the case of security posture you're just putting too much control into an adversary's hands if they get access to your environment the other thing is we talked about the custom rule and maybe this was confusing to some people earlier what I showed here was this is a rule that is not in the default rules list I created it purely for the case of cryptojacking to show how easy it is to you know in five to ten minutes you can create your own rule to you know detect crypto mining so in my case I wanted to have something that would say this exec cve system call which is basically we've seen it throughout the demonstrations if in a pod in a host you know on a container if this exec command is run so if we see here we're looking for any process name that's listed in the shell binaries the shell binaries it was already there actually this list already pre-existed so that's just saying anything that's like bash dash sh those are defined as our process types that we're looking for but in regards to the macros or the list sorry we created this list ourselves so that was where we mostly we just talked about xm rig but if you had like xm rig i i or xm1r or whatever if it's nano minor any minor you could even have wild card minor if you want to do it like that so any kind of you know minor that you can think of put it into a list lists aren't limited to strings in this case like a like the name of a minor but I think of it in the sense that it can be an ip address it could be a hash value a shah so you could say here is the known bad hashes associated malware chuck those hashes into a list and you can use the same logic to say if a process triggered and it has this hash value that we've seen in the system calls yeah we can detect and prevent that as well so in this case it's really easy for us to say I've defined a macro that's saying if the process name is in the malicious binaries list the list is the list of items and then the in malicious binaries are specified here so if it's a spawn process basically it's like exec cve which was the event type that was run if it was listed then in that macro which entails the list then we would trigger an alert that says something generic like malicious binary script was executed in a pod or host we don't know what this point but in the command output which will tell us either it was on a host well it will be a host and a pod if it's a pod running on a host so that's a really cool one but ultimately you can tag it whatever way I tagged it for crypto mining because that was the whole point of today's demonstration was that you can then filter as well for tags you know you could tag it based on its mitre attack tactic or technique you could tag it based on the type of environment that it's triggered in that kind of thing but ultimately the source is based purely on system calls so we can go really deep there's really very little limitation to what we can and can't see we talked about network and connections file update changes permission changes all of that and then the final piece I just want to go on here is that again we talked about the default Falco logic where we have you know certain drivers that are in place so we monitor those system events that are coming from the kernel we have you know our default Falco implementation and then we also have an EBBF driver as well again that's probably better for another session if that's something people are interested in but then we can monitor the Kubernetes audit logs so again we have a plugin for that you've probably seen it in my event output that we can correlate again the container output via system call but also use Kubernetes audit log to see okay what was the deployment what was you know that was associated with that part that was created again you can follow the full chain of activities and that's really important so any new source activity that comes from it we enrich that data and then finally when we talk about that standard API that we provide it means that we can use our same existing architecture to then understand okay what is the logging source of the other tool like SaaS platform whatever it be that we're trying to plug into and ultimately build the same consistent rural language as long as when we go back we just specify our source from syscall we could change it to k8 audit we could change it to aws cloud trail whatever it is that we're trying to plug into and which is really cool and again if you're a developer and you're really interested in making these plugins for a tool that you provide there is no reason why you can't contribute to the community in that sense and so with that I hope everyone kind of found value out of today's session we have like a developers guide for if you're interested in building plugins I've listed the Falco rules because we've rehoused our Falco rules so again you can access those even if you're not testing it today you can read those rules today and understand how they're built and what's the logic behind them from a kind of an educational further learning I've provided a blog for detecting crypto mining I wrote that last year so it's not too old and the rules still apply today and the other one was if you're interested in seeing how crypto jacking works in other environments I've also given a blog article for the Falco plugin for GitHub so that way you can detect crypto mining on a Linux host but you can also see it in a SaaS platform like GitHub which is really cool and also if you want to follow us you know we're pretty much accessible everywhere so we have the GitHub repo and we've linked that accordingly where you can follow us on Twitter I'm pretty sure there's a LinkedIn page as well but either way it's the same name and I also put my own GitHub repo for today's session so if you want any of the commands I ran to do your own crypto mining and ultimately just to have the rules running and detect the crypto mining in your environment feel free to use it as publicly available so with that yeah I think we've covered everything I just want to make sure that we've got time for any more questions if anyone has any Yeah there was a few comments so far and oh a question came in immediately perfect that's good I keep the questions coming in we have 15 minutes for them so no worries for starters there was a attendee commented that they found the Seustake Falco 101 as a great resource for getting started on Falco and linked to that one that was really nice and then we have Andrew commenting alternatives from Minikube just in case you are against running VM on your local machine check out Kubernetes without Kubeflit k-w-o-k and then he gives the links for that if you have any comments on that happy to hear that but then there was also Ashkan commenting or actually asking a question about is there any available supports such as tools or references that can make creating a Falco rule simpler Is there any references to make creating the rules easier there should be so when we talked about our blog or documentation sorry separately yeah absolutely if you go to the Falco rule section and we look at like the basics of rules and how they're created and conditional syntax this is a great starting point like probably the most common one that everyone's going to look for when they want to test off is shell into container so understanding like what are the event types that we're playing with like specify directory container ID whatever it is they show you like the examples here with wild card so on and also things like the priority like this because it didn't specify the source it's by default looking for system calls just like we did today so you know you can see that it says here's the rule it goes through the logic of the conditional set it should have the definition of how like the outlook works so again what is the alert going to be that you find but this is a really great starting point I think for going through understanding the cons syntax so if there's ever anything that you see in a rule and you're just curious you know is it required you know otherwise it won't run correctly but also you want to say like what can I put into it you get a better idea from this table so for instance we talked about source it doesn't need to be specified if you don't it will always use system call actually typically your system call and kubernetes audit depending on context but ultimately if you want to use only one of our other plugins you would then need to specify source equals that plugin but yeah even the logic of macros how they're built how all of this ties into the shelling container example I think this is the the best place to get started but I think also we have a well it's just it provides it it's like an instruct lab so there's I think like I think someone mentioned it earlier about the 101 so you know there's an instruct lab so that means the whole shell is hosted somewhere else on a SaaS platform so you actually just start the you know the session you don't have to put any login credentials or anything and you start you know running some arbitrary commands and it actually shows them in real time so that might be another alternative cool place to get started great if there's any more questions keep please keep them coming we still have a time for them but for now I have a question to ask from my side so if I'm using for example a bucket keeper to ensure all images must be from approved repositories why do I also need to use walk palco as well yeah this is a good question so in the case of like you approve what you can and can put into your environment that's fine but let's say if it's like an employee that's locked into the environment so you know it's an admin and they're an insider threat in this case you know it's fine to say we've said all the guardrails of what can come into the environment but there's nothing to stop the employee who is ultimately you know the one who can do the most from actually running the mining binary themselves or let's say disabling or elevating permissions to actually go and make these changes so it's important to set guardrails like for instance defining what you can and can permit into the environment just like what we talked about setting those privilege set to you know false but we need the intrusion detection as well to say if an employee employee if a user inside an environment is making changes that they shouldn't we want to be able to see them even if they're failing like for instance they were unable to elevate permissions due to a fixed rule you want to say oh we got a detection of someone trying to do something suspicious that way we can you know for auditing purposes you're again you want to be compliant with your regulatory standard you want to have something in place incriminates to tell you okay we have proof auditing proof that you know log activity is there we can see activity unusual activity in the environment and we've alerted on it so so that's one of the main reasons even if they're not able to do something if they're kind of air-capped and what they're able to do we still want to be able to detect unusual suspicious user behavior perfect and another question that I had in mine is so what happens if the outbound connections are going to a server hidden behind a CDN service like cloudfair can you see the mining pool connections no that's a really good point you know I think we touched on that a little bit earlier was how you know if if it is behind some service like that we'll only see the connection to the CDN which in itself is not you know malicious so in our case what we want to do is tell users look if it's the network connection that we can't really rely on we want to be looking at other things like we create a custom rule to there to say if a process is executed and it even has the string listed somewhere to do with XM rig or other mining binaries then we should be able to and again we can show that inside the UI that if we just type in XM rig yeah we get all that relevant context so we can see okay if we can't see outbound connection we know a minor binary was detected maybe we can see XM rig in the context of setting permissions on it or even just a folder you can say like I've seen XM rig associated with file name changes or something in those lines so it's just one of the many indicators of compromise but saying that is probably the most common one people site as something to detect which is network connections so yeah definitely it's about having multiple indicators of compromise in case you're unable to detect those connections great good to know another question from my side so if we have a Kubernetes lab running in K3S running on some IoT devices and edge computing so can you test Falco rules there yeah absolutely so if we're going back to the documentation we had so yeah so if we were to talk about like install or getting started it's important to know like yeah even actually it's covered here at straight so like you can install Falco in pretty much any environment so whether it be you know I don't know you're running on a Raspberry Pi or like an IoT edge device you know even in theory you have a security camera in front of your house if that has like an API that's accessible yeah you could have Falco running on that IoT device you could create a plug-in for that other API and you could essentially stream events for suspicious activity on the camera whether it turns on or off or whatever it be so yeah it's super flexible again anything Linux-based which is you know it's an open standard um yeah you can build your own you know you can run it on all those different operating systems so whether it be I'll go into the install dock whether it be like button 2 sent to S DB and whatever it is we usually have the you know the install documentation associated with it and of course in the case of Kubernetes and K3S you can use the helm charts so it's super easy great and then final question from me so also kind of final call for questions from anyone else so if anyone's typing away trying to get the question in now waste the time to finish the typing and send it in but if an attacker installed the mining binary on the node instead of the container can Falco also detect this and does it require a new role? No and that's you know that's the point here it's like with these system calls you know if we were to go back into DUI here when we looked at the yeah let's say the condition we're looking for was the outbound connection so we can see when we look up outbound connection and these are all our critical alerts in some cases we ran it as that Docker IO generally when we ran Docker run in some cases we ran it within a pod so we can see those you know the alert context what was associated with the activity the play that was different but whether it was on the node whether it was in the container or whether it was run again as a pod deployment or you know in Kubernetes or if it was run as a Docker run we would still get that same context that the the connection was made to the IP irrespective and the same with the binaries we would always see it was XM reviews in each case so yeah as long as we have the context specified in our output of the rule then you get all the context you want to see so you might want to see node name you might want to see container image use because again in each of these cases when we created the Docker run I think it was that run no it was when we created as in the k8s deployment and there was no specific image repository accessible but in the case of Docker IO it was so some context might not be there in the same way it was in another command but ultimately the more context you put into the output you want then naturally you're going to catch it either way but either way the rule is going to be triggered so when we type in outbound or whatever it is that you're looking for in the rule name yeah you will see the same for containers would be on the node great and I don't see any questions anymore here but do you have any final words to remind anyone of any resources or anything that they should be doing next or anything else no absolutely I think just to clarify and I'll double check it there was we have we have I should say an ebook that we have now accessible on the topic here so I could put that in the chat I think it's a really useful resource so I'll just put it in the chat there I think this chat no it's the private chat how do I put it oh yeah it's on the right side panel and I'll share that if you put it through the private chat then we can share it to the public chat that's fine so we have this practical cloud made of security book on Falco so if anyone wants to learn like the inside out and you know you want to implement it within your company definitely a great place to go it's free book you know you don't have to pay for it or anything the other is that we released a project it had Honeybee and again for those who are already you know experienced with Falco and they want to know what the changes are recently with the project we also have that documented so if I was to paste that as well in the chat you can share that with the group so you know we have the new update a new version update of Falco and we're constantly maintaining the project wherever possible so yeah those would be two really useful resources for anyone who's interested perfect perfect ending note for this really great session in my opinion great resources for everyone to check out afterwards so awesome but yeah that's it for today and thank you everyone for joining the latest episode of cloud native live it was great to have a session about detecting cryptojacking in community workloads we really also love the interaction from the and the questions from the audience great to see some discussions happening there as well and we as always we renew the latest cloud dating code every Wednesday and in the coming weeks we have more great sessions coming up so stay tuned for those thanks for joining us today and see you all next week bye