 Hi, I'm Mike Coleman and welcome to this webinar. I am a developer advocate on the Falco project I actually worked for a company called Cystig, but my full-time job is sort of open-source developer advocacy Prior to joining Cystig. I was at Google working on Doing developer advocacy for Google Kubernetes engine before that I was AWS working on some easy cloud services before that I Spent some time at Docker Where I was an evangelist there Puppet before that VMware before that Microsoft before that so been doing this for a long time been in sort of the cloud container virtualization space for About 15 years and so I'm going to try to bring all that experience in and talk a little bit about Falco and how you can Detect behavioral patterns or activities that will lead you to understand when you might be undergoing an attack or an exploit So we're going to do that in the context of five famous exploits So we're going to start with a quick overview of Falco And after that we're going to look at these five different exploits and we'll have a demo for each one Neo4j log4j path traversal attack on Apache Actually path traversal kind of is in all of these are in several of these a web logic console access remote execution again, it's kind of through path traversal and then Jenkins Jenkins exploit so and then we'll wrap up with a quick discussion about the rest of Falco's ecosystem in the context of how you might use it in What we talked about today, and I'll leave you with some additional resources All right, so let's jump in there if we go back a while and we think about How we used to describe security we would almost always say Firewall like our our systems were inside data centers They had this well-defined perimeter and what you needed to do was just keep people people out We weren't sort of in this space at all where we're thinking about zero trust We we looked at most of the threats coming from the outside and we knew that if somebody had gotten into your space You probably had a problem, right? Well, if you compare that to where we are today with the cloud There isn't this idea of a well-defined perimeter, right your your systems and your Devices many of them sit directly on the internet think of Internet of Things all of these items, you know There's still a firewall in place, but there's a lot more exposure So if you want to have fun go put an EC2 instance or a GKE instance or an Azure instance or Oracle cloud instance Whatever any cloud digital ocean. I don't care take one of them put one of those there Virtual machines out on the internet with a public facing Network interface and see what happens. It'll take a matter of seconds before it's inundated With people out there trying to figure out if there's a way for them to get on to the system and then if you think about cloud as well, there's a whole nother layer of Responsibility right you have I am identity and access management. You've got networking policies You've got virtual private networks. You've got network address translation tables You've got a whole bunch of different services That go together to secure your environment and you have to make sure that the teams that you work with are Using those appropriately and that they're following operational best practices, right? They need to make sure they're not doing things like committing API keys into public GitHub repositories for instance, which is something I have done Right, I'll talk about things you shouldn't do and when I talk about things you shouldn't do you can be assured I've almost certainly done them I have done that. I have put my You know public key or excuse me my private key for my public cloud up on the you know I get how repo where people get it and then you need to be able to understand, you know, this these things I had somebody tell me recently in a Conversation that you know the most dangerous person in you know to your system to your integrity of your systems is a disgruntled former employee and I thought yeah, I mean that's pretty dangerous They can do a lot of damage they have keys if they you know if they're upset They can do a bunch of stuff on their way out the door if you don't know what's happening But I actually think the the more riskier person is the overworked systems operator Engineer, whatever this person who makes an honest mistake Right, and they're not trying to cause problems They just make a mistake or they take a shortcut right thinking hey, this isn't a big deal I'm just gonna shell into this container real fast and do something those type of activities you need to be aware of them right because that's where vulnerabilities can be introduced they're where Prat, you know, you've defined rules of the way you operate and when those get deviated from they can either be an Indication of something bad happening or an indication of something That might lead to something bad happening So if you have a wall around your your property or around your business or whatever and Someone gets over that wall or maybe they're already inside because we talked about you know Somebody making a mistake or somebody being disgruntled. How do you know what's going on? Well? You might use a security camera right use security camera to monitor the inside of the thing You could you could be looking for people wandering around that aren't supposed to be there Or maybe people doing something suspicious and you could be alerted to that right and that is you know That's kind of the next level Of protection so even with the perimeter with or without a perimeter You know a security camera is always good to have and if you don't have a perimeter like the cloud It's more important to have a good security camera that is necessary to have a good lock If you've got no fences around your property security cameras are going to be your next best thing So and that's kind of how we look at falco we look at falco as a security camera for your workloads So what we do with falco is we monitor systems in real time Runtime and we call you know open source runtime solution security solution So at real time or at runtime we're monitoring what's happening inside the system And we're looking for threats and either actual threats or patterns That lead us to believe that there could be a threat and we can do this for kubernetes We can do it for containers We can do it on virtual machines and bare metal hosts and we can do it for cloud services right so We are a cncf project. We're in the incubating stage. We have applied for graduation I think we're in the public comment phase right now And we've got about 60 million pulls of our images up on docker hub and we've got about 6.2000 stars on github So fairly popular production project being used by some big companies to do To you know keep help keep their environment safe. So how does it work? So this is this is one slide and I usually take a lot more time on this But I want to I want to get to the the exploits so With falco What we do is we you know at the most basic level we have a Colonel driver a falco driver that sits and has to interface with the kernel We can either do that through a loadable kernel module or an ebpf probe and what we're doing is we're capturing syscalls So everything that happens in the operating system is going to go through a system call So if somebody opens a file or moves a file or accesses a network port or anything like that We're going to know that um, and we'll know that for containers as well because containers share the same kernel as the operating system right so we Instrument the kernel we capture those system call events We put them in a ring buffer and then they get pulled up into user space right so we operate with a with a falco driver in kernel space And then we have a companion component that runs in user space that compares those activities Against a set of rules to know whether or not we need to let you know what's happening And by default we can send alerts to standard out syslog HTTPS gRPC right so that that user space kernel or the user space component Excuse me There's a lot of things but the primary thing is it does rule matching And then it does something called enrichment which is coupling metadata with syscall data to produce output messages that you can read Right, so we'll tell you what the container ID is what the file name was Who the user was What was the parent process? What was the child process all of these things to make it easier for you to go off and do forensics in troubleshoot So at the most basic level we capture events from system calls We compare them against a set of rules if something is out of whack we let you know And this is a falco rule this is at the output of a falco rule right So this is one where we're telling you a shell was spawned in a container with an attached terminal So someone did docker exact minus it You know my container Slash bin slash bash right and we fired up a bash shell in a running container which is You know might be a thing you do But most of the time for production workloads. It's probably not something you should be doing right? So we want to let you know when it happens Um, and then here's one, you know netcat running inside a container Well, that could be indicative of a whole bunch of things many of them bad And we're going to actually use netcat in in a few of our demos today To do reverse shell for instance. So we let you know about that now rules have multiple levels Notices warnings critical errors. I think is one sometimes I forget all of them. Um, and then we And rules are customizable. You can create your own rules. We ship with about 80 default rules You can augment them. You can change them. You can add your own You can disable them all kinds of different things to make sure that you have rules that meet your needs So that is falco I detect events. I compare them to rules. I let you know if something is awry. So how can that be used to help you? When uh, an exploit might be happening predominantly We're not going to tell you if you're running a piece of code that is vulnerable, right? That's image scanning and our registry scanning and all kinds of things will do that What we do and in some cases we can't do that But for the most part what we're going to do is we're going to show you activities That look like an attack is happening, right? So did someone run netcap for instance is somebody Dropping a new binary into a container and running it that wasn't there before We're going to alert you to sort that sort of behavior. That's what we're going to see in these five exploits All right, so the first exploit we're going to look at look at is neo4j and uh, neo4j is a graph database and In versions point through 3.418 when you had your shell server enabled um, there was the expo there was a remote method invocation service an rma service that was exposed and It will this service would arbitrarily deserialize java objects. So what does that mean? So when you need to put things together in java strings of bytes, you can serialize them, right? And you can you can add all these things together and what's called a chain and then you can send that out and then the receiving Code would deserialize that and then it would just go and execute it. Well Because it was arbitrarily Deserializing these objects and it wasn't really checking them You could put whatever you wanted in them, right and they could they could have bad code and whenever it was deserialized It was just going to run and so the issue here is that someone can take create a A set of serialized code they could send it to the neo4j server And that code gets executed. So what we're going to actually do is we're going to Ship some code up to the server. We're going to we're going to get it installed We're going to running and then that's going to execute So you'll see that in the demo and what you're going to see from falco Is that we don't tell you that you're running the wrong version of neo4j? What we tell you is hey somebody copied a file up here. They probably shouldn't do that And and also there's a binary that's running now that was not in your base engine so Now again, this has been fixed. They've they've replaced this this shell object and in later versions So it's it's a known exploit, but there is a defense against it as is the case in every one of these I'm not showing you anything that hasn't been out for a while that there aren't easy fixes available for so Okay, so let's take a look at this now as you can see here I've got the neo4j server running now I've got the neo4j server I'm going to be using something called exploit dot jar and that is a wrapper that I downloaded from Volhub and it allows us to send commands to this server. So let's go ahead And jump into that So the first thing i'm going to do is i'm going to use exploit dot jar to just Touch a file on the system right create a new file so exploit dot jar URL of the neo4j server touch temp dot owned if I go over and I look at the directory structure Of the neo4j server, which is running in a docker container I go into the container I look at the temp directory. There's that owned file So you can see that I was able to send that payload in via the the shell interface now, let's do the same thing i'm going to download A crypto miner here xm rig and now i'm going to extract that crypto miner onto the system and let me clear the screen because it's getting a little bit crowded and Then we're going to go ahead and let's go ahead and actually execute xm rig So we're just going to give it a command until I'll go mind some monero and again If I go back over to the neo4j Docker container and I do a ps against that actually I can just do here on the host I can see xm rig is running here on the host and I've actually killed it So now if I go into sidekick ui, I've got the zoomed in a little bit. So it's easier to read You can see here that is where you know, this is where I copied the file in And this is the the notification we got and this is the notification that we were executing a binary That was not part of that base image The next one we're going to look at is log4j and so log4j pretty famous It's a logging server and log4j uses something called jndi, which is the Java naming and directory interface And jndi can pull objects from directory services. So when An entry is written to the log that entry could actually be hey go pull this information from this directory service and um And then bring it bring it on in and the thing is that those objects can be just information They could be text or whatever, but they can also be code right and um In certain versions of log4j. This code was going to be arbitrarily executed So you would basically, you know, it's it's a fairly, you know complex compared to you know It's the most complex one we're showing today Or there's others obviously they're much more complex than this But this is a little more complex than what we showed in the last one So what we have here is we're going to set up an LDAP server a lightweight directory access protocol server And that's one of the servers that jndi can communicate with and we're going to set up an LDAP server And we're going to have a Object out there that is malicious and what it will do is it will instantiate a reverse shell And so you're going to see me start an LDAP server start up a A listener and then we're going to execute this remote shell back using this object. So You know and what's going to happen with falco is falco is going to say hey, you know someone's sending standard out to a network connection, right? They're redirecting the console and Netcat is also running. So these two things combined are probably not good even on their own. They're not good So let's take a look at that demo And see how this works in action Okay, this is a java application that logs logins attempts to log 4j So when I go to log in what I'm going to do is I'm going to send it a Directory query through jd and ndi to a server that has a bad payload. So let me start by starting that server I'm going to start it on 104 155 96 228. Let me start up my netcat listener here So this is the endpoint for my reverse shell now. I'm going to go ahead and I'm going to create that payload So basically you can see the ip address there of my ldap server and I'm saying take this command which is netcat to my endpoint and run it So when I log in with this it's going to send that command To log 4j log 4j is going to do the ldap query and it's going to start the shell So now if I come over here, I am logged in right so I can look at the file systems I'm root. I can take a look at the processes all of that stuff So how does this look in falco? So let's go back to the falco ui and here you can see I'm redirecting My standard out. I've got netcat running. So I've been alerted to both of these things which could be absolutely problematic So then from there you could decide how you were going to handle it The next one we're going to look at is a path traversal exploit So path traversal is the idea of trying to break out of a directory structure and move Through the directory or the file system traversing different paths right or different directories And normally if something in in the system and apache has this it's called path normalization And path normalization looks for things that shouldn't be there right and You know if it would if it would see somebody going like web root dot dot dot dot dot and try to get up out of That directory structure would be like that's not cool. We can't do that The problem was in this very specific version of apache 2.4.49 There is the ascii character percent 2e that in ascii represents dot the normalization didn't treat it as a dot It just treated it as a string so the normalization Function looked at it went oh, okay, and it just passed that on through and then Then when the web server got it and interpreted the url it knew that percent 2e was a dot So percent 2e made it through normalization and then was executed So percent 2e percent 2e slash would be the same as dot dot slash which would allow you to break out of the web root Now if you had cgi installed on the system and cgi is a very common Um scripting language programming language or interface for apache servers if cgi was enabled You could actually couple that path traversal with remote code execution Right, so you know with just the path traversal you could look at sensitive files But with if cgi was installed you could actually invoke Scripts to run and that's what we're going to do and so we're going to first show How a sensitive file was detected how was access was detected. So we're going to look at pam.com We're going to which is the configuration file And falco is going to tell us that happened and then we're going to use This cgi chaining together with this this vulnerability to get a shell access onto the system So let's take a look at that now Okay, so here's my apache server with a very simple web page The first thing I'm going to try to do is just use curl to take a look at a An icon on the system. So if I do a curl against the apache address And say show me the a.gif it shows up just fine Now if I try to go ahead and get to the pam.com file Using dotted notation relative to the icons directory. So dot dot backslash dot backslash Let's see pam pam.com that fails. That's not going to work But if I take those dots and replace them with percent two ease and try to access it it actually works So that's the problem. So now what I'm going to do is because cgi is installed I'm going to use that same command only this time. I'm going to tell it to fire up a shell So it's done that. So if you see here, it's the same thing bin shell So falco should have picked that up So if I go into falco here end of the ui and I go to the event you can see that we're getting a warning that a Shell was spawned by an untrusted process that apache spawned a shell and that's something we should look into All right, the next one we're going to look at is oracle web logic and now this is actually two Two vulnerabilities chained together the very first one 14882 is a path traversal attack only instead of like in the last one you saw percent two e this one uses uh 25 to e and it it doesn't it Doesn't get normalized out. So that gets treated as a dot and if you send a url formatted with that ascii string You are going to bypass the console authentication. So With just that exploit alone, you can go to a web logic server If you've got access to it, right and you can get to the admin console now There are java methods that allow you to execute remote code And if you chain 14883 with 14882 you can bypass the authentication pass in That one of those java methods and say hey run this code and it will do that Right. So we're going to use this This exploit these two exploits together to bypass the authentication Install a crypto mining rig and run it and then falco is going to again detect a remote file copy And it's going to also tell you hey this thing is running and it wasn't part of the base image That's xm red and we're going to alert you to that so Again, we don't we don't tell you that these vulnerabilities were running And you know explicitly we tell you here's some things that happen that should be looked at All right, so let's take a look at that one Okay, so here's the login for the web logic server And if I go up here and I change the url and I go ahead and you see I'm going to put in those 252 ease to navigate to the console and by doing that I'm going to go ahead and bypass the authentication So here I am I'm on the console at this point and I have full access to the system So what I'm going to do next is I'm going to take this And I'm going to add some commands to it so that I'm able to execute Commands arbitrarily without Authenticating so I'll go in here So I'm going to do that same percent 25 to e in the url But I'm going to use the java language Get runtime dot exec method to go ahead and just create a file on the server So let me go over and check the server and you can see here that that file now exists Okay, so now what we're going to do is we're going to use that same methodology to download xm rig the cryptominer And so I've got that downloaded here You can see I basically did a curl as the command and then coming out of that We'll go ahead and we'll use tar to extract that And then we'll go ahead one more time and we will use the same methodology to go ahead and execute that and get that running So now I have downloaded I have extracted and I am now running the cryptominer on the system So let's go ahead now and we'll take a look and if I look on the oracle web logic server There is xm rig running as expected. So what did falco see? So if we'll go into events And there we go. It's drop and execute a new binary in a container Basically, there's a there's something running in this container that wasn't there before So that's what or that is what falco picked up and that's what you would need to take action on So our next Exploit that we're talking about is in the jenkins server right and jenkins is a cicd tool that is Very widely used for build pipelines testing Automation all kinds of different stuff And so that makes it a target for hackers, right because it's just so widely deployed and There is a cv out there That's based around the stapler framework, which is a java framework that jenkins is built using and stapler allows users to call public methods through a url and There is a vulnerability where Well, typically these these calls are made inside of a groovy sandbox groovy is like a java based scripting language that jenkins uses and typically there's restrictions around what can happen inside of that sandbox it only allows certain methods to be called however there were There was an error or a bug here where Um, that the groovy sandbox could be bypassed and allowed unauthorized users to execute arbitrary commands Jenkins checks the script for errors before executing groovy in the sandbox So the the check operation isn't not actually sandboxed, right? And the attacker can use meta programming to execute arbitrary commands while In this checking step. So, um, that's what we're going to show here in this next lab So let's take a look at that and see how it works Okay, here's our jenkin server I'm going to log in now one of the problems with jenkins is the default username and password admin admin on this server So, um, I'm in the console. I'm just showing it to you. I'm not really going to use it I'm going to actually move into the command line here And the first thing I'm going to do is I'm going to I'm going to issue a command to Just create a file like we've done before and so you'll notice that the command is touch temp.own and we're going to create this encoded string here And we're going to pass it in if you look at the if you look at that url At the end there you can see it's that check script routine that we talked about before So it's in the secure groovy sandbox type of thing and we're going to do the check script And then we're going to pass in that value and then what we're going to see is that if I go over here And you know jenkins is in a container So if I look at the docker container look at the file system, there it is the file that we just created Okay, so let's set up a reverse shell using this exploit So I've got a netcat listener that I'm setting up here on this machine And then I'm going to move back over to the original machine and I'm going to issue a command to download Netcat and put it here on this machine now I've I've wrapped these commands in a python wrapper so that I don't have to have that long command that we used before But I've downloaded netcat. I've changed the executable bit and now it's running So I come over to my original machine and you can see I'm logged into the jenkins server and I can list out the directories So this machine has now been compromised. So what did falco see? Let's pop back over to the events tab and here you can see we're getting the Warning that we've been redid or someone's been redirecting standard and centered out to a network connection Which is the end cap process and that's what's giving us our reverse shell So that's an example of how this exploit can be used All right, so we've looked at the vulnerabilities So you see falco running as you know as we described it originally Which was like look we take system calls and we look at them And we take we do some sort of alerting in this case. We were writing writing out to falco sidekick ui But falco sidekick can actually do a lot more than that and you can actually take input from a lot more places Than just system calls So we have a plugin architecture that allows you to look for events from kubernetes audit logs octa Cloud trail github. We have a new one for google cloud platform that was just released So you can look at things like did someone just publish their their private key up into a github repo Did someone just try to access access a system and didn't use multi-factor authentication or did multi-factor authentication fail a bunch of times So and then rather than just sending to standard out standard error We can send that output to sidekick right, which is actually what was happening the whole time It was being sent to sidekick and rendered in sidekick ui But it can also be sent out to all kinds of different endpoints, right? So you can send it out to chat servers or chat systems like slack or discord or teams You can send it out to logs like elastic You know message queuing systems You can send that output to functions as a service platforms to metrics platforms to alerting platforms And you could even write it out to a storage bucket So with fireco sidekick now you can take that output that information and you can start sending it into systems You're already using to let people know that these things are happening or Even to take action, right? So you get that notification. It comes into sidekick sidekick could send something to lambda for instance Or open fads or whatever. So for instance, let's say somebody executed a New binary in a container that could come in you get that notification You said look if this happens if this level happens We want you to go to lambda and we want to have a function that Relabels that container so that it's not running in production anymore But we save it so we can do forensics and then you know Then kubernetes when that one comes out of service the the deployment set will just you know fire up a new one And and we'll be on our merry way. So that's how you can kind of do Detection and remediation falco does detection sidekick helps you build remediation you know pipelines or or You know chains or whatever you want to call them. So that's you know Plugins and sidekick really complete the falco story now I want to leave you with some falco resources. The first is our website. So you can get us at www.falco.org Um our github is github.com.com slash falco security. We're always looking for new contributors Please you know if you've got an idea if you find an issue If you want to open a pull request head on over there and do that If you've got questions before you do any of those things There's a bunch of different ways to reach us. The first is in the kubernetes slack on the hashtag falco channel pound falco channel On that slack server. So we're in there the maintainers the contributors a very active community members They're there all the time to help folks out. We also host a weekly community call at 4 p.m GMT 9 a.m pacific that is Those notifications are published on our linkedin So you can go to the linkedin to find a link for that community call as well as get updates from the project We're also on twitter. We're falco underscore org on x slash twitter. So you can get us there So with that, I want to thank you for your time today I hope you've found some useful information here. If you have questions. I'm at mike g colman pretty much everywhere I'm mike g colman at cystig. I'm mike g colman on github x twitter linkedin Any of those platforms you can just reach out to me and I'll do my best and of course the kubernetes slack I will do my best to answer your question if I can't get an answer. I will find somebody who can So once again, thank you very much. I hope you found this useful and I look forward to seeing you in the future Thank you