 That's me again. So we have we have got voice that we can start a little bit earlier so that our break will last its normal time. We're going with Marcel Klaasen, Enterprise Sales Engineer from Sysdig. Please give it up for Marcel. Thank you. Good morning everyone. Today I would like to discuss with you a little bit why we do need runtime thread detection in your Kubernetes environment or container environment and how we can do that with FALCO, the open source software that we are using. So first a little bit about myself. So as William already said, Marcel Klaasen, I'm a sales engineer at Sysdig. Meanwhile I also keep myself busy with other things like some energy bundled ring equipment and containerizing that in my K2S cluster that I have at home at multiple locations because my wife has an office somewhere else so I misuse that location to put the cluster of course. Some pumping equipment for my pool because I hated doing it all myself so that's why I'm not automated and of course some security monitoring and as you see there this morning I left my home and with 80% certainty it was me as a person leaving here. But that's not what I'm here for. So basically let's take a look at the agenda. So why do we need runtime security? That's the first thing that we would like to discuss. After that let's go for introducing FALCO. So what is FALCO? What can we do with that? I will tell you a little bit about the FALCO rule engine and then I will explain about how we can expand FALCO with other plugins or sources and output information. So why runtime security? So let's take a look at this departure hall of an airport. So we have a lot of people going in and a lot of information that needs to be processed by us. So what do we need? What do we see? Are we interested in what's happening there? Do we want to keep things safe here? And that's the main reason that we have security at an airport. We want to make sure everything is secure. But what do we do with all those signals? Because we have a lot of signals here. We have people scratching their heads. I don't care. We have people smiling. Do we really care about smiling? No, we don't. Then other signals like is this guy armed? No, that's a false positive, right? We don't want false positive of all our signals. We only want to make sure that signals are correct. So for this guy for instance, that's an armed guy. So this is a threat for our system and we want to make sure that this one is detected. So at the departure hall or in the security center in the airport, that's all arranged was that we lead people into the security lanes and they are threat everyone one by one. But how are you going to do that in your Kubernetes environment or your container environment? So why do we need runtime security? Of course to detect malicious behavior and I know all of you probably are certain that you build your images with all the pipeline, vulnerability scanning, everything is out of it. No critical vulnerabilities left. Everything that is fixed can be fixed. We know all about it, but still, you are running in a deployment and what happened in the deployment if you spawn a container of your image, the image is running right? So we can get some drift. Is your application really doing what it should do? Is there a misconfigured application in there that is allowing an adversary to get into your system using malformed PSP script? And what about zero day exploits or unknown exploits? So you want to know what's happening inside your environment. Next to that, all the images that you're running in your environment are not always yours. So you also are depending on third parties and you don't know what they built, what they've done. So you want to know what's running inside your environment. After that, if you've detected a threat, you want to know what's going on, but you also want to be alerted upon that. So right away, preferably right away when it's happening. So not one hour later or a day later, sometimes you see that or at the report two days later that you see, oh, there was a threat. It's too late. You want to see it now. Now it's happening. Now you want to see the alert. Next to that, after the threat has been raised, you want to do some forensics, take an audit activity, get some knowledge about what happened to see if you can prevent it from futures. And last one, not the least important, is you need to be compliant with a lot of security frameworks. If your information, if your environment is not correctly configured, you are not compliant. If you have certain files that can be written to, you are not compliant. So with Falco, you can detect what's happening on that. If you take a look at the runtime security, we all have that applications that are running and all the applications are spawning their details to the screen. So you all recognize the output of all the different applications that you have, but none of them are confirming to a single source of truth. So they only send out the information that they are programmed to do so. So if there is a malicious activity in that application that is not known as a malicious activity in that certain application, they don't recognize it. They just show the output as this. Someone is getting in a PHP file, but you don't know what's wrong or what's right. So how are you going to get around to all those events from all the different sources? This might not be the right reason, right? It's probably a good start, but it won't help you. This is the time to introduce Falco. So Falco is a cloud-native computing foundational object. It's incubating, actually. And you can consider that as your security camera for your containers and cloud. So basically, Falco is taking a look at everything that's happening in your system on your container level, but also on cloud and other sources. First of all, where did Falco come from? From the people that are long around here. Maybe they know about EtherReal. So EtherReal, the predecessor of Wireshark. It was something around 2000 when I was still young. And at some point, somewhere around 2005, I think, or something, Wireshark was created, and it was basically due to a naming discussion with the owner of EtherReal. And one of the creators of Wireshark, or the co-developers of Wireshark, started a new company named Sysdig. And Sysdig created a deep container, as it's called, container for forensic troubleshooting tool, basically, based on Wireshark, but then specifically for containers. Out of that, Falco was created. And Falco basically was created because he had that static information about container information, about syscalls being done and all the kind of things, but he wanted to have information on what was happening real-time. So Falco is, as they call it right, cloud-native threat and anomaly detection tool. Currently Falco is now incubation project in the GitHub. We have over 7K GitHub stars. I need to check it because every time it's changing, of course. Not live counters, by the way. And over 50 million downloads. And we have a lot of contributors in that space because it's an open source community, right? But you see the names there. This is only a very few of them that are contributing to Falco to being a real-time threat detection engine for your cloud incubation environment. A little bit in repetition, but Falco is the open source for real-time detection of threats, anomaly detection, et cetera. So how does that function? This is a very high-level overview. Basically, Falco, or the grades of Falco think there is only one source of truth in your system. And that's the syscalls. So we can depend on all kind of information, like logging information from NGX or whatever application that you're running, but they miss certain information. So Falco is using all the system calls. They are processing the system calls and providing you alerting output. So what kind of problems is Falco solving for you? So our hosts are containers, something during that they don't. Processes being spawned is something changing in the environment. Our config maps are something like that in a Kubernetes environment being called. Our Kubernetes API calls being called that are not able to be called or shouldn't be called. Our users legitimate are not spawning a shell inside a container. That can be done via the kube control commands, or it can also be done, for instance, from a malicious PSP file, right? Or anything like that. So that's the kind of information that you can on a high level get out of Falco. And that's only based on the Kubernetes environment. In the later stage, I will tell you also about plugins, which will provide you the option to get additional sources into Falco. So if you talk about what is Falco has running. So basically, this is an overview of what the architecture is of Falco. Falco will be installed on your system using a kernel probe or an eBPF module of using eBPF if you don't like any kernel probes. From that point on, it's taking all the events in your system, like host and metrics, data, promises, but specifically for us important security events. It doesn't matter what runtime you're using. If you're using Docker container D or cryo or anything like that, we will see all the information on that specific node. You can install Falco using a single application on a host, which for a cluster is possibly also probably also more easy to install it as a demon set. And we provide options there for installing as a demon set. So your complete cluster is directly protected when installing it. So what do we need to think about when output is being generated? So the initial Falco representation, the default Falco installation is basically only giving you information on an output basis like this, right? So fire syslog, notice a shell was spawned in the container with an attached terminal, warning net cut runs in the container that allows remote code execution. So basically, this is the default set of Falco. In the latest stage, I will tell you also about plugins that will give you possibility to get extended data information and also correlation with other clusters together because basically the negative part between quotes of Falco is that this standard installation Falco is it is running on a single cluster. So if you multiple clusters, you need to go to multiple instances and get multiple data sources to get collected data. So let's take a look. Let me see if that works because some issues with the screen as we do it like this for a moment. Okay, see if I still have connectivity. All good. So what I hear here, I have just a simple installation here, a UNIX terminal. I hope it's readable. And what I have here is basically an installation where I install Falco on Linux, nothing specific, just Falco installation, pretty straightforward. And what I'm going to do now is that I'm going to start a program, a small program that I created to get some events that I want to expand. So basically, you can see all kinds of events here popping up. Let's see if I can move it a little bit more. And I'm going to here open up a tail. Let's do a color tail. It's always nice. And let's say that I want to write below a binary directory. You don't want that in your container, right? You don't want any writing below binary directories or something like that. So let's do that. And I see that there is an event right away from syslog, error file known binary directory name removed. I can also go, for instance, like modify LDP load files, see if that's happening. And basically, there is a lot of information that I can directly get out of it. For instance, also, what's important case for container organization environments nowadays is protect, again, crypto minus. Let's take a look what crypto minus can do. And I can see directly here that critical possible miner running in there. So we may think that this is all set up by me and I'll program some of that, but Falco comes out of the box with quite a number of rules. So if you install Falco, all these rules that you see, all these events that are getting spawned are default. You do have the possibility to create your own rules, additionally to that, but there's an extensive set already available. Let's go back to my presentation for a moment, see what we can get there. A bit more detailed structure. So this is how Falco is functioning. The syscall events that we just saw is going to be one of the Falco sources that you see here, and then is going into the rule engine. So the rule engine is basically determining what's happening in the system. Is this worth of being spawned? It's taking the syscalls, all the syscalls relevant, and that is basically spawning, unalert via gRPC files, turn it out, we're using now syslog, HTTP, but we'll get to that in a minute. So how does a rule look like that? This is how a rule looks like. So we took a rule here from the top, warning, symboling created with the sensitive file. We see the command lnmin sfetcshadow to slash tmp slash marcel. So I'm copying a symbolic link of the shadow file. So how does that a rule then look like? So a rule always has a few constructions, the rule name, description, the condition. This is basically the part where it's all about. The condition is where the syscalls are interacting with the falco engine to get all the details. And then the output, and basically the output is just a resemblance of what do you want to see in this case in your syslog file. We're adding a priority and we can add some tags, and tags can be important for your compliance, because maybe you want to be gdpr compliant and that's one of the reasons that you need to have that, so you can add gdpr tag to that. If you take a look at the condition here, you see that there is a create symbol link and sensitive file names, and there we are using macros. So macro basically can be used to tell, to ease up the use of rules. You can see that this macro creates symbol link. That's the condition that event.type is in, and then you see the symbol link and see the symbol link at as the syscalls that are being detected. Event directory in, so it's input file so we need to have that information. We can also make use of lists, and basically lists are basically giving you the possibility to, for instance, target the sensitive file names in this case, giving you an overview like my sensitive file names are etcshadow, sudoers, palm.conf, and the pbquality.conf files. So basically if any of those files is being touched, as being a symbolic link is being created, you get an alert being raised in your system. So this is basically how you set up this very basic rule. You can extend it very much, but there's also a large community maintaining those rules, so you don't need to set it up yourself. Probably the rule that you want to set up yourself is already done by someone else. So copy it from the community, add it to the local focal rules, and you're done. Have some popular rules here, like for instance best practices, update package manager, modify bin user, I'm not going through all of them because you'll read themself. Privileged container is a nice one. Privileged shell, terminal shell, some compliance parts, like we want to know if kube-control-exec attach is being used. PCI NIST frameworks can be used. Some known vulnerabilities that we know about, like the top one, kube-control-copy are the container escape vulnerabilities. That's all being part of rule set of Falco by default already. From the cloud native stack, and that's basically we see that we have elastic search redis, all that kind of rules are directly supported by Falco. What else can you do besides going into syslog calls? We only added over syscalls, but that's reduced to the local level wide. In some cases you don't even have access to syscalls, but then you need to have auto precautions. But we also have the possibility to extend Falco with plugins to use other sources, like the bottom part, senior Falco plugins. Initially, Falco was only using syscalls and had a few built-in plugins, like the Kubernetes audit logs, was default begin, and the AWS cloud trail was also default begin, but that was basically the only plugins that you could use in the Falco system. Since last year, we built a complete new system that allows us to use plugins in generic. So the Kubernetes audit log and cloud trail, of course, moved to the other sources. The Falco plugins are going into the rule engine, and of course, different rules apply for AWS cloud trail or anything like that, but they are going into the same rule engine. What do we provide for rules now? This is just a simple example, so AWS cloud trail. We have Azure log analytics, GCP cloud log, Kubernetes audit log, as already said, Docker, Twitter, Octa. Basically, every streaming instance can be added as a plugin to Falco. You can build your own rules around it, and of course, the syntax is a bit different, because for a streaming instance, it's different as from a syscall instance, but it's easy to write your own rules. Output file. So I only noticed that we do remote procedure calls, output files, standard out, shell, HTTP, but it's all pretty straightforward. The text that I shown you, right, not very well readable, so that's a little bit difficult. So as we take a look at output files, then we would like to take a look at Falco sidekick, for instance, here. Falco sidekick. Falco sidekick allows you to add external output regulations. So if I go to the overview here, it collects the Falco events, not only from the node that's running on, but from multiple nodes. You can use the Falco sidekick as a collector for all the events that you get there, and you get all the output where you can forward it to, for instance, a Slack. You can do it to Teams as a notification area. We can use also a spiderbed for our Node Red for some SOAR capabilities, or you can do a CloudWatch Slack, feed it back into CloudWatch. So basically, I added a number of them in the bottom part, but the list was, it's much longer. And so basically, it's very easy in Falco sidekick to configure sources. Okay, so let's take another look with that same Falco instance, the same stuff that we did just a minute ago, but then we are trying to get into Falco sidekick. Starting the containers. So basically, this is the containers required for Falco sidekick. I'm doing some reconfiguration with, from, and what I see is, so just showing you what the difference is to enable Falco sidekick as an output mechanism. So I'm going to show you what the original Falco file was, the configuration file, and what the new Falco configuration file is. And you see that basically, I only set the JSON output to true. So basically, instead of bare text, I'm using JSON output and I enabled the URL to some URL to localhost 20.002. That's where my sidekick instance is running. So if I now go to starting that same instance again, you see already at the bottom part, oh, my my color tail cord dumped, always nice, of course. You see already now that what happened is that the output changed to JSON. I'm getting all the information here and if I now spawn a thread, attempt being used to modify directory, and let's see if I now also have my five sidekick running. Where is it? So this is the Falco sidekick console and you see now that information is being spawned inside Falco sidekick. So basically, it's giving you a good understanding of what's happening, it's giving you overview of what can be done and what should be done, and you can filter and search for all the information that is in your, what was the output of your console. So let's take a look at the dashboard. It's giving you also priorities and if I now do, let's say, let's try kill known malicious process or something like that. Events, file below, open for writing, paste bin kill. So apparently the event is directly spawned into sidekick and from here on you have the possibility to also configure your sources like Slack. I think I also set up Slack. I got a notification here as you can see here. You see here that my notification like the file below known binary is also spawned to Slack. So basically this is showing you how easy to set up to get into Falco sources to the rule engine output via sidekick, for instance, to several sources where you can get directly your threat event into different platforms. This is basically what I would like to show you. I would like to thank you for your attention. If there are any questions let me know and for more information Falco.org is your source. Yeah, go ahead. Thanks. Can it just report threats or also prevent them? Sorry, what was the question? Can it just report a threat or can it also prevent it? For example, prevent a sim link. Now Falco is a runtime threat detection engine on its own, right? So we can use SOAR capabilities as I mentioned. So you need additional tools to do the to the remediation killing containers or anything like that. Basically the sidekick of Falco, basically what it is doing, it is a sidecar which produces a RSS feed, if I understand correctly. Something like that or not? So I'm not sure if I understand your question, right? But what Falco sidekick basically is doing that the Falco engine is rerouting all the event messages to the to the processor of sidekick and then sidekick is basically the receiver of the messages and responsible for spawning it to, for instance, Slack. And what I also did and that was this tool that I showed you is sidekick UI and that's basically the follow-up of sidekick. So sidekick is spawning to Slack and to sidekick UI in the example that I've shown. Yeah, so basically it's a feed and then whoever is the consumer they can consume that feed and then get correct. Yeah, correct. Hi, I was wondering if you do not enable EBPF, what are the downsides if you just use it without that? So it's either EBPF or kernel probes. So you don't need both of them. One of them is sufficient. So by default the Falco installation is using the kernel probes. If EBPF is available it will use that one. But you have options to configure that. What types of threads do you usually find and what types of environments? Ah, yeah, okay. That's a good one. So basically Falco is, so I'm working at Sysdig, right? So Sysdig is the enterprise platform around Falco. So we do see the enterprise platform around Falco. We're using it as a runtime thread detection engine in Sysdig. And the main threads that we see is a lot of private shells being spawned inside containers. If you talk about threads from the outside, inside is often a thread, maybe not a real thread, maybe false positive, that's going into cube CTL, someone spawning a terminal shell in container just to take a look if the environment is okay or something like that. But the most threads that we see from outside is more starting with a private shell from the outside, like for instance when starting with Love for J or something you get a private shell and the follow-up actions because the private shell on its own is not a thread, but the follow-up actions like curl and starting a net cut or something like that, that's the real threads. So basically I've shown you single events but in most cases it's a chain of commands. So not only one. Marcel, we have one last question. Is that okay? So is there an option to forward, I saw that Falco can be deployed like a demon set or something to collect the logs from each host. So is there an option to ship the logs to something like a Syslog collector without the sidekick or do you need this as an intermediary to collect the logs and then forward it? Something like Splunk or Data Dog or whatever. Or can you just lose like a Syslog exporter directly? Yeah, so by default it's Syslog and the output that I've shown you. So remote procedure calls, HTTP, so that's by default Falco and for alternative options you need to have the sidekick installed. Thank you very much. Thank you very much Marcel. It was definitely interesting by the number of questions and the number of questions that I didn't let go. Ladies and gentlemen, we are going to have a 12 minutes break. We're going to restart at 10.45 here. No. Sorry, at 11.45. I'm Italian so I got my time's wrong. One thing that I must ask you, there is a workshop at the moment running. Is that correct? So please when you go downstairs, go downstairs and go outside through the smoking area left and there you will get in from the sponsor area. Thank you very much. We'll see you again in a little over 12 minutes.