 Hello, everyone. The next talk is what are your user's kubei controlling into your kubei net cluster by Furio Garcia. Welcome. OK, thank you. OK. A lot of people today. Sorry about my accent first, and then let's try to make this quick and something interesting. The idea is to tell you a little bit how we use Falco to detect what your user are using in kubei CTL in your kubei net cluster. So first of all, let me introduce a little bit what is Falco. Falco is a behavioral activity monitor that allows you to detect suspicious activity based on a set of rules that you can write. We do that using tapping in the kernel module and listening to the syscals there and filtering that with powerful expressions. We have full support for adding context to that syscall from the orchestrator and for the containers and process of the system. And we have a system for notification, very flexible, that you can output that notifications out to the standard log, to the syslog, to webhooks, to whatever you want. And Falco's open source, of course. We recently changed to Apache license, but it was open source from the beginning. And we are really willing that everybody helps us to write better rules or make improvements for the tool. Another thing that we are really excited about is that Falco has recently joined the Cloud Native Computing Foundation, the sandbox level project, but the idea is to bring it up and make it a more mature tool there. The idea is to improve the runtime security for Cloud Native platforms right now, trying to detect a normal behavior there in containers and host and out into the orchestrator's activity. A quick example of how Falco works. In the upper part, you have Falco running. In the lower part, we are doing bad things in a container. We are executing it. We are attaching sensitive files. We are trying to copy the server file, the password file. We are trying to modify files under the binary folder. And all of this is being notified by Falco in the upper part that you can see. So this is the normal use of Falco right now. A little bit talk about the Internet architecture right now to explain how we improve this and modify it to allow the auditing of the orchestrator events. What Falco does internally in the architecture is that it tap in the system event string calls, take that system call and take it to the user space, add more information about the process, the Docker container where it's running the orchestrator and generate an event call object. That event call object goes through the RAL engine that has a lot of Falco rules configured by the user and try to match the Falco rules against the event call, event object call. And if there is a match, then it goes to the output and notifies whatever it means you have configured Falco to do that. How is a Falco rule? I don't know if you see it. Well, I think so. For example, this is a simple rule that notifies you if somebody tries to write below the binary deal. So as you can see, we have the condition where we go to the RAL engine, goes to the system call event and check the event type of the system, the system call, and if it's open, it goes ahead with the condition checking, checks the directory that is being executed, that open call. And then if there is a match, it goes to the output and gets some more information from the event to give more information in the output of the Falco rule. For example, user name that has tried to execute that, the command line that was executed, and the binary that was targeted. So with this in mind, the idea was to try to reuse all these for doing the same with the auditors. So we want to reuse the conditions, the expressions that we have there, the output, even the jammels where we write the rules. So let's see how we did that. First of all, let's talk a little bit about the Kubernetes audit events. This is a thing that is only from Kubernetes, nothing to do with Falco. It was added in 1.11 and provides a chronological set of records that documents the change in a cluster. Its record is a JSON object and you have to enable this. It doesn't come enabled by default but you can select what do you want to be notified about. You can say, hey, I only want to be notified about deletes, I only want to be notified about modifications or about creates or whatever. And then all that can go to a log backend that can be a log or a webhook. In Falco we configure a webhook for that mean. So this is a Kubernetes audit event, a JSON. It was simplified a little bit because there is even more stuff there. So you can see more or less what kind of information came from the Kubernetes to the tool that you configure about the audit events. You can see that we have the birth. In this case, it was a delete. The user that secured that delete. The status or the timestamp when this was secured and unique ID, for example. And even then in the annotations we have the authorization that allowed this execution and that stuff. So we have a lot of information to be filtered and to be used for detecting things. How do we integrate this with Falco? First of all, we create an embedded-with-server inside Falco and make them and prepare it to listen only for post requests as simple HTTP. Then we configure the Kubernetes audit event, the policy and a webhook directly to the embedded-with-server. So whenever a new Kubernetes audit event was generated, it was posted to the embedded-with-server and there it was parsed and added a little bit more information and inserted in the rule engine like the syscall events were. This runs in parallel and with another set of Falco rules, but we have there running the two systems at the same time and the outputs are the same. A little bit more information about how we do that. We create a generic event interface so we have the common parts in one of the objects. We create a Kubernetes audit event object and then we inserted directly the JSON object and created the JSON pointers to track values from there. So the ideas that we have in that event we have all the information that the Kubernetes has passed us and we can work with it. And now another thing that we have to do is to identify the Falco rules so some of them are only for the syscalls and another one for the Kubernetes audit events because they are different. So we add a new property to the Falco rules that is the source. Talking a little bit more about the JSON pointers it's a really easy way if you know XPath it's a really easy way to navigate a JSON and go and check different values inside it so we can, the idea of doing it so we see that you can write that in the Falco rules whenever you are writing and preparing one of the custom tools that you want to write. For example, a more complex example of the JSON pointers using directly one of the Kubernetes audit events. As you can see, it's really easy. You can access the verb. You can access inside two levels or more or even a rise or whatever. So the syntax a little bit when you are using and writing the Kubernetes rules allows you to use any JSON pointer to access any part of the JSON of the Kubernetes audit event. We have created some macros to facilitate the access or for example the verb, the username, the target of the resource can be accessible easily. And you can see all these writing the last command for Falco that is the same like Falco does that's help so Falco does this equals Kubernetes audit gives you all this information directly from the command line. A little bit more of a Falco rule more complex and complete prepared for the Kubernetes audit events. Here we have a rule that allows us to notify a warning about some config map that contains the word password or AWS FS key or AWS is 3 FS key. So whenever some user will create a config map with some sensitive data we will be notified. Talking a little bit more about the Falco rules as you can see we have macros that we can use and list. And in this case for example we can start with a macro for detecting that the config map has that provide credentials. Here we can be more complex but with this probably enough we have another macro to check in if this is a config map and another one to check that we are doing a modify over that config map. And in the condition as you can see we try to check the three of them and whenever we check that the three of them has been fulfilled we go to the output and we check for example and we return to the user that is checking the Falco rules for example the username that tried to do that and what the config map that he was trying to create. And a little bit of demo now. I'm going to show you a little bit how this works in real life if nothing is broken. Here I have Falco running connected to a mini cube cluster and here I'm going to try to execute some things. Let me show you first what I'm going to try to create for example I'm going to try to create a simple config map with my secret key, my access key for AWS here. Let me then try to do that. Okay, and as you can see here I have been notified about that a config map has been created and this will be working better. Let me try to check demo effect. This will have been more information there. Okay, because I am stupid I didn't write correctly. Now yes, okay sorry. Okay here we have the warning about the config map that we the config map with a private credential was created as you can see. We have the information about the user to try to create that, the config map all the information about the config map and everything, okay. So more things, more interesting things that we can detect with Falco. For example let me create a normal deployment for engines. This is really something really easy as you can see here with engines and everything. We are notified about the deployment has been created, nothing special but now if I try to let me do this if I try to exit in the path in the engines deployment path what I wrote. Okay, because I don't have okay but I was okay. Then he notified me that I tried but I didn't correctly succeed because the image was not pulled because I don't have internet connection now probably. So, but as you can see I was notified anyway because I tried and the execution was done even if it was not successful. So there. Connecting internet. Okay, more things. For example if you I can create a service account this is usually normally a problem and you try to bring that service account to the cluster admin to get escalation of privilege or whatever and you are notified about that. There was a cluster role binding to cluster admin about that and you can then act about this or one user or react about that. Another one for example is that okay, this is an overly permissive cluster role create that I have here that allow us to access new resource and create then a cluster role binding for my user okay. So let me create this and now we should see here that there is a warning here about the cluster role was created with the wheel cart. You can check something so so special about it like too much permissions to execute things or things like that. And at the end for example another example about the cluster roles is like this pod exec this pod exec permission for example you can also notify about something so more specific like if you try to give pod execution permissions to a cluster role you are notified about it like you see here so let me return now to the so nothing more specific you have here all the links about the website we have a public slack for sysdig there is no specific slack yet for Falco but you can join there and there is a channel specific for Falco where there are all the questions and there are several developers from the tool there that can answer probably better than me we have some blog posts about it explaining how it works and now you can join us in the github report to help improve this and write more rules and that's all questions? The notifications you showed were just log messages running in Falco do you have ways of exporting it out to other systems? You can configure that in the Falco replace there are several possibilities you can send that to a webhook and then react to that or you can send that to notify a serverless serverless function and then react based on that whatever you want build on top of that is really easy we are going to be in Gaunt next week doing a workshop and we will be showing how to do that exactly and how to with cobales I think we will be doing that Wednesday afternoon in Gaunt after complete management but it's easy to extend better that notification channel and send it to Slack or to whatever place you want Any other questions? So as I understand Falco uses czdc under the hood and czdc recently introduced support of EBPF so now we don't need to add the kernel module right Is it like production ready? Is it stable already? It's in alpha right now we are testing it internally but it didn't reach even QA yet the developers are testing it but we didn't have the time to test it properly yet but it's in the works and probably I think it's for this year for this year it should be ready That's our questions Thank you