 So, hello everyone, welcome our next speaker in Securited Everyone, Lorenzo Fontana, and his talk about Falco Internals 101. Hey, I feel a lot of pressure just because, I mean, it's Saturday and it's lunchtime, so you're all here to listen to this, so it's a very, very high pressure situation. So, we are going to talk about the Falco project and how we've been able to put Cisco to user space, bring Cisco to user space in a very fast way with very low throughput. The thing is that, I didn't really want to talk about the Falco project, which is the GNSF project I work on, but while doing it, actually discovered that we have been making something very interesting for everyone to listen to. So, I actually wanted to tell everyone. If you already knew me, it would be probably because of my book, Linux Observability with BPF, and maybe because you're involved in the Kubernetes space or the Kubernetes space in some way, and also because I've been engaged with them for the past five years probably. First things first, I want to give you some context to let you understand. Falco is a project that basically takes every Cisco that happens while no one is looking at that. Every Cisco is brought to user space, and with those Cisco's, Falco processes a set of rules that again gives you alerts based on what happened in the system. It has been started four years ago, and two years ago it had been donated to the GNSF so that everyone could contribute and make it better. It's written in C++, which already makes it crazy enough. But we wanted to add Cisco's. Cisco's are very interesting mechanism. It's an API to the kernel. It obstructs the hardware to you so that you can just do things without actually destroying this machine and you can interact with the kernel. If you think about Cisco's, whenever you open a file, a Cisco is done, whenever you establish a connection, a Cisco is done, whenever you do anything, a Cisco is done. Even if you use a very high-level language at some point, a Cisco is done. And the Cisco interface might be abstracted by libraries like GNPC, these things. Cisco's are very scary. Whatever library that deals with Cisco, we'll have to actually map every single Cisco to deal with that. And this already makes... This already creates a very high bar for any security implication because if you have a kernel and you have Cisco's, the next kernel version will have maybe a new Cisco, maybe a new parameter, maybe a Cisco-changed version, these kind of things. If I keep saying the words with AD suffix like changing, it's because I'm Italian and I do that and I cannot deal with that, so sorry. Cisco's are very powerful, are very good for kernel space, but are not very, very good for user space. To bring every Cisco to user space, guess what you have to do? A Cisco. So it's already a very, very hard problem. It's like the kitchen area. And if you don't believe me that the problem is that, try to have a system with a throughput of five millions of Cisco at a second. It's crazy. You have five millions of Cisco's and another five millions of Cisco's to actually deal with the Cisco and so on. And it's not 1998 anymore. I mean, it was like five, but the kernel was very, very simple back then. Things like multi-treading and stuff. And you really need to know that when you want to deal with Cisco's and when you do security tool in general, you also have to deal with time. And guess what you need to deal with time? A Cisco. Nice. Or you can deal with things like, you know, registers, like you can go get the time from the processor, but processors are not abstracted. So you have to deal with time for every single architecture you want to support. And it's another maze, like this place. And also, if that wasn't enough, Cisco's are not enough because you get a Cisco that tells you, hey, someone opened this file with these permissions. Thank you. But we are in a different world that ten years ago was like, oh, 1998 is nothing more than years ago. We're a different world than 20 years ago. And you don't have just a system that runs Apache with some PHP. You might have containers where you have a very complex infrastructure. And you want to know in which container this is collected, in which network namespace. If the network namespace was with a mountain namespace or if the attacker actually did something nasty, maybe they changed the C group of the network namespace. So having container information is also useful. And for container information, you have to deal with systems that are very slow because you are going to take the container information from the Docker demon or from container D or something, like with their API. And those are APIs designed to go at, you know, with a latency of 100 milliseconds. But Cisco's are very fast. Five millions a second. And you want to correlate every Cisco to... It's crazy. To every... We didn't solve this problem yet in a very good way. So how to do that? We've been asking ourselves this question. There's a third answer that we didn't even develop yet, but it's secret. And basically two ways. So you just implement your own kernel module, like, you know, it's like when you install VirtualBox in your Linux machine and then you need to, you know, you need to restart. Nice. Or when you upgrade your kernel, when you install some modules, imagine having your company with 100 servers pack updates, restarts the machines. And that wasn't very handy. So we also implemented any BPF probe, which everyone knows what is it, because my talk is not done in BPF and they didn't want to explain. You know BPF? Okay, great. Like 20%. So I will explain it very fast. So with a kernel module, you can just go in the kernel, implement your code, and, you know, it will do whatever you want. Pros, you implement whatever you want. It's very efficient. Cons, it will kernel panic your machine and then as soon as you do a mistake, it will start all down. And you don't have a way to reboot them. And, you know, it's very hard. With an BPF probe, the mechanism is a bit more sophisticated. You can't implement whatever you want because there are some limitations, but you can still go to the kernel and ask for information in a very efficient way. So how does this look like? Well, I initially wanted to do this presentation in the terminal. So they had to do diagrams. So my diagrams are not very good, but, I mean, it gives an idea. So you have the kernel space and user space separated. You have the kernel module on the lower bound, and you have Falco in the upper bound. And in the middle, there are a set of libraries and tools that, you know, allow you to process the Cisco in a very efficient way. Let's start from the bottom. The kernel module is the first implementation. It's still there and still works as Falco. And it basically goes to the kernel and attached to every Cisco that gets executed when it starts executing and when it exits the execution. Like, imagine putting a jump at the start of every function and a jump at the end of every function that executes the Cisco. That's it. When this code executes, it sends it to a ring buffer, which we're going to be talking later, that then is consumed in user space profile descriptor, you know, like a device, like the device of your printer, because this is a kernel module. And then this is consumed by another library that enriches all this stuff with container metadata and stuff which is called lib scenes. We will talk about this more in details later. And then Falco can, you know, match your rules. So this is crazy. First things, a ring buffer. A ring buffer is a data structure that is basically, imagine a link at least, but that links to itself. This is to avoid that you have to... What's happening with the microphone? This is to avoid that you have to actually shuffle back the buffer to the starting point of the execution so that you can continue putting elements and, you know, the last element was overwritten, like a first infrastructure. And in Falco, we decided the perfect size would have been 8 megabytes for it because it fits well in most of registers. It's very handy to use both from kernel space and both from user space. Most users that have very strange use case, they adjust this, but it's a compile-time thing. I personally never had a problem. The kernel takes the events and it mem copy the event to the next pointer in the ring buffer. That's why this is fast. So the memory is never, ever copied. It's just copying the pointer of the memory. Sorry, I did the wrong thing and then contradicting myself. The memory of the pointer is copied to the ring buffer so that you have access to that memory. And at that point, you can load this data from user space by reading the dev Falco 0 files. This is basically how the structs look in its header file and this header file is shared between the user space and kernel space. So you basically, what you're interested in, it's in the head and the tail. And then there's some metadata for you to understand if your ring buffer is performing well. I'm showing this because we are all used to very high-level code, I mean, object oriented programming, all the stuff, functional programming, all the interesting things that I love. But when you have to do these kind of things, you actually have to, you have no choice than designing a struct that has everything in it because you have to make it fast. And so this also has the responsibility, you know, fuck the single responsibility concept, to count the dropped events, count the total number of events, count if there have been preemptions, contests, switches, these kind of things. Dropped events happen because maybe the ring buffer is not fast enough. Remember when I said five million syscalls? Maybe at five million syscalls, events are starting to get in-dropped so there's no memory free in the ring buffer to put them. And at that point, syscalls are not yet ready to be used by your application because, well, you receive a pointer first. Second thing, you go to the pointer to see what's inside. There's memory. And then you look and there's only the name of the syscall and some other, you know, some other data that you don't understand. So we had to make a mechanism that's called fillers that basically goes to every syscall, you know, when I said do this for five millions of syscalls. And it takes, with the fancy functions, syscalls get arguments deprecated because there's no other way yet. It takes the registers and transforms that information to a string or to, you know, in this case. The information was not there, so it just prints out not available. And then, if there have been success, the syscall isn't reached. It's still in the same data structure. So you have to do this for every syscall two times because there's the enter and the exit. And nothing, I just wanted to share this with you all because it's been painful. And at some point, you go to people, you talk with others. You find out that there's EBPF. You find out that people are very upset about having to compile their current module all the times. And you decide to do this with EBPF again. And turns out that it's a much nicer experience. There's no ring buffer involved at all because EBPF doesn't allow you to, you know, to advise and, you know, move memory from current space to user space the purpose of it being safe will be totally, you know, it will be totally useless. But it has a set of concepts that allow you to do that in fancy ways that are called maps. And in this case, Libscap, which was the library that before was just reading the file descriptor, loads an EBPF program to kernel space and says the kernel, hey, kernel takes this object that I compiled for you and the kernel says yes. And then the kernel says, oh, yes, I like this. And then the kernel says, okay, I will send you data. And, you know, all this conversation was the EBPF verifier, which basically makes sure that you don't mess up with the kernel. And at that point, data flows to Libscap through EBPF maps. And guess what we had to do to do the fillers? We had to tell, this is very, very nice, we had to tell Libscap to actually send EBPF programs in the map for the kernel to load them at runtime when they're needed. I mean, you have to see the code. It's very fine. And this makes sure that, again, every syscall is matched with the right arguments. All the blob data becomes, you know, strings and you have, I mean, if you look at the Gmod syscall, you don't want the, you know, extradition representation of the permission. You want to see, you want to see these to 0777 or you want to see the actual name of the constant, like you want to see SUID, right? And to come back to Libscap just to explain better, what this does, it basically maintains the control loop through the kernel module and to the EBPF map because when you read from that file, you know, it's a device, so you continue to read, right? Because you need to consume it or you lose it. And when you do a security software, you cannot lose data, right? You cannot just, I'm not reading it for a while, so that. And, you know, attacker, very, very happy about you not doing that. So it has the responsibility to do that and we had to actually do a library that does that because the attack surface of that library alone would have to be very, very small. It would have to be just some lines of code. Because at that point, since that, you know, does the enrichment to the next slide has a lot of, you know, code that talks with container runtimes and, you know, other parts that, you know, might be compromised. And this also does a very nice thing, which is the SCAP format, which is a file that you basically can write and read. It's a format so you can avoid when you need to test dealing with Cisco, you can avoid to have a real kernel. You can just write and read from this file. And then, again, since that's the container runtime metadata, and I said this is in C++, so we basically had to rewrite the client for every container runtime. Nice. And we're next to metadata and filtering. And this is essentially what Falco is. Well, it's much more than this because if this won't be, the project will not exist. But it's just a main with a while loop that has an event called since pivant here. And every time that we go to the next iteration, the pointer is reset and sent to the next event. And then the Falco engine applies the rules, the Falco rules to this event to match and there was a treat. And if you want to see a Falco rule, is it okay to see? Yes. This is how it looks like. So you have the rule name. It's very fancy. YAML. No one loves the YAML. Oh, my gosh. And that's the description of the rule and then there's a condition. We... Let me find the very easy one. I don't want one with a macro because if not, I have to expand the macro. Okay. Okay, this is a perfect condition. So in this case, we have a macro which we can use to avoid repeating the same thing. And it matches to the event type execv. So whenever someone executes a process exactly in Cisco, we direction to the left so when the Cisco starts executing, it will actually match and go to the next condition. So that's why we needed to do all this crazy stuff. Another important thing, when you deal with Cisco and when you deal with having to bring them to user space, that was one of the mistakes we've been doing earlier was to actually send all the Cisco that was the initial point of my presentation. You don't actually have to. You just send the Cisco's that match the conditions. So what we did at some point was to take some parts of the filtering mechanism and bring them as close as possible to kernel space so that we could actually just take out the Cisco that we needed with the arguments that we needed. Because maybe there are five million Cisco's but we are only interested in opens of this file. And to summarize, you really want to do an VPF probe when you do a project that does something like this because implementing your own kernel module is never a good idea. And having very well-defined boundaries and interfaces is always a good idea because you can work on something that you... And that's very well-defined again. Like, as I said, this cap format is very useful because you can use that format even to send data to another server in your infrastructure. So it's very easier to, you know, make anything on top of that, like a user interface, like Wireshark, right? Wireshark basically use the pick up format. I could make Wireshark-like interface for the cap format, for example. And if you're interested in learning more because 25 million, I guess it's not enough. And also, I speak very slowly. You can join the community or go to the website or find me on my email or don't do anything because you are not interested. And thanks. We might have time for one or two questions if they will not be too far. Hello. Scream. Hello. What is the performance impact on running Falcon machine? Okay. The first question and the second question is if something maps the filter, is there any way to have an action to take place right away instead of a symbol output or something bad happen? Okay. I'll start with the second question. The second question was, there's any way to do an action when something happens, like, you know, doing prevention like Sea Linux or something like that? Well, with custom code. And there's not, actually. Falcon is not a framework to design, you know, prevention mechanism. It's just detection. So it will tell you when something happens. It's in, you know, in that kind of segment of security software where it totally tells you, hey, there's someone poking you, you know, mining some bitcoins on your machine, but it doesn't actually stop the miner from happening. It will send those data to, you know, a server like, you know, a collection mechanism that you use and with those information you can actually take action, like either manually or with a script or with whatever you want. And the first question was, what's the performance impact the performance impacts are very different based on the rules you apply on most of the machines that I personally use because it's an open source project and I don't actually have, you know, myself production machines but, you know, the users have and what I've been noticing in the issue and in the community calls was that it was around one or two percent of the CPU usage. We are almost out of the time so if you've got some other questions please talk. I'm sorry.