 Hey, I'm Melissa Kilby and I work in the services security engineering team at Apple. It rolls right off the tongue. When I first started in cybersecurity, my initial focus was on applying statistics, AI, ML, and big data to threat detection. Over time, I realized that there were many gaps in terms of collecting the right data. I then pivoted and started helping develop low-level Linux kernel monitoring tools and deployed them to production to gain first-hand experience of what is realistic and scalable. Earlier this year, I became a core retainer of the Falco project. Today, I will talk about how to use Falco with EBPF for runtime threat detection and more specifically focusing on how to use the right data effectively with more advanced analytics on the host itself, including theories and hands-on demos. Sometimes I'll go deep into details, but I promise I always pull back out and I'll cover various aspects. So hopefully everyone gets something out of it. In case you didn't know, AI, ML is pretty hot right now. And then you look back at cybersecurity and threat detection and you think to yourself, huh, this is why we can't have nice things, mainly because we don't have accurate training data sets. Still, there must be a better way because the modern cloud infrastructure attack surface is vast and the stakes are high. Attackers could gain unauthorized access to secrets and identities and subsequently steal customer data. They could inject malicious code into the running applications. Or worse, attackers can infiltrate and tamper with your built systems. It's not 100% true that SysCalls or kernel events never lie. The main message of the slide is that having a tool that hooks into the Linux kernel at system calls of interest allows us to know what is going on with the running application, including being able to detect if someone attacks the infrastructure for a wide range of attacks. Let's see how it's done in a quick hacking demo. Once upon a time, there was a Jenkins server. You know, there won't be a happy ending. Suppose you found a vulnerability that allows you to get a shell back by exploiting the crew we script console. The Java parent process spawns a child process that creates a base 64 encoded file in the temp directory. Subsequently, this file is being decoded and this attack, it is an elf binary. Next, permissions are changed to make the binary executable. Finally, executing this implant results in sending a shell over the network right to the attacker. Over this reverse shell, the attacker can run post exploitation commands, possibly elevating privileges, stealing secrets, or a pivot further in your network. This demo also introduced the important concept of the Linux process tree that I want to explain in more detail as it's going to be important later. In the animation, you saw how new processes are spawned and now you see all of them displayed together as if we locked all exactly E-system calls over time. However, the Linux kernel defines the process tree in terms of how it looks right now, considering only processes that are alive. Falco mirrors the kernel's view of the process tree and also allows you to traverse the parent process lineage in real time for each incoming event. A little shameless black, this is a really slick feature of Falco. Under the hood, Falco keeps a cache of processes or more precisely of threads. We store a struct for each thread that contains all the properties we're interested in, such as command arcs, the TTY, the executable path, environment variables, the secret that belongs to and many, many more attributes. We purge the entry on exit from the table and because things are never perfect, unfortunately, there will be broken links sometimes, partially because of tooling limitations, but also because of edge cases in a Linux kernel that are harder to keep track of. Looking back at what Falco can detect today, the quick hacking demo already introduced remote code execution RCE. RCEs are very critical because they effectively open the door to attackers to do more damage to the infrastructure, such as lifting application secrets from disk, local privilege escalation, gaining higher Linux capabilities, eventually escaping the Kubernetes pod sandbox, pivoting through your network and generally speaking, unauthorized access. That's a lot of things we can detect today with the right Falco rules. The only downside is that you need to know what to look for. Do you feel that light breeze let you step out the door here in Chicago this morning? Despite our constant progress in making cloud native security better every day, threat actors are outpacing our innovation, constantly finding new ways to turn NATO past our achievements. What does doing nothing cost you? It could be everything, multiplying a lot of things that are done right by zero is still a zero. What we can do, however, is constantly raise the bar and slow attackers down. Some of the current primary pain points include the challenges with suppressing noise. This rule shows that for Falco alerts that monitor a file opens, you need to limit the scope of the rule to subdirectories or file naming patterns that you know beforehand. Depending on your environment, you may also need to add many exclusions. Unfortunately, this traditional approach puts you at risk of missing out for an entire family of attacks, namely arbitrary file reads. As the name suggests for this, you need to be able to write rules against any file. Sounds crazy? Let's do it. I compiled a test binary that contains the new algorithm I have yet to introduce. In the left terminal, you will see the Falco alert outputs. Suppose we're alerting on any file open because I want to show you how it looks like when you try to monitor all file opens. Most likely you will get a lot of events that happen all the time. Let's run the same test binary again with the new automatic filtering capabilities and very quickly, we're suppressing the obvious noise. With a feature like this, we could alert on potential application secrets lifting, even if we don't know what they're naming conventions. Therefore, implementing frequency counting of Linux kernel events will be useful for security to build future anomaly detection. It can be a first step to what's shifting the information symmetry in favor of defenders. In theory, we are the ones who have access to all data, so we should take advantage of our ability to learn applications normal behavior. I must admit it's not gonna be exactly that easy. For example, what about memory attacks? Cisco's related to memory or IO are so high frequency compared to even file opens that on busy servers it will be very difficult to process all events in real time without either a massive performance set or starting to chop events. Therefore, just keep in mind that much more engineering work at multiple layers is required to expand monitoring capabilities in this regard. However, the arbitrary file re-demonstration showed how accessing and encoding all the rich information readily available on the host can be a game changer. Because it's one example use case you definitely cannot solve of host in a data lake compute environment. In summary, rule-based detections focus on what we think attackers will do, not on what they are doing. Attackers simply don't play by the rules. They do the opposite, which is why saying ahead in Linux runtime monitoring is so hard. Having data rules and monitoring in place for known attacks is a great baseline, but it's often not enough. This approach is also very time and engineering consuming. So why not stop playing by the rules ourselves and speed up novelty discovery and adaptation? Using mathematical models to determine a pattern of past behavior has the added advantage that even if an attacker knows about it, it will be much harder to plant in and bypass such monitoring. As a result, we may be able to slow them down and drive up the operational costs of their attacks. Switching gears, everything I'm presenting now, including the first early prototype demo from before, is a work in progress for Falco. Let's dive into the frequency counting approach we could use in Falco and real world production to solve some of the challenges I just presented. The background image already sets the stage. Mathematical approaches always involve matrices or tensors in one way or another. By the way, The Matrix is one of my favorite movies. The question for you now is, do you take the blue pill or do you take the red pill? Just kidding, follow me. We're about to enter The Matrix. The North Star is to derive a smart mapping of endpoint processes to Euclidean space such that similar events are close by in that space and unusual or abnormal patterns stand out enabling us to monitor and analyze any behavior outside of past behavior in a more abstract and generic way. In simpler terms, we are talking about data compression. For Falco, we have some interesting requirements. The rate of kernel events on real servers is so high that performance must be prioritized over accuracy while still providing the value you need. We need to achieve frequency counting of large data streams in sublinear space using constant time complexity. I believe in using established algorithms. Furthermore, we need to be able to deal with different data types and create a versatile solution that can be applied to our special field of cybersecurity. As a first step, I proposed to use countment sketch which was invented 20 years ago, an algorithm for probabilistic frequency counting of buckets. I will introduce the base algorithm in the next slides skipping all the variants that exist by now. A countment sketch is a two-dimensional array with d rows and w counters each. The rows will be d independent hash functions. They should be non-cryptographic. 64-bit should be fine. They should perform well at run speed. And have, for example, favorable avalanche properties, meaning slight changes in the input do not result in large changes in the output. The number of buckets is much smaller than the universe of possible values. Hence, this is why it's called data compression. The way to shape the dimensions is providing failure and error tolerance rates. Values are probably going to be something like 0.001. For example, you could end up with seven hash functions and 27,000 buckets. Assuming 64-bit counters, the sketch would be 1.5 megabytes. When we receive a new value, we loop through the hash functions, compute the hash value, and take the modular to the number of buckets w to get the vector index. We then increment the counter-added index. To get the current count estimate, follow the same steps. But instead of incrementing the counter, only look up the counts and return the minimum value across rows. Taking the minimum count from multiple hash functions is the trick that makes this algorithm more robust. Often, the goal is to maintain a cache of the top K heavy hitters. For Falka, I think all that is needed is to make the count estimates available to the rules expression language. Countment sketches are prone to overcounting, but they never undercount. In Linux runtime application behavior, we have heavily skewed distributions, therefore having an algorithm that performs well and determining high-frequency counts adds a lot of value to threat detection. Of course, countment sketch will not automatically find and be able to tag an anomaly that is malicious, but that is not the goal. We want to automatically remove all irrelevant data so that we can be more adventurous and broader in our monitoring scope using normal Falka rules. The key takeaways are that unlike hash tables, we use less and fixed memory to kind of achieve the same results as overcounting will be within an acceptable error. The biggest win will be the safety boundary for Falka rules that could blow up in production but now won't anymore. Lastly, dear SREs, we want to make you happy, so this is for you and for my own peace of mind. I thought a lot about this next step. How many sketches per running Falka do we need is one enough? Probably not. Initially, I believed that we should have separate sketches for each container since it seemed logical to treat each container in isolation. But over time, I remembered that this is what people commonly do in data modeling at first only to realize that it is not as maintainable or as convenient as having more generalized models. For example, Netflix first had separate movie recommendation models for each country but later collapsed them into a single global model. Therefore, I think we should use shared sketches across containers, making them larger instead. That being said, I think there should be separate sketches for the values we're counting. Okay, what are we actually counting? The big leading question is what defines the context of a Linux process in the best and most robust manner or how to do counting in a way that matters and results in actionable information for threat detection? I believe this is why advanced modeling is so challenging and why we probably have fewer breakthroughs yet compared to other domains because you still need a lot of domain expertise and sure enough, you're fighting against attackers who will definitely use any loophole they can possibly find. The short answer is that there is no single combination of signals that will work for all use cases. Instead, you need to encode your data in different ways and slice and dice them a little bit. This is why I propose using multiple sketches. Your first sketch could use the container ID process name TTY to encode interactive activity, the executable path plus parent process lineage, plus names of the process group and session leaders to count how often we have seen the same shape and form of process origins before. You would string concatenate all these properties together and then hash them. To find the vector index, the bucket whose count you would increase as explained in the previous slides. Your second sketch could encode the same signals as the first, plus the file descriptor names, which is the file name for open related syscalls for the use case of abnormal file opens. Later in your file code rule, it may make sense to look at both of these counts for the cases where, for example, the file name may contain random timestamps. In that case, the process context alone may help to still show that this is normal and recurring activity. Next, let's take a look at the process command line or proc arcs for that matter. You could encode in your third sketch. This is where it gets really interesting. Many folks coming into the domain of cybersecurity from other domains attempt to treat the command arcs as English sentences, and then try to train models on them. However, there are inherent challenges. First, often there are no command arcs. Second, the more arcs you have, the more information you have to work with. Think of the good old information theory invented by Claude Shannon in the 1940s. And it's true, some attacks may inject a payload over the command arcs. Really though, this is just a subset of attacks. Command arcs can also be problematic, just like file names. For example, Java processes often have many arcs. Some of them are random looking. This may trick you into thinking that this is new behavior when in reality it is not. So yes, we do want to use the command arcs because they can be very useful for finding normal heavy hitter app behavior, but sometimes it might fail miserably. What I'm about to show is non-animal experts, but to some of you it may come as a surprise that what you type into your terminal as command is not necessarily what comes out as process command line arguments when auditing the kernel via tapping fork and exact syscalls. On the left, you have the full context. This is what I would type into my terminal. This is an input to my shell process. On the right, you have what would be logged as arcs in exactly these syscalls. As you will see, they're pretty different. That's because the left is a full programming language that is being passed and evaluated by the shell, which does its own processing that results in the actual fork and finally the exact syscalls that you will see logged on the right. On the left, you have a program that executes bash and tells the shell to open a TCP socket to the provided IP and who bashed standard in, standard out to the socket. On the right, you see that all of the TCP setup is missed. The next two lines on the left are programs that invoke shell built-ins like echo and read. These are implemented by the shell itself, so no external program is needed. As such, no external exact syscalls are made that would allow us to see these critical intermediary steps. In addition to built-ins, the shell evaluates environment variables and pushes them into the child's process environment map before executing the command. As such, hooking exact VE will only see the occur command being run. Both examples are wonderful examples while you need to be more sophisticated and tapping into the right kernel signals. Logging only open or logging open syscalls or using LSM hooks for that matter and logging environment variables will make that information still available. Finally, for pipeline commands, the shell is going to create a number of pipes in memory four or a few times and exact each command individually with the input output file descriptors pointing to these pipes. Looking just at process listings, we cannot tell which commands are hooked up to which other commands, so we lose a lot of semantic understanding as to what the sequence of commands is doing. To complicate matters further, if you simply run a script, we do not look into the file itself. Therefore, looking at command arcs is more interesting for a command injection attacks. By the way, to spot new executables and containers that should be immutable, it is better to use very precise kernel signals that Falco features. For example, Falco can detect if something was written to the upper container file system layer at runtime. In summary, the terrain completeness of shell inputs and other aspects of Linux makes advanced data analytics for cybersecurity so hard. It's not that algorithms don't work, it's that security still requires a lot of domain expertise to use them effectively. The statement extends to the decision of when to use modeling approaches versus when to better invest into super robust kernel signals like the methods I mentioned for detecting new executables. Here, I would like to reiterate that I think the frequency counting proposed is best used in combination with additional Falco rules filter expressions. So don't try something like this in production, but remember what I showed in my first demo? It's still crazy that we can now potentially detect any abnormal file opens without needing to know the file naming conventions for custom application secrets. It's now time for another demo. Again, on the left, you will see the Falco alert outputs and similarly to the first demo. I will show again how heavy hitters can make monitoring and data collection more risky. The container and the background cats at sea shadow all the time. Now running the same test binary with counterman sketch powered filter expressions will quickly perform probabilistic frequency counting and silence the noisy output. When I interactively drop into the container and cat at sea shadow again, almost doing the same thing, we still catch it because the string we hash contained the TTY, which indicates interactive activity. We see the same effects if I now run some dummy command that mimics random command injection against the vulnerable application, for example. I may be biased, but I think that's pretty cool and can be a game changer in production. Here are a few recommendations for how I tend to go about contributing more significant paradigm shifts to the open source Falco project. Working in the open, showing early proof of concepts as I'm doing today is very important to avoid being that isolated monkey hanging off the furthest tree branch just to see if you fall. The other maintainers have been around to block, so you would be foolish not to take advantage of their inputs to ensure that this solution is not only useful, but also maintainable in the long term. The early release phase is very interesting because now you have to ensure that the software is user-friendly and that you help newcomers to the field learn how to use the new capability. In addition, the framework should be extensible so that other contributors can expand it and improve its analytics capabilities over time. If accepted as a stable feature, the community will have deemed it useful and feasible for real-world production and more investment will be made in the future. In this talk, we have discussed how learning, velocity, scalability, and cost can be improved in security. By learning the normal high-frequency behavior of applications and accessing more information on the host, we can increase the chances of detecting unknown attacks. This is important because traditional security approaches are often unable to detect novel attacks. A big explanation mark here, we still need human domain expertise. We can now adapt to changes in application behavior and detect new attacks more quickly. Then traditional solutions. They can also scale to meet the needs of large and complex environments. We can reduce the cost of security by avoiding the need to manually tune and maintain traditional solutions. Additionally, they can help to reduce the cost of data storage and processing by avoiding the need to send all security data to a data lake for analysis. Overall, learning-based security solutions offer a number of advantages over traditional solutions. Okay, thank you for having me here. And I thought I would be going close to the end of the presentation, but it seems like I do have some time for some questions. Do we have a microphone? Okay. Outside of the detection, do you have any remediation options with that, too? Is it just mostly just detecting? Yeah, so this talk was only about detections on the host itself. There probably would be many other talks about how you would go about similar algorithms or procedures in a data lake compute environment or, as you said, for incident response or for other aspects of security. So again, this was only focused on do more advanced data analytics on the host itself because you can tap into data that otherwise you could not. This is why I was highlighting the file open example because it would be crazy to send off all open syscalls of the host. You basically cannot do it. Okay, no more questions? There is one more. Hi, yeah, I had a, wow, that's loud. A quick question on, basically, what you're doing effectively is anomaly detection because you're learning some amount of the normal behavior and then pulling out the anomalies. The question I have is how that scales because when you get, when I think of anomaly detection at Apple scale or Google scale or something, there's so many events happening that anomalies happen all the time and so you still end up flooding the team with alerts. So how do you handle that? How do you do the next layer of filtering to filter out the anomalies you detected that they're not at all interesting? That's the kind of essence of scalability challenges. Correct, I totally agree with you. This is why I recommended to not use the simple accountment sketch filtering while monitoring all file open. So definitely don't do this. And I believe the first next step would be to use these probabilistic counts to, for example, if you look at the upstream file code rules, there's one for sensitive files and that's also the one I showed here. But it only looks at a subset of the Etsy directory. So maybe with this probabilistic counting, you could expand your monitoring to all of the Etsy directory. So I believe in going to small steps and see how it works in production, I would never turn this on for all file opens. And to your point of, there's always going to be anomalies, yes, common entire patterns or SIEs doing ad hoc debugging. People maybe being more new to Kubernetes, they don't know how it works yet. And they also do a lot of interesting debugging. And as you said, at a large company, at a large scale, there will always be a lot of data coming in. But I also believe it's not a problem to over-collect a little bit and capture some more data. And I think the main message of this talk is that you don't have to log all of the exact VE system calls. You don't have to log all of the network system calls or file opens to be able to do very comprehensive forensics, for example. So again, it's really about that you can be more adventurous in your file code rules and expand your logging without plowing up your data lake and production at the same time. I think we have one more question right here. Okay. So you talked about monitoring system calls in particularly file descriptors. But what if more system activity or host activity is added? Wouldn't that lead to more unique events, so to speak? So let's say network calls are, you know, just pairing it up with other activity on the system and wouldn't that lead to more alerts in general? So is it just if we are monitoring one type of system calls or is it possible to scale all into every attribute of a process, let's say? Oh, so I mean, since I'm presenting here about Falco, Falco can hook into, I think by now it's over 300 system calls. The way it works in a kernel, we hook into the system call enter and exit events and you can create your custom rules and monitor any system call you want. In addition to that, you have more options in the kernel like Linux security module hooks or another option to, for example, monitor file opens, there's added benefits to it. And I didn't quite get the other part of the question. I mean, basically it detects one common pattern, right? And then that way it's able to reduce the alerts. But if we add more monitoring, wouldn't that lead to more unique scenarios or cases leading to more alerts, so to speak? Okay, yeah, I bet. So this is why I always recommend to be very conservative in production and first try one little thing and then go from there. And what happens today, if you have Falco rules that are, let's say, considered to be more data collection, they're not expected to fire just on malicious activity. Sometimes what can happen is that it works well for months and then it totally blows up. So I would really like to stress that the biggest advantage of this will be the safety boundary, that these super heavy hitters, you can just automatically tune. And in addition to doing this manually, it's a lot of engineering time and effort. So I would be, since I deployed to a very large environment, there's gonna be more time left for me to work on these features and I constantly tune the rules. So pretty new Falco here. If you could tell me, maybe using a tool like S Trace that's a lot more lightweight versus, what's the advantage that Falco brings and is there any overhead added or if you could go a little bit into that. So I would recommend to you, S Trace is for attack debugging and Falco is for comprehensive monitoring and large production environments. My favorite part of Falco is the rules expression language. It's very powerful to really tell Falco what to monitor on the host. And in combination with that one, I think we can really shift the information asymmetry just a little bit. Cool, thank you so much. Melissa, do we have time for one more? That's 38 seconds left. Okay, this is our last question. Real quick, does Falco have any way of mapping processes to the pods that are running that process so that if a pod moves to a different host, you can maybe track that behavior? Okay, so track when it moves to a different host. That's an interesting question, but yes, we do hook into the container runtime socket and we fetch all the common Kubernetes and container information such as container name tag and also name space and pod name. And by the way, we're currently reworking the K8 client to tap even into more Kubernetes fields. Okay, we're down to zero seconds. Thank you.