 Good afternoon, everyone, and welcome to Falco at Apple. I'm Melissa Kilby from the Apple Services Security Engineering team, no hybrid conference without a hybrid format. We're really good at this. So I'll be virtually joined by Eric Sage, a member of the Linux kernel team at Apple. Today, we're going to show how we deployed Falco as one of our security monitoring solutions. But first, we would like to address why the Linux kernel team at Apple loves EVPS. The Linux kernel team manages and improves the Linux kernel across the fleet of servers that power Apple services. EVPF provides us with an incredibly powerful and flexible tool for adding features into the kernel that provide many improvements over traditional methods of extending kernel functionality. Thanks, Melissa. To go into more detail, traditionally kernel modules have been used at third party code to the kernel, but they come at the high cost of giving full access to kernel data structures and access to the system's hardware. For example, the Falco kernel module may concern itself only with tracing, but is the ability to do anything at once and load it into the kernel. From a security perspective, it does not follow the principle of least privilege. EVPF programs, on the other hand, are limited to a set of curated function calls called EVPF helper functions, depending upon the EVPF program type. EVPF programs allow us to categorize the types of programs loaded into the kernel so that each program be given only the data and functionality it needs to complete its task. The new Falco EVPF probe, for example, is specifically a tracing program and only has access to tracing helper functions and data. In this way, the EVPF virtual machine provides a safe sandbox for the third party code to run in without giving full access to the kernel. Even though it runs in kernel space, it still follows the principle of least privilege. EVPF overall reduces the dependency complexity. There are many reasons why software needs access to kernel data or direct access to the hardware, which has led to frameworks and modules that bypass the kernel to process data in user space or load large amounts of code into the kernel to greatly extend kernel functionality. For example, there are many packet processing frameworks designed to circumvent the cost of the kernel network stack to process packets at line rate. EVPF programs alleviate the need for these drastic solutions by allowing user space to provide different types of EVPF programs to solve specific problems and gives the EVPF code access to very low level processing pipelines, such as tying directly into a network driver before the network stack is invoked in the case of XDP programs. When it comes time to debugging and profiling, EVPF also reduces the need for many complicated add-on solutions. Because EVPF unifies adverse set of frameworks and modules with a single solution, many tools can be created that specifically target the EVPF ecosystem. LibBPF is a user space library that provides functionality on top of the EVPF's call. LibBPF's common set of library abstractions make tool development quick and easy, greatly reducing the work one needs to do to build on top of the EVPF foundation. For example, BPF Tool is one such program that uses LibBPF to provide operators with information about loaded EVPF maps and programs. It can show map contents, jittered instructions, the amount of locked kernel memory, number of programs, invocations, and much more. The Linux kernel team at Apple is always improving our testing and deployment strategy. EVPF has added compile once run anywhere, which allows us to write EVPF programs, which can be targeted to many different kernel versions. This is a drastic improvement of a kernel modules, which must be compiled and tested against specific kernel versions. Compile once run anywhere uses BTF type information to adjust the EVPF programs on the fly to work on a variety of kernel versions. EVPF is also kernel native. Which is a feature that cannot be oversold. Many of the solutions that EVPF replaces do not have direct support from the normal kernel development channels and must work to maintain compatibility with the direction of kernel development. Having EVPF be a part of the kernel means having the peace of mind that EVPF will always be supported by the primary kernel developers. For example, many tools are already being built with LibBPF, which is maintained in the kernel source and mirrored on GitHub. Users of LibBPF can get direct support from the kernels net dev and EVPF communities with libraries actively being developed. Last hybrid slide, let's hear about some of the ways EVPF could evolve in the future. We still believe there are many improvements that can be made to EVPF. We would love to see better tooling around analyzing the performance of EVPF programs to make it easier to identify bottlenecks and inefficiencies in EVPF program design. We would also love to see better tools and documentation for operators that may not be familiar with EVPF just yet, with the aim of simplifying EVPF so that it is easy to understand what each program is doing on a running system. Finally, there are many security features that could be added to EVPF, such as using the same kernel signing mechanisms used to sign kernel modules to sign and verify EVPF programs. That said, we can't wait to see what the amazing EVPF community comes up with next. Thanks, Eric. Now we dive into how an organization can leverage FALCO to supercharge its runtime security monitoring. We will cover a lot of content very fast, such as insights, detections, or data modeling implications. And we are in LA this year. Hollywood magic happens just around the corner, so there will be a magical offensive security demo with special effects, and then we show how you can use FALCO to detect such attacker behavior, which is pretty cool. At the center of the stage are running instances of a program in memory, and they make requests to the kernel via system calls to interact with a piece of hardware. System calls are a highly effective source of truth for a broad spectrum of security monitoring use cases. One could say that system calls hold the potential to capture entire classes of vulnerabilities and attacker behavior often chained together to a nice kill chain. If there was one takeaway message, then it would be, syscalls never lie. Consider this your open invitation to run off and make it a false statement. When a system call is made, it is the metadata around the system call, most notably the process execution path, command arcs, or file descriptor name. That is essential to determine if the observed behavior is unusual or even malicious for your environment. Here's a list of the most important system calls from a security monitoring perspective. They are important because they cover behavior observed in many explorations. FALCO rules are written in a basic expression language. How it works, the rules specifically built for Linux, Kubernetes, or Tognative are instantly evaluated on the host. FALCO ships with standard rules created by the community that you can easily build upon. FALCO can also lock container events or has a Kubernetes integration. Today, however, we continue to focus on system calls only. A subset of FALCO rules falls into the category of very specific security detections. For instance, SSH outbound connections could channel a shell over non-standard ports. Perhaps configurations are such that this should never happen. In this case, such a rule can also serve as assurance. If the alert fires, we can say with high confidence that this is unusual. A larger subset of FALCO rules falls into the category of less specific. They are derived from high-level constructs that are known to be security critical. For example, spawning a shell by a non-shell application or modifying files below a set of system standard binary directories. These type of FALCO rules generate a whole bunch of signals and it is hard to map them to attack our behavior. Such rules can also assist in developing new organization-specific detections. In any case, enormous work is required to analyze and triage them. This is why FALCO logs are usually sent to a centralized data lake where the environment is profiled over time in order to fine-tune the rules. It's now time for a story worth sharing. Once upon a time, there was a web application. What can possibly go wrong? Over the next couple of slides, we will show some of the ways you can exploit the top three consistently exploited vulnerabilities and misconfigurations. That is arbitrary file read bugs, remote product execution, and misconfigurations leading to privilege escalation. I am a hacker now. My terminal is ready to go. Suppose we found a third-party PHP application and then discovered that the app contains a local file include vulnerability. This means the app takes a parameter from the user in the PHP request and then goes to that file to evaluate it as PHP code. Interesting here is that we can use such an LFI to trigger an arbitrary file read. We just successfully read the world readable Etsy password file. More interesting is how do we get remote code execution out of this? The way we can achieve this is to upload a file we create over the LFI and then invoke that file using the local file include vulnerability again. The file we create would typically contain a payload that gives us a shell on the victim machine. In this example application, there was an authenticated endpoint that processes uploader files. So we simply include this admin upload functionality PHP file with the LFI in order to upload our script to the web app. Let's sneak you. And we also know where that file ends up. After setting up the listener on the attacker machine, we make another request to include the just uploaded file with the LFI and that way invoke the payloads. Now we have remote code execution on the box. I now switch roles and am a defense magician. One by one, we will build up the sub-process tree that will contain all the events collected by Falco for the suspect processes. Falco can log the open reach of the file containing the payload. Then Falco detects that the non-shell application spawned a shell. Subsequently, the payload unwraps. And finally through the doop system call, best standard in, send it out, is hooked to the TCP socket effectively providing the attacker with a reverse shell. Over the reverse shell, remote commands can be run on the victim machine. Besides vulnerabilities, misconfigurations are a common concern. In this scenario, the RPM manager is terribly misconfigured. The current user account can run the RPM binary, literally anything as root. I get it, it's not actually that glamorous, but I hope it gets the point across. The pseudo process executed without password enforcement, the attacker now has full control on the system. Falco also has a generic rule that detects if a package management process is run inside a container. This demo made it look very easy to accurately detect all malicious artifacts. In reality, systems are very noisy. And Falco logs, yes, they can drive you crazy. This is why we're now going to dive deeper into some of the challenges in my face once you're ready to get even more value out of Falco. The application is to start a file named Etsy password with a technique called directory traversal. Etsy password is quite literally read all the time on Linux systems, so generically modifying or monitoring this file would be very helpful. But this particular artifact is pretty suspicious and indeed could indicate that the app is vulnerable to an arbitrary file read bug. But what happened to the Falco logs here? The directory traversal signal is not in the file descriptor name. This will make a good and bad Etsy password read look exactly the same. Some source and make file modifications are necessary to unlock the entire debugging engine. So we start our debugging journey in kernel space. BPF printk will show that the file descriptor name is captured as expected. Following the control flow in user space will show that the FD path is being sanitized. Modifying the sanitization function can fix this. Ironically, user input sanitization function, or let's say better like they're off, are often the root cause for such arbitrary file read bugs we're trying to detect here. Next, making sense out of command arcs or file descriptor names is the tricky part that can drive you crazy. This is because one, you rely too much on prior assumptions to match for two specific strings and are at risk of missing potentially malicious behavior. Or two, you have a hard time tuning your rule and constantly play catch up with filtering out benign activities specific to your environment. The hard part really is that shell input is touring complete. Combined with the inherent variability in exploits, there is an unknown amount of semantically related process behavior that will look very different in all of these system call metadata. To make matters worse, many benign commands contain exotic elements. Think of hashes, IPs, timestamps, that can make normalization and profiling a daunting task in and of itself. Furthermore, similarity or rarity alone is of little help to know if the intent of the process is malicious. Therefore, knowledge bases that encode domain knowledge are super critical as well. Finally, depending on your threat model and the type of activity you seek to detect, you also have to be aware of ways that obfuscate behavior. For example, when reading a file over shell built-ins, no external exactly ease system call is made that would allow you to see those intermediary steps. However, when touching a monitored file, FALCO has the power to accurately log such activity regardless of the level of obfuscation or exploit variability. This file descriptor or monitoring capabilities are beautiful and one of my personal favorite features of FALCO. In summary, the community as a whole still has a lot of work to do to solve both the semantic similarity challenge and the intent attribution challenge. Moving on to another important aspect. There is a growing body of research on correlating signals according to the MITRE attack framework in order to derive a higher fidelity alert. Correlations can also help tremendously in the alert triage process. Popular tools are big data technologies such as Presto, Spark or Elasticsearch. As far as methodologies are concerned, there are endless ways. One simple approach could be to simply group and aggregate the signals. Grouping could be done on a per container basis or for a particular sub-process tree. When performing sub-process tree analysis, there are a few tricks that can help reduce computational burden. For example, you can check for the same virtual process group ID to correlate all FALCO rules in a connected sub-trees. Or you can check for the same TTY or login shell ID if the use case is to track down activity in the same terminal or interactive shell. One point of caution here, again, making too many prior assumptions regarding how correlated signals are expected to look like can be tricky. In reality, we just don't know. Things always look very logical in retrospect and puzzling in prospect. And this is especially true in security. Personally, a big fan of first focusing on high value single events one at a time in line with what we have shown up to this point before performing correlations. One big question remains in the room is FALCO rights for my organization? We demonstrated that these type of tools hold the potential to capture exploitation artifacts for entire classes of vulnerabilities. Within one day of deployment, you can get a lot of value out of the tool as the standard rules that were created by the community are very well-tuned. FALCO is production-ready and battle-proven, featuring a sophisticated, yet easily extendable kernel instrumentation. Finally, the tax you pay in terms of CPU and memory usage is appropriate considering FALCO's capabilities. Earlier, we touched on some of the possibilities EVPF in general could evolve, but also for security monitoring focused extension, there are a lot of possibilities that have yet to be explored. One projection could be that more high-value kernel signals could be discovered. This is also in line with what we heard earlier this morning that perhaps the way we do tracing at the moment with FALCO is not what we should be doing in the future. And this also includes new ways of deriving high-value kernel signals, also enhanced encoding of the enriched metadata around process collection, which would address all the challenges we touched on earlier when making sense out of command arcs or file descriptor names. Finally, advanced pre-filters in the kernel could be in enticing proposition. Advanced here could mean that the semantics or intent of a process is encoded as a state vector for more dynamic AI ML-like reasoning right in the kernel. And I am aware we're very, very far away from this, but remember a lot can happen between now and never. Wait, what if I didn't pay any attention at all? It happens. Start with a very simple rules created by the community, customize FALCO and FALCO rules to suit your environment and threat model better, then slowly venture out eventually into deep space to start getting even more value out of the tool. And for which sophisticated data modeling approaches can be of help. Perhaps in the future we'll be given the opportunity to dive even deeper into modeling implications for now. Thank you for having us here and big shout out to the FALCO developers and the FALCO community.