 So we have Noah Crowley, who will be talking about BPF in practice. And if you folks have any questions for them, just post them, post all the questions in chat and they'll try to answer them. Thank you for waiting. Hey everyone, welcome to BPF in practice. I'm Noah Crowley, I'm a developer advocate at Tigera, the makers of open source Calico, which I'll talk about a little bit later in this presentation. I've been a developer advocate for about four years now, primarily focused on DevOps methodologies and observability. My first exposure to BPF was actually as a DevOps engineer. So we were working with an API that wasn't properly instrumented, right? And which insisted that it was performing perfectly well while the other services that depend on it kept hitting request timeouts and throwing errors and being generally unhappy. So our goal was to figure out what was going on. And the SRE I was working with at the time led me in the direction of Linux's perf functionality. But he also mentioned that there was this new approach using BPF, low level technology originally for packet processing that was being adapted for wider use. So the second time I encountered BPF, those predictions had begun to come to fruition. I was working at influx data where a pair of engineers Leonardo Di Donato and Lorenzo Fontana were working on instrumenting third party infrastructure components with BPF. And this gave them the ability to gather telemetry from our infrastructure that they hadn't been able to capture before and in doing so gain a better understanding of where performance improvements could be made. So those two would go on to build kubectl trace, which is a program for scheduling BPF trace programs into your Kubernetes clusters. And they would work on as maintainers on the Falco project, which is another project I'll talk about a little bit later. So let's back up for a second and talk about what exactly BPF is. BPF stands today as a relatively new facility within the Linux kernel. So work began on the project in 2014, which in the SAS world might sound like an eternity ago, but in the world of kernel development is actually fairly recent. But the roots of BPF go back even further. So in 1992, a paper was published entitled the BSD packet filter, a new architecture for user level capture. And this paper lays out a problem. So at the time, many Unix systems had packet capture capabilities for network monitoring, but they were implemented in user space, meaning that the packets had to cross the user space kernel boundary and incur performance penalty. So BPF was designed to filter the packets in place by providing a BPF pseudo machine embedded in the Linux kernel and designed to work with register based CPUs. So those two things combined resulted in significant performance improvements. And this became the primary method for packet capture in Linux based systems. So if you've ever used TCP dump, for instance, you've already made use of BPF code since TCP dump commands are compiled to BPF programs when they're executed. So TCP dump makes use of the original BPF. The project that started in 2014 was called extended BPF or EBPF. And the goal is to extend BPF beyond its original capabilities and turn it into more of a general purpose virtual machine. And there were a few reasons behind this push, right? So the first is that being able to modify the kernel is valuable. There are performance penalties involved in switching between kernel and user space. And that becomes important for functionality like packet processing. And the kernel is the mediator between your hardware and your software. So it provides this perfect vantage point to see what's going on, making it a good fit for observability and security use cases in addition to networking. But there was already ways to modify the kernel, right? You could check out the source code and change it yourself or you could use a kernel module. But there are downsides to both of those approaches. What about using kernel modules? Well, you might not be able to run a kernel module. Maybe you're in an environment where it's simply not possible to install one, such as if you're using a Hoseku Burnetti service. And if you can use a kernel module in your environment, you're installing a piece of code that is full access to the kernel and to your hardware, and that is not inherently safe. Now, ideally, the module developers will have done a good job, but you'll still need to audit that code and bugs and vulnerabilities do happen, and that can take significant amounts of work to do. So let's start talking more about the features of BPF and how they address these issues. So unlike kernel modules, which have full access to the kernel, BPF programs are executed in the BPF VM, and they require helper functions to access kernel functionality. And the way that BPF programs are run is that they're attached to execution points within the kernel, and when a kernel event occurs at that execution point, the BPF programs are fired. So when you create a BPF program, you can define which execution point you want to hook into, and this in turn defines different helper functions that you can use. So different execution points give you different kernel capabilities. And this limits the scope of your programs. It keeps them small, and it fits neatly with the principle of least privilege. So what are these different execution points? What kinds of programs can you write? So this is a list from user include Linux BPF.h from the 5.14 kernel, and it's pretty long. There is a fair number of things here, and they're somewhat inscrutable. So let's look at just a few of these in particular. So the first program that was added to the Linux kernel is the BPF program type socket filter. This attaches your BPF program to a socket, and it gives you access to all packets flowing through that socket, but you can't make any changes. So this is primarily for observability use cases. BPF program type K probe allows you to dynamically attach to execution points within the kernel, running your code both before and after a function executes, allowing you to instrument any function on the fly. This is extremely powerful and lets you look at all sorts of things, but because it's dynamic, it has the downside of not providing a stable interface to the kernel. So a program that works on one machine might not work on another, or a program that works on one version of the kernel might not work on another. In comparison, BPF program type trace point attaches to trace points, which are guaranteed to be stable. So your programs will work even with new versions of the kernel. We'll talk about a few more of the program types when we dive into the practical examples in a few minutes. So in addition to limiting the scope of your programs based on where they attach to kernel events, BPF ensures the safety of your code by running it through a static verifier to ensure that it won't interfere with the operation of the kernel. So the verifier ensures that your BPF programs don't have issues like invalid pointer dereferencing or exceeding the maximum call stack, but this imposes a few limitations. So for example, early on in BPF's development, loops were not permitted entirely because an infinite loop might end up locking up the kernel. Today, there is some limited functionality for bounded loops that's been added, but you don't have the same flexibility that you do when you're writing normal code. And these are the tradeoffs you have to make to be able to run program BPF programs within the kernel, right? The last thing you want to do is take on your entire machine with a badly written program. So the final component of BPF programs and the thing that ties everything together are maps. Maps are how BPF programs exchange data with user space. Without maps, much of what folks are doing today with BPF wouldn't be possible. I'm not going to dive too deeply into maps here, but they come in a variety of forms. There's basic stuff like hash maps and arrays, and then there are specialty maps like BPF map type program array, which is used to make tail calls from one BPF program to another, chaining them together. And that's actually something that is used in open source Calico. This is a list of all the maps once again from the BPF.h header. And there's a lot of stuff here, and you might be looking at this and the program type example from a minute ago. You might be saying to yourself, okay, but I'm not a kernel developer, and I'm probably not a BPF developer either. Although writing BPF code is easier than modifying the kernel itself. But one of the limitations of BPF today is that the project is still very much in progress. And because it interfaces with the kernel at a low level and because it uses dynamic instrumentation, programs can break from one version to the next. That means if you're developing your own tools, you need to stay on top of updating the kernel itself. And I think that's one of the limitations of BPF today is that the project is still very much in progress. And because it interfaces with the kernel at a low level and because it uses dynamic instrumentation, programs can break from one version to the next. That means if you're developing your own tools, you need to stay on top of updating them to make changes. So not only are you not a kernel developer, not a kernel developer, but you have to keep doing all this extra work. So the question is, what's the alternative? How can you get value from BPF if you're not writing code? And I think the answer is to use other people's tools, right? The benefits of BPF are not just in being able to write code, but being able to use code that others have written. So I'm going to talk about three projects. They're all open source. I'm going to cover Falco, which is a runtime security engine. I'm going to cover Calico, which is a CNI plugin for Kubernetes. And I'm going to cover BCC, which is the BPF compiler collection, which is a tool chain and a set of tools and applications. So first off, we'll talk about Falco. Falco is a runtime security project. It was originally created by Sysdig in 2016 and then was donated to the CNCF in 2018, where it's now an incubating project. Falco functions by monitoring the Linux system calls at runtime and comparing them against a set of rules for what is normal, acceptable behavior, and what is anomalous, potentially threat-invocating behavior. And Falco will fire off an alert if those rules get triggered. And one of the things I find most interesting about the Falco project is that it's a tool and not necessarily a solution. So Linux Syscalls paint a really clear picture of what's going on in your system, but it's up to you as an operator to understand that system and to apply engineering and security principles to how you monitor, alert and eventually react to incidents. And that can vary from environment to environment. Falco does ship with a default set of rules and they check for things like namespace changes or executing shell or SSH binaries, mutating core Linux functionality, things like that. But as I said, you can do some modification on those rules. And the Falco team, the maintainers announced at the beginning of this year that they're actually also working on a real Falco rules language with a compiler and everything they say. So that's going to open up a lot of interesting opportunities in terms of flexibility and what you can do with this software. We'll look at a few examples of rules in a moment, but first let's talk about generally how Falco works. So Falco consists of two components. The first is the Falco process, which runs in user space. And the second is the Falco driver, which interfaces with the kernel. So Falco actually provides multiple drivers, both the traditional kernel module driver and the BPF driver. And the developers have talked about how one of the motivations for this BPF driver was this idea that I mentioned earlier where you want to add functionality to a system, but you don't necessarily have the ability to install a kernel module, such as if you're running in a hosted Kubernetes environment. Today we're going to focus mostly on the BPF driver. So the BPF driver makes use of two program types. The first is BPF program type trace point. And the second is BPF program type raw trace point. And both of these hook into the kernel at those static marks that I mentioned earlier and parse the sys calls as they occur, passing them to the rules in engine for evaluation. And the rules themselves are written in YAML and make use of cystic filters in one particular section. So these are the fields, the key value pairs that are required to define a rule. They've got the rule name, they've got the condition, which is the cystic filter. They've got descriptions. You can change the output, the log messages, different things like that. So this is an example of one rule. This rule alerts whenever anyone tries to write a file below the Etsy directory. This kind of thing is super valuable obviously in your system. If somebody's writing to the root directory or someone else, there's a good chance that you don't want them to be doing that. So this is a relatively simple rule. It just works by, you know, on its own. Falco also has some capabilities, like I said before, for customizing those rules. So this is a more complex rule that uses a list in addition to a rule. So Falco has lists and it also has some macros involved. So this is a more complicated rule. Falco gives us both lists and macros to give our rules more flexibility. And here we want to be able to have one rule that accounts for all of these different binaries. So we have a list, shell binaries, and we have a rule, run shell untrusted. And anytime someone attempts to spawn a shell below a non-shell application, such as, you know, if they're running a web server in a container or something like that, then Falco is going to throw an alert. So Falco already provides a lot of functionality with these built-in rules and your ability to customize them. And additionally, you can have Falco parse Kubernetes audit events and alert on anonymous behavior in your Kubernetes cluster. And hopefully soon we'll start seeing the work that's being done in terms of the Falco language. So if that sounds interesting to you and you want to get involved in Falco or try it out, you can visit their website at falco.org. You can follow them on Twitter at falco.org, Falco underscore org on Twitter. Or you can also join the Kubernetes Slack and talk to the maintainers in the Falco channel. So the next project I want to talk about is Calico. Calico is an open core project originally founded at MetaSwitch. Originally for OpenShift networking, but it's since evolved into managing networks within Kubernetes. So it does things like assign IP addresses, manage network routing, enforce network policy, things like that. And Calico has a modular architecture. So different components are responsible for different things like IP address management or configuring routing, configuring BGP configurations, and even the data plane itself. So this architecture means that Calico can support multiple data planes. And as we all know, all software has its own trade-offs, right? So different data planes are more appropriate for different use cases. Calico's default data plane is based on Linux IP tables. It's battle tested. It's well understood. But Calico offers additional data plans to support environments like windows or to provide different affordances than IP tables, such as a data plane based on BPF, which is what we're going to be talking about today. So the Calico BPF data plane makes use of BPF programs of the traffic controller type. And there are two of those. The first one is BPF program type scheduler classification. And the second is BPF program type scheduler action. These programs attach at both the node ingress and the egress point of the Linux networking stack just before the packets get to IP tables. So because Calico accesses the packets early on in their lifecycle, it can handle them through a fast path that bypasses much of the networking stack, avoiding the packet processing that the kernel normally does and resulting in a significant performance benefit. The BPF data plane can scale to a higher throughput. It can use less CPU per gigabit than the standard. While using less CPU per gigabit than the standard IP tables data plane. So the way that this works is at those TC hooks, we're able to intercept the packets. We do a look up in what we call the CT map, which is equivalent to the contract table in the traditional IP tables data plane. And then if we've seen that flow before, we can bypass the whole IP tables and send it directly to where it was supposed to be. And being able to modify the packets using BPF also gives us some Kubernetes specific functionality that we don't get with the standard data plane, such as preserving external client source IP addresses and supporting direct server return for more efficient routing. So the logic for this functionality is compiled ahead of time and uses a set of maps to store the NAT information. So there's a map for storing the metadata for a service and a second map for storing the IP addresses of the backing pods. But moving packets around isn't all that Calico does, right? It can also enforce network policy. And network policy is one of the core components of network security. It allows you to control what network traffic is allowed and what network traffic isn't. So Kubernetes policies let you filter traffic by specifying selectors and those end up filtering any traffic that matches to the selector. So here's a pair of network policies that matches traffic based on a pod label. So if you can see, there's two labels, one red and one blue. On the left you have our ingress policy and on the right you have our egress policy. And you also have a specification for the ports that this traffic is going to go over. But Calico actually offers both Kubernetes policy and Calico policy, which has an expanded feature set such as being able to define policy based on service name. And because of the complexity of Calico policy, we run into issues flattening the policy into a format that can be easily stored and worked with in BPF maps. And so we had to come up with another approach. So unlike the packet processing and load balancing logic, Calico policy is converted into optimized bytecode and uses BPF maps to store the IP sets that are matched by the policy selectors. And then these programs are chained together using BPF tail calls. So we'll go from the main program. If it's an established flow, we can send it directly to the virtual Ethernet interface and into the pod. But if it's not, then we can send it through this policy program to figure out just exactly what we're supposed to do with this flow. The BPF data plane also provides connect time load balancing using BPF. So when there's an incoming connection to the Kubernetes service, Calico will intercept it and reconfigure the socket to connect it directly to the backing pods, the IP address instead, removing all NAT overhead from service connections. And actually BPF is also used in the standard IP tables data plane, making use of the XDP program type to provide denial of service mitigation. So the XDP program type allows you to insert BPF programs very early in the packet processing pipeline. In fact, XDP programs can be attached at a few different points, including the kernel, the network driver, but even the network interface card itself. So Calico will pick the fastest hook to use that's available to it, and it'll set up its XDP programs there. And what that means is you can drop connections with XDP drop really early on and without tracking the connection, which ends up bringing up a lot of the CPU that's required to deal with the flood of incoming data. So if you're a Kubernetes user and you need to enforce network policy, Calico is a great choice. You can give it a try, you can go to projectcalico.org, you can follow projectcalico at projectcalico on Twitter, or you can join the Calico user group Slack at slack.projectcalico.org. We also have a monthly community, meaning if that sounds appealing to you. So the last project that I wanted to talk about today is BCC. And BCC stands for the BPF compiler collection. And this is a toolkit for building BPF programs, but it's also a collection of programs and examples that you can use today. So if you're looking to write your own BPF programs, then this might be a good choice. You might also want to look at libbpf, which is an alternative project with a slightly different interface. But this talk is not really about writing code. And the idea here is that you can already use a lot of the tools that are being provided by the BCC project. So this is a list, well, not a list, but a graphic that shows the various tools and utilities that are available to you. There are a lot of them, and they cover everything from specific applications to system libraries and system calls to your I.O. stack, your networking stack, virtual memory. And depending on the type of investigation that you're looking to do, you might use one or two or more of these different programs as you're probing the performance of the system. So let's jump into a quick demo here, and I'm going to show off just a few of these programs very, very quickly so you can get an understanding of what they can do. Right here, I have a virtual machine. It's running Ubuntu 2104. I've installed the BCC tools using the repository package manager or the distributions package manager. And so they're already on the system. And Ubuntu is a little weird because it appends hyphen BPFCC to all of the program names. So if you see that, don't get confused. So the first tool that I wanted to look at is execsnoop. And execsnoop is a program that traces the execution of applications on your system. And this can be really useful for looking at intermittent applications that can intermittent programs that can potentially eat up CPU, but it might not show up if you're looking at something like PS or processes and long running processes. So I can come in here and now that the program is running, I can cat this file that I have, you know, are prepared. And once I cat it, it will show up and it will say, hey, look, you ran this program. I can also list the directory and see, you know, that that fires off the BPF program as well. The second example that I wanted to show is opensnup. So opensnup functions similarly to execsnup, except instead of looking at process execution, it looks at files being opened. So if we run that application, we see that we're already getting results because files are constantly being accessed on your system. And if we look at this here, this is a little noisy, like how are we going to be able to find what we're looking for in this output? So one way to do that is to give opensnup a name argument. So in this case, we're going to give it cat again, and we're going to run the program. And this time you'll see there are no files being opened. But if we come down here and we type cat cat me and run that, then we see all of the files that cat accesses when it runs, as well as the final file there at the end cat me that it actually concatenated onto the shell. This is really useful for looking at things like misconfigurations if a program is trying to write log files to the wrong place. If it's looking in the wrong place for configuration files or things like that. So the next application that I wanted to show off is called bio latency. And this is not about biology that stands for BPF, IO latency. A lot of these tools existed in different forms but for BPF. And what this tool does is it actually starts to trace block device IO and generate a histogram of response times. So you can get really specific information about how your IO is functioning that you might not get from the higher level metrics in your system. And after that ran for a couple of seconds, we can interrupt it and we can look at this histogram and we can see that response times here for this file system are below 255 microseconds, which is very, very fast. So that's looking great. But this can let you rule out or hone in on issues with device IO and, you know, either say it's not or it is that and either focus on other problems or figure out how to fix it. So the final program that I wanted to show off is called TCP Connect. And TCP Connect does what it sounds like it does. It tracks TCP connections. So this can be really useful for seeing if programs are trying to access the wrong services and they're trying to reach out and connect to things they're not supposed to be. But if we come down here and we curl Wikipedia.org, we'll see that result show up in the TCP Connect program. So it gives us the process ID and the command that initiated the connection. It gives us a source address, the destination address, and the port and a lot of information that you can use to dig sort of deeper into these applications. So that's all I have for now. I think BPF really is exciting. It offers a lot of new possibilities. It's being used all over the world at places like Netflix, Facebook, and Google. They're even working on an implementation of BPF for Windows. There are some limitations. There are some shortcomings. But like everything in software, there is always trade-offs. So I encourage you to get out there to try it yourself to see if you can gain some value at your company from making use of these applications. So if you have any good BPF stories, please reach out to me on Twitter via email. Thanks again, and have a good one. Windows. Oops, sorry about that. We have Noah. He will try to answer your questions. Hey, everyone. Any questions for me? Looks like there's one in the chat. Yeah, I believe there are commands to find out who's logged into a server. You can definitely trace the various syscalls that go on the scene behind the scenes when that happens. I'm not sure exactly which command it is, but I will look that up for you. And if you reach out, I will let you know. So I think in terms of using a Unicarnal to circuit the packet path is kind of a bigger question than just is a Unicarnal better than BPF. There are so many other considerations that you have to take into account if you decide to go to a Unicarnal. One of those things might be that you might have a Unicarnal with BPF built into it. I don't know if that exists right now. But one of the complaints that I've heard about Unicarnals is the difficulty of debugging them. And that's sort of counter to the idea of BPF where we want to make things easy to debug. So I think the concerns that you have to address when you're thinking about those kinds of situations sort of go beyond just using BPF or not for network processing. Yeah, I'd love to check that out. Thank you. All right, looks like that's it in terms of questions. I'll be around the event the rest of the day. So if people want to send me a direct message or anything, please feel free to do so or reach out to me on the other contact map that I specified. Awesome. Thank you so much for the presentation, Noah. Yeah, my pleasure. Sorry, y'all, for being late. I appreciate your patience.