 Hi everybody. So this is going to be a little different. I'm not going to be presenting a brand new tool with Uber capabilities and all of that. I'll focus more on building a custom tool. And today we'll be building a Kubernetes tracer, an ABPF Kubernetes tracer that will watch file access. Basic, but again, the focus is on what it takes to build something. Why do you want to do it? Why? Because, well, at some point you outgrow the tools you start with and possibly you need custom business logic. So what are those tools? BPF Trace is probably the number one tool you'll start with. It provides a high level interface to EPPF capabilities and it's awesome. It allows you to hook into a lot of different parts of the kernel and the system in general, but its functionality is limited. There's only so many functions that provide you and if you want something more, you need to look elsewhere. So what about Kubernetes? So with Kubernetes, you have cube control trace that pretty much provides an ability to run BPF Trace on Kubernetes, but actually one of the newer capabilities in cube control trace is an ability to run other things. So in a way, one potential design option is to take cube control trace and create your own tracer and hook it up together. So that can be a shortcut if you want to go that way. But before we look at the Kubernetes side of things, we'll take a look at the fundamentals and the key EPPF concepts developers would care about. There's a couple of things you'll want to explore when you're building an application. First, where do you want to hook in? You need to decide what part of the system you need to hook in and how. And that's covered by the different program types available in EPPF. Then you also need to figure out the IO. How are you going to talk to the kernel part of your application? How are you going to provide the config parameters? How are you going to get the data out and all of that? So usually when you build an EPPF-based application, you have a kernel piece and a user piece. And then in the kernel, you have a number of hooks. For us, it's going to be tracing hooks. Then you're going to have a map or a set of maps that you would use for input and output, mostly configuration and basic stats on the way out. And for any kind of dynamic data, you probably want to use perf buffers or ring buffers. And ring buffers are perf buffers plus plus, much nicer, easier to use, single queue and all of that. And then you might also, if you're using newer versions, you might also use global variables for basic config parameters. So there are quite a few program types, a lot of different places where you can hook in. And most of the program types, they fall into these high level categories and the biggest categories are tracing and networking. And today we'll focus on tracing, but there's a lot you can do with networking and lots of great talks about that. So with tracing, you have a choice to make. There are different program types to choose from. And they allow you to do static tracing or dynamic tracing. So with static tracing, the nice thing about it, you use stable hook points in the system that will not change from one version of the system to another. So that's great. With dynamic tracing, you have more flexibility, but it's more brittle. In the dynamic tracing bucket, you have your K probes and K red probes or U probes and U red probes. And there you can hook any function you want. The problem there is that if all of a sudden the function goes away, you're out of luck. But with the static trace points, there's a well-defined interface that's stable. So if you want a tracing app that's stable and it will run from one version to another, you definitely want to pick static trace points and do static tracing. And that's going to simplify a lot in your application. And then with trace points, there are a few different options. For example, in the kernel, you have trace points and row trace points. And depending on how advanced your application is, you might want to pick row trace points. It's pretty much trace points, but without parameter pre-processing. So if you need direct access to data, you'll use those. And Tracy, for example, uses a lot of those. So once you have a general idea of your system architecture, you need to go, well, you need to figure out what components to use, what kind of code and libraries you'll want to use. And this is actually one of the biggest challenges you'll have to deal with because there are a lot of different libraries and lots of different languages. And if you're new to eBPF, you're going to be lost. So many things to choose from. Which one do you pick? And you don't want to invest in one thing and then hit a dead end. But C and C++, the most known library is PCC. So if you're not brand new to eBPF, you know that it's probably the biggest library out there, the biggest ecosystem with lots of examples. It's most known for its Python wrapper. So a lot of talks this year, previous years, any talk about eBPF, almost any talk about eBPF, most likely included Python snippets. The main gotcha with PCC and the wrappers around it is that you need to compile the eBPF program on the target machine. And Grant talked about that, the whole idea behind compile once and run everywhere is kind of the solution to the problem with PCC because, you know, when you distribute your application, you don't want to bundle the compiler with it and you don't want to deal with the brittle environment because you don't know what's going on. So but with eBPF, you address that problem with this new concept of compile once run everywhere. It's low level, it provides a lot of nice capabilities and the skeleton it generates is actually super useful. Even if you're not a C expert, it makes it easier to work with eBPF. eBPF Pub is a new library. It was created for OS Query. It's a simple wrapper for LibBPF designed for tracing. Haven't used it much, but it looks interesting. The code is pretty clean. Lots of Go libraries. The first one in the original library is GoBPF from IO Visor. It's a wrapper for BCC and you get the same benefits, most benefits of BCC and the downsides from it. It's a good option for tracing and a lot of tracing tools in the cloud native space have been built with this library. There's a library from Celium. It's a pure library, pure Go library. It's interesting. One of my personal complaints is that it tried to invent its own terminology, collection specs. I don't know. It's confusing. There's enough confusing stuff there with eBPF. Any more confusing stuff is going to make things more complicated. But one of the nice features there is the BPF2Go module. It allows you to embed the eBPF program, compiled eBPF program code into your Go code. It provides a nice wrapper to load it and to work with that. They're also working on the compile once run everywhere support. They support a lot of it already. I've seen samples, but they're almost there. There's a library from Dropbox. It's pretty nice, much nicer than the Celium library in terms of its structure and how easy it is to understand. Unfortunately, it doesn't support enough program types. So if you want to build tracing application using trace points, you're out of luck. There's a library from Aqua, eBPFGo, and Tracy is using it. This is actually one of the libraries we'll be using today. It's nice. It's a thin wrapper. It exposes a lot of capabilities in terms of what you can attach. I was surprised to see an ability to attach traffic control programs. That's actually not something you get with a lot of libraries. One thing I wish it had was an ability to auto-attach or default attach programs. You kind of need to explicitly attach the program that you defined and we'll see what it looks like in the code. With Python, the BCC wrapper is most known to anybody out there working with BCC. Nice for prototyping on your local machine. That's probably the first thing I would do easy. I like PyEBPF because it tried to introduce this idea of Python-only BPF experience where you don't need to write C code. It needs more attention. It needs updates. With Rust, there are quite a few libraries. There's a lightweight wrapper, LibBPFRS. It's pretty much the official wrapper. If you look at the LibBPF bootstrap repo, you'll see Rust examples. I think that's the library that you used there. RedBPF was one of the original LibBPF wrappers. It has a number of other capabilities. The most exciting Rust library is Aya. The author of Aya, at least one of the primary authors, used to work on RedBPF as well. It's pure Rust, including the BPF program code. It means that you don't need to write any C code. If you're a Rust person, it's super cool. It's still a little early. The tooling ecosystem looks promising. Yeah, a little tough. I have a few samples to show there. There are a few other libraries, but they're mostly BCC wrappers. The ILLUWA wrapper is the official wrapper. Now that we looked at a few fundamentals in the libraries, we can take a look at the Kubernetes side of things. The first thing you probably want to figure out is the KubeControl plugin. For the Kubernetes side of things, you'll need to have two parts. The plugin, technically, you can work around it and just use KubeControl and manually set up resources, but it's nice. That's what KubeControl trace does. Then you also need to figure out the part that you run in the cluster. With the KubeControl plugins, technically, you don't need any libraries. You don't actually need any programming language to create a plugin. You can create a simple shell script, put it in one of the bin folders, and it will be picked up. As long as it starts with KubeCTL- and your plugin name, it will be picked up. You can use libraries if you want to. There's a nice helper library for go, CLI runtime. That's what KubeControl uses. Internally, I think it was extracted from it. I didn't find it super useful because I don't use Cobra for the CLI options. A lot of the helpers are around that, but the client configuration helpers, they're actually nice. Then with the KubeControl plugin, you also need to figure out how you'll interact with Kubernetes, what client library you'll want to use. Technically, the language you use for the KubeControl plugin and the language you use for the tracing engine, they don't need to be the same. They can, you know, the KubeControl plugin could be go. The engine could be C, but there are a few options. There's obviously the official Client Go library and there's an official Python library. It has a lot of examples, pretty nice. There is an unofficial Rust library, KubeRS, also nice, probably has the most examples I've seen. Today, I'm not going to show examples in each language, possible language, but it's nice to know that there are options. There are a number of options for other languages, but yeah, feel free to explore. I'd stick with those three. So once you figure out how you're going to build your KubeControl plugin and how you're going to talk to the Kubernetes cluster, you'll need to figure out how you'll run the tracer application in Kubernetes. There are a few options. The two major options are using jobs and Demonsense. Jobs is what KubeControl Trace uses. So when you want to run KubeControl Trace, when you type KubeControl Trace, run something, something, something, then the KubeControl plugin will schedule a job on demand and it'll figure out where to run it and then the BPF Tracer or the other tracer will run on the selected nodes where the job is running. So when, but when you're using jobs, you need to figure out how to, where to run the jobs because if you need to create an application in a certain pod, you need to make sure that the job runs there. There are a few options available there. The most basic one is to set the node name in the pod spec, but it's brittle and it's not recommended. The next best option is to use node selectors and that's pretty much the most basic option that you can use. If you don't need any kind of fancy way to select the nodes where you want to run the job, that would be the way to do it. And the last one, the last option is to use node affinity, but node affinity that provides more advanced node selection capabilities. And this is, I think that's what KubeControl Trace uses internally, node affinity. The other option to use is DemonSets and it's actually the option that a number of other tools use. The tradeoff there is that with jobs, you get tracing on demand. So if something happens and then you decide to run KubeControl Trace, and it's nice on one hand, on the other hand, it's, you know, it might have gaps because by the time something happens and you run KubeControl Trace with a job, what you need is already gone. For example, if you need to trace how an application binds to a pod, using a job isn't really a good design there because by the time you decide that, hey, I want to run KubeControl Trace, the application already called the bind call and all of that. So that's when DemonSets are nice. Kubernetes takes care of running your pod on every machine. The only got you there is that it's on you to keep track of the containers that you want to track. And tools like Inspector Gadget have a number of tricks to detect that. For example, with Cryo, they use pre-start hooks. So they get notified when a container is created, but it's not running and then they can configure their gadgets, their tracers to trace the application before it even starts. So we have a few minutes left and I figured it's a good opportunity to take a look at the code. So this is the, I'm going to make it a little bigger, this is the KubeControl plugin for the sample application. And it's using Go, and like I said, it doesn't need to be Go. It doesn't even need to be in a real programming language. But here it has this install command and the install command connects to Kubernetes and then it launches a demon set. And most of the time, and a lot of samples I've seen that they use KubeControl and they use YAML and all of that. So I figured it would be a nice example to show how to create a demon set programmatically. It's pretty much the same things you'll do either way. There's a couple of things you'll need to make sure you set. One of them is privileged mode because you're going to have a tracer. It needs low level access. And then you might need to mount certain host file system paths. So with this sensor, I have a couple of, right now it's configured to use the C-sensor. And I'll have a little demo in a few minutes. So yeah, one, let's see. So here's an example of KubeCTL. This is actually a shell script. So if I run KubeCTL file, and that's the name of the plugin I created. And it ends up being in one of the path directories. So if I run KubeCTL file, install, it's going to install the application on Kubernetes. And then I have a demon set right there. And if I look at the pods, I see that I have a pod. And if I look at the logs, this is what the sensor is doing. It's the C-sensor. And it just shows me what different applications are accessing. And then, so let's take a look at the application. So this is the go application, the go sensor that gets scheduled in the demon set. And so that's the main code there. When you started, you create a file monitor. And then when you initialize it, you're going to load the embedded, and in this case, it's embedding the BPF bytecode using the new embedding feature in go. And then it loads the BPF module, and then attaches the trace points. So this is what I was referring to. You have to, for each trace point you have, you need to call attached trace point explicitly. With the LibBPF library, there's a single call that you can make to attach everything. So it would be nice to do that. So once you attach everything, it sets up perv buffers to get the events from the kernel component. And then in the run loop, it consumes events from the raw event channel. And this is the channel that was passed to the perv buffer setup call. And that's one of the nice things in the LibBPF go library. It provides a go abstraction to interact with the low level primitives. And the data that you get, you need to decode it. So the program itself, so the other side, this is the map that we use for internal storage. And I'll get to it later. And that's the events perv buffer that we use to pass data to the user space. So the trace points we have is the sys enter open trace point and the sys exit open trace point and sys enter open at trace point and sys exit open at trace point. Pretty much we hook the entry and the exit to the file operations. So we can get the status codes on the way out and get more information. And this is what the map is used for. It's used for temporary storage. So when we see the entry trace point, we save the data into a structure. For example, here we create this info infrastructure. We save the process ID, the thread ID flags, the file name. And then we save it in the map. And on the way out, we look up that record and then we add stuff to it. For example, we add the user ID. And then we also get the process name. And then you can also get the actual file name and all of that. But that's meant to be pretty straightforward. And the C version is pretty similar. So there's also a temporary map. There is a ring buffer. And that's the nicer version of the Perf buffer. And the coded side looks almost identical. The user side of it looks different. And here's the call that I like the most. So you open the BPF program, the BPF module, then you load the module and everything in it, the programs and the maps and all of that. Then there's a single call to attach everything. And there, internally, LibBPF uses these section definitions internally to do the auto-attaching magic. I know we're out of time, so.