 Hello, everyone. I'm Anna. I work at Isovalent as a software engineer. I work there on observability-wide, where they're speaking. And we work on observability tools using EEPF directly and indirectly. EEPF is not new. It's technology that has been here for a while. It's used in production for a while. And it has got a lot of traction already in observability space and in wider cloud-native space. But it is a technology that most of us use indirectly only through user interfaces of various tools. So it still causes a lot of confusion, misunderstanding when I talk to people about EEPF. I see a lot of that. This question, what is EEPF? Pretty basic question is still asked pretty often. So let's start with quick introduction. What is EEPF and what I will talk about today? EEPF is a technology that, in short, makes the Linux kernel programmable. It is the commonly used comparison is that EEPF is to Linux kernel, what JavaScript is to the browser. So it allows you to inject programs into Linux kernel, run them without modifying the Linux kernel itself, without recompiling it, et cetera. Now how it looks like. Here we have a simple diagram. Normally, software engineers write code running in user space. Most applications will just run in user space, but practically all code that anybody writes, anything, any tool that does something, relies on kernel on operating system to do a ton of stuff. And by a ton of stuff, I mean really a lot. This is usually done via syscalls. So the application calls, system calls, to ask kernel to do things on its behalf. Like file operations, open, retry to file, network calls, various other management things, like running processes and resource management, things like that. This happens a lot. I think most engineers from what I see kind of realize, but not fully how much this has to happen. Like even simple operations like opening a file, usually mean like 100 system calls or something like that. And with eBPF, we can inject something into the kernel space, inject some code that will be automatically run on certain events. So when something happens in kernel, our eBPF program will run. eBPF programs are written usually in C, recently also in Rust, but because Linux kernel is written in C, this traditionally, eBPF programs will be written in C. And they get attached to these events. What are these events? Let's talk about that. So eBPF programs can be attached to many different events. In observability context, there are a few that are most important, I would say. First, K probes and K red probes from kernel probes. These are basically function calls inside Linux kernel. So Linux kernel defines functions and they get called. Similarly for U probes and U red probes, these are functions in user space. So these are events that mean that some user space function got called. U red probe will be similarly similar event, but when the function returns, perf events. These are interesting ones. So Linux kernel has this built-in profiler, perf. It's been there forever, for a very long time, to use very commonly to gather system level data. And it is used pretty commonly these days by various user space tools to do something with this performance data that we have from perf tool. Another event, the common use in observability, are tracepoints and user space tracepoints, so-called use ditties. These are very similar to K probes and U probes. The difference is that tracepoints are statically defined, same like user space tracepoints. What means two things, basically? Means that they are more stable than K probes and U probes, because function names can change between different versions. Tracepoints are considered more stable, but also they have to be statically defined, so there's a limited number of them. There are many, many more attachment points for EVPF programs, like, for example, socket events used commonly in networking or in security space, but from the observability perspective, these are, I think, the most important ones. And this is when the EVPF programs are run. Now, to be actually useful, any computer program doing anything needs something to store its state. Like, user space applications typically will connect to a database where the whole state of the world is stored in the database. In EVPF world, the role of the database is served by BPF maps. BPF maps are basically key value stores stored in kernel memory. There are many different kinds, like really many different kinds of BPF maps. For example, hash, table, perfing, buffers, but many others. That's all are basically key value stores. And they can store some state of the world. They are used for communication between different components. And this sounds obvious when you realize that, but I think that it's important to use EVPF for observability. You can't really just run something in the kernel, and that's it. You need to communicate that to user somehow, right? This is what BPF maps are used for, for communication between kernel space and user space, so both if you have code running inside the kernel and user space application can use same maps and communicate with each other that way. And also, BPF maps are used for communication between different EVPF programs, for example, attached to completely different events so that we can observe not only that something happened in the kernel, but we can analyze in the BPF program what happened and gather this data from different events. All right, so that was a short introduction to EVPF. Last thing I wanted to show about it, EVPF verifier. So this whole concept of running things in Linux kernel is not new at all. There are kernel modules, which allow that for quite a while. What makes EVPF different is the verifier that verifies that the program is safe to run. By safe to run, we usually mean that it's guaranteed to complete and it won't access memory that it shouldn't access and a few other things like that. But basically, that it won't crash the kernel. What is a risk with kernel modules? People are generally often pretty scared about just installing kernel modules to the Linux kernel because if kernel module contains the back, then not just the program crashes, the whole machine crashes the crashes. So that would be very bad. With EVPF, this is not really a problem because not really a worry because all the code is verified by a verifier and then only then run. All right. So why EVPF is useful for observability and especially in cloud native space? Well, here at KubeCon, we run things in Kubernetes, right? I run a lot of stuff in Kubernetes for quite a while. And well, in Kubernetes, we have many applications running on same note. They are scheduled by Kubernetes. As a developer, we don't really need to care which machine the application is running. It's just scheduled and then rescheduled by Kubernetes. This is great from resources perspective, from management perspective. But one thing that still remains the same is that although there are many applications running on any host and we don't need to care which applications are running where, still one node, one host, has always one Linux kernel. And this Linux kernel, oh, by the way, I'm always saying Linux kernel, this could be Windows too. The EVPF is supported in Windows to these days. But let's stick to this assumption that everything is running on Linux kernels. So all these applications running on one node reach out to Linux kernel to do things, to make network calls, to open files, to run processes, everything. And that means that if we put EVPF programs in this Linux kernel, they see everything that happens to all these applications. While we don't need to do any code changes in applications themselves, we need to have some code in user space that will load these EVPF programs, but no code changes to applications themselves, while EVPF programs can see practically everything. One thing that makes it so promising in observability space is that there is, well, one thing is that there are no code changes. But second thing is that there is very low performance overhead in all of that. EVPF programs run in kernel space. And of course, there is this kernel user communication happening all the time. But we don't need to run everything in user space. We don't need to do that many calls to kernel if we do some of our observability logic in the kernel space. So this is why people in observability space often get so excited about EVPF. And why it's so, so promising? Because, well, observability, instrumenting for observability always adds this overhead that nobody likes. OK. Let's move on to look at some of EVPF observability tools. So as I mentioned at the beginning, most people won't write EVPF code themselves. It's C code. It's not language that many people really feel super comfortable with. It's code running in the kernel. To load it, you need high privileges. These days, there are many, many tools that use EVPF for security, for networking, and also for observability. So let's take a look at some of these tools available today. First thing I wanted to show is this. And this is a diagram that I think appears in probably like 80% of talks about EVPF. Maybe I exaggerated. OK, maybe half. But a diagram that those of you who are tracking this space probably saw a lot, the author of the diagram is Brendan Gregg. And it shows various tools from so-called BCC, BPF Compiler Collection, a set of tools that's for monitoring many, many different things in Linux kernel. So we have networking tools. We have monitoring of user space applications, resources like memory and storage, a lot of them. Now, all of these tools are available in the BCC repository. This screenshot is a screenshot from the GitHub page of this repo. It's just a very small subset of the tools because the list is very, very long. By the way, if somebody wants to learn EVPF or try out, I can highly recommend to take a look at this code, at these tools, because each of these tools is just two or three files. They are pretty small, pretty self-contained, and they cover a lot of many various things, like networking, resources, everything. So yeah, great examples for somebody who wants to learn EVPF. Now, there is this EVPF exporter tool from Cloudflare, which, how do you describe it, is BCC tools as Prometheus metrics. And that summarizes it pretty well. This is a tool that can expose everything monitored by BCC tools as Prometheus metrics. How it looks like, you have to provide a configuration file. In the configuration file, you define what metrics you want and what labels you want on this metrics. This is pretty much necessary with observability tools using EVPF, because the thing about EVPF is that it produces completely ridiculous amount of data. And we need to define this pretty granularly what we actually want from it. And here on the side, you can see the EVPF code that is generated from this config file. So it basically creates a trace point, attaches to a trace point that will count timers fired. All right, our next thing I want to talk about is something that we work. The main project of Isoval and the company work at Silium. Silium uses EVPF for networking. It's a Kubernetes CNA plug-in, a very rich CNA plug-in. It also has service mesh and other mesh that features a lot of security enforcement. But this is not a talk about security and about networking. Silium also has an observability layer. And this is using EVPF for observability sort of indirectly. Because, well, Silium is using EVPF for networking. So instead of doing networking traditionally with IP tables, we inject EVPF programs that handle network flows in OX kernel. But if you have this EVPF programs that do that, then you can as well just ask these EVPF programs to provide you information about what network flows are flowing in the system and observe that. So this is what Hubble does. Hubble is observability layer for Silium. Hubble provides CLI that allows you to query all network flows with many, many, many different filters. And also, if you prefer an aggregated view, and you don't need that much granularity, then Hubble exposes prometheus metrics that can be ingested and can be queried from Grafana. For example, here's Grafana screenshots where you'll use query network traffic from various different network layers with regular prometheus metrics. Another thing, a kind of project under the Silium umbrella is Tetragon. Tetragon is security tool. It is a tool that mostly security people would be interested in, people who need to do audits, for example, or want to detect attacks. Tetragon uses K-Propes and TracePoints to monitor events that happen in the kernel. To use it, you define tracing policy. Here you can see an example of a tracing policy. And what we can see here, we see that the tracing policy defines K-Propes, that the program will attach to. The K-Propes is FDET installed. This is Linux kernel function that basically will be called on file operations, roughly speaking. And the tracing policy also defines what output you want to get. In that case, we want to get file operations that happen inside the ETC folder. And if we inject that to the Linux kernel, to the program generated with this config, then we can get this output from Tetragon. So we get very granular, like very granular events that are happening inside the Linux kernel. In this case, we see that I think it basically somebody writes a password, changes a password. As we can see, it's not just one event, but from a Linux kernel perspective, this is like there is some process started, then file is opened, then write, then close, then exit. So there are a few operations out there happening around it. All right. Moving on from security space, one thing that is pretty hot in observability space these days is continuous profiling. Earlier today, Frederick had talked about file formats. And here, we can see a screenshot of the icicle graph from Parker. There are a few of these continuous providers using EVPF these days. Parker, Pixie, Pyroscope, they work in pretty different ways. But I looked at the code of these tools. And well, this is the code. This is the EVPF code from Parker. It is heavily truncated to fit it on slide. But at the bottom, you can see inside there there are links to the full file. And this is some code, I guess, barely readable for you. So let's put some arrows on it. What we can see in this code is it defines the ProfileCPU function. This ProfileCPU function takes as an argument some struct. This struct is defined in Linux kernel headers. And this will be struct that is passed to our event. Then there is some reading from BPF map and writing to a BPF map and helper function that basically is called to gather all this data to form a profile. Now, this is some BPF code. To actually use it, it needs to be loaded into the Linux kernel from user space. And this is code that does it. The parka user space goal code. Again, probably very truncated and probably barely readable. But what is important here is that parka uses this lib BPF Go library from Aqua Security, the Go library for operations on BF programs. And we can see that this code basically defines the same ProfileCPU function. And then it attaches it to Perf event. So previously, when I was going through various attachment points, mentioned this Perf profiler in Linux kernel. This is what is happening here, basically. We are attaching to Perf event events from the Perf profiler. And this function, this BPF function from Preparous site is run continuously to gather the full profile. Another thing that is pretty hot in the observability space these days is distributed tracing. Here's a distributed trace, a Yeager screenshot. So we can see, well, this is something that everybody wants. But it's not easy to get because for distributed tracing, you basically need two things, right? Like to export traces, this can be done somewhat automatically, but also you need to propagate context. This is not easy. To propagate context, this is always language specific. For some languages, there are tools that do that automatically. But for other languages, this is really not easy. But there is a project that is very early, a very early stage. Under OpenTelemetry umbrella, there is Go Auto Instrumentation project that uses EVPF to do exactly that, to propagate trace context. So what we can see here is EVPF code. Again, let's put some arrows. We can see that it defines Uprobes for Go GRPC functions. And then it writes trace headers to a BPF map. And again, this is code that does automatic trace context propagation with EVPF. But it needs to be loaded into Linux kernel somehow. So here is the user space go code doing that. It uses different user space library, Selium EVPF library to define this EVPF program in user space. Here you can see that it uses EVPF annotations. And we are attaching the Uprobes, the EVPF Uprobes to the EVPF programs to Uprobes, which are Go GRPC functions. OK, so can we EVPF everything? One problem, do you remember this picture? One problem with EVPF is that, well, when user does an action, then kernel sees it. But kernel really sees it in a very, very different way. Kernel sees file operations, network connections, a lot of that kind of stuff. So yeah, observability with EVPF is challenging mostly because EVPF doesn't have business context at all. It's not exactly technically impossible to give this business context, but it's not practical. Like we can't copy your whole database to a BPF map. It's not really practical at all. EVPF observability is still young. Like this is distributed tracing of the instrumentation is very promising, but very, very young, to be honest, but very promising. And the data volume can be, well, we had a few talks about scale of observability today. It's always large, but with EVPF, it can be not only large. It can be really absurd. So EVPF tools, observability tools using EVPF generally have to be heavily configured to gather only data that you need. So here's a summary of what's possible today with EVPF. Systematrix is no-call changes, network and security visibility for sure with Serium and Tetragon and Hubble. Continuous profiling tool. There are a few of these profilers these days. And I really hope that distributed tracing auto instrumentation will mature in the next years because this is a really promising space. And there is something already for that. Thank you very much. To learn more, feel free to visit EVPF.io. There is also a book written by my colleague Liz Rice, Learning EVPF. You can download it from Asevalent website. But there is also a book signing tomorrow if you want to grab a paper coffee with an autograph. And that's it from me. Thank you very much.