 Good morning. So my name is Michael. I work for Keenfolk. We are a small company from Berlin. We do mostly Linux software. And among many other things, it's EBBF today. So the talk is about GoPBF. It's a fairly new library to work with EBBF programs from Go. And it aims to help you to access the EBBF subsystem from your Go software. The talk is basically two parts. It's the first part on EBBF in general. So what is it? How does it work? The basic principles behind it. And in the second part, we are going to look into GoPBF, which is also split into two parts, depending on what your use case is. And just see how it works in general. So EBBF is a so-called bytecode virtual machine in the Linux kernel. Today it's used for tracing, networking, so software-defined networking, for performance analysis, for security purposes, and so on. The classic BPVF, or the Berkeley packet filter, is much older, so the functionality to use it for packet filtering has been in the Linux kernel for a long time. And it's used by, for example, TCP dump in 1992. There was no unified way to filter network traffic. And the BSD packet filter paper brought such a system, which then was implemented for the Linux kernel as well. And today it's called Enhanced BPF, because it has grown a lot more functionality over the usual packet filtering use case. You can attach to kernel events. You can modify network packets, or to drop them, or to route them. You can use it to do syscall filtering and a lot more stuff. A nice property is the safety guarantee you get through the BPF verifier. So when you load your BPF program into the Linux kernel, it first gets verified, because the kernel needs to make sure that your program isn't malicious. It doesn't block the kernel forever, and so on. So after your tool gets through the verifier, you can be sure that there are no loops in there, that there is no invalid memory access, and so on. And of course, you are running in the kernel, so it's very fast. And it's also cheat-compiled on some architectures. So yeah, fast is a requirement to do, for example, networking. So it's good we have that. So from a high-level perspective, it looks like this. In Gizaspace, you have your program, and from there, you use the BPF syscall, or some library, which makes your life easier to interact with the Linux kernel and load your program or your programs. You can have many of them. Those live in kernel space, of course. And then you have eBPF maps to share data between the eBPF programs and to share data between the eBPF program in kernel space and your program running in user space. So first, a couple of details on eBPF programs. They can be attached to Sockets. That is, again, the traditional use case. So if you want to execute some eBPF program whenever a packet arrives, you can use that feature. You can attach to kernel trace points. The Linux kernel already does define a lot of trace points, so you often, you can just look into what you want to trace. You can check out the kernel source tree, see where the trace points are, what is available there, and then maybe just attach to a trace point. If that doesn't work for you, K-probes maybe do. You can use K-probes to inject a program as a handler at a certain address. So for example, if you want to trace a syscall, a Linux syscall, you could use a K-probe which then always triggers when the syscall is executed. And there are U-probes that's basically the same as K-probes. The only difference is that those for user space function calls, so if you want to trace something there, you would use U-probes. Then we have maps. Maps are the data structures to share data between kernel and user space. When you create a map, you can say what data you want to put in there. So it's up to you if you want to have an integer value in there and you only need to store one. You can create a map with size one and define that key and value is of type integer, for example. And there's one special map. It's called proc array. And this is sort of a lookup table in the eBPF system which contains entries to other BPF functions. So if you want to replace your currently running eBPF program with another one, you could look up one there and call it. And you should be aware that the main page for BPF is rather out of date, so it can be pretty hard to get information on the available programs or map types and you should look into the source code. So I think right now maybe four map types are in the main page and in Linux 4.10 are eight or 10 map types. So it's not super up to date. Yeah, that's the list of existing maps. Right now we have a hash which actually works like you would expect from a hash or a map key value. We have an array and then we have different types for special use cases. I will show you one in detail and that's this type perf event array. So it's an array of file descriptors with perf event data and the nice thing about it is that the kernel can just write into a ring buffer and doesn't have to wait for you in user space and the user space program can then map the memory of the ring buffer and read from it asynchronously. And that's also a type of map that you often would use when you use K probes to trace something and we will see that in one example later. So to give you an idea how BPF looks like when you really write BPF assembly or pseudo assembly I will show you a short code snippet but you don't have, usually have to do that. So that's a very small program you could load into the kernel as is to count F join at sys calls. So whenever the sys call gets called it increments the counter by one and exits and yeah, as you can see it looks rather tricky and it's not very approachable. So fortunately we don't have to do that but we can use tooling. Around enhanced BPF and modern tracing tools on Linux there is the IO Visor project, you can find it on GitHub and also on IO Visor.org. So if you are into that and want to learn more about the available tools, find documentation and so on that's a good place to start. So now on go BPF, it's as I said before a library to create, load and use BPF programs and it's actually two sub packages. It's one which allows you to use Cego and the BPF compiler collection and a second package which allows you to use pre-built alpha object files and we will have a quick look on both parts. By go BPF, when we started working in that area there was no library yet which did what we wanted to do but we found a Hoover project from IO Visor which wants to be, they call it data plane so I think the idea is that you have a demon on your system and then you ask the demon via an IPI for system data and the program then loads for example some EVPF program to provide you with the necessary data. And second reason is we work on Weave Scope which is a tool to do cluster monitoring so for example Kubernetes and this is written in go and it also needs to gather a lot of data from your system. So for example, processes, containers, network connections and right now their approach is to parse proc and check the contract tables and this is rather difficult to do because you have to scan a lot of data and that vary fast and when you have a lot of processes or a lot of containers it can be tricky to do that reliable and fast and depending on the use case it's much nicer to have EVPF there because the kernel is doing a lot of work and you just have to read the events then. So when you use the BCC package from GoPPF you can write your program in C. It's a modified C language for BPF back end so it feels like C you have to be careful about a couple of constraints because in the end the BCC will build an eBBF tool out of it and it has to pass the kernel verifier but it makes things a lot easier. You have a lot of helper functions interacting with a map is much more comfortable than the assembly code we saw before and so on. LibBCC actually is not only the library which helps you to write your program but it's also a compiler so it works like you pass it your program as source code either you read it from a file or you just have it in your Go code in some constant for example and then it translates your program into BPF bytecode so we can actually verify it or BCC verifies it for you before it gets passed to the kernel and BCC also provides you with a lot of headers before you often had to import kernel headers to have all the necessary definitions for BPF and all that is in BCC so it's really a nice library and you should check it out if you want. So I will show you a bit of source code. This is a snippet from the example tool in the Go BPF repository and as you can see here we just define a source variable and put our C code in there, two important things here. One is we define a struct of type JoanEvent and then we define an array and it's of type PerfEvent as we saw before where we are going to write those events so that's the map which we then can read from user space later and the Go part looks like this. Here we load the source code then we get a reference on the table then we get the reference as a PerfMap and we pass it a channel and then we can read from the channel the events so we have a loop we just wait for data and whenever there is data we print it and in Go BPF BCC it looks like this we open a Perf buffer and we give it a callback cookie and then we have a gateway function which takes the cookie, looks up the handler data and sends back the event data so I will quickly show that to you so here we have the example program and since you need root permissions for the BPF Sys call I am root here and when I now do some action here you can see we get an event and we get correct reports on what was happening and also the return value so I was not allowed to do the second operation permission denied and here is the return value not zero, yeah so the second package in Go BPF is elf so if you don't want to translate your program before you actually use it or if you cannot ship the BCC, lip BCC on all your systems because maybe it's too large and your containers should be small and tiny then you could think about using pre-compiled object files. This feature relies on elf sections so in Linux elf files you have sections and by defining by setting particular section names so here for example K-REC broad for return probe we allow Go BPF to detect from the elf file how the contents should be loaded. If you are interested in that you could have a look on the TCP tracer BPF package from Weefworks it's as the name says a TCP tracer and we plan to use that in scope so it does exactly that it's a very small C program it gets compiled to an elf object file and then we then later load it with Go BPF to trace TCP connections. Yeah, you can create such object files with Clang I think the BPF backend has been in the static compiler for quite some while so you first produce bitcode object file and feed that into the compiler and then by selecting architecture BPF you generate the BPF bytecode for later use. Nice or tricky feature of Go BPF elf is that you can create your program for multiple kernel versions usually you have to tell the kernel when you load a program for which version it was written because the kernel will only accept the program which was written exactly for the version you are running and this can be tricky when you have patch release of the Linux kernel you maybe don't want to recompile your program so we built in this feature when you set it to this constant we will try to guess the current kernel version during runtime and then just say to the kernel yeah we built it for the version. You can do that, it can be tricky because sometimes kernel structures change functions change so depending on what you do in your BPF program it could break afterwards so you should know what you can expect from the kernel or what you rely on. And one last word, continuous integration is also tricky. You need root access, you need a modern kernel you often don't have that so if you ever need to test Linux kernel features have a look on stage one builder on the stage one builder project with here we used Rocket for custom we use Rocket with a custom stage one image which allows us to test any workloads on CI systems and time is up I believe so that's it. We have time for a question because Ron needs some time to build up anyone having a question? Yeah actually I have two questions first I didn't get the media behind using BPF in the context of tracing as it's firstly used in networking especially for filtering packets so what are the new features that give it this good reputation? Second in the context of tracing what about it's overhead compared to other tracing tools like data or system tab and how we can measure it? Thank you. So I think I will start with the second question so I think the main advantage here is that you now can have it in mainline kernel you don't need to have any extra modules it's supported by upstream regarding the first question so a lot of the functionality wasn't possible before so maybe you know Brandon Craig from Netflix who has been doing this for a long time and de-trace from Solaris always was quite powerful to trace systems and Linux for a long time didn't give you the same features to trace programs. Regarding networking one feature from EBPF is fast data path XDP it's used for example by the Sillian project so they do container security and networking with EBPF and this is something you couldn't do before. Any other questions? No, okay, thank you very much.