 OK. So, welcome to my talk. Thank you for attending. So this talk would be about my experience in using eBPF on resource constrained embedded devices and have an agenda. So at first we will talk a little about me, the company I work for and what we do. And then we will go into my experience with eBPF coupled with embedded devices. We will see different projects and different approaches which try to use eBPF on embedded. And then we will see what lies for the future of eBPF on embedded in my opinion. OK. So I work for Colabora. We are an open source focused consultancy. We help other companies like use and leverage open source without getting too much in trouble with it. And by that, I mean not ending up in a situation, for example, where they have like a kernel branch with tens or hundreds of thousands of downstream patches which they can't upgrade. We had a keynote today about how important it is to upgrade your kernels. Don't want to put yourself into a corner where you can't upgrade your product. OK. And it's a real pleasure to work with companies who actually get it and know how to maintain the projects and how to actually leverage the open source in insider organizations. So I am an embedded systems developer. I really like projects like Open Embedded and Yocto. And I'm always looking for new technologies which can help us develop easier and debug and even analyze the systems in production. This is how I ended up researching EBPF for embedded devices. So I was learning about new technologies which I can apply in this domain. I'm not an expert EBPF user. So I consider myself just a user. And yeah, I like open devices, devices which you can actually run your custom code on. So I do understand there are certain areas where you need lockdown devices, for example, in safety critical applications. So today's embedded systems are very different from the ones like 15 years ago. So we don't have a simple microcontroller which drives an LCD display. Today's devices are smart. They are all connected. So they are much more powerful. Today we have in simple products like, for example, a bicycle. You can ask how many processor cores do you have inside your bicycle. And this question actually makes sense. If you ask this question 20 years ago, people would have laughed at you. And besides the hardware complexity in embedded devices, the software complexity is rising a lot. So there are products which the embedded interface is programmed in JavaScript. And most of the embedded logic is programmed in JavaScript. So you have the embedded device which brings up something like the Chromium embedded framework. And it's basically a web application running on the device. Yes. And this complexity, both software and hardware, brings with it obvious security concerns. So even though our devices are much more powerful and capable, they are still very hard to develop or even harder due to the added complexity. And why is this? So my answer to this question, I mean, I don't know any embedded developer from today who can complain that developing an embedded system, especially an IoT connected smart embedded system, is very easy. So, yeah, it's very hard because we have the increased complexity coupled with the resource constraints. So we tried to make these embedded devices as cheap as possible. We tried to fit Linux distributions, network stacks in very low memories. I mean, you have lots of trade-offs which are made in the embedded space. And you can add your own pet peeve here. I mean, there are lots of reasons. These are just some of the reasons which went through my mind. There's a lot of problems with licensing. There are a lot of products who don't allow GPL v3 software on them. And, yeah. So I actually like Busybox. So Busybox is a really cool project. It gave me a lot of headaches over the years. But overall, yes, I don't have anything against Busybox. So, yeah, these are all resource constraints and trade-offs which the embedded system developers make. And, of course, we, as engineers, come up with solutions against these constraints. So we have remote debugging. We have, we're booting grute fesses over the network. We do lots of random hacks for embedded systems. And we have, in general, a pretty big toolset for embedded systems development. And this is where I think EBPF also fits in. So it's one of the tools. So EBPF is not like a silver bullet which can replace, like, all the tools you're using. It's just one of the things which you can use in addition to all the other stuff which you're already using in embedded development. Yeah, so it won't knock down those jars. It will be besides them. That's the logo of the VCC project. Okay, and this sounds like a solution which is in search of a problem. And it kind of is. So EBPF was not developed for embedded devices. And we are basically trying to put a square piece into a triangle hole. So, yeah, it was developed for cloud. Applications and for analyzing big iron servers, not small embedded devices. But we have seen, we have precedence over the years where technologies were developed for the server space which ended up being used in the embedded space like the multiprocessing code in the Linux kernel. Now it's used on small embedded devices. So the same thing, even though the EBPF virtual machine into the Linux kernel, it was not added for embedded systems in mind, it could be used there. So we will talk a little bit about EBPF and how it works. I can't give an in-depth introduction to EBPF due to time constraints, but I wrote a block series, a five-part block series which is targeted toward embedded developers which take you from zero and bring you up using EBPF projects. And it's on the Collabora blog. I have links to that at the end of my presentation. So we will be analyzing a little bit the EBPF VM which is implemented inside the Linux kernel and the bytecode and how user space uses it. So EBPF is a virtual machine which runs inside the Linux kernel. So its purpose is to run code which can analyze the system while running safely without hanging your kernel, without crashing it, without negatively affecting your performance. And that's a pretty tall order. I mean, you can crash your kernel if you insert a normal kernel module written in C but by inserting code which is EBPF, you have a very good safety guarantee that nothing bad will happen and you can actually introspect your system in production without affecting it. This is what cloud companies are doing. They are running their big servers for streaming services and analyzing them at the same time while they're serving customer requests without actually affecting them. So we can do this also on embedded devices but not in all use cases. So the VM pokes both the kernel space and the user space. So you can look at applications and inside the kernel what's happening. So how this works? We have the general workflow here. So user space has access to a system call in the Linux kernel and it calls that system call to load BPF code which gets compiled to native machine code and that native machine code gets attached to a code section to a running process. And because it's natively compiled, it's just in time compiled by the virtual machine, it's inserted at a very specific place and it doesn't really affect negatively the performance in that place where it's inserted. When the process normally runs, the code which is hooked into the process produces data and makes that data available to the user space application, generally the one which inserted it via data structures which are shared with user space like maps or usually maps. But you can also write, for example, to the trace ring buffer and yes. So user space collects data from the code it hooks inside the kernel and that code can be hooked both to applications or to kernel threads, yes. So we have here an example. It's a very simple example. So user process calls the BPF load system call. There's a verifier inside the kernel. This verifier is very important because it verifies the correctness of the BPF program which it loads. So in order to not crash the kernel and to be safe, eBPF programs are very restricted. They cannot have like more than, there was a limit on the instruction count which you could have in an eBPF program. It used to be 4,096 instructions, maximum, but now it's like 2 million instructions because the VM is more efficient. And you cannot have like unbounded loops because you don't want to have the halting problem. So even though you have these restrictions on the eBPF code which you load in the kernel, the eBPF code, even though it's so small, it can be very useful because you just want to fetch some data or analyze some data and small programs can do quite a lot, yes. And you will have the just-in-time compiler which compiles code natively to the processor architecture which runs the kernel and it gets, we attach it to the handler for the open system call. And when the other processes try to call sysopen, the process which added its hook into the system call, they can see, hey, what are these other processes doing, like what files are they reading, what data are they reading and stuff like that. So this is how the byte code for eBPF looks like. It's very similar to machine code. It's a virtual machine. And this is loaded by user space. This is created in user space and executed in kernel space. And this, some applications, they actually store it directly in the program. So they precompile the eBPF or they write it byte by byte if it's a very simple program and they just load it in the kernel. But that's very hard because there's been a trend in eBPF programming. The complexity of the programs increased and it's very hard to write programs that way. So in user space, the clang compiler has a backend which can compile a subset of the C language. It's a restricted C called the language which can be compiled to eBPF. And that can be loaded in the kernel and you don't have to write an assembly like language. But it's still very hard to interact with that program. Okay, you compile the eBPF program with clang, you load it in the kernel, but what do you do then? Then you need to call other system calls to read the data, then you need to do a lot of manual management. So the BCC project was created to is this interaction. So this allows you to write eBPF programs and interact with them from a Python user space. The BCC project automates as much of the process of interacting with the eBPF program as possible. And you have Python, Luango bindings. This is an example. This is what I presented before in the diagram. So we have as a Python string the source which gets compiled to eBPF. This is a very simple function, which in Python we tell BCC to attach to the dosis open function in the kernel. And we're using the kernel ski prop back end for this. And this code snippet of restricted sea is very simple. We just read the file name and we print the file name into a trace ring buffer which is accessible from user space. Now the structure, we have one parameter. That's the context structure that contains the registers of the function called when it happens in the kernel. So if you access the elements of the context structure, you get the register values which are the function parameters which were passed to the open system call. And we read the first parameter which is a pointer to the file name. And this is basically what we did in the diagram. When you run this program until you kill it, it will print the file names of the other... Of the files, it will print the file names accessed by the other processes. So it's very simple. Now, yes, this is what I explained. So we have the two parts. BCC automatically calls clang for you. So you don't have to think about, okay, I need to call clang, I need to call the sys bpf to load the bpf, all of that is done automatically. And there are quite a lot of tools. So I just posted a trivial example before, but the BCC project comes with BCC tools subset which are already production ready tools which you can use and there are quite a lot of them. And they are quite powerful. So there's even a book on the subject which I preordered it, but I don't know if it's available. And yeah, it's a huge ecosystem. So it's very useful. Now, how do we leverage this on embedded devices? There are quite some general problems because the ecosystem is very new, eBPF, the technology itself was added to the Linux kernel quite recently in terms of history and it's still heavily developed. It evolves a lot. There are multiple approaches. We will look at project advantages, disadvantages, and yeah, there's no silver bullet, so there's a huge space for innovation. Okay, so the first problem is portability. So the eBPF VM is 64 bit. If you might have noticed from the bytecode which I posted, so because the VM is always the same, it uses 64 bit registers on all architectures and I obviously wanted to use it on 32 bit systems. Yes, and I had problems accessing stuff outside the virtual machine because even though you're running in a VM in 64 bit, the entire user space are like 32 bits. So the VM can do 30 bits of register addressing. If you feel just half of the register, you can like the reference pointers to 32 bit addresses, but you would have to do that like with custom pointer arithmetic and you don't want to do that in restricted scene. And yeah, so this is an area which is like heavily developed right now. There's a project called Core. We will talk about that next, which basically means compile ones run everywhere, which tries to make eBPF portable. And it does this by adding a type system to eBPF. So in general when compiling eBPF, you also add type information from the kernel headers inside the eBPF program. So you know, for example, what are the offsets in the structures which you are accessing. So you know exactly at what data are you looking at. Yes, so this is Core. This is a work in progress. So an important chunk of it was merged for kernel 5.4. The eBPF maintainer posted a patch series for kernel 5.5, I think, and he titled it revolutionize eBPF tracing. I mean, yeah. So there are some pretty big changes happening right now. So the idea is to stop using clang on the fly. So right now BCC, when you run a program like the Python program, which I exposed before, it calls clang at runtime and it compiles that code on the fly and loads it in on the kernel. This means you need clang on your device. And BCC links to lib clang, which is pretty big and it consumes quite a lot of RAM. So, yeah, by having this project which makes eBPF portable, we get quite a lot of benefits including not having to ship clang every time you want to run an eBPF program because you have all the type information in there. Yes. You still require kernel headers to build the eBPF program. There's no way around it because the type information is in the headers which describes the structures. But starting with kernel 5.2, a change was made to include kernel headers inside the kernel as an archive. So when the eBPF program gets compiled, BCC can just access the kernel headers and, yeah. But this is kind of unrelated to core, yeah. Okay, another problem is standardization. So we kept talking a lot about clang here, but GCC also added eBPF backend and it edited very recently, like weeks ago or a month ago. And it's still in, like, it's very new. And the majority of tools and projects like BCC and BCC tools are written for clang and there's no, like, standardized elf format right now. So basically the standard is whatever a clang produces and whatever the kernel accepts. And GCC tries to, like, emulate that. So having, like, more than one compiler is a good idea. Yes, but having actually portable code is, like, more important than actually trying to standardize stuff. Another problem is that you need to be root to call the eBPF system call. So in order to run any of the BCC tools, which are a lot of them, you need to root privileges. And that's a problem, especially if you're having a security model which is based on separate users. And, yeah, there was a really good article which I linked here on LWN and it explains the problem in a lot of detail. So, yes. Usually you need, like, other security mechanisms besides user separation in Linux. That's the conclusion. And another problem, which surfaced quite recently, so I used the preempt RT patch and I used the eBPF and I used them independently. I never tried to use eBPF on preempt RT systems, but real time systems are quite important for embedded. And it turns out that, yes, eBPF is not very friendly with RT systems, so it disables preemption which is a big no-no in RT systems and it can induce unwanted delays. And these delays depend on the size of the eBPF program. So if you have a very big eBPF program, like two million instructions, it will add a bigger latency spike. So, right now, what a patch series was proposed is to disable in the configuration. When eBPF is compiled, you can't have, like, the preempt RT full and vice versa. There are solutions for this, so it all boils down to locking and in the future this will be solved. It's not an impossible task, but it's something which doesn't work right now. So if you're having, like, a real time system, you probably don't want to run eBPF in production. Okay, so we're gonna talk about multiple approaches and multiple projects which run eBPF. So the simplest thing is to, like, just have your C program or whatever language and have it called a system call and load the precompiled eBPF program. This is the lightest weight, so you don't need a lot of flash memory, you don't, like, need a lot of complexity for very small applications, but if you have, like, a big application which tries to do a lot of stuff with eBPF, it can get hard to maintain. And, yeah, we have examples of very simple use cases like this in the eBPF kernel source. Another approach is to use BCC, but BCC, as I mentioned, is very big. It adds 300 megabytes to your root FS of dependencies and that's not acceptable. We have customers who tell us, hey, our entire system needs to fit into eight megabytes. You can do, like, a full BCC installed. But other, like, embedded products, like Android, they're okay with this hit because they have a lot of space. And there's a project, Androidab, which gives you BCC and the full power of, like, eBPF on Android. It's very useful for development, and, yeah, they are okay to take this hit. For the advantage of having all the BCC tools available. Another project, which I tried to upstream in BCC, but failed and got kind of abandoned, but people tried to keep it alive, is BPFD. So this adds a demon to the embedded device, which is a small wrapper around the system call. So what this does is serializes the system calls over the network to talk with the host BCC application. So you can use, like, a medium, like SSH, telnet, or whatever. Android users were using Adeb, the Android debug bridge, to communicate between a BCC, full installation of BCC, which ran on the host, which had Clang and Python. Python is another dependency, so you need to ship Python on your embedded device if you want full BCC. But by using BPFD, you had just a small binary. The problem is latency with this. So you add the network latency for the system calls, and you had, like, a lot of complexity, and you had to translate, basically, every operation from BCC to the BPFD wrapper, which runs on the embedded device, and every change you did in the BCC internals, you had to mirror that change on the embedded wrapper. There was, like, no standardized communication protocol, and you need versioning, and, no, that didn't fly with upstream, and they just said, no. Yes. So another really cool project is Ply. This compiles a custom embedded domain-specific language to eBPF. So we have an example here. This attaches a K-probe, which prints the stack for an I2C transfer, and it's this simple. So you just call Ply with this string, which is the domain language, and you can get all the information from the system. So this is a project which gets compiled on a host and gets put on the device, and everything happens on the device, which is very interesting. BCC has a project which inspired this, called BPF filter, and that has a very similar approach. It uses a domain-specific language to simplify development. The main advantage of this is that it's very high level. The domain-specific language is higher level than trying to write your own restricted C code to compile it to eBPF or your Python user space interaction. But making it higher level is also more restricted. You don't have the full freedom which you get with BCC, but still it's a very cool project. It's still under heavy development. It's a one-man project, so there's a bus factor there, but still it's very cool. It was useful. Another project, so there's a Go BPF project, which provides bindings for BCC, and this leverages the advantage of Go of creating static binaries. And you can bypass a lot of the dependencies on a host. And this can be used to replace the Python dependency. So you still need LLVM to compile the BPF program, but you don't need the entire Python installation on your embedded device. You can just compile Go binary and ship that on the embedded device. Yes. And this is really cool. They are re-implementing tools from BCC tools. Not all BCC tools have equivalent in Go, but slowly they're getting there. It's a really cool project. And I linked there to a re-implementation of execs, which like traces the execs in a system. OK. So, ways forward. Core needs to be successful, and this is like the main focus of the EBPF community right now to make BCC lighter. So the idea is to have BCC tools on all the systems and have portable EBPF. Like you can compile it and run it on multiple systems. You don't have to compile it each time for every system. Yes. And projects could continue the pre-compiled approach, which if you have like a simple network filter which you want to apply, then you can pre-compile the EBPF and just ship it in your own source code. GoBPF can eliminate the Python dependencies. PPFD is kind of dead. And ply. Yes. It will continue and hopefully mature as a project and become really useful. And yes, we can use EBPF on embedded devices today. Even though there are problems and there are limitations, but it's quite useful. And yes, there's quite a lot of work remaining. Like a standard library for EBPF programs would be very nice. So right now each EBPF program writes its restricted C from scratch. Usually the restricted C part, which gets run in the kernel on the virtual machine, is not very big. So it's not a big problem. But it would be really cool to have a library of functions available, which can be shared between different projects or tools. Yes. And this is all like maybe we will get there someday. I mean I'm sure we will get there, but it will still take some time. But even though it's a work in progress, it's still very useful. And yes, having the ability to tap into the EBPF powers on embedded devices, that's really cool. Yes. So I have added some links here. I have to say that Brandon's blog. So Brandon, he wrote a lot of tools. He wrote even the book on EBPF. And it's a very, very good resource if you want to learn about EBPF in general and how it works and how to use it. And here's a link to the blog post I wrote. So you can start with a blog post, which takes you from zero and gives you a pretty good understanding of how EBPF works. And then you can continue either with the book or with Brandon's articles if you want to deepen your knowledge of EBPF. Yes. Thank you. And this was it.