 Thank you very much. And thank you everyone for coming to this session. Hopefully this is going to be quite interactive. I've got some like tutorial labs that are running online. You can do them on your laptop. You can probably even do them on your phone, I'm told. So hopefully you'll be able to actually get hands on this afternoon, which will be excellent. I work for a company called Isovalent, which is the company that originally created the Cilium project. Does anybody here a Cilium user or familiar with Cilium? Yeah, a few of you. So, Cilium is a networking project that uses EBPF very heavily, and we have a ton of people that I'm really fortunate to work with who are extremely experts in EBPF and actually develop it within the kernel. In fact, some of my colleagues are somewhere upstairs having a kind of private discussion about the next things that are coming in EBPF. So, I'm in this really great position where I get to learn a lot about EBPF from my colleagues, and I hope to share some of that with you today. I wrote a book, this book. I've got a few copies of it here. So, people who ask the best questions, basically, I'm going to give you copies. So, that's your motivation to ask really good questions, all right? But you can also download it for the price of your contact details from isovalent.com slash learning EBPF. The interactive labs that I just mentioned, they're all basically taken from the examples from this book. So, you'll be able to kind of use that lab. If you want to, after this talk, you'll be able to sort of go away, use those labs, build on them, and do a ton more with the examples from the book. We're not going to cover everything in the book in this afternoon's session in an hour and a half. I think that would be a bit ambitious, but hopefully it will be enough to give you a sense of what EBPF is to get a start on EBPF programming and to get a feel for why it's so powerful. Now, those interactive labs take a couple of minutes to warm up. So, if you want to get your phone out or your laptop, go to iso go.to slash EBPF dash Vancouver and just kind of click on the kind of get started button. And as I say, it just takes a couple of minutes to spin up the instance. And while those instances are spinning up, we'll just have a little bit of a discussion about what EBPF is and why it's so exciting. The way I've tried to structure this afternoon's thing is I think half a dozen different interactive labs. I'm going to talk a bit, then give you a bit of a time to just sort of work through those labs and talk a bit more, you know, take questions as we go along and so on. I will put that link up again shortly. Hopefully, it's pretty easy to remember iso go.to slash EBPF dash Vancouver. All right. Is anybody having any trouble following that link or is it? Is it all? Somebody's giving me a thumbs up. OK, awesome. So, what is EBPF? It stands for Extended Berkeley Packet Filter. And you can pretty much put that out of your mind now because that acronym is pretty meaningless. It does so much more than packet filtering. Just don't even worry about it. So, what EBPF is, is an ability to run custom programmes within the kernel. We can change the behaviour of the kernel by writing EBPF programmes. I should quickly just get a show of hands. Has anybody here actually written any EBPF code before? OK, yeah, got a couple of experts I'll be calling on you for help when it all goes wrong. No, it won't go wrong. OK. Why is dynamically changing the kernel a new thing? Why is this interesting? So, the kernel is involved in everything that we do that's interesting on a computer. Every time we do anything that involves hardware, whether it's writing to a file or reading from the network, or allocating memory, all of these things require assistance from the kernel. And the kernel, it's under constant development. The kernel is constantly evolving. But if you want to change the kernel, first of all, you've got to convince the whole kernel community that your change is a good idea. And then you've got to make the change, which, you know, there's millions of lines of code in the kernel. It is not a simple thing to make a change to the kernel. You've got to convince everybody that your change is a good idea. And even if you get a patch accepted into the kernel, it will be literally years before people are using that change in production. Because people don't run the latest and greatest kernel, mostly in their production environments. They're usually running a kernel that's three, four, five years old. So if you want to change the behavior of the kernel, you've either got to wait for a really long time before that's something that you and your colleagues can run in production, or you can run a very bleeding edge kernel, or you can compile your own version of the kernel. But if you want to distribute that change to other people, that's pretty hard work and you're in it for the long haul. With EBPF, we can just write a program, load it into the program. We don't even need to reboot the machine, and we can change the way that the kernel is behaving. So that is fundamentally why this is a game-changing technology, because the kernel is involved in everything. So usually when we write application code, we're writing in user space. And our application is making system calls to ask the kernel for assistance. As I mentioned, anything to do with hardware, it's going to require the kernel to do something. Most application developers don't really think about system calls, that it's all abstracted away. You usually have some kind of open or something for opening a file or opening a socket or writing something to the screen. There's probably some kind of print command. But all of those things underneath at a lower level are making system calls. Within the kernel, we can attach EBPF programs to pretty much any event. We can actually attach to events in user space as well, but mostly I'm really interested in what we can do by attaching them to the kernel. Anything, literally any function call, any event, any perf event, any network packet, network packets arriving at various points in the stack, can be used as events that we can attach to an EBPF program. So whenever that event happens, and we'll see some examples of this very soon, it will run our EBPF program in the kernel. And yeah, there are lots of different types of event that we can attach our programs to. And that slightly influences what kind of things we can do with those programs. So K probes and U probes are essentially functions within kernel or user space. Trace points, there are trace points in the kernel. You can also attach to trace points in your application. You can attach to network packets, as I say, arriving at different points in the stack. You can attach to the Linux security modules. Anybody used things like APARMA or SE Linux? Anybody enjoyed using APARMA or SE Linux? OK. Those security modules use this Linux security module API, aren't they? And we can use that to trigger EBPF programs. Perfect events that we might use for measuring, well, performance really, all sorts of things. Anybody here using Kubernetes? Yeah, quite a lot of people. So I just want to quickly talk about how using EBPF is really important and influential in the world of Kubernetes. So in Kubernetes, we run our application code in containers, and those containers run inside pods. But however many pods we're running on a virtual machine or on a bare metal machine, there is only one kernel. The containers all share the same kernel. And they're all using that same kernel whenever they want to do anything interesting, whether it's networking or accessing files, or if Kubernetes wants to create a new container, that's going to request assistance from the kernel. If anything wants to change privileges, that requires assistance from the kernel. So the kernel is involved in everything that all of our application pods are doing. And that means if we can instrument the kernel, we can use that to see what's happening in all of our pods. We can use that to potentially change the way that the kernel is behaving in relation to those pods. Maybe we're going to prevent some things from happening for a security reason, or we're going to influence the way that network packets are delivered around the system. So I think that's enough talk and time for you to do some action here. Hello world in eBPF. So in that online tutorial, you should find some instructions on one side that will show you what it is you need to do, and some terminals that you can use to actually run those commands. Basically going to just give you a little bit of time to go through that hello world. Any questions at this point? Okay, I see a few people. Okay, the question is, is it a shared environment? So we're using a thing called instruct, which gives us all these different sandboxes. So you've all got individual sandboxes. So hopefully you will not be able to interfere with each other's instances, but it's quite nice. I mean they are using containers to provide this kind of little sandbox. And if we have questions, just wait until I give you the microphone so that people at home can hear it and for the recording, please, thank you. Yeah, we're using a platform called instruct, which is really great for selling up this kind of thing. All of the labs that we're going to use today, they're all going to have the same kind of file structure. The examples from the whole book are there in all of the labs. So if you want to, you can kind of go explore other examples if you feel like you have enough time. I see a question here. Hi, so do you have to be an expert in kernel events to write these? That is a really great question. Yeah, so everything that we'll do today is pretty basic. At some point, actually if you look at that hello world function, you get past a context parameter, and that context depends on what kind of event has triggered the program, what kind of program type it is. And at some point you might want to interact with that context. You might want to interact with the network packet, for example, and you need to know the structure of the network packet. Or if you were, let's say you were attaching to the Linux security module API, you kind of have to know what the consequence of allowing or failing that call is going to be. You can get to kernel data structures, so you might need to understand how those kernel data structures work. Yeah, it is true that at some point this is pretty much kernel programming. It's safer for reasons that will come to you later than writing your own kernel module. And you get some helper functions to kind of assist with a lot of this contextual information that you might use. But yeah, it is basically kernel programming, so it can get pretty detailed pretty quick. But what I hope you'll get out of even just doing hello world type exercises is a sense for why EBPF is really interesting. So even if you go away and you never write another EBPF program again, you'll have an understanding of why it's such a good platform for building things like observability and security tools and networking functionality. I see another question. Thank you. I have a question about the user area, because when we are writing programs in kernel level, there are some different performance speeds in kernel level and application layer. Is there any best practices or design patterns that help us to achieve some goals? Because I have some conflicts and some issues on managing getting data from the kernel because of the difference of the speeds. How can you help me with that? Yeah, so I think what we're talking about there is the kind of cost of context switching between kernel and user space. So every time you do that transition between user space and kernel or back again, it takes some time. EBPF gives us some maps which we're going to see in the next section which helps us pass information between kernel and user space. And it allows us to do stuff in the kernel, put information into a map and then deal with it in user space asynchronously. So we don't have to just keep doing this transition between kernel and user space. It's also the case that the more you can do inside the kernel, the higher performance your tool will probably be. So a lot of what we do in, for example, there's an EBPF security project that we have as part of Selim called Tetragon. And one of the reasons why it's super high performance is because we filter loads of events within the kernel. So minimising the amount of information we're sending to user space. Did that kind of answer that? The next thing with maps might help with that. Has everybody pretty much been able to do Hello World at this point? Yeah? I see a few thumbs up. Anybody want me to sort of stop? I'll move on. Let's move on. So what you've seen there is Hello World, a very simple programme that's just generating some trace. And every time you triggered that programme, in that case it was through the execve system call, it should have generated a line of tracing. I already just briefly mentioned this context parameter. So depending where you've attached your programme, you might get different contextual information. The context might include things like what is the user space process that caused this event in the first place. So for a system call, a user space programme process had to make that system call. And you'd be able to get hold of the information about that process with a helper function. In fact, the kernel tracing gives us some of that information. So what is the executable, what's the process ID, timestamps and all kind of other information that gets included as part of that tracing message. This is just Hello World, right? And it's not terribly useful using this kind of tracing mechanism because you can run multiple EBPF programmes at the same time and typically an EBPF-based tool will run lots, like for Cillian we have an EBPF programme for every network interface and there's one for every container. So there will be lots of EBPF programmes. And if they all wrote information just to the same tracing location, which they do, that's not terribly helpful. So this is just a kind of illustration of the fact that you can trigger something from and it will cause your EBPF programme to run. So tracing messages all go to this single location. This is not an efficient way to get information out of an EBPF programme. So instead, we're going to use these things called maps. A map is a data structure that you can access from an EBPF programme. You can also access it from user space. You can write information depending on the type of map. You can write information in user space and read it out from an EBPF programme or you could pass it from one EBPF programme or use it in multiple EBPF programmes to share information between them. And there are lots of different types of maps. A lot of them are hash tables that will have a key value pair and that's the example that we're going to see in the lab this afternoon. There are also some Perth event and there are actually some more modern ones that I don't think are in this list that allow us to stream event information into user space and get information out in a very efficient fashion. There are also some kind of special purpose maps that we can use for things like having a map that lists other EBPF programmes. That's a whole other topic that's probably not worth going into at this point. So the next lab, which is also in the Chapter 2 directory, it's going to show you an EBPF map so it's going to give you an illustration of using a map to pass information from kernel space to user space. So feel free to go ahead and do that second lab and if anybody's got any questions, I'm sure Eric will come. For the second one, when I had it running and then I was running commands on the other side, I wasn't seeing a one-to-one increase in the counters. Is there a reason for that? So there's a few things that can be going on there. So one is we're still attached to the exec v system call. In the second terminal, you're triggering that system call by running things but there could also be other things that happen to be running on that virtual machine so maybe there's some kind of system d demon doing something and if it happens to call that system call, then you might see that event triggered. And actually that's a really good illustration of why EBPF is so powerful because you didn't have to change anything about that terminal, the shell it was running, what commands it's running, the system d thing that might be triggering exactly, all of those things, they are instantly instrumented as soon as you load that EBPF program and attach it to the event. This is one of the really cool things about EBPF. We don't have to change anything about our applications to get visibility of them from within the kernel with EBPF. See another question? So I see all these programs are basically C programs which are strings and then gets passed via Python. Is there a reason for that? We could probably go directly and I guess the other question is what happens if my code, I'm sure all this code runs but I make a mistake in the C code. I'm assuming it gets compiled somewhere and it blows up. That is actually a really good question. So for these first couple of examples I've used a framework called BCC and BCC is essentially a Python library and it does a lot of heavy lifting for us. So it's taking that C program. It's running the compiler. It's loading the program into the kernel. There is some Python code there to tell it to do it but a lot of the detail is taken away from you. We'll see some examples later on where we'll just have the C code and we'll compile it directly and load it into the kernel separately. BCC I think is a really great beginner tool. And if you want to just write a simple script to instrument something, BCC can make that really straightforward for you. I would say that in this day and age there are more efficient and better ways to create code that's more portable. One of the interesting things is that it's compiling that code for you like on the fly. The reason why it does that is if you compile on the system that you're going to run that EBPF program in, you can have a lot of confidence that the code is compiled with data structure information that matches the kernel where it's running. If you then take that EBPF program and send it to somebody else who's running maybe a different version of the kernel, data structures could have changed and sort of in the olden days or in the past that was a problem. Your EBPF program might not be compatible with different kernels. There is now a thing called compile once run everywhere which I haven't covered in the lab today but there is a chapter in the book that is all about making your programs run on different kernel versions. Not only do you compile the code but it comes with information about the data structure layout where it's compiled and some really complicated stuff has to happen to adjust the code when it's loaded into the kernel. This is one of the things that BCC just kind of takes out of the equation. We don't have to think about it at all and I kind of think it's a bit complicated for day one. So that's why I haven't covered compile once run everywhere in this lab but if you want to dive into it, it is in the book. Can you make another question right here? In a co-bonada scenario there are multiple nodes and you have kernels in each of the nodes so you kind of load EBPF into it so my thinking is can EBPF potentially replace or play the role of a service mesh? Yeah, it's like you're some kind of sillion plant. Yes, so every node is running its own kernel. I showed a picture before with containers or sharing a kernel but you're absolutely right every node has its own kernel but it means we only have to instrument the kernel once. We will see an example later of just some simple things that we can do with network packets. We can change the way that networking behaves on a node. This is essentially the basis of sillion networking but we can intercept network packets, we can understand, sillion can understand the Kubernetes identities that are involved in that traffic so rather than dealing with things like IP addresses your pods, every pod has its own IP address but as pods come and go their IP addresses change dynamically so it's pretty hard to debug a networking problem using IP addresses in Kubernetes. What you want is what's the name of the pod, what's the node it was running on, what name space was it in and all that information we can be aware of in sillion. The basis of sillion is Kubernetes-aware networking. Service mesh, there's a lot of what do you mean when you mean service mesh? Do you mean MTLS? Do you mean load balancing? Do you mean canary rollouts? There's all sorts of different things that people imply. Layer 7, sillion uses the Envoy proxy to handle layer 7 termination which is required for things like layer 7 load balancing understanding what service I'm going to send different requests to. Today there is a little bit of layer 7 passing that's possible within EBPF but mostly we call out to Envoy proxy to do that for us in user space. It's a pretty complex behaviour. The answer to can you implement a service in EBPF is kind of yes and no because probably 80% of service mesh functionality is handled in EBPF in sillion but for the bits where we need to actually pass a layer 7 packet we mostly call out to the Envoy proxy to do that today. That was probably quite a complex answer to a... How's everybody getting on with the EBPF maths? Yeah? Thumbs up? All right. Okay, so this kind of comes back to that question we just talked about around compiling code so under the covers BCC is compiling that program string which is C code is compiling it into an object file and that object consists of EBPF bytecode so this kind of gets into a little bit of like what is happening when we run an EBPF program? It's essentially a virtual machine so there's an instruction set that looks a little bit like machine code that object file is a series of bytecode instructions that look pretty similar to machine code if you're familiar with that and EBPF implements a software virtual machine so there are 10 general purpose registers numbered registers 0 to 9 this is an example of an instruction if you're familiar with machine code of any type this is kind of pretty straightforward it's much simpler than looking at, I don't know x86 machine code but you have these pretty similar like assigning a value to a register or assigning jump instructions comparison instructions, that kind of thing so the compiler creates an object file that defines this series of EBPF instructions that make up the program and it also defines any maps that the program is going to use and there's a BPF system call that's used to load that program and any maps into the kernel goes through a process called verification which we're going to dig into a little bit later on but verification is what makes EBPF safe to run it's what differentiates EBPF programs from kernel modules because EBPF programs can't crash because the verifier makes sure they don't so once your program is loaded into the kernel it gets attached to an event you have to say what is the event that's going to trigger this program to run and then we can use system calls to get information out into user space from any maps alright so the next couple of labs are all about a tool called BPF tool which is like a kind of Swiss Army knife for manipulating EBPF programs and EBPF maps and it's also going to show you an example of bytecode for those of you who are interested to see it I don't feel like you necessarily need to understand every single instruction but it's there if you want to so that next labs are going to show you how some of the things that you can do with BPF tool BPF tool can perform a lot of those system calls for things like loading programs into the kernel reading information out with maps BPF tool can do this kind of stuff for you and it's a very useful tool for seeing what BPF programs are on your system what the contents of maps are you can manipulate maps all sorts of things with BPF tool so yeah go ahead and try that out my colleague Quentin Monet did a really really good Twitter thread that was I don't know 50 messages long or something with all sorts of different things that you can do with BPF tool see a question again Do you have a good reference for is there like a place that we can go to if we're not familiar with kernel programming to see all of the different hooks and like specifically what they referred to kind of like a jump start to pour the hook in and what to like the pieces that went into that so there is some rudimentary documentation in the kernel docs and I say rudimentary I think it's improving all the time and in the book there are links to where the best places are so for example there is some kernel documentation about the instruction set but I think there's actually an unofficial documentation that's part of BCC project that I think is actually a bit easier to read and better there's a site called ebpf.io which is a kind of community site for lots of ebpf things and you'll find lots of great links on that I think it would be fair to say that partly because ebpf is continuing to evolve it's not brilliantly and perfectly documented anywhere completely comprehensively but I think over time the kernel documentation will get better and better it's also slightly complicated because the set of things that you can attach to depends on the version of the kernel that you're in because it's still continuing to evolve so it can be a bit like here is a list but it only applies to kernels up to 5.18 or something and then you might be interested the source code is the really kind of comprehensive documentation I see another question here this might be related to a previous question that was asked but I see a number of see group events on this terminal is that because of the environment like if I ran the same thing on a laptop I'd just get the program that's attached to the K probe partly I think it depends on the kernel version but I think in modern environments you pretty much will see a program attached to every or you will see a series of see group programs I'm not 100% sure what those programs are and it's something I keep meaning to look up but yeah they it may be related to the I don't know network environment I should really know the answer to that but I don't one of the reasons to maybe force yourself to think a little bit about the bytecode instructions is that when we get to verification the verifier runs on bytecode instructions it doesn't run on the C code and I think that's the source of a lot of you hear people talking about how fiddly is to get ebpf code verified and the messages that you get back from the verifier can be a little bit critic but I think one of the reasons for that is because the verifier is looking at bytecode and you have to do the job of relating that to the source code that you gave that got compiled into that bytecode so that can be one reason to kind of get to grips with the fact that there are these registers and these instructions that are acting on those registers I see people still working their way through by the looks of that Any other questions at this point? So I'll just follow up on what you just said so it looked like we were compiling with Clang so are we able to generate a debug mapping to map the bytecode back to the C code that generated them? If you compile with the debug flag then when you inspect the bytecode you will get the source line code information as well and you'll see that in the verifier output too So it's not a complete like try to figure out what the compiler did on her own? It's not complete but I feel like there's a hint about this is the line of source code and this is the bytecode that was generated from it but the yeah it's sometimes it's clearer than other times I think particularly because of the way the verifier is keeping track of the type of information that it thinks every register is holding and sometimes there's a point at which the verifier says no that's not what I was expecting you kind of have to go back and look through the bytecode for where was that value set and that might be in a previous source code line that doesn't necessarily seem to relate to where there are it does but it doesn't at first glance always look like it How are people getting on with that? Go on to the next one EBPF and networking So far you've seen EBPF programs essentially just attached to K-Probes but as we said we can attach them to points in the networking stack and there are some really great very simple reasons why we might want to do this this example I think is particularly great so this is the concept of a kernel vulnerability where a particularly crafted network packet can crash the kernel Hopefully there aren't any of these live today but there have been a couple of cases in the past where let's say the network packet has some kind of length field but if you set that length to the wrong value the kernel and it passes the packet and maybe tries to access some memory invalidly or something and then crashes and if you were an attacker knowing how to craft these packets that the kernel will crash on would be what's called a packet of death and when these kind of vulnerabilities have occurred this is a kernel vulnerability so in the olden days without EBPF if you wanted to patch your kernel that's going to involve installing a new version of the kernel and rebooting your machine which could be pretty painful if you have a large fleet but with EBPF you can just load a program dynamically just like you've been doing already and that EBPF program can check the format of the network packet see if it has whatever the packet of death sort of format is and then just drop those packets and that means you can just mitigate this kind of vulnerability without disrupting anything that's running on your machines you can literally just go around all your machines install an EBPF program and it is no longer vulnerable you didn't have to upgrade the kernel I feel like this is a really powerful illustration of why EBPF is such a great platform that has all these really interesting mitigation properties of course we can do other things we don't just have to discard packets we can modify packets, we can redirect them we can create things like Cillium that do really interesting things with network packets but what we're going to see in the example in this lab is the ability to drop a packet and essentially there's some code that looks for ping packets if the packet has the ICMP which is the proper name for ping then we can drop that packet and what you're going to see in the lab is you can dynamically change that program so while your ping is running you can change whether or not those packets are going to be dropped or passed through I mentioned Cillium a few times I want to just talk a little bit about why EBPF is so great for container networking and Cillium in particular so typically containers get their own network namespace and that is typically true I'm saying typically because it is possible to run containers on the host namespace almost nobody does that in real-world scenarios so mostly your containers and your pods in Kubernetes have their own network namespace and the container is connected to the host network namespace through a virtual ethernet connection so if you're not familiar with the idea of a network namespace it's like a network stack so from the perspective of the pod to the IP address and if it wants to talk over the network to something outside of its own pod it has to use this network connection to do that that network connection happens to be virtual rather than physical so a packet that for example comes from the outside world is coming through the physical connection into this machine and it can be pretty convoluted there's quite a lot of things that the kernel does with network packets and then it goes to the virtual ethernet connection to then get sent into the pods network namespace and vice versa for packets coming out of the pod they've got to traverse these two network stacks and that can be pretty inefficient particularly in the world of Kubernetes where as we've mentioned before these IP addresses are coming and going and changing dynamically IP tables are used to create rules how are we going to handle these network packets what are we going to do with a network packet where are we going to send it and IP tables rules whenever you make a change they actually the whole set of rules has to get rewritten the more rules there are the slower this can get and the fact that IP addresses can be changing and pods can be coming and going can make it quite noisy the way these IP tables have to get updated so much more efficiently we can use EBPF to intercept packets as they arrive on that physical ethernet connection and essentially insert them straight into the pods networking namespace without going through the entirety of this host networking stack because we know here's a packet it's destined for this IP address we can pass it to that pod directly there are actually lots of different places in the networking stack where we can attach programs this includes there's BPFC groups the programs that you sort of see in that long list of pre-existing programs we can attach to sockets we can attach to the sort of the top of the stack sort of as a packets received from an application into the networking stack or just before it's passed to the application we can intercept at various points within the stack they have different qualities and different reasons why you might want to do that the example we're going to look at today is my favourite which is XDP XDP stands for Express Data Path and this is imagine a packet arrives on a physical network connection and before the kernel has had a chance to do anything with it this is where the XDP hook is the reason why it's before the kernel has a chance to do anything with it is because it's actually possible to run XDP programs on network cards or inside network device drivers so this is called XDP offload and if you imagine you can have a network card that can run these EBPF programs that can handle things like packet of death mitigation or firewalling or all sorts of load balancing for example and those packets can be handled on the network card and never even have to hit the CPU they might never have to be handled by the kernel so this can be really like high performance networking not all network cards support this XDP offload but you can think of it conceptually this way don't necessarily have to get all the way to the kernel's networking stack that can be manipulated by these XDP EBPF programs and that program can do what it wants to do to the packet it can look at the contents of the packet it can make decisions about whether to pass it on or drop it or send it to a different address it can change the contents of that packet so if we wanted to send a packet to a different address we'd probably have to change the addressing inside the network packet we don't have to go into the details of that today but it's just kind of cool to know that you can what we're going to see in this example in the lab is the idea that either the XDP program is going to return the value XDP pass which says to the kernel carry on as you were with this packet I'm finished with it, pass it on as you would have done or XDP drop which is just get rid of the packet just drop it throw it away and we're going to use that to intercept some ping packets in the next lab this one is in a different directory you need to go into chapter 3 this time rather than chapter 2 I think it's pretty cool that Instruct lets us run these kind of things on a phone quite a good way of getting to see these things without having to set up the whole environment well spotted, thank you sorry, I can't remember how long they're going to be up today there are some more labs we're going to make this available in the full set of ISOvalent Labs, I'm not quite sure whether that will be straight away but it's not later this week very soon there's a ton of other labs so it's ISOvalent.com slash labs but yeah, I'll try and keep these up at least for this week so from an enterprise adoption perspective currently we deal with images and then we have some enterprise related stuff on top of the image that we have like peter redhat or ubuntu for example and then we ship out these images for the this is machine images, virtual images so if we talk about bringing an ebpf into an enterprise adoption kind of a thing then there's obviously a lot of things that's possible and how do we ensure what's the code quality of these ebpf programs that's going into these images is there anything available so the verifier which we'll come to in the next lab answers some of that question it ensures that unlike a kernel module ebpf isn't going to be able to crash the system but I mean I've shown you ebpf doing some pretty powerful things like intercepting network packets and changing them potentially the ability to load an ebpf program is it's superpowers it's not something that you should give away lightly it's certainly like root access plus plus so you do need to not give ebpf privileges to just anything be very careful about what and who can load ebpf programs and you have to be sure of the provenance of those programs I mean some of this it's like any software do you trust your vendor to have given you appropriate code open source project you can read the code there is it kind of begs the question of signing like how do you know that your ebpf program is the program that you expect it to be and that is actually a much more complicated problem than it is in most software because I mentioned compile once run everywhere and this is one of the reasons why it's kind of complicated it's even without compile once run everywhere you kind of end up with some relocation of the code as it's being loaded into the kernel that means that you can't just run a hash over all of the instructions and say is this exactly the code I expected because it might have changed by the time the kernel actually gets to see it some of that relocation is done in user space so there is a there is worker foot in the ebpf kernel community to come up with a way of signing ebpf programs so that you can have this kind of validation of where it came from and that it is the exact program that you expect it to be but it's quite a lot more complicated than it is for user space programs so today there is not a fantastic answer to that question other than the fact that you're going to get an ebpf program delivered to you probably as part of a project that involves user space code as well and that can be signed ebpf program itself it's hard to sign it's also still quite common to compile programs in place and if you're compiling it in place you can't sign it question there hello how would a distro maintainer use ebpf during a root of s creation let's say we want to generate a root of s for a distro and we want to integrate some program with ebpf how will we go around that? Will we be using a system d service or whatever the unit system is to run it at each boot? I think the answer to that is probably yes I think there's certainly some people who've been doing some interesting work and even boot loaders I'm not super familiar with exactly how they've done that I think system d would be the sort of that would be my natural way of doing it or some kind of in it so there's no unified configuration file that it can be added to so that it can get loaded on boot I don't think there's a standard no it's typically after the fact I guess part of the point is that you can change it after boot and that you can dynamically change the set of ebpf programs rather than fixing it at the start of the day but I don't see any reason why you can't do it as part of any kind of in it script or system d how are you all getting on with the xdp? some thumbs up I don't know if you might be doing your email for all I know so we mentioned the verifier a few times and this is basically the last of the labs for today is to look at some of the things that the verifier will do for you so the verifier runs over those ebpf bytecode instructions analyzes every possible path through that ebpf program keeps track of what the possible values could be in every register and then uses that to assess whether or not it's safe so for example making sure that you're not going to dereference a null pointer and that means that in order to get code to pass through the verifier you have to explicitly check that any pointer is not null before you dereference it so just because code might be valid C you know you can dereference a pointer in C and it will compile and you can generate the bytecode but at the point where you verify it the verifier will object because it will say I needed you to have checked this pointer before you dereferenced it and there are several things that the verifier is doing another thing that it's doing is checking that your program will run to completion so analysing all these possible paths through the program and it essentially keeps track of how many instructions it's processed the instruction is called the complexity limit essentially saying how many different instructions have I passed through on this analysis and if it reaches a million then it's going to say I don't have confidence this program is going to actually run to completion so I'm going to reject it that complexity limit used to be 4000 but in more recent 5 point pretty much anything kernels a million instructions so that you can run some pretty complex code in EBPF programs these days but that's one of the things the verifier is looking for it's also checking that you're using appropriate helper functions so you saw the helper function for printing out trace messages there are helper functions for getting hold of information about the currently running process I think I mentioned that before there are helper functions for helping you access network packets but you can't use a helper function for accessing network packet from a program attached to a K probe because there is no network packet involved you can't use a helper function for finding out what the current process ID is or what the current running command is you can't use that in the context of processing a network packet because there is no user space process involved so the set of helper functions that it makes sense to call depends on the context and where you've attached that program and the verifier will check that kind of thing for you so the next lab just gives you a few examples of things that will break the verifier I'm going to warn you that even when you run this lab the first time it's going to generate the verifier output for you and it might look a bit this is a bit overwhelming but if the program doesn't stop if it just sits there then the verifier has passed if the verifier fails the program will stop and you'll see your prompt and then you can read back and look for whatever the error message is so the idea in this lab is you start with some code that does pass the verifier and you modify it in a few different ways that give you some examples of breaking verification there's a ton more examples in the book and actually you might see in the file that you're editing that there are some comments that can give you some clues for other ways that you can break that or cause this example to not pass the verifier I see a question Just more tactical things since this lab is actually asking us to use an editor I could just use a basic text editor because I can't figure out how do you save with this editor Towards the top of the screen there's where the file name is it's kind of like a tab there's a little disk symbol and if you click on that it should save but yes you can use any editor I'm not sure what's installed probably by another question I may have missed this but how do you figure out how do you pass the information about the process that triggered an event So it's kind of done automatically for you by as a result of the type of program it is where you've attached it So let me see if I can find So this example where it was an XDP program the context variable by convention we always call it CTX but it's this I can't quite see from here but it's this sort of network packet metadata structure that because we're attached to XDP that's what you get if we are attached to a K-Pribe I'm actually called it void start here because I wasn't using it but you would get a different structure that represents information about the process and it's a kernel data structure so you can access it directly or using the helper function so a lot of the helper functions will take this context parameter as an argument so the answer is it depends another question over there Right now with the programs we are playing we are getting the output in the terminal but how do we go about using this in the real world like is it like a visualisation on top of it because so that we can make more sense out of it or just it's always going to be a terminal based technology Yeah, no it can be anything so in that the second lab the one where it was the eBPF maps depending on how familiar you are with Python code you can see the Python code kind of accessing the map and printing it out to the screen but you wouldn't have to do that because it's much more complex that extracted that information and sent it to Prometheus you can do anything you like with that information once you've got it into user space it's like if you read information out of a file you could print it to the screen or you could do anything else you like with it I'm tracking off his question for like practical application of this so let's say I have like a 100 node Kubernetes cluster and I want to actually deploy a program across all the nodes like is there an ecosystem that exists for the management and deployment of these programs to more than one kernel keeping track of all of them The short answer to that is no there isn't but there is a good reason for that is the examples that you've seen here are all kind of standalone here's just one EBPF program that does a thing and there are lots of really good use cases that behave like that actually as part of the BCC project there's a ton of observability tools that you can use for kind of measuring or observing all kinds of different events Brendan Gregg was a real pioneer in that and just this huge family of observability tools and those typically are one or two or maybe four EBPF programs something that quite often happens particularly if you're looking at things that involve system calls is you'd have one EBPF program attached to the entry to a system call and a different one attached to the exit of the system call because at the entry point you've got information about what was part of the arguments part of the system call and the exit you've got the results and those EBPF programs maybe use a map to coordinate and kind of combine the two because by the time you've exited from a system call the arguments are no longer present so you kind of have these two different EBPF programs so even with those pretty straightforward examples we're already talking about two EBPF programs if you want to do something a bit more complex you might have many different EBPF programs silly you're going to have it depends what functionality you're running and how many containers you're running and that means that there's no kind of really standard way to manage those EBPF programs and you typically going to have, I mean you saw a EBPF tool EBPF tool is a way that you could manage but it's pretty low level with a more sophisticated I'm going to say silly but anything, you know Falco anything Parker, anything that's you know a sort of production ready EBPF tool will come with a user space agent that handles the life cycle of the kernel programs for you another bit of complexity here for example so with Cillium there's a user space agent that in a Kubernetes environment it's going to run as a demon set so there's an instance on every node and that's going to take care of loading those EBPF programs into the kernel and updating them and what have you but by design if the agent stops running for any reason the programs remain in place and the corresponding maps that they have they remain in place in the kernel and that means your data plane still exists it might not update the new pods until the agent is back in place but it means that if for any reason the agent gets disrupted your data plane is still in place all this being a very sort of long and complicated answer to say there is no really standard way of approaching managing that kind of life cycle because it really depends on the application of those EBPF programs I hope that made sense another question there following up on your comments about there are going to be separate EBPF programs one for the entry one for the exit and when you talk about multiple nodes there are going to be many more things so how do you stitch the things together like how do you stitch all the calls in particular operation or transaction together when it's being emitted by multiple of these programs which are being batched and sent to the agent which we might be running in that particular node which kind of listens to those things the mechanism is going to be maps because you can access the same map from multiple different EBPF programs the sort of design patterns around that very much depend on the application so I wouldn't say there's like a standard way of doing that also because EBPF is evolving some of this is kind of continues to develop things like how we get a stream of events out of the kernel there are newer ways of getting perf events that become more and more efficient depending on the version of the kernel that you're using you might see code that was written three years ago that uses one type of map and that if you were re-implementing it today you'd use a different type of map just because it's more efficient there's no kind of standard for that and it very much depends on the application unfortunately you do once you get past some pretty packet drop hello world things do start getting quite complicated quite fast was that another question? as things are getting more complicated what does the developer workflow look like in terms of do you have IDs that already know how to interpret EBPF C codes so you get nice syntax highlighting and warnings do you have CI pipelines that are spinning up a dozen different kernels against different environments? in the Cillian project we do have a number of those things I'm going to say pretty new frameworks for things like getting code coverage on BPF programs there's a couple of different things for that that I'm aware of which is called something like BPF Cove and another one is I think part of the Cillian put it into the Cillian maybe BPF actually I think maybe it's its own repo on the Cillian but yeah so there's a few different tools for and again this is something that's kind of developing those code coverage tools are being created within the last year things like CI CD again it's going to depend on the life cycle but I would definitely say projects like like Cillian, like Falco, like Parker probably Facebook have a project called Catran that's a very mature networking project load balancing project so I think all of those are probably good places to go and look at examples of how things are handled but yeah there isn't like a kind of like sort of commonly accepted practices I think yet it's still a bit early for that A really question here so question is regarding can we use Rust for example to write this code in that case the verifier can be we can make verifier simpler it's a really great question yes so the answer is yes you can the BPF virtual machine is still going to operate on bytecode but your compiler Clang, GCC and the Rust compiler are all today capable of compiling to EBPF bytecode Other compiled languages it maybe doesn't make a lot of sense so anything that has garbage collection that wouldn't be compatible with the EBPF virtual machine so yes Rust is a actually pretty common language today I think for some projects to write their EBPF code and if you're a Rust programmer that has the advantage that you can write your kernel code and your user space code in the same language you can write it in C I think today C programmers you know I don't know C programmers are really happy with writing in C but a lot of user space programmers will want to write in something else so there are things like Go libraries that are good at the user space part that kind of user loader interface Python as well you've seen BCC as an example but that can only handle the user space part in terms of improving the verification I don't actually know if there's been any kind of research into that because I think you'd still have the requirement to do things like checking explicit checks on pointers it wouldn't make any difference to the helper functions and so on The reason I ask is I look at this the reference example because I think in my opinion in Rust you don't have that kind of problems that's why I'm saying maybe it's still got to go via the bytecode though so I think this kind of comes back to when the verifier is running over the bytecode it doesn't know what the compiler was it doesn't know what the original language was So I see the book is available in a PDF is there ebook friendly formats like EPUB or something like that I'm going to say no I mean it's it's available on Kindle for example but that's really kind of a question for the publisher so I don't get to control that unfortunately which does remind me that I have five copies of these I've been paying attention to people who've been asking questions so I'm going to bring out a few of these Would you like a book? Yes, absolutely One over here I'm afraid I don't have one quite for everybody who asked a question That was a really good Rust question and apologies that I didn't have enough for everybody who asked a question What did you all think of the labs? Was that useful? I'll try and keep those up this week and we'll probably try and keep them actually put them on to the labs page for longer and if you enjoyed that there's a ton more detail in the book and we still have about ten minutes if you have any more questions Thank you Liz