 Okay. Welcome to the last session of the conference. I'm surprised there's this many people in the room, so hopefully I can live up to the expectations. This presentation is just buzzer real bite, EBPF and cloud native. If you don't know me, I'm Bill Mulligan. I'm a community pollinator at ISOvalent. I say pollinator because EBPF, we like to talk about bees and I help communities grow and flourish, especially around EBPF and psyllium. Who recognizes this logo right now, like Intel inside? When you see this logo, you think of Intel inside. This is a great marketing thing, but how many people actually understood what the technology meant? When you're giving somebody a laptop and you say, oh, there's Intel inside, how many people actually understand that? Not that many, but because it's a brand and people think that Intel inside makes it better, people want to buy that. It's not about the technology, but it's about what they think the technology will do for them. I would actually argue that EBPF is quite similar. It's something you're going to want because of what it enables, what it allows you to do, what it's giving to your technology, but you don't necessarily have to understand or care about what's underneath the covers. This session will give you a brief introduction into what EBPF is and some of those advantages it will bring, but you're not going to have to write in code to take advantage of those benefits. You don't need to know exactly how the hardware works to get the benefits of Intel inside. Having EBPF inside will allow you to do things like faster networking or low overhead observability, but you get all those benefits without having to understand all the technology. Hopefully after this presentation you'll understand a little bit what's going on, and that's the goal of this. To set the stage of where EBPF was coming from, there's this paradox between kernel space and user space. If you're in the Linux kernel, you have complete system awareness because you basically have the interface between the software and the actual hardware, but because of this it lacks the flexibility. It's a very important thing. The Linux kernel is deployed on billions of devices worldwide. It has a slow innovation cycle and it doesn't really have the flexibility of user space. In user space it's extremely programmable. It's where we do all of our things, but it doesn't actually have access to those kernel structures or resources. In user space you have this flexibility. In kernel space you have performance and visibility, but you don't really have both of them anywhere. There is kernel modules, but they come with their own set of problems where they're difficult to code. They have the potential to crash the kernel and they're not always stable, which obviously you don't want if you're running a mission-critical system. How can we overcome this challenge and get the best out of both? Can we enable programmability in the kernel and how will this benefit the cloud-native world? That's where EBPF comes along. What is EBPF? EBPF at an extremely high level makes the Linux kernel and actually a lot of kernel systems programmable. It's beyond just Linux now, but it helps provide that flexibility in the kernel. What it does is it makes it programmable in both a secure and efficient way. I'll dive into each of these aspects later on, but I think that's the next thing. One example that was popularized to explain what EBPF is doing is that what the JavaScript is to the browser, EBPF is to Linux kernel. If you think about before JavaScript came to the internet, you had these static web pages where you're essentially just consuming information off of them, but once we had JavaScript, suddenly we had an interactive environment. You could have things that you could click on. You had forms that you could submit. All of a sudden, these pages came to life and you were able to interact with it. That's exactly what EBPF is doing to the Linux kernel too. It's making it interactive. It's making it programmable so we can use it in new and interesting ways that we couldn't do before. This quote from Linus, who obviously started Linux, is like, BPF has actually been really useful. The real power of it is it allows people to do specialized code that isn't enabled until it's asked for. It's this programmability aspect. What it allows you to do is to dynamically at runtime change the kernel behavior. This is a nice cartoon of what it's like. This paradox I was talking about between kernel space and user space before we had EBPF, what it would be like. An application developer is going along and they want to have a new feature to observe their application. They go and talk to their friendly kernel developer. He said, hey, I don't want to get involved in Linux kernel mailing lists. Would you be able to add this feature to the Linux kernel for me? The developer is like, sure. It's just going to take a year to convince everybody in the community that it's a good idea. One year later, you finally have it merged into the upstream kernel. It's like, great. Then you actually need to come from the upstream latest kernel to the one you're actually running in production. For some people, maybe five years later. You're looking at a six-year development cycle to get your new feature into the kernel and to have it into production. By that time, if you know anything about technology, everything is changed. This isn't a very good way to get new features and functionality that you need into the Linux kernel. Now, with EBPF, because we can dynamically add new functionality right at runtime, your application developer can say, hey, I want this new feature to observe my application. The EBPF developer can say, hey, no problem. Let me just quickly solve this with EBPF. A couple of days later, they can add the program into their running kernel system. It will atomically add it, and you don't even have to reboot your machine. Suddenly, you have a completely new functionality added into your system running in production, and you can have it almost immediately. That's really what the power of EBPF is. It's shortening this innovation cycle on the Linux kernel, which is a 30-year-old technology and bringing it forward into the cloud native world into a faster pace of technological change. EBPF programs, what they do, we were saying that they're running, so they're running on events. They hook into different points within the kernel, things like K-probes or U-probes, system calls, and when that is invoked, then the EBPF program runs. It's watching everything that's happening, and it's running a specific program based off of those triggers in the kernel. That's kind of like the whole idea behind EBPF. As Brendan Gregg likes to say, this really gives you superpowers to Linux because it allows you to add new functionality at any point in the kernel whenever you want to. The Linux kernel, as I was saying before, is a very important system because it's controlling this interface between software and hardware, so you don't really want to mess it up. Causing your kernel to crash or panic is really bad. The important part is it's also safe and performant changes to kernel behavior. You're not just changing it as you want. You want to be able to do it in a safe and efficient manner, too. EBPF kind of knew that, and built things in around that, too. The first and probably most important thing is the safety. Before your EBPF program can be loaded into the kernel, it has to pass through the EBPF verifier. Basically, the verifier checks that the programs will run all the way to completion. They won't make the kernel hang. They won't crash or harm the system in a way that is irrecoverable. The verifier helps make sure that your programs aren't going to harm the kernel or harm your whole infrastructure. The second part that is really important for a lot of people is the performance. If you're running something in the kernel, you want it to be very performant. What EBPF does is it takes the bytecode of these programs and compiles it into a machine-specific instruction set. You're running it in the kernel. These programs basically run as efficiently as natively compiled Linux kernel code. Instead of having to run something in user space, you can now run it directly in the kernel. That speeds up a lot of things. Also, you're saving the cost of having to switch the context from user space into kernel space and back again. You get a lot of performance benefits out of EBPF too. You're not just making user space go away with EBPF, so you still need to interact with it. EBPF also has the concept of maps. Essentially, what these are is shared memory between user space and kernel space where they can pass information back and forth between them. As your EBPF program is running, there's a observability and collecting information on events that are happening in the kernel. It can then pass those events into user space for processing, for saving, for whatever you want to do. The great thing about maps is they aren't losing data and they're more performant too. That's the extremely brief overview of EBPF. Obviously, there's a lot more if you really want to dive into it. Now, I guess this is a cloud native conference. How does EBPF and all of this relate to cloud native? When we're thinking about what cloud native is, it's bringing all these resources into running on a single host. We're running lots of containers on a single host. When we're talking about EBPF, it's these multiple different applications are all connecting from user space down into kernel space. Things like running processes, reading files, or making a network call. It's all doing it on the same host. The kernel is aware of everything that's happening on that host. Everything that needs to happen needs to go through the kernel. All of these pods that are running are accessing everything through the same kernel. This is what's cool about EBPF. Since everything is going into a shared kernel, if you put an EBPF program on this host, it can see everything that's happening. It has complete context of everything that's happening in your Kubernetes cluster. It's not going to be missing anything because everything has to go through the kernel to do that. If you're using something like a sidecar in user space, it's only connected to a single pod. If you, I don't know, a malicious actor comes in and starts spinning up pods that are outside your cluster, you're sorry, containers that are outside your cluster, a lot of user space things are going to miss that. Since everything is going through the kernel, EBPF has complete visibility over that. The other thing is you don't need to do any changes to your code and configuration because it's running in kernel space. Anything that's calling into kernel space will be seen by EBPF. It's kind of no instrumentation, which is a really big benefit too. What EBPF does by moving into kernel space is it allows you to use all the benefits around performance and visibility that the Linux kernel has, but now finally add in that flexibility aspect and speed up the innovation cycle and make EBPF ready for the cloud native world. It provides safety, performance, observability, programmability into this 30-year-old technology and the great thing is now EBPF is available on most kernels that people are running in production. Let's dive into some of the cloud native use cases that we have. The first one is networking. EBPF is actually from a packet filtering background and it's not just for networking anyways, but this is really the roots of the technology. The two big projects around that are Cilium and Hubble and this is providing high performance networking based on EBPF. We're seeing other projects start to pick up networking with EBPF too and it's great to see that innovation happening. The second really big use case is around observability. EBPF has complete context and visibility into everything that's happening on the host. You can see, understand what all of your applications are doing or even the applications that aren't yours that are running on the host. There's different projects like inspector gadget, Hubble for network visibility, open telemetry, just add an EBPF collector, packet where are you for debugging network traces and also Pixie. If you want to check out how EBPF is related to observability, I would check out some of these projects. The next big area is around security. I think this is a really rising use case. Knowing the ecosystem, the most use cases, the most projects I've seen so far have been around networking and a lot around observability. I think we're really starting to see the innovation happening around security right now too. The reason why that is is because I think a little bit based off of the observability piece where you're able to observe everything that's happening on your system. That's how you secure a system is you need to know what's going on otherwise you have a bunch of unknown unknowns. With EBPF you're able to understand what's happening on the whole host and based off of that you can secure your systems. We're seeing a lot of interesting projects that you can create security profiles based off of what you know that's happening from EBPF programs and you can also do things like runtime enforcement. Say there's a malicious program or actor on your machine and you can see it based on the profile you're collecting, then EBPF since it's running in the kernel can kill that process or stop it. I think security is a very big space to watch in the next couple of years around for EBPF. Then the last big area is in profiling and tracing. Being able to track everything that's happening in how applications are working, you can basically track everything that is doing on the system and through that understand exactly how the application is performing or also isn't performing and optimize that. There's a lot of projects that are doing continuous profiling like PARCA, Pyroscope and Pixi that enable you to understand the performance characteristics of your application and also improve them. I thought one really cool collaboration that happened is the PARCA project actually profiled Cilium and they were able to reduce our memory overhead by like 30%. It's quite cool to see the benefits that you can have from that. It's not just about the projects. Like I said, I'm a community pollinator so let's talk a little bit about the community. What's happening in the EBPF space right now? If you go to ebpf.io, there's a landscape of all the different projects that are happening. I wasn't able to dive into every single project in this short presentation. It's just a brief overview of the use cases but there's lots of projects happening in the EBPF ecosystem. These are the major applications as we like to say but there's even more emerging ones. I know in Cloud Native we like to joke about the CNCF landscape. EBPF isn't quite there yet but I think it's well on the trajectory to get to somewhere because I'm adding projects to this landscape all the time and a lot of new and interesting use cases too. Besides just like the applications, there's a lot happening on the infrastructure side. Obviously like the Linux kernel, different compilers for your EBPF programs and different libraries to load and unload EBPF programs into and out of the kernel too. If you're interested in actually coding of the programs, I would check out some of these projects. The kernel community, EBPF is actually one of the fastest growing subsystems in the Linux kernel mailing list. This is actually the original email submitting the patch set to first merge EBPF in. Daniel Borkman, who is one of my colleagues at ISA Availant, is super nice and he's one of the co-maintainers with some people at Meta2 and there's a lot of active development in the kernel community. In terms of like the market perspective and the acquisitions, we're seeing acquisitions across all these different use cases from security, observability, profiling. I think we're going to see this pick up more and more as more companies are both leveraging EBPF in their products and larger companies that want to innovate more too. There's also, as I was saying before, it's not just about Linux anymore. EBPF can be running in a lot of different contexts. EBPF is actually being ported to Windows right now and they're actually also working on standardizing the EBPF instruction set in the IETF. The discussions are just beginning for that too right now. However, it's not just a magic bullet. I don't want to say it's all like sunshine, rainbows, fairy tales. Obviously no technology is perfect. EBPF is just a technology. There's things that it's good at, there's things that it aren't bad at and it's not applicable to every single use case. A lot of the quote unquote like EBPF projects could be done with more simple technologies. It's the same thing like not everything needs to run on Kubernetes. There's different technologies for different use cases and when you're thinking don't think everything has to be in EBPF just like not everything has to run on Kubernetes but there's a lot of great reasons to do it. If you're looking at EBPF projects or EBPF technologies, look at the outcome rather than the technology. I think the technology you're going to be seeing driving a lot of the use cases coming up in Cloud Native will be EBPF but that's because of the benefits that it brings rather than kind of the underlying technology itself. I don't think you should think of EBPF as a check box. There's right ways to do it and the wrong ways to do it just like any other technology. If this got you a little bit interested in EBPF, where should you go next? One is Wikipedia, the favorite source of the internet. The EBPF Wikipedia page recently got added and it gives a nice overview of it. There's also as I mentioned before, EBPF.io. That's where you can find the project landscape and also a little bit more in-depth overview of what EBPF actually is if you want to dive in more and that also provides you links to resources for further reading and further learning. EBPF.io is also being translated into different languages. You can see the French translation right here. I'm actually looking for help right now for translating the website into Chinese so if you're interested in helping out and learning about EBPF and getting a jumpstart to contributing to the ecosystem, come talk to me. I'd love to see you help translate the website into Chinese. There's also labs to get started so if you want to learn a little bit more about programming EBPF labs, there's labs on the EBPF website and there's also actually four years now of EBPF Summit. All the videos are on YouTube, lots of great talks from lots of different speakers seeing how their company or their project or their technology is benefiting from EBPF. Shameless self-promotion. I also run an EBPF newsletter that goes out bi-weekly. You can sign up on the website and this is actually not publicly released yet so because you're here today you get a little bit of a sneak preview but we're actually going to be doing a documentary on EBPF and it's going to be launching at KubeCon, CloudNativeCon in Chicago. The actual trailer for it is going to be launching on Monday so you get a couple of days early just because you're in this room right now. I've seen the trailer, it's pretty awesome, very biased but I think it'll be really cool and I think the story behind EBPF is really compelling too. At KubeCon I'll also be doing an illustrated children's guide to EBPF. If you saw the one for Kubernetes it'll be very similar. I'm excited about this because I'm writing this with one of my coworkers. Yeah, so those are some of the resources of where to go next. I guess to wrap the whole presentation up I'll bring it back to where we were at the beginning. This whole vision for EBPF inside. We want new technologies not because of what they are fundamentally but because of what they enable. People wanted Intel inside because they thought it would give them a better computer, something that would be able to allow them to do more things with it, have a better system, a more performance system, a more scalable system, whatever it is. I think that's where we're going to see EBPF going. It's going to be the technology inside of a lot of projects, a lot of technologies, a lot of companies that's really driving the next wave of innovation in cloud native and not because it's of the underlying technology. EBPF isn't just a marketing thing. I didn't say it's a silver bullet for every single solution but it does provide a lot of benefits. It provides this programmability, the safety, these performance benefits and because of that people will be leveraging it in their applications because of what it enables. I guess a similar thing is when you're thinking about your mobile phone. Before we had smartphones, you just had whatever software came on the actual application on the phone, whatever the manufacturer told you this is what you're going to need. But then when Apple came out with the iPhone, suddenly you had an application store where you could decide what applications were useful for you to have on your phone. As I came here to CubeCon China, I downloaded AliPay so that I was able to pay people here. I could make a decision about what things were. I think that's the great thing about EBPF is allowing us to add new functionality to a decades old technology, allowing us to do a lot of interesting things. I think people love their smartphones because it's able to let them do new things that they weren't able to do before by choosing what software is on it. I think that's what EBPF is unlocking in the cloud native ecosystem and why it's going to be so important going forward. Thank you for coming today. If you want to check out more, go to ebpf.io, connect with me on Twitter. We also have two books, what is EBPF and learning EBPF, available for download on the ISOvailin website. With that, I think I have a couple of minutes for questions. Thank you. Thank you. Thank you for a great talk. Just a very simple question. On some slide, you said that this program, the byte code for the EBPF is first verified and then it's compiled or I'm just wondering like how it differs from the kernel module when you can put every single instruction and then break the system. So this verification is, how is it doing that you cannot break the Linux kernel? Is it running in sandbox? Is it a virtual machine or is it just a binary code that can do anything but it's verified somehow? How does it work that it cannot break the kernel? Yeah, so the Verifier is a very complex piece of software and it could be a whole presentation and conference on its own. I think the big thing is making sure that you're not like, so you can't have unbounded loops in there. The program has to run to completion. It doesn't, I guess like leave any memory like hang. So it does a lot of checks and I guess that's the role of the Verifier. Yeah, okay. I understand. But for example, let's say you have a malicious EBPF program that is sometimes, only sometimes based on something doing an endless loop, for example, or something like this. So your validator is not catching this and then when you return from the system call, for example, something bad happens and you can handle the system. So like, how can you ensure that this is a safe solution? Because this runs in a kernel space. Yeah, so make sure every single, like there are limitations on the complexity of programs to avoid exactly that issue. So the Verifier will actually unroll like every single like decision for like the whole program and it will make sure every single one ends. Yeah, and the one final thing. So the final, the final code is Compil and this is a native code or is it executed by some internal virtual machine? It's by an internal virtual machine. Okay, that's all. Thank you very much. Yeah, thanks. Thanks Bill for that presentation. It's simple and to the point. I have a couple of questions for you. So can you please touch on some of the US cases that using EBBF is not a good fit? And my second question is to echo his concern from the field. Some many end users are concerned about the security. So is it safe to say, you know, whatever code we're going to run based on EBBF would be as safe as the underlying Linux host? Because if the checkers are going to stop those calls that has to be built in the Linux kernel itself. Okay, yeah. So I'll take those one at a time. So the first one is, is there any use cases that EBBF isn't good for? I would say like, when I was talking about like the Linux kernel tech is like, you know, a decade's old technology. So it's not like people haven't been aware of like challenges and limitations and they haven't tried to build tooling around it, right? The kernel has a lot of tooling built in and there's things that you can already use. So if there's a tool already there, you don't need to write your own programs for it, right? That's, I think like part of the tweet is, right? Like a lot of these things could be replaced by simple things like shell scripts, right? And so if you have like a more simple use case, like you don't need to go to EBBF. There's a lot of challenges that are already solved. And it's, I think when you're going to more of these complex use cases, like, right, so Cilium is built on EBBF because it's trying to bring networking into the cloud native world where things are a lot more dynamic, they're a lot more scalable. And so that's where the benefit that EBBF kind of brings, where things like traditional things like IP tables wouldn't work, right? And so it really depends like on your use case and what you're trying to do. And so I'd really look at what the use case is. And then on the security aspect, I guess there's a lot, sorry, can you repeat that? Because there's a lot of different parts of security. Yeah. So in the field for, you know, organizations that are on your team to adopt EBBF, the always first question would be how about the security about this? So I try to kind of educate them this, there is a Linux checker, so the system calls are not just going to go and run right away. It's going to go through, you know, the Linux kernel checker. So that is, if something already you write code or application that's going to pass that already from a particular security hole in the Linux kernel, it's going to already pass. It doesn't matter if you have EBBF or not, right? But that's those are the challenges I see in the field for people pushing back against BPF. Yeah, I guess on the security aspect, I would say like general security advice is like, don't run untrusted code on your machine. Like don't just like let random things. And I think the same advice applies to EBBF too. So by default, like, like EBBF is like a privileged, only privileged users can do it. And so, like, you need to have like root or like admin access to be able to load and unload EBBF programs. And if you're loading EBBF programs, right, you're loading something into the kernel. So you should think about what you're loading into the kernel. You shouldn't just be, I'm not trying to advocate for just like loading a bunch of EBBF programs and seeing what happens. Like, you should know what you're putting into your kernel or like a lot of people are relying on higher level applications that they trust, right? So, Sillian is doing the loading and unloading of the EBBF programs because it's a like application that you trust. And so, you should basically only use EBBF programs that you trust. Yeah. Thank you. That's all. Yeah. I'm a double senior and I have a lot of work to troubleshooting the host and the containers. And you know, we made a lot of network issues if we have 24x7 high availability system. So, this is use cases if we want to troubleshooting and what's the programming based on EBBF you will recommend it to me. Yeah. Yeah. So I guess it depends on what you're trying to troubleshoot. So, there's I guess a bunch of these different observability technologies. If it's not working, packet where are you is probably a good place to start because that will trace all of the packets going through the kernel. If it's other thing, inspector gadget allows you to see what's happening in your application. I guess my recommendation would be like, I think most people aren't going to be writing their own EBBF programs. What they're going to be doing is leveraging applications that are running EBBF, right? And so, like I would look for a tool that's using EBBF because it allows you to have kind of like this complete observability to start debugging your issue rather than thinking about like, what's the EBBF program that I can write to solve my issue because it's not easy to write EBBF programs, but you're going to want to be using the tools and the technologies that leverage EBBF because of the benefits in observability that it will bring and it'll make it easier to troubleshoot, debug, and solve your issue. Okay. Thank you. Hi Bill. Thanks. I'm curious. Are there maybe some kernel build flags or like kernel config options that might be kind of, there might be a trade-off or superior implementation that you could maybe benefit from by, you know, maybe turning this off and then implementing that in an EBBF program? Are there examples of that kind of thing? It's just listening to you has me thinking about that. So you're talking about like performance tuning of the Linux kernel? Yeah. I mean, it's like, it's not easy to write an EBBF program, but it's also like incredibly difficult to like modify the Linux kernel. Yeah. And I'm kind of wondering if there's existing kernel features that have sort of been, I mean, at the high level, right, like Cilium's like a great example of just like reworking the entire networking stack to publish a bunch of interesting information and map it out to other systems. But like, I'm kind of curious if there's other maybe more trivial things where it's like, there was a device flag, but it had a really awkward interface or it didn't work properly. And instead of modifying the Linux kernel, you know, they just wrote a device driver and EBBF or something. Yeah. I mean, there's a lot of interesting use cases like this. Like one example would be like human interface devices. So I know that I think it was like the Microsoft surface like touchpad, they weren't able to like implement like some of the stuff in the device drivers. They add an EBBF program so that some of the input functionality that was available like on the actual device, what it could actually be like supported by the Linux kernel, right? And that was an easier way to do it that rather than trying to like rewrite the whole driver. So that that's one way to do it. Yeah. That's hilarious and ridiculous. Thank you. Yeah, exactly. Right. And so, I mean, EBBF is being applied in a lot of different use cases and yeah, we've seen it a lot of different places from like Nix to human interface devices. I would actually, if you want to dive more into this, Brendan Greg gave a talk at EBBF summit, one of the keynotes that this past year, and he is kind of talking about like performance tuning and how EBBF is enabling to do that. And he's talking about, it's called Fast by Friday and it kind of like walks through the different things you're going to want to think about if you're trying to performance tune an application running on Linux. Okay. If there's no more questions. I guess this is the last session too. So thank you for coming to KubeCon open source summit in China and see you maybe in Chicago or anywhere else. Yeah. Thank you.