 But because this is the first talk this morning, I would like to briefly cover what is eBPF. For those of you who have never heard about it or are curious, and this is essentially your first touch point, I think many of you may have seen a slide similar to this, kind of eBPF being able to execute code in the kernel when some event happens. And one very common one is when a system call is happening. And then instead of the system call being performed automatically by the kernel, we can execute some code, such as shown here. This is a common example on how some of the system call tracers would generate an event in the kernel to extract visibility. So if you run BCC, if you run Tracey, if you run Falco, they're using similar techniques like this to run a piece of code in the kernel to extract visibility. But that sounds very, very abstract. Let's take it a bit back and actually look at it a bit more broadly. I think what makes most sense to me and what is the easiest angle to understand eBPF is that eBPF is to the kernel. What JavaScript is to the browser. What I mean by that is when you think back 20 years ago, many of us probably remember we had to upgrade our browsers, our Netscape Navigator, our internet explorers, whatever browser we're using, Opera, whatever, to view certain websites. And if you often see this message, oh, you cannot view this website because your browser is too old. That has disappeared because of JavaScript, because JavaScript and other programming languages that are able to run sandboxed in the browser made the browser programmable so web double operas could innovate and write applications without having to build something into the browser first. And all of a sudden, we no longer need to upgrade the browsers constantly. This led to massive innovation. Most of us would have probably never guessed that something as complex as an entire word processing tool, Microsoft Word, would get potentially replaced by something that's running in a browser. But it has happened. And it has happened because of this programmability. It has happened because web application engineers can safely innovate as part of the browser. EBPF is exactly the same for the Linux kernel. And hence, it will lead to a very similar wave of innovation. And that's what we're exactly seeing on going on right now. We're seeing a massive innovation of networking, security, observability, service mesh, performance troubleshooting. All of these fields are going through a massive leap of innovation because of this programmability. I was a kernel developer for about 20 years. And this was a massive problem. And it shows the reason why innovation was so challenging in the kernel. But it's often in this situation that somebody would ask me a question, can I get this into the kernel? We need this feature. I need to observe my app better. I need this. And yes, that's cool. And we did it. And we added it to the kernel. But it took years and years and years for those kernel versions to make it into the hands of users, in particular of users who are using enterprise Linux distributions. They would lag behind many years. This made it essentially impossible to innovate in the kernel because you were lagging behind so much. Because of the dynamic programmability of eBPF, this is solved. And all of a sudden, we can innovate and give new code, give new functionality into the hands of end users, essentially on a daily basis. Very similar to how JavaScript has done this to the browser. And this is leading to this massive innovation in the kernel space that was not possible before. Some of you may have seen the eBPF landscape that is available on eBPF.io. For today, I've created a new landscape. It's not an official one. That's just mine. I've just put it together for educational purposes here, with a very strict cloud-native focus. So I think not all of the projects that you will find on eBPF.io are in here, but the ones that are found are very compelling for a cloud-native world. On the lower level, we have the platform layer. We can see that eBPF can run on Windows and on Linux. Then we have the SDK and packaging layer. This is the layer that makes consuming eBPF easier, because eBPF itself is a very low-level bytecode kernel level. So this middle layer makes it easier to actually use eBPF. There's a variety of programming or language bindings, Go, Rust, C++, Bumblebee making package easier. And then application observability, networking and service mesh, and security. And you can see a variety of projects. Some of them are in the cloud-native foundation. This is Pixie, Cilium, Chattergon, Falco. We'll talk about Chattergon in a little bit, for many of you will probably hear about it for the first time. But the requirement, not all eBPF projects need to be in the cloud-native foundation in order to be useful, of course. If you look into that low level a bit more, because that's what most people regard as eBPF, it's this runtime. So how does that runtime actually look like? I only have one slide here, kind of on the technical side. This doesn't look exactly the same for all operating systems. So eBPF for Windows looks a little bit different, but the concepts are the same. On the level, essentially a set of development tooling. In this case, I'm using Clang as a way to compile pseudo-c code into eBPF bytecode, but you could use another tool chain as well. Essentially, what you need is bytecode that you can then inject into the kernel. You then use typically a library, whether it's libppf or goppf for the Rust or IO, whatever it is, to actually load the eBPF code into the kernel with a system call, so there's a specific eBPF system call. The kernel will take that program, run it through the verifier, determine whether it is secure to run, and if so, give it to the just-in-time compiler, compile it into native eBPF code, and load it into the specific hook that you have decided you want to run your eBPF code on. For example, execute this program when a certain system call is being made, or when this network packet is being processed, or when new data is available in the socket, or when a TCP retransmission happens, or when a disk is being accessed, and so on. Then we have eBPF maps, these are data structures, and a great way to extract visibility from kernel space into user space, so these are hash tables, maps, and so on. So to summarize kind of the eBPF, what is eBPF? eBPF gives us amazing programmability, and it's leading to this innovation very similar to what JavaScript is done to the browser. I think most of us see the excellent visibility and control as a cornerstone or everything. We can see a lot more things with eBPF. We have deeper visibility, but we also have a lot more control, and we'll see a couple of examples of this. Control, I mean, we may have control into lower levels of the operating system, but also inside of applications. And I think what's also very fascinating is the massively low overhead of eBPF, allowing us to do observability, where it was not possible before because of overhead, new networking use cases, service mesh use cases, and so on. So a bit of eBPF history before we go into the future. eBPF started in 2014, same year as Kubernetes. This was the year when the first kernel patch was merged into the kernel. A year later, the LLVM compiler received its eBPF backend. Back in these days, eBPF was really, really, really low-level, so it was only meant for kernel engineers, kernel-level operas. So having an LLVM background, oh, we can write pseudo-seaco, that's great. That's a high-level language, like for kernel developers, that's a high-level language. Also, the CLS-BPF was merged. This was the ability to run eBPF programs for network packets. For example, Cilium and others are using this hook today. And also BCC, and BCC was the first higher-level end-user project around eBPF. It was a set of Python-based scripts that would allow you to introspect system, applications, and so on. And today, even today, BCC is massively used, has a wide variety of use cases that it can cover. A year later, XDP was merged. XDP is the high-performance networking layer of eBPF, allows eBPF to compete with technologies like DPDK, user-space networking, for example, and Cilium was announced. So in 2016, we announced Cilium. A year later, Brandon Greck, some of you may have heard him talk on eBPF, is a massive figure involved in the evolution of eBPF. Cointe-term Linux eBPF superpowers, and it was very, very fitting. eBPF is a superpower in the operating system that allows us to do things that were not possible before. Also, same year, Facebook came out and basically said, hey, world, this is how we do low-balancing now. And it's 10 times faster as IPVS. And by the way, we're using eBPF on the world. It was like, wow, how did that happen? Like 10x improvements are very rare for mature production infrastructure. This code is known as Katron today. It's available open source. Also, Cloudflare started migrating all of their DDoS use cases, that these DDoS codes over to eBPF. And I think in this year, it became clear, okay, eBPF is going to be a thing. Like there's massive players betting on eBPF. eBPF is going to be a technology that changed the world. eBPF Trace was announced 2019 as a replacement or as a evolution of BCC, couple of Cilium releases with IPV LAN support, encryption, Q-proxy replacement, and so on and so on. And also Hubble. Hubble is the observability layer of Cilium that gives us network visibility metrics and so on. And also Brandon Gregg published eBPF performance tools, probably the book today, the most detailed book on eBPF that is available today. 2020, Cilium 1718, and then I think another massive milestone. Google went out and said, you are going to change all of GKE networking and it's going to be eBPF based and we're going to bet it on top of Cilium. You may have seen data playing V2 in GKE, that's Cilium based, it's eBPF based. I think major milestone for eBPF or like a major cloud provider is making their products based on eBPF. Cilium 19 with mag left, low balancing support and the first acquisitions happen. So 2020, Neuralq acquired Pixi in the observability space and Pixi is today a sandbox project in the CNCF. 2021, Amazon is following Google's footsteps, announced that for EKS anywhere, they are going to use eBPF and again, Cilium. Cilium makes it into the CNCF, it lives today in the CNCF as an incubation level project. I think this was the second eBPF summit that we had, completely overrun with over 3,000 attendees virtually, massive event, Cilium 111 with topology of routing, BGP, open telemetry support and so on, like lots of innovation going on, which leads us to this year and the year is not even that old yet, but we already have a couple of very newsworthy items. Cilium service mesh, we announced this year and Tetragon, I will have a slide later on on this, we're announcing this week and eBPF summits 2022 will happen in September, virtually again, the CFP will open shortly. That was a brief history and if you look back, kind of it started in the same year as Kubernetes and eBPF has gone comma very, very, very long way and we're not only seeing the big hyperscalers, Google, Facebook, Netflix and so on, leverage eBPF internally. Today, eBPF is used in enterprises and users use eBPF directly and there is a wide set of end user projects that are available if you look back into the landscape. Let's look a little bit into the future as well, like where will eBPF go? Like it has already taken massive space, observability, networking, security, most vendors have some eBPF strategy now, where will it go from here? I think a big part will be service mesh and I think the appeal here is very, very interesting. Service mesh is, for those who have never heard about service mesh, service mesh is a concept where essentially code that used to live in applications themselves to secure communication, to perform client-side low balancing, to gain introspection, to do retries, to move that code out of the applications themselves and move it to somewhat transparent so that code doesn't have to be written for each application separately and the most appealing first way of doing this was to run a so-called sidecar and run this code in a proxy. So the same proxy could be placed in front of every application and run this code. For example, perform MTLS or retries or layer seven low balancing, retrieve telemetry and so on. But if you look at service mesh and the desire for it, it's actually close to TCP or IPsec. It's like we want visibility, we want security, we want connectivity, it's just a much better version of this. So if we're talking about, for example, HTTP retries, it's conceptually very similar to a TCP retry, right? We want reliable consistent communication. So why can't that layer, why shouldn't that layer be part of the operating system and be transparent to users? If that's what all applications want, why don't we move that into the kernel? And that's the focus, that's the goal of the EBPF service mesh. So I think one way to look at this is not necessarily to think about that, all the functionality that is currently done by a proxy has to move into EBPF. I think the concept should be move some of it into the kernel and integrates the proxy with EBPF and collaborate. I have one example here where it shows the impact. On the lower left, you can see performance for HTTP based visibility. And the numbers shown here are latency numbers, so lower is better. You see a bar for no visibility, you see a bar for EBPF based HTTP parsing and a bar for a side card or a proxy based HTTP parsing. And you can see the massive difference. And I think this is appealing, the complexity reduction is appealing. I think this will be a major focus point of EBPF overall. Observability, I think we've already seen a lot, but we've only just begun. So there have been acquisitions, there have been new tools announced, but in terms of potential and what can be done with EBPF, we've just barely scratched the surface. I've listed several projects up here that will provide observability that are providing observability. I would say, let's watch this space very, very closely. Not all of these tools provide the same visibility. We have tools like Pixi focusing really on application monitoring and application visibility. Also, for example, performance troubleshooting, where is my memory used? Where is my CPU being used up? BCC, it's a bit more focused on the system level visibility, what files are being opened, what connections are happening, who is doing right system calls and so on. Cube control trace is bringing BCC tooling to a Kubernetes specific environment because BCC is not Kubernetes aware. Ply, tracing or function tracing. Tetragon will talk in a bit. Inspector gadgets, similar to Cube control trace, making visibility and EBPF very approachable on Kubernetes. Parker, I think similar to Pixi, new company, Creator Parker, definitely something to watch out for. Tracy from Aqua system call tracing. So I think there will be more and more toolings here and I think we'll see the space explode in terms of observability that is being provided with low overhead. All right, so to me, what I'm most excited about in terms of EBPF future is actually the security angle because I feel we can make together a hugely forward in terms of security. So you will see an announcement go out later today about Tetragon. Tetragon is the knowledge of the Cilium team of like EBPF creators of Cilium creators apply that same knowledge to the runtime security and security observability space. So Tetragon will provide both observability and runtime enforcement. It's part of the Cilium family. So it's essentially part of the Cilium governments, which means we automatically also contributed to the CNCF. And Tetragon will be able to do two things really, really, really well. One is to provide fantastic observability across a lot of layers in the stack. And I'm listing lots of examples here. So it runs as an agent in user space and it can connect to Prometheus, to Fluentee, Grafana, Elastic, Open Telemetry and so on. And it instruments the kernel to extract visibility from very low levels like monitoring which disks are being accessed to network layers, HTTP parsing, DNS, TLS and so on. The namespacing, like what sort of privilege escalations are going on, what processes are breaking out of their namespaces, file access, TCPIP, the whole stack, for example, tracing TCP sequence numbers, obviously the system call layer, system calls that are being done, process execution and all the way into the application. BPF can do function tracing, BPF can generate stack traces, BPF can even monitor the code that is being executed. So I think first of all, the first kind of part of Tetragon is extracting all of that visibility in a very low overhead and make that available as locks, as metrics, as events and as traces. And the second, and I think that's even more fascinating is that EDPF gives us a new approach to doing security enforcement. When we look at enforcement today, typically enforcement either happens strictly at the system call, this would be something like seccomp, where you are allowed to do this system call yes or no, right, this application could do this system call yes or no. And that's great, it reduces the attack surface. Or we have security systems that, for example, use P trace or a kernel module extract visibility and then asynchronously kill the process if something bad happens. With Tetragon, we're using EDPF enforcement inside of the kernel, which means that instead of enforcing just at the system call level, where we need to understand what is good and bad, we can monitor what are undesirable outcomes, such as I don't want this container to ever break its namespace boundary. When that ever happens, kill it immediately. Don't allow it to run. And if you look at the upper right, you see exactly this. This policy was loaded here that essentially all it says is if any process matching these criteria, for example, these pod labels, if they break out of the namespace boundary, immediately kill it. And it did. And it did so without actually understanding that this was exploiting CV 20, 21, 22, 555. It doesn't actually need to understand the attack. It can monitor the scope of the process changes and it gains privileges or it changes namespace boundaries or it's breaking out of the container. And I think that's very fascinating will lead us to a fundamentally better security posture and it's thanks to EBPF because all of a sudden we can enforce in the kernel itself when it happens, instead of just observing and then killing process or stopping processes or pausing containers from the outside. What's even more fascinating is that it's extremely in line with BPF LSM. So I'm 80% excited for Chattergon and 20% excited for LSM, LSM BPF. LSM is the security model or framework inside of the Linux kernel. It's giving an opportunity to load BPF programs at the variety of hook points as well and then code logic in there. The downside of LSM BPF is that it requires relatively recent kernels with 5.7. Chattergon runs on all the kernels. That's why I'm slightly more excited about Chattergon but Chattergon and the LSM BPF can work together and LSM BPF can make Chattergon even better. So with that, I'm super excited for the future of BPF. I think most of the real innovation is yet to happen even though we have done a lot of innovation together. And before I open it up for questions, I do wanna call out, we have formed an BPF foundation together with many players in the industry. If you want to collaborate, feel free, approach the Linux foundation, the BPF foundation is up for membership. There's lots of great technical discussions, lots of collaboration going on between all the players involved in the BPF space. I think we have time for questions. Questions? We do have time for questions. So does anyone have, yes, I have a hand raised already. Thank you, Thomas. I want to ask, there were many discussions about the kernel, not just Linux kernel regarding design questions, going from a monolithic kernel, maybe to a microkernel, such as we know, for example, from Torvalds and Tannenbaum. So I wanted to ask, do you see BPF basically redesigning the kernel from a monolithic towards a microkernel and enabling all those things to basically get out of the kernel code itself and be kind of like plugins or add-ons? Yeah, it's a fantastic question. I think that's exactly what's happening, but it's obviously unrealistic to rewrite the entire Linux kernel, which means that BPF actually gives us a practical approach to achieving this. We can replace certain parts of the kernel with BPF logic. We can take side cuts or shortcuts. We can make the kernel more modular, which is exactly what's happening. It's even going a step further. Some of the big companies out there who have own kernel teams and who rely on writing kernel changes that need to go into a production kernel very quickly, they are essentially have started writing new code in BPF instead of natively because it's more secure, because it is subject to the verifier and so on. So I think what you said is absolutely correct. I think we are moving to a more micro-kernel-based architecture, but rewriting the kernel is not an option. So I think BPF gives a way to essentially work around that and start to refactor without rewriting. Thank you, Thomas. Do we have other questions? Oh, yes, we have another one here. Thanks, Thomas. So once we have Celium installed in the system, how is TetraCon different to Hubble? Because even Hubble provides observability, right? So Hubble is strictly dependent on Celium, so it's visibility based on the networking layer of Celium. TetraCon is 100% independent. It does not require Celium in any way. It uses, you don't need to run Celium. TetraCon is independent. And obviously the focus point is a little bit different as well. It's primarily about runtime, but it can actually also extract network visibility, but it does that without parsing that or packets, for example. Difference, essentially. More questions, folks? Looking around. Okay, for some of you who were EBPF curious, not the super deep technical versions, do any of you have questions? Oh, yes, we have another. Fantastic. Hi, I'm not so deep into the EBPF technology, but what I'm interested in is when I have observability for my program with EBPF tools, that's absolutely great. Can I have observability, what resources the EBPF program inside the curl is using, and how much the overhead is? Yes, that exists. So there's actually EBPF for EBPF. There was a fantastic talk last, in the last EBPF summit last summer, where you can use EBPF profiling technology to actually understand how much memory, which parts of the EBPF program are consuming how much CPU, and you can essentially create flame graphs of which part of my EBPF program consume memory and so on. And also I think in terms of other consumption, EBPF programs are subject to C groups and everything else, just like any other process as well. So the process that installs the EBPF programs, they own the memory, they own the CPU as well. So whatever C group that process is associated with, we'll do it for it. Did I answer your question? Yeah. Excellent. Other questions? It is seriously fine to ask questions that are early exploratory. Hi. I was wondering if EBPF supports hardware chipsets like a KVM to do TLS termination and things like that. So there is EBPF offload to a variety of smartNIC providers. I'm assuming you're talking primarily in the working use case, given the TLS context. It does exist. I cannot tell you exactly which hardware vendors offer the TLS offload part, but because EBPF is super generic for a current level technology, the offload is possible and all the big smartNIC providers have announced and will tell you that they offer EBPF offload. But for more specific questions, you'll have to ask them. Okay, others. Oh, good. I love that I asked the exploratory question questions. Hi, if I am new in EBPF, how do you recommend to get involved? Excellent, yes. There is lots of opportunity, right? So I think we often quickly talk super low level because it's super fascinating, but it's actually a great opportunity to contribute to EBPF on all levels. I would start with go to ebpf.io, but has, I think, first of all, material to get started. It also has a link to a Slack channel which has, I think, by now over 8,000 people involved from all sorts of projects, and you can quickly find an EBPF project to collaborate on. I would say if I go back, the best way to contribute to EBPF is essentially top down. So I think in the beginning, collaboration on EBPF was all on this runtime kernel level and you had to be a kernel developer to contribute to EBPF. Now you can essentially work on end user projects or you can work on one of the libraries, for example. So let's say you are a Go developer or you can do Rust. All these projects, they need help, they need collaboration, and you will learn about EBPF as you contribute to these libraries, but I think even the understanding of that other language is sometimes even more important than having in-depth EBPF knowledge. So that's a great way. Obviously, also to work on projects like Falco or Sillium or BCC and so on, that can also be a great way. Thanks, Thomas. We've got at least one more. Hey, short question. What could be a good way to introduce and advocate EBPF in the company? So not from a term, I want to participate, but they are different, the operation folks and then the application developers, they don't know anything and maybe you have an idea. I think what usually works well is I think let's give a fantastic talk around what is EBPF last year. I think sharing that link is an awesome first step because it gives this entry level, what is EBPF, what are the benefits for everybody? And then I think quickly going to the projects that bring value. So instead of talking, like EBPF is fascinating on how it's being done, but the actual value is in the projects themselves, so in what they provide. Oh, here is how we could do, like if you talk to your app teams, look at this demo on how you can figure out which parts of your app are using how much CPU and their eyes will lighten up. But when they see a flame graph for the first time, they will say, oh, that's great, I want that. And then EBPF is more of an implementation detail that allows this or, hey, we need to run networking at scale, we can point them at psyllium, or you can say, I want visibility, you can tell them Tracy Falco, whatever, Tetragon. That's a great way to get the buy in, you don't need to understand EBPF, your company doesn't need to understand EBPF to make use of it. We have more. Yeah, thank you. Actually, I have a couple of questions. The first one is that, like I had the feeling when you talked about the proxies in front of applications, I thought directly about STO, and I thought it's just going to be replaced entirely like using some custom scripts with EBPF SDKs, like GoLang or Rust. This is the first one. And the second one, perhaps I will let you... Yeah, let's do the first one and then you can ask the second one. So unfortunately, I cannot cover everything in this talk around service mesh, but in a nutshell, we don't want to replace service mesh control planes. We are data path people. So, and we also don't want to replace Envoy. There's actually, we'll have a talk tomorrow at service mesh, if you're there, with a bit more details. We want for users to be able to pick whatever control plane they want, whether this is Istio, whether this is SMI, whether this is just Kubernetes, or I just want to use Ingress or Gateway API, it doesn't matter. And then we provide the most efficient data path with a combination of Envoy and EBPF together and move that into the kernel and essentially get rid of sidecars. That's our motivation. We're not interested to create yet another set of CRDs and make the whole space more complicated. Yeah, crystal clear. The second question is like, I'm more like a DevSecOps and I'm so interested in these scripts using Rust. And I was wondering how do you do the CI stuff, like make it deterministic, like using CI pipelines and making sure that it works with different kernels and so on. Is there some good practices, some recommendation to follow? Yeah, absolutely. So I think obviously all the projects, all the bigger projects have these CI pipelines that think a good way, for example, Syllium is doing this with GitHub Actions very successfully. But actually one of the properties of EBPF is that it's very portable. It's much more portable than a Linux kernel module. Like the API that an EBPF program uses in the kernel is stable. And with BTF, we can even access data structures for observability in the kernel in a state banner as well. And that actually reduces the need to test every single kernel version. And with something like Cori, like compile ones, run any...