 Hi, my name is Liz Rice. I'm Chief Open Source Officer with ISO Valent, which is the company behind the Sicilian Networking Project. And until recently, I was chair of the Technical Oversight Committee at the Cloud Native Computing Foundation. And in that role, I had the privilege to see lots of technologies all across the cloud native landscape. And something that has particularly caught my eye over the last few years is EBPF, which I'm going to speak about today here at Kubernetes Community Days, Chennai. So I want to explain why I'm so excited about it and how EBPF gives us superpowers for cloud native tooling, particularly for networking for security and for observability. So let's start by seeing what EBPF stands for. So the letters stand for extended Berkeley packet filter. But to be honest, I'm not sure that's terribly helpful because EBPF can do so much more now than packet filtering. What we need to know is that EBPF allows us to dynamically run custom programs in the kernel. So let's make sure we're on the same page about user space and kernel. As application developers, we mostly write our applications in user space. And we're given abstractions that protect us from the system calls that need to be made to the kernel. So for example, if our application wants to do anything that interfaces with hardware, maybe that's writing to the screen or receiving a network packet or writing something to a file. All of these things require access to hardware, even accessing memory. And user space can't do this directly. It has to ask for help from the kernel. And the kernel provides that interface between user space applications and the hardware they're running on. And it also coordinates multiple user space processes that are running simultaneously. So user space applications make system calls to ask for help from the kernel. But we typically don't write system calls directly in our programming languages. We're given higher level abstractions. For example, reading and writing to files will map to read and write system calls at the system call interface. So the kernel, when things happen, there are events in the kernel. That could be network packets arriving. It could be a user space application making a system call. All sorts of events constantly being triggered within the kernel. And we can attach EBPF programs to these events so that whenever the event happens, our EBPF program can run. Let's take a look at a concrete example. So here is my very basic hello world example in EBPF. The code that's going to run in the kernel is here. It's a very simple C function. EBPF programs are functions. And all it's going to do is write out some tracing. Let's change this. So it's going to trace out, hello, KCD Chennai, whenever my EBPF program is triggered. And I'm going to attach it here. The rest of the code is Python. It's using a Python framework called BCC, which is quite a nice way to get started with BPF programming because it makes it very easy to load programs into the kernel and attach them to events. So here I'm attaching the EBPF program to the system call called execve. And execve is what gets triggered when a new executable is being run. And as you'll see on the virtual machine that I'm running on, there are lots of new executables being run all the time. So I need to be rude to run this because you need privileges to load BPF programs. And we immediately see tracing being generated by system calls that are running on this virtual machine. In another terminal window on the same machine, I can run, let's say PS, and we can see the process number 74282. If I find that in the output, there it is. So bash with the process ID 74282 triggered execve and it wrote out the line of tracing as a result. And we also got information like the timestamp and various flags. That information gives us some context about the process that was running that triggered that event. Other types of event might give us information about a network packet or the socket buffer being passed from an application into the kernel. And we get contextual information about whatever it is that triggered the event that our BPF program was attached to. So we have some code that runs in the kernel, and we also have some user space code that loads the BPF program into the kernel. We might also write user space code that can communicate with BPF to extract things like metrics. So, if BPF is often used in observability, we can attach a program to increment a counter every time an event happens, and then read that counter in what's called a BPF map, so that user space can read that information and display the metrics. So, even with my tiny Hello World example, we dynamically change the way that the kernel behaves. And this is really quite a seed change in what we can do in kernel technology. Normally, if you want to make a change to the kernel, it takes a very long time. Not only is it complex to change the kernel, it's 30 million lines of code. But also, making a change to the kernel requires that the entire kernel community are on board with that change and think that it's a good idea. Not only that, but getting the change into the upstream kernel. It still takes typically years for that kernel version to become part of the Linux distributions that we run the distributions like Red Hat or Ubuntu or Debian or Arch. All these different flavors of Linux are distributed with kernel versions that have typically been released one, two, three, four, five years earlier. So it can take a really long time for features in the kernel to make it into production environments that enterprises are using. And this is why eBPF has suddenly become so popular and we're seeing a lot of tools being built on eBPF because the eBPF functionality that's required within the kernel to enable this whole platform. That is now sufficiently stable that it has been in, it's in the kernel releases that are typically being distributed today. I think there's all the Linux distributions are now using enough eBPF technology or they have enough eBPF platform built into them that we can, for example, run Selium. Two or three years ago, quite a lot of production versions of Linux were running an older kernel that wouldn't have sufficient eBPF capabilities. So this is why we can use eBPF for tooling today. So it takes a long time to get changes into the kernel, but with eBPF we can make changes instantly. All we need to do is write some code, load it into the kernel, and we can change the behavior of that machine. This means we can have bespoke behavior very easily, don't even need to do a reboot. This can be really useful for all sorts of reasons and one interesting application is for mitigating kernel vulnerabilities. There's a class of vulnerabilities called packet of death where a vulnerability in the kernel means it's unable to handle a network packet that's been crafted in a particular way. And with eBPF we can easily mitigate these vulnerabilities when they when they're discovered. So if the packet of death vulnerability exists when the host receives a packet that's crafted in a particular way, it's unable to handle it correctly and the kernel crashes. And this is a very bad day because it brings down the whole machine. With eBPF we can write a program that hooks into the event of receiving a network packet. Look at that packet and see if it's crafted in the format that the packet of death vulnerabilities are required to exploit that vulnerability. And if it is a packet of death, then our eBPF program can simply discard it. And that means the kernel never gets to process that packet and the vulnerability that's unable to process that packet never gets hit. So the packet is is harmlessly discarded. Let's take a look at how easy it is to discard network packets. So I have a an eBPF program here. This is the C code that we're going to load into the kernel. It's called goodbye ping. And at the moment, all it does is trace out whether we receive an ICMP packet. ICMP is also known as ping or TCP packet. So don't worry too much about the details of this code. It's looking at each network packet to find, first of all, whether it's an IP packet. And if it is, it will look at the protocol type and trace out a message if it's either ping or TCP. Also currently, in either case, I'm returning XDP pass as the return code from this function. So what that says to the kernel is just carry on handling this packet as you were going to do if there was no eBPF program here. So I'm going to run this inside a container. And that container has an address that has an interface called f0 with IP address 172.17.0.2. And I can start pinging that address from outside the container and we'll see every second a response is coming back to that ping request. So I've got to make file that it compiles the C code. And it detaches any previously existing program on that f0 interface and container. And then it loads the version that we've just compiled onto that interface. So every time a network packet is received on the f0 interface, it should trigger my eBPF program. And I'm going to start cutting out the trace output. And you can see here that every second we're getting a message telling us that a ping packet has been received. And the sequence numbers incrementing every second and we can see tracing being generated. So now let's modify the program and instead of saying that we're going to pass the packet up, let's just drop it. So in other words, every time we see a ping packet, it's going to just discard it. So I can make that and that will load my new version. And we instantly see the sequence numbers have stopped incrementing because the ping is being received in the container dropped and no response has been generated. So the ping application here is never seeing that response. But we continue to see the tracing because we trace out the fact that the packet has been received before we drop it. And I can very easily turn it back on again to enable packets to be received. Make that one more time. And we instantly see the sequence numbers starts incrementing again because we've changed the behavior. We allowed them to pass then we drop them and now we're allowing them to pass again. So, so again, we've been able to change the behavior of the kernel dynamically. We didn't have to change that didn't even have to stop ping from running, we could instantly modify the way that the kernel handles those packets. One thing you might be wondering is how is this safe. The EPPF code has to be safe to run, because if it crashed, or if it looped indefinitely that would stop the kernel from working and that would essentially cause it to what if it crashed it brings down the whole machine. If it hangs, all of your applications will hang. So there is an important part of EPPF called the verifier. And when we load an EPPF program into the kernel, the verifier checks it to make sure that it's going to run to completion, that it's memory access is all safe. It's not only that a given EPPF program can only look at memory that's appropriate for it. So for example, if it's triggered by one process it can't go off and look at memory and by another process. And it also checks that we never dereference a null pointer in an EPPF program you have to explicitly check that your pointer is not null before you dereference it. So the verifier is used to make sure that our EPPF code is going to be safe to run. And this is one reason why sometimes EPPF gets called sandboxing. To some extent that's true it's sandboxing our programs to make sure that they are safe. But I also think sandboxing can be a little bit of a confusing term to use in the cloud native world because we also sometimes talk about sandboxing for containers and EPPF is not a replacement technology for containers it's kind of entirely orthogonal. So let's have a look at what it does mean to run EPPF programs in a container environment or particularly in a Kubernetes environment. So in any given host, whether that's a virtual machine or a bare metal machine, there's one kernel. And that kernel looks after all of the user space applications, whether or not they're running inside containers inside pods, Kubernetes environment they typically are running in containers inside pods. And when after those pods, the application code within those pods want to do anything interesting, like accessing the network or reading or writing to files, or even when Kubernetes wants to create more containers on this host. All of these things require support from the kernel. So the kernel is involved and aware, whenever the pods do anything. And that means if we instrument the kernel with EPPF programs, they can be aware of everything that's happening inside those user space applications running within the pods. So we can write observability tools that can see events happening regardless of what pods triggered those events. And this is why EPPF is such a powerful tool for observability. EPPF programs have this view across the entire node, and it enables really deep observability tooling. So here are a couple of examples. One is from Cillium, where we're using EPPF to connect network endpoints together. We can observe every network packet that's flowing to or from different pods. And we also have this awareness of Kubernetes identities so we can map not just what IP address and the port that packets are going to and from, but what's the pod name, what's the service name, what's the node, what's the name space. In a cloud native environment, that's much, much more powerful. In a cloud native environment, in a Kubernetes environment, IP addresses don't mean much for very long. A pod can be created and destroyed dynamically, and that IP address could be reused for a different pod in the future. So if you only know what IP address a packet went to or from, you're going to have a hard time figuring out what application was involved. What you really want to know is what was the Kubernetes pod that was involved in that network communication. Another example of really deep observability that EPPF enables is Pixie, and this enables all kinds of different observability measurements. This is just one example, a flame graph, and it's showing how CPU is being used for all the applications across. In fact, in this case across the entire cluster because it can coordinate information from multiple nodes. So cloud native is benefiting from really great observability tools and they're highly performing because they run in the kernel, giving us these deep insights to how our applications are behaving. We can also use this view across the entire node to enable some really efficient networking connections. So let's look at how networking works in a traditional pre-EPPF environment. So each of our pods is typically running in its own network of namespace, and that means it's running a network stack that's separate from the host's network stack. And the pod is connected to the host through a virtual ethernet connection. So a network packet that's coming into this host and destined for that application. First of all has to traverse the network stack on the host, and then it passes across that virtual ethernet connection through the network stack in the pod. And then finally it reaches the application. Now with EPPF, we can take responsibility for connecting all of the different endpoints. And when we receive a packet on that physical interface, we know that it's destined for that pod because we're aware of the Kubernetes identities and the addresses involved. So we can take that packet, pass it straight to the pods networking namespace. And this makes the path for that network packet dramatically shorter and makes for faster networking. We can see this both in a flame graph. This is taken from a blog post that we did last year, some benchmarking work that we did, where you can see that some time is taken when a packet is received. Some time is taken for it to be processed in EPPF, then it gets passed directly into the pod. And in the pods, we can see time being taken to traverse the network stack and then the socket. What this results in is more efficient networking. And this is true for Cilium. It's also true for Callico in EPPF mode. The blue line on the left is a baseline of node to node host to host traffic without any pods without any containers involved. And we can see that we achieve nearly as fast networking speeds using EPPF because we're able to bypass so much of that additional networking stack. Whereas in the legacy mode, so not using the shortcutting process, both Cilium and Callico have less, they're able to handle fewer requests per second. So EPPF is making a significant improvement in the speed at which we can process network packets. Another really important aspect about EPPF is that not only does EPPF programs have this ability to see across the entire node, they can do it without having to make any changes to the applications. We don't have to change the way the application is configured. We don't have to write any code within the application. The EPPF program running in the kernel immediately gets visibility into those programs. Even if the application was running before we load the EPPF program, it's visible to EPPF tooling. Simulclad did this really great cartoon about how we can use EPPF for much more efficient instrumentation than the sidecar model. So what do we mean? Why is EPPF more efficient than sidecars? Well, every time in the sidecar model, we have to have a container inserted into every pod so that we can instrument that pod. Whether this is for logging or tracing or security tooling, you have to have that sidecar injected into the pod so that it can share the name space of that pod and see what's going on in that pod. And in order to inject that sidecar container, it has to be defined in the pod's YAML. That probably isn't done manually. You probably have some automated process to inject the sidecar, perhaps in admission control, perhaps even in your CICD system. But if the sidecar YAML isn't there, then the sidecar won't get created. It has to be added through YAML. So what if something goes wrong? If the YAML doesn't get injected correctly, then the sidecar will not have visibility over what's happening inside that pod. And that could be a misconfiguration. It could be a bug. Something that causes the sidecar to not be injected means the pod is invisible to that tooling. With the EBPF, we don't need to modify the YAML at all. We simply need to load the EBPF code into the kernel and it immediately gets visibility over all of the pods, all of the containers within those pods. Not only that, but if there is something malicious running on the node, EBPF tooling can see it. So if you have an attacker who perhaps they've compromised the node, they've started running some malicious workloads in pods, not in pods, doesn't matter. It's going to be visible to EBPF code running in the kernel. Whereas if you're relying on the sidecar model, your attacker is probably not going to instrument their pods with your observability tooling or your security tooling. So EBPF tooling makes it much more likely that you will see any malicious activity happening on the node. The first very common use for sidecars is with service mesh and EBPF is enabling service mesh models that don't require sidecars. We launched the Cillium service mesh beta towards the end of 2021 and we've had hundreds of people sign up to use it and the feedback has been phenomenal. People are very excited about it for a couple of reasons. The first is we don't need a sidecar in every pod and that reduces the complexity and the resource usage. So take for example, well, if you're going to inject a network proxy into every pod, if that proxy has to have routing information, that routing information is duplicated in every sidecar. Whereas if we have one network proxy per node, then we only need one copy of that routing information. It also makes it much less complex to manage and we've had a lot of feedback from users who really want to avoid the administrative overhead of dealing with a sidecar in every pod. The other aspect that's really significant improvement is network performance. So if we're using a sidecar, a packet coming from the application has to go through the loop back into face within the pod's name space so that it can reach that network proxy that runs in user space. And then the proxy can send the network packet out through the, well, out through the pod's network name space and then through the host's networking stack. It's quite a convoluted path for every network packet. With EBPF and the sidecarless model, we can dramatically shortcut that. So if traffic doesn't need layer seven termination, it can be sent very much like the non service mesh case. It can go from the application through the EBPF network connection directly to the physical interface. In the case where there are packets that do need to be terminated, they can be sent to that network proxy running in user space on the node. But it's again a much shorter path because we don't have to traverse network packets in both user space, don't have to traverse network stacks in both the pod and the host network name spaces. So service mesh is a great example where EBPF is enabling us to think about cloud native problems in a in a new way in a in a more efficient way. EBPF is enabling a range of really powerful tools in cloud native. So on the landscape today, there's psyllium as a networking plugin it's it's the only incubation level CNI project. And within psyllium, there's Hubble observability which I showed you some examples of before to collect network flow information and build up service maps for example. There's pixie which I showed you the example of a flame graph but again very powerful observability tool that can give you all sorts of insights into how your applications are running. A new sub project in the psyllium family is tetragone, which enables security observability and runtime enforcement using EBPF. And that falls into the security family of CNCF tools, along with Falco which also uses EBPF or a kernel module to provide insight and observability for detecting malicious or suspicious security events. So there's this wide range of very powerful tools and what they all have in common is you don't need to change your application code to use them and they instantly get insight and control over all of your cloud native applications running in the cluster. So EBPF makes the Linux kernel programmable. Linux is not the only operating system out there and EBPF is now being created on Windows. We've recently seen the first demos of some psyllium functionality running on Windows through the EBPF on Windows project. So we expect to see this powerful tooling capability extending from Linux but also into the Windows world so that you'll be able to run really powerful tooling on Windows just as you can on Linux today. So I hope that's given you some insight into why I'm so excited about EBPF and why I believe it is the foundation for this new generation of cloud native networking and security and observability tooling. If you want to find out more, there are two free to download reports that we've published through a Riley one is what is EBPF which is an introduction that I wrote recently and another report on security observability with EBPF written by Natalia Ivanko who is a colleague of mine at ISOvalent and Jed Salazar. So both of these reports you can download from the ISOvalent website. If you want to check out psyllium, it is like all CNTF projects available on GitHub and very welcoming. We'd love to get new contributions and there's a very, very active Slack community that you'll find if you go to the psyllium website and follow the links to Slack from there. You'll find a community full of people who will help you out, answer questions and get you started on your psyllium journey. So with that, thank you very much for hosting me today. Covenette's Community Day Chennai. I hope you've been having a wonderful day and I hope you have some great questions for me. Thank you.