 excitement I introduce you to a remarkable technologist and influential leader in the cloud native ecosystem, Liz Rice. As a chief open source officer at Isovelin, Liz has been instrumental in shaping the future of container security, cloud native applications and eBPF, which you will learn more about it in a minute. Liz is the former chair of the cloud native computing foundation's technical oversight committee, open UK board member, AWS container hero, Google developer expert, and so much more to introduce her reasonable would be the talk by itself. In short, she is a dedicated advocate for open source technologies. Her commitment to share her expertise earned a reputation as a trusted voice in the Kubernetes and container security communities. Liz's talent for breaking down technology into understandable bite sizes made her more than well-recognized in the open source world and beyond. She is also the author of Kubernetes security, a go-to resource for understanding and implementing best practice and container security, and will publish soon next month her new book, learning eBPF. And here she is, the influential and ever inspiring Liz Rice. Thank you so much for that very kind introduction, Max. Wonderful to be here. Thank you everyone for joining us today in this event in support of the people in Ukraine. Shout out to anybody who's watching us from there. So, Max has given me an amazing introduction, so I don't think I really need to say any more about myself. Let's make sure I'm on the right screen. There we go. Max also mentioned my new book. It's actually already available electronically and the physical copies are at the printers right now, so I haven't seen a physical copy myself yet, but it's on its way. And it's called Learning eBPF. I've been really fascinated by this technology for quite a few years now, and today I want to share some of that, I guess, enthusiasm as well as knowledge. I've got a ton to get through in this talk in a relatively short time, so please do if you have questions, ask them in the chat. If I don't have time during the talk, I will try to answer them all afterwards. So, the two maintainers of eBPF within the Linux kernel, Alexei and Daniel, were kind enough to write some really nice quotes about my book on the back, and I wanted to particularly highlight this quote from Daniel, that eBPF started a new infrastructure movement in the cloud native space. And that's really what I want to talk about today, how we can make use of eBPF to really get a whole powerful new set of tools for managing our cloud native infrastructure and observing our applications and securing our applications. So, what is eBPF? So, the acronym kind of stands for Extended Barclay Packet Filter, but it has so much more that it can do, that has nothing to do with packets and filtering, that these days the eBPF acronym doesn't really carry all that much meaning. eBPF allows us to program the kernel. We can run custom programs within the kernel. So, let's just make sure we're all on the same page about what that means when we talk about the kernel. Our operating system is divided into user space and kernel, and if we're writing applications, we typically write in user space. Whenever our applications want to do anything that touches hardware, needs to ask for support for that through the system call interface to the kernel. Now, most of the time, as developers, we're blissfully unaware of this because our programming languages take care of this for us. But every time you want to read or write to a file or communicate over a network, allocate some memory, write something to the screen. All of these things require support from the kernel and requires your application to make a system call to support that activity. As well as managing the interface to any hardware, the kernel is also coordinating all the different user space processes that might be running on that machine. Now, with eBPF, we can insert programs into the kernel and attach them to events that happen in the kernel. That event might be a trace point in the kernel code or, in fact, in user space code. It could be the entry or exit from a function. It could be a network packet arriving. There are all sorts of different events that we can attach our eBPF program to. We can literally instrument the entire system with eBPF programs. So, this is a very simple example of the kind of hello world program that you will see if you look at my book and if you want to know more about eBPF programming. If we have a line here that will generate the trace, hello world, we attach this to the execv system call. That's the system call that's used when you want to run a new executable. That's an entire eBPF program. In addition, you need something in user space to actually load that eBPF program into the kernels. I'm just going to skip over that for today. But if once we have that program loaded into the kernel, if we run anything, any executable on the same machine, we would see a line of trace generated. That tracing would also include some information about the context of the event that triggered that particular eBPF program. Here the context is an execv system call and we get some information about the process that actually made the call to execv and we get the process ID and so on as well as the trace generated. So, another example, and this one I'll show this live. So, hopefully you can see my screen and I hope it's big enough. I hope someone will shout if it's not. So, I could run an executable called OpenSneep which uses eBPF to monitor any file opening events. And if I run this on this virtual machine, where I happen to be running Kubernetes, it's actually a kind cluster. I've got quite a lot of pods running and as you can see, they're opening files a lot. I'm just going to stop that and let's just concentrate on seeing any files being opened by bash. And that's interesting. Okay, that is live demo excitement. I don't know what it's complaining about but that's going to take a minute I think to finish. I'm going to kill that terminal. Yes. Okay, I'm going to create another terminal. I think I've got another one already. Okay, I see what's happened. My pod might have finished. Let's see if I can run. Let me meet this terminal up. Okay, let me quit at that. Let's try this again. Okay, so that's working and I'm just interested in bash. Okay. So, if I were to run bash in another terminal on the same virtual machine, we can see that bash opens a bunch of files. But I also want to show what happens if I execute bash inside a pod. Pod running called KCD and if I run bash in it, I'm going to just clear the screen so you can see what it generates. Oh, my alias isn't set up for cube control. And there we go. So, it doesn't matter whether I was running bash inside a container in a pod or whether I was running it directly on the host, it's visible to this open snoop executable. All right. And that's one of the real powerful aspects, characteristics of EVPF that makes it so useful for cloud-native tooling. When we run applications under Kubernetes, we run them in containers inside pods. And those containers are essentially running in user space. But they all share one kernel per host machine. So, your machine or virtual machine that's running a Kubernetes node has a kernel that's common to all of those pods and all of those containers that are running on it. And whenever your applications want to do anything interesting, whether it involves hardware, whether it involves creating new containers, all of those things are going to involve that common kernel that's shared by all of those containers. So, the kernel is aware of and involved in pretty much everything that your applications are doing, regardless of what pod they're running in. If they're on that node, they're using the same kernel. And that means that if we instrument the kernel with EVPF programs, those programs can be aware of and can influence what's happening across all of our applications running on that machine. And we don't have to make any changes to our applications. We don't have to change their configuration for them to be or for their activity to be visible to those EVPF programs that are running in the kernel. And we can do all sorts of interesting things with those EVPF programs. We can use them to spot when events are happening and report on them. I just showed you reporting on file open events. We can be triggered by a network packet arriving at a certain point in the network stack. And we can use that to manipulate network packets. And we can even take security actions from within the kernel. So, that allows us to use EVPF to build observability, networking, and security tools. Now, one interesting thing I want to convey today is that although I've written a book that's basically about writing EVPF code, for most of us, we don't need to like EVPF programs ourselves, because there are already some really amazing projects out there that can do really powerful things using this technology. So, I'm going to show a few examples. One of them is Inspector Gadget, which just got accepted into the sandbox in the CNCF. And I hope I can run an example of this. Let's see whether I can. Is that got that? Yes, that has. So, I think I can do K Gadget trace open. And I think I can do this in all namespaces. Yeah. And that's very similar to the OpenSnoop example you already saw, except that now it's giving us some additional information about the Kubernetes identities involved. So, we can see the node and the namespace and the pod name, as well as things like the process ID that's involved. So, that Kubernetes information is crucial if we want to use EVPF tools to manage or secure cloud-native applications. We're going to need this Kubernetes information so that we can correlate events to the pods and the applications that are involved in that activity. So, Inspector Gadget is one example of a way of getting Kubernetes information from using that in conjunction with EVPF tools. Another example of something that is Kubernetes aware is Cilium. So, Cilium provides networking and security in the form of things like network policy and network encryption. So, it's a CNI, a networking plugin for Kubernetes. I'm not going to run through all of its various capabilities other than to say that it is a very powerful, full-featured way of connecting your cloud-native applications to each other and to the outside world through things like egress gateway and multi-cluster support and so on. We'll talk a bit about some of these features a little bit more later. One of the things that we can do with EVPF is to manipulate network packets. I mentioned that earlier. Now, this diagram shows the path that a packet has to take in traditional container networking to get to a pod. So, by design, a pod will typically have its own, it will have a network namespace of its own. It will not share the namespace of the host. And that means it's connected to the host's networking stack through a virtual ethernet connection. So, a packet that wants to come from the outside world to reach that pod has to come in through the host's physical ethernet connection, traverse the host's networking stack, then go across this virtual ethernet connection into the pod where it goes through another copy of the networking stack to reach the application. So, that's quite a convoluted route for a networking packet to take. And the longer this path is, the more it's potentially going to affect latency. With EVPF, we can intersect the packet as it arrives on that physical interface and redirect it straight into the pod's networking namespace. So, we get to bypass a lot of that host's networking stack. And that creates some really important performance improvements. We also want to be able to see, as operators of a deployment, we want to see what networking is happening. And Cilium has a component called Hubble that also uses EVPF to report on each individual network packet and to build that up into a picture of how different services are communicating with each other. You can see it in this kind of UI view. There's also a command line view. And it generates Prometheus metrics. So, if we combine Cilium, Hubble and Prometheus, we can get some really powerful visualizations of what's happening in our networking. We can also integrate that into Grafana's stack. And this is an example showing how we can use tempo tracing. We can get exemplars from Grafana and use that to see individual exemplars of individual network packets. So, this open source collection of tools can give you this very powerful insight into how networking is behaving in your cluster. Another really powerful observability tool in the CNTF landscape is Pixi. This runs kind of similar to how Inspector Gadget runs EVPF programs and kind of associates them with Kubernetes identities. Pixi also runs EVPF programs and scripts and gives you visualizations of the output. This is just one example showing how CPU usage is behaving across an entire cluster. So, this is aggregating data from multiple nodes. And this is just one of the different views that Pixi can provide using the power of EVPF. If we turn to security and we want to observe security events, one of the well-known projects in this space is FALCO, which has an EVPF mode for detecting particular events. Much like my Hello World example, this particular example from the FALCO docs is attached to the exact VE system call. And having detected that a binary is being executed, there's a policy here that checks whether or not it's running Netcat, which is probably not something that you want to see running in your production clusters. Cillium has also recently added a security observability tool. I said recently, it was a year ago now. And this uses EVPF to observe, not just observe the events, but also to filter events within the kernel. So, in this example, we will observe file activity associated with the FD install function within the kernel. And we're going to filter those events so that only activity related to the files in the et cetera directory will actually generate an event. And then we can look at the output and see if someone is opening, writing, reading from a file in that directory, it will generate a tetragon event. So, hopefully that's given you an overview of the kind of powerful tools that we have already in the cloud native landscape that are based on EVPF and using it to really provide incredibly useful and performant tools. I want to talk a bit about how that differs from sidecars. Because a lot of that kind of tooling, logging, tracing, even some network capabilities have previously been implemented using sidecars. Before we even had sidecars, we had libraries. And the problem with libraries is if you want to have the same common code running across all your different applications, you would need a library implemented in each of the programming languages that you use in your applications. So, if you want to do, let's say, logging, you might need a Python library for logging for your Python applications and a Go library for logging for your Go applications. With containerization, we're able to pull that library code out into a sidecar container. And because those containers are isolated from the application container, they can be written in any language you like. So, we can just have one common container that's shared and used by all of our different pods regardless of what application. So, we can get common functionality implemented in a consistent way across all our applications running in that cluster. The problem is that a sidecar container by design is running inside a pod and can only see that pod. And in order to get there, there has to be some YAML configuration that configures that sidecar container. You probably don't write that YAML by hand. There's probably some automated process that injects the sidecar YAML into your application YAML. But if, through some kind of misconfiguration or error, if the sidecar doesn't get injected into any given pod, then that pod is not instrumented. And your tool does not have visibility over it. If it were a logging library, you would not be getting logging if the sidecar, a logging container, logging sidecar. If that sidecar wasn't there, you wouldn't be getting logs out of that pod. In contrast, if we instrument the kernel, as we've already seen, that means it has visibility to all of our different pods. And we didn't have to make any changes to the pod in order for the EBPF program to have visibility. What's more, if some kind of malicious process starts running in your node, whether that's directly on the host or within a pod, it's visible to EBPF programs because EBPF doesn't care whether a process is in a pod or not. Sidecars also can require pretty significant resource usage. Because they are by design isolated from each other using the pod model, they each need their own copy of any configuration, routing information, that kind of thing. If we're using EBPF, we don't have to have duplicate copies of that configuration or whatever other information. We can have a single instance of that information and access it using a mechanism called EBPF Maps that allows us to share information between a user space process and the kernel EBPF applications. This allows us to remove proxy sidecars from ServiceMesh. So the traditional model for ServiceMesh is to have a proxy container running inside every application pod. Just about a year ago, we introduced Cillium ServiceMesh which still uses Envoy, in fact, as the proxy component. But rather than having one instance in every single pod, we can have a single instance per node, per host, and connect the network stack via that proxy when necessary using EBPF. That problem of having multiple proxy pods inserted into every single one of your applications has long been recognized as a problem for ServiceMesh. And it doesn't just cost resources, it also adds network latency because the path that a packet has to pass through to get through ServiceMesh proxy inside every pod gets even longer. And if we use EBPF, we can bypass a lot of that stack. Even if we have to go through a layer seven proxy, we have a much shorter path than if there's a separate proxy in every pod. This becomes even more true if we're communicating between two different pods, if we're able to replace that with a single proxy, if those pods are co-located on the same node, the path is dramatically shorter. We know that users have been really keen on the benefits of removing sidecars, not just removing the latency overhead, but the reduction in complexity that it provides. And Cilium is not the only ServiceMesh implementation that has started to move away from the sidecar model. Istio have also introduced AmbientMesh, which again avoids using sidecars. We don't have quite the same implementation, but I think it shows the direction of travel. One question that comes up with this is what about encryption? So for many use cases, transparent encryption, which has been part of Cilium for a long time, is sufficient. Transparent because you don't have to make any changes to your application. You simply have the kernel at either end of the communication do the encryption using the nodes identity for the encryption keys. A lot of people have used MTLS and ServiceMesh so that they can have layer seven specific application identities for encryption. And that is now something that you can support in Cilium ServiceMesh using a next generation MTLS implementation where we can inject identities, specific identity certificates into the kernel and use that transparent encryption protocol to secure the connection, but using a layer seven identity. So is EBPF going to mean the end of the sidecar model? There's this fantastic cartoon that Nathan LeClaire did a few years ago. I think it's wonderful. My other sidecar is a kernel. And while I think it is likely to be a better approach in many cases, there are a couple of reasons why you might still use sidecars. One is if you're writing your own code, you may have your own sidecar container. You don't necessarily want to reimplement that in the kernel because EBPF programming requires kernel knowledge. It might be an unnecessary high bar to jump to avoid having sidecars. The other question is whether or not you actually have access to the node to install EBPF. If you're using a kind of serverless model or if you're using something like Fargate and you don't actually have access to the underlying nodes, the underlying machines, you may not have permissions or ability to configure EBPF on those machines. But I think the real reason why there will be a push towards EBPF is the improved performance that it gives us. This is an example of layer seven parsing using EBPF versus using a user space proxy to perform the same parsing. And as you can see, EBPF is almost as fast as if we just weren't doing any parsing at all. So I hope that has in a very quick run through given you an idea of why I'm so excited about cloud native EBPF superpowers. As I said, you don't have to write EBPF code yourself. But if you do want to, well, if you don't want to, I'm going to advise you, if you want more detail, I wrote a short report that's much sort of higher level about how EBPF provides such powerful platform for new tooling. And if you do want to write EBPF code, then let me point you at my new book. Either way, there are lots of practical examples you can try out for yourself on the iSurveillance Labs site. And you can also download those books for the cost of your contact details from iSurveillance.com. With that, that's been an incredibly fast run through I think of what I think EBPF is bringing to the cloud native world. I hope you will have some questions. I'll be online here. I'm also, if you want to contact me on the internet, I'm Liz Rice pretty much everywhere on the internet. So thank you very much for having me here today. Thank you very much, Liz. Awesome to have you. I'm just checking shortly the Q&A and the session chat if there's already some questions dropping in. Maybe one short one from my side is what do you think will be the next big step with EBPF? What are the next reason development this direction? So it's really interesting. Working at iSurveillance, I get to work with people like Daniel Borkman who is maintaining EBPF in the kernel. And they've been doing really interesting things with things like support for, it's called big TCP, like big data packets, which is going to enable some much faster, well, higher performance networking. There are some interesting things around the way that the EBPF language itself is evolving, or the C that we use for EBPF language, EBPF programs is evolving. So yeah, the interesting thing is that it takes quite a long time for the new features in the kernel to actually reach production. So the versions of the Linux kernel that everyone's now using in production have certainly sufficient modern versions of EBPF capabilities that they can use all of the tools that I've mentioned today. But for things like these super efficient networking capabilities, we're probably going to have to wait for a year or two before those kernels are available for production use, unless you want to go and use a cutting edge, bleeding edge kernel. Awesome. Thank you very much. I don't see any other question coming up except, thanks very good, which is not a question, but like a statement, I would say. Well, I appreciate it, even if it's not a question. As Liz said, you can reach out to her anytime or drop some questions here. Because time-wise, we need to keep it rolling. Thank you again, Liz. Have a great day. Thanks for being here. Thanks. Bye, everyone.