 Welcome to the last afternoon of KubeCon. Somehow, they always schedule my KubeCon talk in the last day, but I'm very excited to present here and to introduce ourselves later. But the title of the talk is EBPF plus wasm. And I'm sure you have all heard of EBPF or wasm. Wasm is the next generation, isolation format. It's small. It's light. It's fast. It can work together with containers. And EBPF sort of works in the parallel universe. It provides observability for container workloads. It can also be deployed by containers and all that. So how are those linked together? What are the benefits of linking those two largely separate bytecode format, one in the kernel, one in the user space? So that's the topic we're going to discuss today. We're going to give you a new open source project on it's, I wouldn't say it's a new open source project, because it has been under development for over a year. But a new project that allows us provide a better way to manage EBPF agents and provide a better way to validate and secure EBPF agents through using wasm. And also, by having a user space agent running in the wasm, that's provided data analytics capabilities for the EBPF agent. So that's, I'm very excited to present this to you. And my name is Michael Yuan, and I'm the wasm in this talk. So I'm the founder of CNCF's project called Wasm Edge. So it's a web assembly runtime that's in the CNCF sandbox. And please. My name is Yishan Zhang, currently maintaining a small community called Inomia BPF and trying to make the BPF easy to deploy or use. And exploring some new technologies with the web as an EBPF runtime or tool chance. Thank you, Yishan. So I'll start. And to have an overview or agenda of what are we going to talk about in the next half an hour, right? So first, we have a very light introduction of EBPF in wasm, which I'm sure most of you guys know at least one of them, and probably both. And then we're going to dive into the topic of how wasm improve EBPF developer experience. It's all about developer experience. So we're going to go through the topics such as non-intrusive deployment into Kubernetes part using wasm to deploy EBPF agents. And the decoupled from application workloads, declarative security checks at deployment times using wasm application to check the security of the EBPF application. And then more interesting support user space EBPF, meaning there are EBPF-like applications that does not need access to the kernel, hence does not need access to the privileged user. You can use it in the user space to monitor things like network traffic and things like that. And downstream analytics of the EBPF data, that's another when an agent collects data, it goes feedback into some analytics tools and wasm provide a way to do the transformation of the data and analytics of data and allow data to be pushed to downstream analytics tools. And then that will be the bulk of the talk. But then perhaps the last five, 10 minutes, we're going to talk about how the other way around, how EBPF improves wasm developer experience. I know there's quite a few web assembly developers here. And one of the biggest challenges web assembly has, we talked about at Wasm Day at the beginning of the group count, is our way to guarantee security for wasm application because wasm provide isolation format that's similar to VMs and containers. Is our way to guarantee security and policies for wasm applications. And then the second is, how do we enable better debugging and better tracing for wasm applications? And we believe EBPF agents embedded into the wasm application, especially embedded into the system interface was it would provide a very good potential solution for those problems that we encounter at the wasm side as well. So this is a quick agenda. So here's a brief introduction of EBPF. I'm sure most of you guys, being this big cube count, you already knew that. You probably have already been using that. So there's use cases where then you develop applications, you develop using tools, and both development tools and deployment tools in the user space to develop EBPF agents. And then you deploy those agents to the kernel. And then it's start to do this work in the kernel, do the monitoring, security enforcement, and things like that. And then the data comes back and then you can analyze the data. So that's the general very high level overview of how EBPF is being used today. And then on the other side, on wasm side, EBPF is an agent you deploy, mostly commonly deployed into the Linux kernel. But wasm is almost exclusive. It runs on the user space, and it tries to run business logic, like database workload and application workload, and things like that. What we typically know as microservices, however, the chief benefit of wasm is a lot smaller than a container-based solution. So if you have a container application that connects to the database and turns the database result into some kind of web page, you could easily look at 100 megabytes, 200 megabytes of container size. But for wasm, you are looking at 1 megabyte or 2 megabytes. It's 100 times saving in terms of space. And it's also especially pronounced when you have computing intensive environments. For instance, we've seen a lot of wasm applications that use the AI ML pipelines, replacing Python, essentially. So instead of having three gigabytes PyTorch container image, you would have wasm application that is only 10 megabytes, 20 megabytes. And you would be able to run a large variety of, say, PyTorch models, or even larger language models like Lama, things like that. So from the surface, there are very different technologies. One is deep embedded into the infrastructure and to provide mostly ops. And on the bottom side, it's user space technology that handles business logic and application logic. So how do they connect together then? Let me. OK, so let's get to the first topic. So I'll provide a high-level overview first, and then you'll do the demo. OK, yeah. So that's the first big section. How wasm improves the EBPF developer experience? So to answer that question, we have to look at how EBPF is deployed today. So there's two major ways of deploying EBPF today. So one of them is tightly integrated into a control plane. So it's bundled into a larger application. So you have different frameworks that you can do that, Selenium and Pixie, and things like that. And then there's another way, which is what we call decoupled, or more like a sidecar EBPF deployment, where your EBPF runs its own process and communicate with other applications running on this host to sockets. So you have two chains like a bumblebee and things like that, a BPFD. So those are the two most popular EBPF deployment methods today. So the problem, the issue with the integrated deployment, however, I think the biggest issue is securities. Because regardless of what you want to do, EBPF has to be deployed in the kernel. So you have to give it a very broad set of permissions and authorizations, essentially. It has to run as a super user. That creates its own set of policy and security and those issues. And also, the EBPF program is typically very small, but you sort of need a very large framework just to deploy it. So it is not very efficient, especially if you talk about cases like in Edge Cloud, things of that nature. So then there's the other part of the integrated development, the other problem of the integrated development is on the development side. Because that was, so it is also somewhat difficult because there's new framework involved and things like that. It's somewhat more difficult to develop those EBPF applications. So then that leads to people exploring the other deployment models. That's a decoupled deployment model. So instead of having a large framework that manages all the EBPF agents and modules, you can have a decoupled setup, meaning that you run a separate process on your host machine that used to manage and deploy EBPF applications. So that's a process you can communicate with the rest of the containers in a host that can communicate with that process through sockets. So that gives rise to sockets and RPCs and things like that. That gives rise to, like I mentioned, a project like Bumblebee and BPFDE and things like that. But they also have somewhat challenges because those are a separate piece of software that you have to maintain that is outside of your main application workloads. It's not integrated by definition. So you have to have a separate way to maintain, update, and patch any problems that the community may have discovered with those. So it requires more work. So that's the second way. That's the decoupled EBPF deployment. So the solution that we want to present in this open source project is Wasm plus EBPF. That's really our goal is to have best of both worlds. So instead of having a Linux container that deploys the EBPF agent, we use a Wasm application, or we call the Wasm container, to deploy EBPF applications. And because we had an announcement in last year at KubeCon in Detroit with Docker to show that Wasm application can run side by side with container applications in Docker Compose and Docker Setup. And since then, there has been a lot of development to make Wasm a first class citizen in the Kubernetes cluster. So in the Kubernetes cluster, you can have virtual machines now. You can have container. And then you can also have Wasm applications that run side by side. So from the Kubernetes point of view, those Wasm applications behave like they are containers. But they are just Wasm applications. So their sandboxing is very different than the regular container. So the benefit of those is that they are much lighter and much easier to manage. So that's the feature we want to take advantage of, is to have the EBPF agent embedded in the Wasm application and being deployed by the Wasm application and have their data communicated back to the Wasm application. Instead of 100 megabytes, 200 megabytes container, that does this work. We want a very small 1 megabytes, 2 megabytes, couple megabytes lightweight Wasm application to do this work. So I think that's the same thing. So this is a project. So you can find that on GitHub. But I think, perhaps more interestingly, we're going to have a demo. So oh, perhaps we could do that. And you could do the demo first. And I'll go back to talk about other things. Is that OK? I don't want to have a demo. I'm sorry. My computer has some issues. So I didn't prepare for a demo here. So I just make some chief. This is a simple chief or a video demo of how to use the container tools like Podman to run WebAssembly and eBPF as it shows. We can use a Podman container to start and run eBPF programming WebAssembly, which you can trace the run to your latency. This is scheduled as a histogram for the new system. This program is supported from the BCC tools and compared in WebAssembly with our tool chance. You can fetch data from BPCF hash maps in a kernel, do some post-processing in your space, and then print the result through a console. The eBPF program is integrated into your space WebAssembly application, which can be packed into an OCI image and studied as a container with your own time. We can also easily seek in Podman WebAssembly containers and see a container we just studied and like a user Podman to list it and count it. Let's go to the developer experience. eBPF is widely used framework for developing eBPF programs in C and C-PAPS plus. This developer experience in WebAssembly is similar to that of the eBPF workshop, including automatically generated eBPF code framework and type definitions, just like a BPCF tool and live a BPCF, both have tasks. Right as the auto-generated skeleton from our WebAssembly eBPF tool, left-hard user your space code of load and attaching the kernel eBPF program. It was written in C, which can be compared into WebAssembly. Here are some basic examples. Let's take a moment to watch through it. First, we have like your problem of mobility and tracing. This is like setting up a watchtower inside your applications that you keep an eye on how functions are running without modifying them. And this is the tracing part. And XBB is for networking. This can be used to process, pick a lowest level before you reach the network stack in the kernel. Allowing you to fit or direct our drop packets as you need. And then we may have an LSM for security. This can allow you to set rules on what the system can and cannot do, like drawing a process from assessing a file or some network pod in the kernel. Let's take a look at how WebAssembly works. This project essentially wants to trigger what's on sandbox as a native usage runtime space on top of OS, which means we can use WebAssembly to develop and deploy eBPF programs as ordinary eBPF programs, but with the added benefit of security and ease of moving from one system to another. In runtime, a WebAssembly model can manage multiple eBPF programs and allow dynamically load eBPF programs from WebAssembly sandbox into a kernel, set the desired events like Uprobb and Kprobb test points to attach to, and our attention, control a complete lifecycle of multiple eBPF objects and support most eBPF program times. It can also communicate with WebAssembly and eBPF. It set up paths to backhand force conversation with kernel use eBPF maps like a ring buffer or perf event. The eBPF can send messages from the kernel state to use this via ring buffer and perf event or assess hash maps from the WebAssembly visual machines. They can also share memory between kernel and WebAssembly runtime. Now let's move on how we can create eBPF applications with some eBPF list called tool trends. To develop a eBPF program, we first need to compare the corresponding source code into eBPF objects like a eBPF get all using a client or LLVM tool trend which contains the eBPF by call and the corresponding data structure definitions, maps and program definitions in the eBPF format. This is where the eBPF works. It's inside the eBPF get all. Then we can use eBPF info to generate skeleton and bindings for your eBPF program development. This approach is similar to the component models in WebAssembly in which we can use like a WIC, WIC-Pengen, and another tool to generate bindings in C, Rust, and Go. Then we can develop your space programs in this language and compare and pack them to a WebAssembly model. With the support of big CF and code generation or communications between the kernel eBPF and your space WebAssembly do not need to go through serialization and deserialization mechanism. At the same time, the eBPF develop experience is just like a eBPF tool and live eBPF or stuff does. We can compare WebAssembly and eBPF models which is typically like the BCC tools like 100 kbits and they can be dimerically load and excluding kernel like 100 million seconds. It's optimized for rapid development and institutions. We have overcome some challenges before we can fully use eBPF capability within WebAssembly or Kubernetes. First, we have got some libraries for C, CPP, Rust, and Go, each enabling eBPF interactions in their respective languages. So they are for Rust, C++, and Go. We need to transfer this library to the WebAssembly version to enable and develop eBPF programs in this language with WebAssembly. And also another challenge is data layout. The WebAssembly is like three to bits and eBPF is like six to four bits. Data layout is different. So we use a tool transfer to generate bindings and all of these serializations between them. The last trend is the kernel compatibility. And the eBPF requires the kernel versions to support. We may use like compare once and run everywhere technology to enhance the portability of eBPF programs between kernel versions. And we can also look at some use-based eBPF runtime for optional or compatible layers for different kernel versions like we can for a ring bar frame per event. This is what a use-based eBPF does. We are also examining, this is a new project, we are also examining a new developer in system of survivability with interaction, the combining of WebAssembly and with use-based eBPF runtimes. WebAssembly with kernel eBPF unload a lot of potentials, allow us to engage deeply with the language kernel. However, it gets to come with the need of specific kernel versions, like a high kernel versions to support eBPF and our privilege to load the eBPF into a kernel. The kernel also need to config and enable the eBPF runtime. And the eBPF time, our new approach to eBPF like operating in your space. I mean, we can deploy this in eBPF tracing programs without special permissions and without depending on a linear kernel version, even not limited to a linear system. eBPF time may allow us to use eBPF tools with like YoProp and CISO tracepoints to monitor and tracing applications in your space. This tool are lightweight and don't require like stopping and compare the eBPF applications with eBPF runtime. YoProp are like your space counterparts to K-Prop, this is a YoProp task. It's inside a kernel application which allow us to trace your space functions in eBPF. The kernel also supports YoProp. However, the kernel YoProp is slow due to the overhead of context switching between kernel and YoProp. We have time to solve this problem by running entirely in YoProp with the advantage of being up to 10 times faster than kernel YoProp. Plus, eBPF time plays nicely with the same eBPF tool trends. It supports inter-process eBPF maps in your space or interacting with kernel eBPF maps and you can run family tools like BCC and eBPF trace entirely in your space without any change to their code. So we are looking maybe a more flexible way to gather inside and manage systems and also exploring potential inter-grading with technologies like eBPK and network tests with low-service eBPF. It's a step forward in making a powerful system tools more accessible and efficient. So let's come to my call. Okay, thank you. Yeah, so if I may summarize the point, so before we move on to the next topic is that, so why do you use Wasm to deploy eBPF? You know, three things, faster, easier and safer, okay? Faster also means smaller because instead of using a container that is hundreds of megabytes use Wasm, which is only less than 10 megabytes of the entire so-called container to deploy the eBPF application, you know, that's faster and smaller. And easier is that because the eBPF is now embedded in the Wasm, so now you can manage eBPF applications like OCI artifact because, you know, because Wasm applications can be stored directly in OCI registry and be managed by Kubernetes and Docker and such, right? So with that, you can have eBPF applications embedded in one, so essentially you can have OCI registry to manage eBPF application as well. And I also know there's a lot of work about Wasm registry, you know, like component model and such, you know. So, you know, that would also allow eBPF applications to be managed in those repository being versioned and be managed there as well. So this is easier. And also that allows, especially with the user space eBPF, that allows you to develop single applications embedded in Wasm that's, you know, portable across different kernels and also user space because the kernel eBPF is, there are some, you know, it's strongly tied towards kernel versions, you know. So there's, the portability is somewhat limited. However, if you just want eBPF to monitor, say network traffic and, you know, things like that, things that happen in the user space, you can have a lot better portability by, you know, having those eBPF applications embedded in Wasm, right? And then Safer, you know, that's also, the user space eBPF does not require kernel privileges. So you can deploy it as a user space application. So that makes it safer out of the gate, right? And also, because the eBPF is managed by, is deployed by Wasm, so you can have Wasm security model that applied to the eBPF stuff, right? You know, so, because Wasm has a, you know, what they call capability-based security. So, you know, so all the operating system access to the Wasm runtime has to be declared and configured. So you can have, you know, a very, a very declarative and a very specific set of security policies of what this Wasm application is allowed to do and not allowed to do in terms of deploying eBPF. And you can also, yeah, that's, you know, that's, I think that's enough benefits, right? You know, that's, you know, there's faster, easier, and safer, you know, that's, you know, that's why you need to, you know, I'd highly encourage you guys to try it out, especially if you already use eBPF today. You know, you know, instead of deploying from container, deploy it from Wasm. You know, that's, I think, you know, it's, you know, I wouldn't say it's a better way, but it is a better way, you know, it's a better way to do this, you know. So, I have five minutes left, so, you know, I'll go really quick about, because I think most of you guys care about eBPF, less about Wasm, so, but I also want to talk about, because, you know, because we have a Wasm runtime, so I want to show how eBPF internally improves the Wasm developer experience, right? You know, so, this is a survey the WebAssembly community has conducted last year, you know, to talk about what's the, you know, the need in the WebAssembly ecosystem, you know, if people want to write microservices on the embedded functions, you know, whatever, in Wasm, what do they need? You know, so, there's a very pressing need for better debugging tools, because right now, you know, without debugging tools, Wasm would always be more or less like a toy, you know, because, you know, you only find out problems once you deploy it, right, you know. So, this is actually one of the things that eBPF can make it, you know, well, it's debugging and security tools. So, I talk about those things separate, right? You know, one is eBPF makes Wasm more secure, because eBPF, it's, you know, it is the same as it's already do in the container world, right? You know, you have policies that you can use eBPF to filter out packets that, you know, that's, you know, that's violate certain security policies. You can do that, the exact same thing, with Wasm interface towards operating system. So, you know, so you can have declarative policies where, you know, even more fine agreeing than say, Wasm's own declarative policy. You know, so it's, it complements, you know, the Wasm security policy restricts what type of eBPF application it can deploy. And eBPF, when it's running, it can provide runtime and dynamic checks of a lot of those security policies, right, you know. So, eBPF makes Wasm applications better isolated and more secure. And on the other hand is that eBPF could provide debugging tracing for the Wasm runtime. So, you know, not only things that happens, you know, because a lot of the debugging needs come from, you know, when you interact with other systems, not just, you know, a stack trace of your own, and, you know, a function calling stack, but, you know, it's to say, you know, that's when you call host functions in a system. So, for instance, you know, we are using Wasm to do AI inference. It's frequently needed to call underlining the PyTorch library or the, you know, large language model libraries and, you know, things like that. And if the GPU access has a problem, you know, just, you know, to, you know, then it becomes really hard to debug because what you see inside of Wasm runtime is segmentation fault. You know, that's so, you know, you have illegal CPU instructions or you have illegal GPU instructions. You know, that's so, you know, I think with eBPF and you can manage those, you can look into those traffic and you can manage those traffic, I think it would be a lot, you know, it would give a much more compelling developer experience for web assembly developers as well. So, the main takeaway is that, you know, the first, that's the Wasm container, or Wasm application is a lightweight way to deploy eBPF on hosts in Kubernetes, right? It has the benefits of both the tight integration and the decoupled approach. You know, in a way it is tight integration because it's integrated into Wasm. However, it is also decoupled from your main workload, right? You know, so it's, I think it's a good compromise and it's a good use case of a lightweight sandbox like Wasm, right? And Wasm can use user-space eBPF. I thought that was very interesting because that's, like we said repeatedly, user-space eBPF does not require root access, does not require a whole set of security permissions. You know, it allows those, you know, the application that does not need kernel access to behave much more security, much more security and much better, right? And then started eBPF could become part of the Wasm implementation, you know, to enhance security and traceability at least in, you know, when Wasm is running in the Linux environment, which is, I think, you know, in the cloud, it's maybe 90% of the case, right? So, yeah, I think that's, we are right on time, you know, so that's good, you know. I'll go back to put on the link to the project. So, you know, we'd really love you guys to check it out and, you know, if you have questions. I think we'll have one minute, you know, so if anyone wants to ask a question.