 All right, now we are ready to talk about Kubernetes observability with the EBPF step-by-step. Let me quickly introduce our speakers. So here's Denis. He's the director of field engineering with Solo.io. A fun fact about Denis is all the demo he's going to demo today are all written by him, by himself. I'm the director of open source with Solo.io. I'm a very long time contributor to the Istio project, one of the founding maintainer there. A couple fun fact of me, I recently become a CNCF ambassador. I also run a HOOT livestream every week to chat about Istio envoy GraphQL, EBPF, a little bit about Solo, in case you don't know of our company. We are wearing our Solo shirts. It's founded by Edith Lavigne, who sits right over there. Our headquarters is in Boston. And we are the company specialized in application networking, API connectivity, and also... I was just saying, if you go there, you can see it, it will be... OK, thank you. We are the industry leader in that space. We're growing really fast too, and we are hiring. Oh, so what is EBPF? Given this is the EBPF day, we're not going to repeat to you. What exactly is EBPF? But I want to share with you my learning journey about EBPF. First thing, the first fourth thing that really caught my eye, when I started looking at EBPF last year, it's a sandbox environment. And it's extended. Thomas talked about EBPF allow us to extend the kernel, like JavaScript allow us to extend the web pages. And it has different hook points, which enables us to do the extension. And last but not least, it's supposed to be very safe. Like I mentioned to you, we have a strong background related to E-Steel and Envoy. When I look at EBPF, I'm connecting it to WebAssembly. How many of you know WebAssembly? Yeah, a few hands. So essentially what EBPF provides to the kernel is what WebAssembly provides to Envoy proxy to allow you to extend the Envoy proxy to do your own customer extension points. So EBPF have hook points. Envoy proxy has different faces for the proxy wasm, so you can extend your wasm binary into different faces. Then the next thing, as a user, I started looking at EBPF is why? Networking observability security, right? On the layer three and layer four. And this is very interesting. If you look at E-Steel and Envoy technology, you know E-Steel Envoy is also solving the exact same category, but we're solving a layer seven layer, right? Such as how do you do traffic shifting? How do you do RBAC control, service authorization? How do you do observability at layer seven? When I first started learning EBPF, I started with BCC, BPF compiler collection. How many of you actually wrote an EBPF program before? Yeah, a few of you, many hands, very good. So this is a great toolkit to help me get started with my first program. But I find a couple of issues with this. First, it's the user space program and the kernel program. They both written in Python. For me, I'm a Python developer, so the first thing I actually did is on Mac, I'm a Mac user, I tried to run that program, Python, hello world, that P-Y, of course that didn't work, right? Because it actually required the system have BPF program, have the right kernel, it has needs to have the right kernel header. So finding the wrong time, it's actually, it's a lot harder than I thought. I think I spent a few hours on this and finally doing Google search, finally finding the vagrant file that allows me to run my first hello world program. The next thing I look at is actually the presenter before. So I was talking about DBBPF and compile ones and run everywhere. How many of you love Java and Go that allows you kind of take the binary to run it everywhere? Yeah, many of you, right? So what's nice about this DBBPF is a program loader, right? It allows you to kind of write the kernel and use a space program in C and then you can compile them in advance. So as long as you have like kernel level, you can run them and execute the program. So the compiled binary has your eBPF program compiled so it's ready to go. With that, I'm going to pass on the mic to Dini to show us a demo on this. Yeah, thanks, Lin. So I'll try to do with one hand. I'll tell you. So obviously the goal here is, oh, where we go that? That's fine. It's okay. So the goal is not to do like a full demo of libbpf, right, but just because we are going to show you how we can do observability in Kubernetes, with eBPF, what we wanted to do is to just start by what is the foundation, right? If you want to do observability, you want to know which service talk to which service, you need to get the source IP and target IP of the network communications happening. And there is like this TCP connect example that is quite straightforward that you can use to get this data. And the first way you can do it by using like this tooling like libbpf and Cori. So I'm going to start with this and then Lin is going to show you how to explain what we contributed to the community with this Bumblebee project. And then I'm going to do the same program with Bumblebee and then we are going to do step-by-step ability from there. So what we have here is these two files, right? We have the kernel file here, right? That's this, what we want to, this is the eBPF program that we want to load in the kernel. And as you can see, there is a hash map that has the key is this struct. So the source IP and the IP address, value is the number of requests that correspond to this pair basically. And then what we want is just displaying this information, right? We want to display source IP, destination IP, number of requests. And just to do that, you need to write this complex program in C as well, right? So it's a lot of lines of code, if you think about it, just to display what I get from the kernel program. So I'm going to do that so that we have one working example. And after that, I'm going to, we are going to do the same in another way. So let's start by this. To copy these files that have, so let me go to this directory here. I'll do a lot of copy and paste so that should be good with one end. That will be my first one-ended demo live, right? So let's try. And so I just take these files in my current directory. And when you have, obviously, in this environment, I have all the prerequisites, so that seems very easy to do. The most difficult would be probably to have all these prerequisites, like the kernel editors and all the stuff. But I get them. So here, I just need to do a make TCP connect and just build this binary that has the user program and the key program that the user program has in it, and then executing this binary. And I should start to see some source address, target address, and the number of calls, as you can see here, right? And it will display every... So it's really basic foundation for what we are going to demonstrate about this QBNT's observability. So up to you now, Lin, to explain that we can do it different ways than what we explained with LibBPF, right? Can you guys hear me? Is the microphone good? Better? Hopefully. Thank you. So this really got us thinking, right? What if we could only write the kernel program and don't need to worry about the user space program? How many of you like Docker and the Docker experience? Well, Docker CLI can use push, build, right? So only four people in the entire room like the Docker experience where you can push, pull, build, and so on. That's what I can see, right? How many people using Docker today? Don't be shy, right? Okay, a little bit more. Docker is container to us, right? So this is where Bumblebee comes in. By the way, I know the screen is a little bit tiny, so if you scan that QR code, you will be able to visit bumblebee.io. So essentially, Bumblebee provides Docker-like experience to EBPF to you, to enable you to easily build your EBPF program, to be able to publish your program to your OCI registry and be able to have somebody else or could be you run the program by pulling the images from the OCI registry. So essentially what Bumblebee does is allow you to focus on writing EBPF program and Bumblebee provides the user program for you and generate the permissive metrics for your user pro, for your metrics, if it permisses format automatically for you. So it's a really nice way for you to get started with EBPF and start learning EBPF. Here at Solo, we find out learning EBPF is hard, and we spend a lot of time debugging EBPF program, so we decided to open source Bumblebee. In fact, a couple of maintainers from Bumblebee sit right in the table in front of me. Next, I'm going to pass to Dini to show us an interesting demo about Bumblebee. Thanks, Lin. So yeah, I think that's what Lin was highlighting is the fact that you have obviously some cases where writing a user program makes sense because you want to have like a bi-directional communication with the kernel program, but when it comes to observability, it's not that true, right? In terms of observability, we've seen a use case before where you can filter out some of the stuff in the kernel, but whatever you read in the user program, you just want to display it or to process it in a simple way, right? So that's why this idea of having the user program built for you. So let me go back here and go to the next steps. The way we use like instruct for that, and we have like a workshop where you can go through all this one by one, and we go through that, deliver that workshop very often. So if you're interested, just take a look at our website. So now, first thing I'm going to do is just getting this Bumblebee CLI. Oops. So just copy and paste this. And so the first benefits you will see is that I don't need to worry about prerequisites, right? I don't care about, oh, yeah, can you hear me? Yes? Okay, cool, thanks. So you don't need to worry anymore about your prerequisites, right? What you need is just Docker. It's quite easy, right? Obviously you have like some kernel requirements when you load it, but to compile it, it's really, you just need Docker, right? And you get this CLI, and then you could run a B in it to just have a skeleton created for you. I want to have a C program for network use case, and you will get a skeleton. Here, we are not going to do that. Here, what we are going to do, we are going to use another file that is very similar to the one before. So you remember before we had this kernel program here. And now we are going to use this one just here. It looks the same, right? And to show you that it's really similar, I'm just going to do a diff here of the two files. So the file that we used with libbpf before, at the file we use right now. So let me try to paste this, no? No, that's fine, thanks. That should be, thanks. Just trying to understand why it doesn't do like a copy. That's always this funny. I got like some strange demo effect sometime, but never like the copy and paste not working, but no worries, we'll find it. Perhaps it's the Wi-Fi that is not very happy again. So the good news, I have a plan B already in place. So give me one second. I'm switching to my phone, and that should be better. Yeah, it's always the Wi-Fi. Yeah, smart switch, let's go there. Cool, it will not be as fast, but will be more reliable, I think. Okay, so I need to export this again, paste it, and then I should be able this time to do a copy and paste, hopefully, copy and paste. Oh, yeah, it was really just long, okay. Cool, so what's the difference, you see? Really minor, right? So what I want to highlight is that you can take whatever source code that you have for LibBPF. You take just the kernel program, you forget about the user program, you take the kernel program, and you just have this naming convention that if I want the user program to treat this hash map as a counter, I just add dot counter, that should be, everyone should be able to do it, right? So it's really straightforward. So let's do it now. I'm copying this file to my current directory. I love this network. Let me try to do in this one, yeah. And I'm going to run this command. You see, it's a B build. So basically it's like, you build this file. Oh, yeah, I am not having this in my, pass here, copy, bear with me, we'll do it. So you see, I just say I want to build this program and you can see the format of the second argument is like a really like a Docker image, right? Like an OCI image, right? So localhost 5000 means that I run my registry locally and solo is my repository and TCP connect my image name and V1 the tag, right? And if it's like localhost 5000, it because obviously I have my registry running locally in my machine, right? So hopefully my network will be able to give me, move me to the next step. That will be challenging with this network. Let's try again. So, okay, export this again. Now it is built anyway, so that's fine. So the next thing I want to do when I build it is to push it. You know, it's like the Docker experience, right? You do a build with a Docker file. Yeah, you do a build with like a BPF program and then you do like a B push, which basically will push this to my registry that is running in my local machine here. And finally, I can do a B run that will simply run this program from my image. And you can see very similar to what we've seen before, right? A little bit nicer UI, but I will not tell you that value is to have this UI. The value is just that I didn't write any code, right? To get this UI. And the thing that I really like about it, in fact, is not really the UI, per se is really the fact that it exposes automatically all this data as promoter's metrics or promoter's format metrics, right? So now I can go to this endpoint and I can see the data, the exactly the same data that I can see in my UI here available as metrics, right? So you remember, we said the goal is to show you this Cubase observability from the beginning. So we have achieved the first piece, right? We have now a program that we can load from anywhere and that can give us information about the source and the target IP. So what we are going to do next is pretty simple. We are going to run like a demon set that is going to deploy this program in all of my Kubernetes nodes so that I don't gather just the information from my local machine, but I gather the information for all the different Kubernetes nodes. Then we are going to use promoter's to capture this metrics and we are going to display a nice graph at the end. So to do that, we just deploy like a demo application. A lot of people are probably familiar with this Bookinfo application. If you are not, it's very easy. It's just like an application that is composed of multiple microservices that talk together, right? So just if you want to show a graph of services communicating together, you need to have these services. So that's why we use this example. And if I look at these Qubectl get pods, I use the minus O wide. It's just to show you that we have these containers that are deployed in multiple nodes, right? You see master worker one and worker two. Just to show that we are going to be able to display a graph with these communications happening in different nodes. And the machine where I am right now, you see it's called root at master. So I am right now in SSH of the master where I have this image that I pushed to the registry. You remember, I told you about the registry. So if I do a Docker PS, you can see here that I have my registry running, right? So my BPF program is already loaded there. So next step is deploying promoter's because I want to persist my metrics, right? I just don't want to have them in real time. I want to persist them so that I can build a graph as it is now, but I could build a graph as it is in the last like one hour or two hours or whatever like that, right? So I'm going to really do a very simple deployment of promoter's, right? So really the basic community version of promoter's. And after that, I'm going to deploy this Bumblebee image as a demo set. But what will be surprising for you at the beginning is that I'm not going to run the image I built because it's not a Docker image that I built. What I built is an OCI compliant image that contains my BPF program. So I'm going to run a demo set where it's a Docker image that has this BCLI and the argument of this command that it's called B is where it can find my image, right? So it's not that you run the BPF program as a Docker image, right? It's really like the B program itself runs as a Docker image, right? So you can really distribute your, you can build your BPF program and then you can distribute it through registry. So if you have OCI registry, which is the case of most of the people today, then you can use it to distribute your program across many different machines. Okay, that's not what I want to copy. So it's like a second act of this copy and paste problem. Let's try it again. Yeah. So I run it as a demo set and I create the pod monitors for people who are familiar with promoter's. It's a way to tell promoter's, go and scrap the metrics from this pod, right? So I have this demo set, so I have one pod per cluster or per node and it expose these metrics like I have shown you before, right? It expose these metrics natively. So I'm just going to tell promoter's to go and capture these metrics here. That's kind of the modern way to do it. A lot of you perhaps are more familiar with like a annotation that you put in the pods. That's like the old way and the new way is really like you create a pod monitor and you declaratively say what you want to scrap. And finally, I'm going to generate traffic, right? So I'm just going to go into my front end of the Bookinfo application that's called product page and this product page will call the other services, right? So just generating some traffic because now that I have my EBPF program loaded on each node, I have my promoter's cluster that capture this data. Now I want to generate traffic, right? So that I have data in my promoter's cluster. The last step is I'm going to, I built a very small program, a small UI that does two things. It's connect to the promoter's cluster to get all these metrics. So it will get things like source IP, destination IP, number of requests. But the problem is that nobody cares about source. If I build you a graph that says this IP, talk to this IP, you don't really care, right? What you want is a pod talking to a service, right? Because if you look at the source and target IP, the source IP is a pod IP in the Kubernetes world and the target IP is a service IP in general in the Kubernetes world, right? So what this program will do is that it will also connect to the Kubernetes API server to correlate all this data, right? So that's why I create this cluster role binding to tell that my program is allowed to talk to the Kubernetes API server, basically. And finally, I can deploy this demo application that is going to make the magic happen, hopefully. So I paste this here. I can take a look at my pods running here. So it's still creating it. But you can see already the Bumblebee program, you see three pods because I have the master, the worker one and worker two, okay? So again, just waiting for it to be ready. It's running, so now I can go there and that works. So I can see here, basically, that I have like, so let me like change again. And in fact, what's funny is that if I refresh, I will have even more data because I will have the program itself trying to get the data from Prometheus and so on, right? So you can see here, for example, for people who are familiar with the product page, I can see the product page talk to, or the product page pod talk to the service review, which is composed of these two pods that call to the rating service. And I can see, for example, here, right, that I have my KBPF program, the one that displays everything, that talk to the Prometheus server here and talk to this Kubernetes API server to be able to display this graph, right? So that's it. I don't think we have a, do we have a closing slide? I'm not sure. Let me check, but I think that was mostly it, right? That's what we done and across nodes that just give you an overview of it. And as I said, we have this workshop where you start from zero and you do that by yourself. So don't hesitate to register. We run that like every month across different time zones. Thanks, everyone. Yeah, thanks, everyone. Great job. Shout out to Dini on an awesome demo. If you are interested to do the demo on your laptop, check out solo.io slash events. We offer these badges every month. Also, check out Bumblebee using the QR code. Thank you very much. Right, who has some questions for Lynn and Dennis? Question, question. Must be some questions. All right, I'll ask a question to you lot. Has anybody here already tried using Bumblebee? There's a hand. Yeah, okay, good. Thank you. All right, last call for questions. We got one question. Oh, great. Hello. Are you familiar with the EBPF exporter from CloudFair? No. Yes, a little bit. We've read it at the GitHub page. I haven't personally tried it. Yes, I think what this really adds on top of that is the idea behind packaging it as an OCI image and pushing it to an OCI registry. But I think the underlying concept seems to be very similar to me, and I'm really excited to see this. But one question that I had on this was, when you're deploying it as a demo set, do you have to, for every single program that you write, do you have to create a separate demo set for it, or can you package multiple ones together and say pull this set of images and just deploy that as one whole? Can you hear me? So I think that what we did here, like the demo set and being able to show you the Kubernetes thing was just an excuse to show you a real use case instead of a program that would just display some boring information, right? So the goal of the project is not to simplify the way you are going to deploy in production or managing the lifecycle of this program. The goal of the Bumblebee is really to simplify the way you can go from writing your code to building your code to making it available for production, right? And what you're asking about is probably what's happening just after, right? I have now my image in my registry. What tooling I use to run these different things in parallel or whatever, but this is not something that is in the scope of the project right now. It's like if you look at Docker, right? You will push your image and then Kubernetes came to try to solve all these other problems, right? So it's kind of the same. Perhaps there will be a Kubernetes of EBPF because there will be so many EBPF program running in the same machine that you want to tackle all these things, but that's definitely not the scope of the project today. Another question? Yes. Thanks. I really love the way you do with the OCI compliant image for distributing, but my question will be some kind of mechanism to check them, maybe to sign them to say, okay, so it doesn't become something like I can download whatever thing malicious EBPF programs, for example, to say, will there be something provided because now you are runtime and I can already imagine my developers going crazy, downloading stuff like in the Wild Dogger times, things from strange dude 200 and run it on a production cluster. Yeah, that's a good question. I think that's why we, basically what we expect is people will use like a public registry when they play with it and they want to try out stuff in their laptop, right? But we don't expect people will allow users to run packages coming from a public registry and it's quite easy to block that, right? So we expect customers to use their own local artifactory or local registry that is OCI compliant and to manage that kind of policies, right, yeah. I feel like I should also add that there is some work going on in the BPF community around the signing of BPF programs. I think that's definitely a work in progress. Just to add to that, because we use OCI, you can sign it like any other image. It would be nice when the runtime would check this and say, okay, is it signed? Who signed it? And then to be sure, as you said, only corporate signed stuff can be run, so you protect this in before. Yeah, I think that's what the kernel BPF signing work is about, yeah. Question, any more questions? Yes. Do you plan to support any other languages than C? Yeah, I think we discussed about Rust as something that was asked, but I mean, it's definitely an open source project. We want people to participate, so feel free to open issues there if you have preference and so on, right? That's definitely the idea, I think. Yeah, you wanted to add something? So maybe I would add, I'm indeed the founder of Solo. I mean, this project idea was to help the community, right? We, using it internally, we thought that this will help a lot to the community so we open source it. The purpose is to bring it to the CNCF. We are in the process right now to basically put it as a project in the CNCF. We are not planning to keep it for ourselves. So as I said, please come help us make it successful, okay? Any more questions? Okay, well, thank you very much. I think that's all of them.