 Do you all have a laptop with you? OK, so if you do have a laptop, this is a workshop. You are welcome to follow along with me. If you do not have a laptop, there's no problem. You can watch me run the workshop, and you can watch me live, which hopefully it's as close as you do it yourself without a laptop. All right, let's get started and learn some EVPF together. So before we get started, I'd like to quickly introduce myself. My name is Len Sa. I come from the United States. I live in the east coast of the United States. North Carolina is a state. A little bit down Washington DC, Virginia, somewhere between New York and Florida, if you know where New York and Florida is. I have been an Istio contributor and one of the founding members of the Istio project. So I've been working for the past six or seven years on the Istio project. I actually wrote two books about Istio. So right now I work as solo, leading the open source development team. So by the way, apologies, I've been standing right here. And part of the reason is there is an elevated section. I don't want to fall back and forth. So I'll be standing here. Apologies, I actually don't like standing behind a podium. So I wrote two books about Istio now working as solo. Before I joined solo, so when I joined solo, that was a little bit over two and a half years ago. And before I joined solo, I worked at IBM. And I was a master inventor at IBM, a fun fact about me. So right before I leave IBM the day, I took a screenshot of how many patents I contributed for IBM. So that's 207, which means I was a co-inventor for these 207 patents, which, of course, IBM owns them. But it's nice to have my name on. So that's a little bit about me. How many of you know of a small company, Solo.io? All right, 102, thank you. So it was founded by Edith Lavigne in 2017. So I joined solo in 2021 right in the middle of COVID when we were talking about resign from jobs. I was part of the movement as well. I decided to try something new. So I had working at IBM for 19 years. I decided why not try something new? Because stuck at home was very boring during COVID. So we are a very fast growing company focused on API gateway, application network connectivity. So now we're going to jump into EBPF. So how many of you know what EBPF stands for? All right, EBPF stands for Extended Berkeley Package Filter. EBPF is known to be very flexible. It can execute your extension, custom logic inside of the kernel as a sandbox program, which is super cool. Because if you think about Linux kernel, the innovation of Linux kernel is not as fast as many people had hoped. And also the fact of the fix landed in the Linux kernel doesn't mean it's going to be landing in your Linux distro, right? So that's normally like a few years gap. I remember somebody did a statistic on the PR for Linux kernel. I think only a third of the PR actually gets merged into Linux kernel because the maintainers have a high bar to get those fixes of new feature enhancement into Linux kernel. So EBPF really play a critical role here to allow you to easily extend beyond what the kernel can offer. And without needing to wait until the PR gets merged into the kernel, without needing to wait until it landed into your favorite Linux distro, and then you have to upgrade to that version too. EBPF is designed to be very safe. The code is verified by the kernel as it's been loaded to make sure it doesn't crash, make sure it doesn't harm the kernel. That's why it's running within the sandbox environment. And it's supposed to be very fast with JIT compiled and native speed. Many of you probably using EBPF at one point. How many of you have used TCP dump? All right, I'd expect most of you, if not everybody, right? So that's the humble original of EBPF. EBPF these days have been used in a variety of user cases, right? Folks to start using EBPF for security to make sure the certain communications are secure for tracing observability to observe what's going on. In fact, we're gonna cover some of that in our lab. For network policy enforcement, right? Some of our most popular CNI, for example, Cilium, and even Collicle recently also added EBPF support to do network policy enforcement. So a lot of functionality related to EBPF to innovate around the kernel before the kernel has these functions. This is one of our favorite diagram around the EBPF. It really explains really well the difference between the user space and the kernel space, right? Typically, EBPF program has a user program that allows you to generate the BPF bytecode and to be able to load that bytecode into the kernel. And then once it's loaded into the kernel, on the kernel space, kernel is going to verify. Remember we said EBPF is designed to be safe, right? So there's a verifier, verify it will not crush or hang the kernel. And then you can attach your program to different hook points. So the kernel is going to attach to different hook points such as K-probe, U-probe, and trace points. And then part of the connectivity corridor between the kernel and the user program is the maps. The maps that you designed to be relevant to your EBPF program. So the kernel would put some of the useful data you may need in the map. And then the user program can read those maps and maybe display to you or log it out for you for whatever format you want to consume it. So that's how in a nutshell, the difference between user program and kernel program, kernel space program and how they interact with each other. So why is this important, right? Why is the EBPF so important? I think early on we kind of touch on the speed of innovation before your fixes, the features can be added to the kernel. The other thing about EBPF, it's really designed to accomplish some of the tasks that's not possible to be done by outside of the EBPF. Because before, for example, with IB tables, right? You have to kind of do a lot of stuff in the user space and also have some interaction with the kernel program. So there are certain tasks and scaling issues related to that. And EBPF is also designed for multiple personas, right? So from an application developer to SIE, to developer engineer and network operators, EBPF will have a role in there, right? Application developers can write EBPF programs for developer engineer and SIE to consume. Maybe, for example, observe what's going on within the kernel and some of the kernel programs. And the network operator can use EBPF programs to enforce network policies. So it has a wide range of benefits to different personas. If you have ever started EBPF program, the first resource you probably run into is the BCC. You build your first EBPF program using with BCC. How many of you actually try to write a simple EBPF program before? Awesome, yeah. Were you using BCC or something else? Okay, yeah. So BCC is Linux BCC EBPF tracing tool. So it has a bunch of toolkit that you can use and you can call. It's the first thing most people starts to learn as they learn EBPF programs. One challenge, I also went through this two years ago. I find out using EBPF program with the original BCC tools. It's a little bit confusing. Essentially, most of the example shows how you can write using a simple Python program. I think you can also write in C as well. But if you're using a Python program, essentially in that program, you're gonna have the user space program and also the kernel program. And what's the difference though is the user space program is written in Python and then the kernel space program is written in C. You might be wondering how would that work, right? How would the compiler work? So what essentially has is the user, the kernel program written in C is kind of like a string. So that's how the Python treats it. So you can't do much validation or compiling with your compiler with that kernel program in C using the Python compiler. So during execution of your simple EBPF program, typically it calls the Khan LLVM, which performs a kernel header lookup. You know, check if it's the right kernel version and then compile your code at runtime, which is very expensive, right? Because it's resource heavy. You actually, before you run it, you have to compile it right at runtime. So this is very resource heavy, especially in the production environment. Now the newer ones that comes just recently in the most recent few years is building your EBPF program with BPF query. Query stands for compile once run everywhere. So what this means is if the developer actually write an EBPF program and then make it offer to you to run it, you don't have to compile it, compile it, right? So it would compile it ahead of the, the developer can compile it for you and then you just take the binary to run it. It also provides the BPF as a user space BPF loader so you can use it as a library to load your EBPF program. It continues using Clon as the compiler and it has BPF type information in the VM Linux headers. So this means the kernel and user space program are both written in C and you can execute it on any system. While the Gatsch is, the system has to, on a particular version that supports BPF query. I think it's the next kernel of 5.6. If I remember correctly, but don't quote me on that. So do a little bit of Google search. You are now for sure. I think it's 5.6. So now the next question is that, okay, so you still have to write user program, user space and also the kernel code, right? Imagine what if you only write the kernel program so you don't have to worry about the user space code, right? Would that be possible? So this is where I want to introduce a open source project called Bumblebee. So if you go to ebpf.io slash applications, which we took a screenshot of that website and that essentially lists all the EBPF open source projects out there. Bumblebee is a project started at solo.io which will contribute to the open source ecosystem to help user to learn EBPF and also bring the Docker-like experience to all the EBPF developers. So how does it work? So essentially with Bumblebee, you can, by the way, you can visit Bumblebee.io to visit our website. You can build EBPF program and build it as OCI images and then you can also publish your EBPF programs as the OCI images into any OCI compliant registries and then you can also run your EBPF program as OCI images using our runner. So that's exactly the favorite Docker experience you guys are familiar with, right? So you build your images, you push to the registry and then you can run your Docker images. So that's why we believe what Bumblebee really provides to the EBPF community is this Docker-like experience to you, for you to learn and build and publish and for maybe your other friends and families and coworkers to consume your EBPF programs. So at the nutshell, Bumblebee helps you to build run, distribute your EBPF programs as OCI images. Also last but not the least, Bumblebee can also expose the events as metrics for you so you can hook it up with premises and potentially even visualize your metrics with other observability system. Bumblebee leverages the EBPF kernel space code and take existing EBPF kernel space code into metrics. That's how the metrics works, which we will also cover in the lab. Right now, Bumblebee relies on the query we were just talking about, BPF query. So you have to use Bumblebee on relative neural NINX kernel that supports our query. All right, so let's talk about how you can potentially migrate some of the existing EBPF program that was originally written to work with BCC to leap BPF and work with Bumblebee. So in the lab, we're going to talk about home care, out-of-memory care program. So in this program, remember we talk about with BCC, you typically have the user space code, right? You also have the kernel space code. So the first program is the user space and the second one is the kernel space code. Also, in this program, our engineer was written to use HashMap, which is not as perform as RIM buffer. So we're going to take a little bit, small surgery to the program to migrate the HashMap to RIM buffer. And also, the initial was using the perf buffer and then we're going to also migrate to RIM buffer. This is because RIM buffer is more performant. All right, let's take a little bit, dive into perf buffer was RIM buffer, right? From a resource usage perspective, RIM buffer allows you to use shared buffer. So instead of using buffer per CPU, you can use shared buffer. So it's more, it can be saving you some resources. From an event ordering perspective, you can also use shared buffer. From an event ordering perspective, the BPF events, the events from the kernel sometimes have a mini seconds of delay when you have multiple CPU running. Sometimes it could be a little bit out of ordering. So with shared buffer, you no longer have that problem even when you have multiple CPU. RIM buffer also provide a nice reserve and submit API. So that's pretty easy and developer friendly to use. And the only requirements is Linux 5.8. So I probably said it wrong on the requirements earlier. Yeah, so 5.8 is the requirements to use RIM buffer. If you're interested in the details, check out the links. So let's jump into the workshop. Before we get started, I want to give a quick overview of the workshop. So the first one, we're going to build and deploy BPF applications with Bumblebee, right? So we're going to build the application using Bumblebee and then push it to the OCI registry. And then we're going to run the programs. And then we're also going to explore the kernel events and view it in the premises UI. So you can view the metrics that we've collected for you through Bumblebee. I think it's the third level. We're going to debug and compose multiple kernel space code and ship them as a single OCI image. So we're going to talk about some of the pros and cons why you want to combine multiple EVPF programs into a single OCI image. And then we're going to jump into some of the DBPF helpers, which we probably don't have time on, which is the last lab. All right, let's get started with the lab. So what I want you to do, if you do have a computer, go to bit.ly-bit.ly-develop-bumblebee eBPF-apps. So when you click on this link, what's this going to do is it's going to ask you for your user name, company, and email that we can send you additional information about the program. And also if you do a certification badge, which is completed free, we can send a badge to you as well. So once you provide that information, you should be able to get into this view, which have the content of the lab. By the way, a gentleman there from Newstack was just asking me if he doesn't have a Linux machine, can he still do the lab? So the answer is yes. If you don't have a Linux machine, what the lab does, by the way, let me click on the stop button first. So what the lab does is we basically contract a third company instruct, which we pay for them to provision VMs in different clouds. So they are smart enough to think out which cloud have resources that's close to you and then deploy a virtual machine for you. So you essentially, as long as you have a browser, you are interactive with a virtual machine in the cloud. And then that virtual machine would have the right Linux version for you. Is that a question? Oh yes, good. The link, yes. Let me do that, great question. Yeah, so bit.ly slash develop dash ebpf dash apps. Yeah, let me launch it on both screens. So you guys have it too, I can also monitor here. Wait for a few minutes, just making sure everybody, we will wait a little bit, just making sure everybody have an environment. The way the lab works is I will run the lab live with you. I know some people likes to do the lab kind of as I was talking, but you don't have to, at the end, we will give you a little bit of time to do the lab if you prefer to kind of watch me first and then do the lab yourself. So I will add a few minutes at the end, just making sure, because I know some people prefer to do the lab. All right, does everybody have the URL? Raise the hand if you still need the URL. All right, so it looks like everybody have it. I'm going to jump into the screen then. All right, so this is what you are going to see at the beginning after you type your information. And click on the start button. You can click on the start right here. That would jump right into the lab. So I'm going to wait a little bit as you guys have it. All right, does everybody kind of click the start button to kick off the environment? Yes? It's broken for you? Okay. Yeah, the Wi-Fi is very slow. Yesterday I was in a room doing a live demo. I ended up using my hotspot. So if you do have a hotspot, I would recommend you to use it. Today I learned my lesson, so I actually have a dedicated cable now. But I don't think they can offer that to everybody. So I'm going to go ahead and do a quick demo. But I don't think they can offer that to everybody. But if you do have a hotspot, I would highly encourage you to use it. The Wi-Fi, it is very slow. So do you also have a problem? Okay, so I think the Wi-Fi is a little bit intermittent. Maybe it was not so slow, but it wasn't consistent to me. So how many of you actually click on the start button to get the environment going? All right. So good luck for you guys, folks. I'm going to get started then as some of you are waiting for environments. Okay, so in this lab, we're going to detect out-of-memory kills with the help of EVPF. Is the size okay for folks in the back? Good. All right, thank you. I appreciate that. So out-of-memory manager, the original dates back to the old days of Linux and the out-of-memory manager. So when your instance is running low on memory, older pages will be reclaimed. If this is not possible, a function called out-of-memory will be called with a single task, basically to avoid running out-of-memory. In the case of Kubernetes, the basic out-of-memory also exists. So if you are running, I come from a cloud-native environment. So in Kubernetes, sometimes we see out-of-memory parts go out-of-memory, and Kubernetes keeps restarting the parts. So that's also because part reaches the limit for the container. Or maybe the node that the part is running gets over-committed. So monitoring out-of-memory events, it's actually really critical, right? Because that enables you to operate your workloads reliably and making sure you are giving the right CPU requests and reserve the right CPU memory for your parts or applications. Catching out-of-memory, on the other hand, is not trivial. This is where EBPF can help. So in this example, we are going to first take a look at the upstream deep-BPF tools from BCC, which they have out-of-memory scripts. So this is the code, which is really, really simple. It has a basic struct. And inside of the struct is have a perf event, and it has the key and value for it. And it's hooking up to the out-of-memory keel process. So whenever there is a out-of-memory keel process event happening, it's going to try to log the process ID, the group ID, and the pages. When that happens, the page is being used by the program and then kind of write the contents out into the map. So that's essentially what this program does. The kernel version is very, very simple. If you actually go to the user space portion of this program, it's actually not so simple, because it has to load the code, the EBPF code into the kernel. It has to handle the events. It also has to think out how to print it out or give some output to the user, so the user can think out which program has the out-of-memory. So that's the basic code about out-of-memory keel for the EBPF program. So in this example, since we're working with Bumblebee, what we are going to do is we're going to do not using the user space program, which we just showed that's more than 100 lines of code. It's a little bit more complicated, and what Bumblebee does is kind of save you from writing the need to write a user space program code. So you can focus on the kernel code. So let's get started. So I'm going to go ahead and download the out-of-memory keel BPF program from the BCC website. So once it's downloaded, I am going to look at the difference between the out-of-memory keel program and also the same program that we modified for Bumblebee, the data slash steps slash om keel directory. So this would show you the difference between the two programs, which the primary difference is around the changing form that we covered early, the change from perf buffer to ring buffer because ring buffer is supported by Bumblebee and also it's more performant than perf buffer. So if I launch the program with WIM, hopefully it's a little bit easy for you to read and for me to read too. So on the left side is the om keel BPF program we downloaded from the BCC site. On the right side is our new BPF program which is also om keel, but it's optimized to use the ring buffer. So let's go over the difference quickly. I think for the include, we skipped that because what we include was simple. So we just inline it in the code here on the right side as a struct. And the key difference on this struct here is it was using perf buffer. Now we're changing it to use ring buffer. So as you can see, the perf buffer we are using with ring buffer we're using max entries and also the value of the data. One thing about the struct I want to point out is we actually changed the name of the struct before it's called events, now we call it om keel. And the reason we do that is because when you have a meaningful struct name, Bumblebee can show it much easier so you know which well is the metrics come from which map. And Bumblebee supports map.counter which is what we are using. I believe Bumblebee also supports map.gouch but for now counter it's our purpose and scroll down a little bit into the BPF K probe method. The change is essentially just switching from perf buffer to ring buffer. As you can see with ring buffer you can reserve what you need which is really nice and you can then submit. So that's the reserve and submit API we talk about. The other differences I think we are working with a pointer here for the struct so that's some of the semantics a little bit different on how to read the data. So that's the key difference between the two. What I'm going to do next is I'm going to exit out of the two the WIM diff and what we are going to do is install Bumblebee. So let's go ahead and make sure I didn't miss any steps. To install Bumblebee it's really simple so you can download Bumblebee from our website the latest version and then you can put it into your user being local directory so Bumblebee is in your path. Now what we are going to do next is we are going to remember we reviewed the data steps on omkill and omkill.c program so we are going to copy that program to the local directory so that we can easily build it with Bumblebee. Remember we talk about Bumblebee can help you build EBPF program so much easier so right now we are running the build program to build the particular program EBPF program omkill.c and we are going to say I want to build it for as this image name so that I can push it to my registry which I happen to have a registry local. All right so right now it's compiling the EBPF program using Cori that we talked about earlier so this has a dependency on EBPF and also relies on a newer version and it's kernel 5.8 and you can see it successfully compiled the program and yeah and it did create this OCI images for us the next thing we can do is let's go ahead and push the OCI images into our local registry which runs on localhost 5000 yeah we did push it into our local registry now we should be able to run it let's go ahead and run it all right so this is the Bumblebee simple UI so what this UI essentially shows you is display the omkill remember that was struct that's the ring buffer in our program it's going to display everything going on with saying that that ring buffer so right now we don't have any out of memory events so let's generate some so in order for us to generate all the memory events what we are going to do is leveraging we're going to try to run this in Kubernetes so we're going to deploy a demon set so first we're going to deploy Bumblebee into our Kubernetes cluster as a demon set and inside of this we're going to configure to run without the UI which is this node that's TTY and we're also going to configure to run this ebp of OCI images which is build omkill version 1 and we're going to configure to as a demon set it's going to deploy into every single node of my Kubernetes cluster which you can see I have three nodes two worker and one master so let's go ahead to deploy this demon set into my Kubernetes cluster and if you do get a pause you should be able to see the status of Bumblebee it's rich running already so it looks like our demon set is deployed the next thing we're going to do is install premises remember we said Bumblebee can help you generate metrics automatically in this case it would generate metrics related to omkill so we're going to install premises so you can view these metrics in premises so we're going to use helm to install cube premises stack which have the entire stack of premises and what helm does is essentially you know download the right chart and install it as Kubernetes deployment services inside of your Kubernetes cluster so if you run getPause-A again you can see right now it's having a bunch of premises related contents because we install the entire premises stack alright so now what we need is actually the pod that can generate out of memory so before we do that we need to have a pod monitor to scrape our Bumblebee pods so the metrics can be script properly in premises so let's go ahead and deploy the pod monitor so as you can see the pod monitor is configuring slash metrics and every 15 seconds and it's scraping the Bumblebee pods which has app equals Bumblebee as the label alright now let's go ahead check premises UI right so right now we don't have any data to curate yet that's because we don't have any events related to ARM keel so now we need to finally deploy a test application so in this test application we're using a memory leak can image from AWS solution and we purposely fully set a very small request and limit the memory right so 64 mag and 128 mag because this image that container runs this image is going to continue leak memory so it's going to hit out of memory pretty fast in theory alright so we got this guy installed let's take a look at the pods on the Kubernetes system so you can see we already have out of memory keeled events and we also have a restart so essentially what Kubernetes does is it have detected out of memory keel events and also try to restart a pod see if it can recover after it starts right so now if we go back to the Bumblebee UI you can actually see some events actually happening which is super cool right so it actually tells you the PID ID of the process that's causing out of memory it also tells you how many pages it was using when out of memory events was generated what about premises can I view the metrics in premises so let's go ahead to see if we can query the premises so let's see alright so this is the expression box and that's execute ARM keel events so you can see remember that EBPF ARM keels that ring buffer name we gave so that's how you know when you have multiple ring buffer or maps that's how you know this is related to ARM keel so it's important to give a meaningful name and it captures the problematic part right so it says this Bumblebee container is reporting in the default name space it's reporting this this program called stress and this is the PID ID of the program and this is the page memory page when it has the problem so it captures all that information for you which is super cool and if you see right now we have one, two, three, five of them and if you do a part here we have four of them and four restarts so that matches five out of memory here it also matches what you see on the UI which is also five of them all right so that's how the ARM keel program works in action so what we did is we kind of we kind of using B program to build the ARM keel program we changed slightly updated for Bumblebee so this is without the needing to have user space code so we built the program and the post registry we run it we also deploy the Bumblebee in Kubernetes we as a demon set and we kind of deploy premises and premises part monitor for Bumblebee so it can report metrics and script to endpoints and reports the out of memory events metrics on the premises so you can see it on the UI with that I'm going to wait a few minutes so you guys can do the lab as well questions yeah so I think if I heard the question correctly is what does Bumblebee offer that premises didn't okay what does premises offering that Bumblebee didn't offer right so in this example we run Bumblebee by itself which I didn't have to in fact we kind of run it on the because our Kubernetes is also running on this particular node on this particular VM that's how we were able to observe out of memory events right so essentially what Bumblebee provides is the ring buffer that captures these events from the kernel right what premises provide is to be able to scrape these endpoints and be able to kind of get it on the UI so you can view them premises would not have these metrics without Bumblebee yeah so in theory though you don't need both in my example I happened to launch Bumblebee because I was showing you guys how to run Bumblebee but in this example for instance right now I killed my other Bumblebee right now we have five restart so that means I'm expecting six metrics so if I do execute here I have six metrics so you will know without the other Bumblebee program you know I can still visualize everything in premises that's because I am actually running Bumblebee inside of my Kubernetes cluster remember we kind of took Bumblebee and said we deploy it in Kubernetes cluster and when we launched the Bumblebee Demonset we said I want you to run this own KL EVPF program as you run Bumblebee and I don't want you to run the UI for the Bumblebee because Bumblebee UI takes resources for you so yeah so that's what happens does that answer your question yeah yeah yeah exactly I just happened to show it to show how you can easily run it without Kubernetes but if you are working with Kubernetes environment I would definitely recommend you just run it in Kubernetes it's just so much easier alright that's folks one more question so 5.8 should work with Bumblebee the older ones I think that's when it's having problems so if you go to Bumblebee website I think Bumble if I can type I think we document the version right on the website hopefully somewhere let me see yeah the reason why older version would not work is right now when we initially wrote Bumblebee it only works with the kernel that supports Kory and Kory has a version limitation we just didn't feel because that's the future we didn't want to support older version with our limited resources let me see if I can find the version requirements I'm surprised we don't have it on the site here but yeah definitely it depends on the DBBPF toolkit the Kory exists which I believe is 5.8 right any other questions do you guys need more time or should I continue question or continue I think the question is how can we measure the impact of the DBBPF program on the kernel is there any measurement tools for that that's a great question because I think right now when I run it in Kubernetes you can measure how much your pods are using and all that but that's not counting the portion that's loaded into the kernel I actually don't know the answer for that I have to get back to you does somebody in the audience know the answer for that I think that's a very good question so hopefully I do have Christian also online so hopefully you can tell us the answer Christian actually wrote the workshop and he has unfortunately he had a surgery so couldn't be there in person so hopefully he knows the answer for this question alright I am going to continue given the time so once you finish the lab click on the green check button I think it does a sanity check just making sure you are in the right stage then we can continue with the next challenging so raise your hand you have finished the first lab excellent your I'm sorry did you say your screen is locked? yeah thank you I know on different operating system it's different so thank you for that I typically Windows users always have problems because their copy paste is different did you say the answer is shift okay so shift and insert is the magic for Windows okay yeah shift and insert if you are working with the Windows environment thank you alright so let's load in the next one in this one we are going to learn a little bit about debugging EVPF code so I'm going to click on start our button here and we are getting started on this okay I did by the way just to answer the question the gentleman had Christian did provide me some feedback thank you he said the EVPF program are very lightweight the overhead is like 1-2% in terms of GPU Bumblebee will consume a few hundred of megabit because it handles user space as well so that's typically how it works and he also mentioned the older one will only work in the program the older one working in the program is using hash map as ring buffer needs 5.8 so if you are using older one that supports query hash map if that answers your question on that alright okay so in this lab we are going to go with OpenSnoop as an example we will learn about debugging EVPF program using OpenSnoop as an example so what does OpenSnoop do OpenSnoop traces open system cause open system so with OpenSnoop you can easily check where your application where your configuration files are if you have into a situation you couldn't find lots of your application OpenSnoop can help there too so let's get familiar with the code so first what we are going to do is we are going to go to the repository to download the OpenSnoop BPF code using WGAT and we are going to take a look of the difference between OpenSnoop we downloaded which is on the left side to the debug program on the right side which is provided by this lab so this shows the difference and let's take a look at the difference using WIM cause it's just so much easy to look so if you type WIM if that would highlight the difference so you can see the Bumblebee OpenSnoop is leverage a RIM buffer instead of a Perf buffer in this example here so this is RIM buffer this is Perf buffer on the left side and this is what's downloaded from BCC and the the F name I believe is the name of the program and the P is the P of the program so this is and then we config a MaxEntry which is same as the other program right let's check out the key difference here one thing about key difference is in the initial program it has a trace allowed method which uses filters this Bumblebee doesn't have kernel side filtering support yet so what we did is we essentially skipped that method in the function of Bumblebee version of the program the other thing we did is okay in the open so in the open method the trace point open method essentially the key difference is using a RIM buffer in this case and also trace allowed because it's not supported in Bumblebee so we remove that portion and same as open net we remove the trace allowed portion and in the trace exit we are using the reserve and submit API to work with the RIM buffer yep that's pretty much the high level differences here so with that I'm going to get out of the WIM program and what we're going to do is we're going to copy this debug program to the current directory so we can easily build it with Bumblebee so to build with Bumblebee we just run B build command which would build this into an OCI image that we specified here and it's compiling and it's building the image and let's go ahead push this image to the OCI registry it's local here now we can go ahead and run this program so when we run this program notice here we specify dash F in this case we particularly want to reduce the noise by filter so we're going to use this to filter these events and call DNS just to show these so let's go ahead run this right now we don't have any events but hopefully some events will come in I think you can easily generate some events if you're getting Kubernetes parts here so alright so we do have some events coming in now okay so we have events related to core DNS so what we're going to do is let's validate if these events are correct so if you run psaux to show the list of the process and the grab core DNS you can find out okay this is the core file and this is running 5051 so if we go here look at our core file notice here it's using a different PIT number so it's 5098 or 5096 which is quite different than what we see here so something is going on that's not quite right so let's take a look at the code so in order for us to look at the code one thing we can do is maybe add some debug statement so let's go ahead you can use the vi or you can also use the file editor to add to add the debug statement here so let's see the files we're working with is openSnoop let me hit refresh button and we're working with openSnoop bpfprogram.c right so the first thing I want to do actually it's just the right file let me just verify this looks different so the first thing we're going to do is in the openNAT method we're going to do some printing just making sure our PID because PID is what's having problems so we're going to making sure our PID ID is correct so instead of here we are going to add a print statement and making sure it's right and then save it let's see what else should we add okay in the trace exit let's also print out the PID ID because that's the one having a problem so alright so let's making sure well to edit okay right after my copy and right before I think we emit the event alright so in the trace exit okay it's open exit event data structure okay get current PID alright why am I not finding my copy here let's see if I can search for it my CPY alright let me make sure this is the right file route okay so if I go here make sure the folder I am in okay so the image okay so it's open snoop.c that's why forgot that's the name I copy sorry about that alright so okay so in the open net let's do this one more time okay so this is the one we want to do in the open net after the PID number let's want to print this out and also in the trace exit right before we submit the event to ring buffer we also want to print out the trace exit for that alright okay so we got it let's go ahead save it alright so right after you save it one thing to do is actually didn't do this yesterday so let's make sure you build the program one more time right to pick up the change so to build the program you run be build and you can then push it to the registry so you can run it easily now if we do run the program now using the same command hopefully right now we are going to get a little bit more data from the trace pipe file to see yeah something is coming in so the core DNS as notice there's an interesting thing going on the open net and trace exit is actually using different PID number right that's odd I would imagine it's the same PID number for both right because it's using the same it's the same call right to open and exit so something maybe it's going odd here this maybe also explains why we are you know having different not the correct number the PID number showing up here right so we might be showing the one that's incorrect so let's think out what's going on here alright so let's take a look at how we handle PID number in either places just making sure which one is right so this is the PID number we are getting the current PID in trace exit this is open net we are getting the current PID okay I see the difference now the difference is this guy has a shift 32 right shift 32 places to get rid of the thread group ID well the other one that's not right in order to fix this what we are going to do is to fix this all we need to do is shift it by 32 like the other one right so let me see so in here instead of having event PID using BPF get current PID here let's go ahead shift it by 32 like the other one so hopefully with that we will get consistent results between open net and exit let's go ahead click on save button to save it and the last thing we also need to do is let's go ahead build it so we need to build the program and we need to push it and then we need to run it right alright we got it running okay now the next thing we are going to do is let's open up our debugging command to look at the trace pipe just making sure they are consistent okay so this might be from the older one let's see 5051 okay this is the new one so you can see it's consistent now and if we do run the other command hopefully that's okay yeah that's also 5051 so that's consistent now and also it's consistent as what you are seeing here as the PID number so that means this debugging session is working we find out the problem where it is we were able to you know print out more additional debugging information unfortunately with the EBPF the debugging information print is pretty original that you can't do a debugger easily so you just print out the problematic thing and then you compile the traces and then once we make the code change yeah we take the code to bumblebee to have the command to recompile the code republish the image and then we rerun the images so that shows the whole life cycle of things with that I'm going to click on the check button and give you guys a few minutes to either ask questions or finish the second lab question I think you are just asking for clarifications that we basically build the images locally and then the image is packed as local host 5000 right which is the prefix for the local registry and then we push the image to the local registry and in the Kubernetes cluster we also config the Kubernetes cluster to use the local registry that's how you can either run locally using be wrong with the image name or if you are pointing to the local registry or you can run in your Kubernetes cluster by pointing to wrong with that particular images which your Kubernetes knows because it's configured to use that registry so that's what's shown here you could potentially do is using the Google registry any other OCI compliant registry or you set up your own registry and then push it there the only reason we showed local host is local host is easy in a lab format but you don't have to use it the idea of Bumblebee is really designed to use with any other OCI compliant registry that's already being out there that pipeline might already be familiar with so provide that consistent experience for you that you already familiar with does that answer your question great question, thank you so much Christian also just confirmed what I said was correct so that's good do you guys need a few more minutes you guys are ready to go to the last lab good good sounds like most folks are good, I will continue then because we have about 19 minutes so I want to make sure I explore the labs and alright so in this lab we're going to explore advanced user cases by chain multiple EBPF program together and with Bumblebee before we start the lab I want to kind of talk about why you want to do that should you do that right the good way is of course you can do that which opens the option but on the bad side it opens the decision tree for you should you do something like that because how many of you use monolithic applications in your workplace where you might be having a lot of function within a single application imagine you have checkout imagine you have your UI and imagine you have your business logic portion of your application all in one single application so that's monolithic which we have before we come to cloud native we break the monolithic into microservices so monolithic has some problem because it doesn't deliver very fast it's not very agile there's a lot of coordination and you have to do just to get a release so it typically takes a lot longer to get a release out so with EBPF chain multiple EBPF programs you would also have to consider that this EBPF program naturally do they perform seminal function are they owned by different teams so the people perspective is also very important because we talk about coordination delivery speed so that's super important you have to think about doesn't make sense to combine into chain in multiple EBPF program into one so it really depends on your business need organization structure and the functionality of those EBPF programs so in this lab we assume you already done the study and the test you are convinced you do want to consolidate and chain needs two EBPF programs together and we want to show you how to do that and do it with Bumblebee alright looks like I just got a connection that was error that's not great alright I just refresh it hopefully that fix it so so this time we are going to go through some code so basically we will combine two small EBPF program and we're going to deploy it as a single OCI image created by Bumblebee so the first program we're going to take a look is this exit and exact.c and let's check out the contents comments of the code to see what it's doing I should have this in the data steps if I want to okay let's find out it might be easy to view it here in the data steps advanced exit and exit.c awesome alright so what it's doing in this program is okay we want to know the details for the exact course so we kind of have a struct for it we also want to know the detail for the exit cost so we created another struct for it also created a ring buffer to the exact events and we gave a meaningful name and we're using counter here and we're using ring buffer here notice we're using the same same type ring buffer here for the exit events also and we're calling it exit events with the same type and also with the counter it's just different names same type of ring buffer and then this is our exact hook point where we attached the logic to all the system enter exact cost essentially what we have is we reserve the size space we need for the exact events and then we populate with the PID the name of the process and then we submit the exact event so that it could potentially be used by any user space called for example bumblebee and then similarly for the exit event we are attaching this logic to the exit trace point cost so this is a very similar code where we reserve the space for the exit events and we also submit the data into the ring buffer map and then before that of course we populate the exit cost PID and also the name of the process so you have that in the map in the ring buffer and potentially access it through our user space code which is provided by bumblebee alright let's go back to our terminal we didn't make any code change here and what we are going to do is that's our combined two EBPF program we talk about multiple ring buffer multiple attach hookpoint to the kernel now we are going to copy this program from root data steps into the current directory and then we are going to build an image from our code alright now we are going to push it again using local registry but you don't have to use local registry it can be any other OCI compliant registry the only thing is if you are using a different registry for example you are using google registry you want to use the b tag command which I will show you a little bit here so there is a tag command so you can use a different name it has its own prefix so you want to make sure you are using that before you push alright let's go ahead run this image as you can see we have the exact events and exit events both of the ring buffer map with the display here in the exact events we have some events coming guess what those events are it's probably me right as I mentioned to you this lab provides an environment in the cloud and inside of the VM we run our Kubernetes cluster using K3S K3D which is a really lightweight Kubernetes environment so with me interacting with our K3S we have shell terminals we have premises UI so there are actually a bunch of exact events being logged with the process ID you can see it's different process ID now we don't have any exit events because I haven't really done anything regarding exit so let's check out the premises integration here so what we are going to do is we're going to check out is there any metrics on the premises side that's interesting so 1991 I believe is the D4 premises ports for metrics so if you grab all the EBPF metrics which are produced by bumblebee with your EBPF program running here you can see a bunch of exact events which is similar as what bumblebee shows on here between K3S and Bash program now if you search for exact you see a bunch but if you search for exit I don't think there's any exit let me make sure I don't miss it solo actually I can just grab exit might be easy to type so there is nothing so let's see if we can generate some exit events down the road too alright so I know you guys probably a little bit confused right so we have this thing in premises we also have this in our Kubernetes cluster with bumblebee we also have this bumblebee UI oh by the way I just have an exit coming oh that's because I did a grab okay awesome so now if I do it I actually do have an exit and it actually report that's from the command right grab which is very cool so you can see what EBPF program is working by capture both exact exit and I should be able to see it here too so very nice alright so just talking really quickly about when you should use either one the UI or running bumblebee in Kubernetes so I think the UI is really nice for demo and local development it's very easy to see things but it's not optimal for production environments so the resource usage of the UI is much higher and and also it's harder for you to consume the output in a programmatic way as human we can read it but if you have some toolchain with it you probably want to automate that generate alerts or send a text message to you you need some automation in place for that the UI is not great for that so this is really when you want to leverage running bumblebee in your Kubernetes cluster leverage these premises metrics so that you can efficiently consume the output of your scripts and use it in any automation system you have and remember at the beginning when we run bumblebee we run it with no TTY so that's essentially running bumblebee without the UI so you don't have to pay for the cluster for the UI if you're not using it the other thing I want to quickly highlight, I know we highlighted this before it's just the naming really matters so when we write these structs for the ring buffer we want to make sure give a meaningful name this is important for metrics because if you do premises metrics here if we are grabbing AVPF you can see having this name so that you know exactly what's actually what's exact it's just so helpful to you as you get in these data from the map in your program to be able to leverage in your automation so that's very critical to have alright so in this lab just really quick recap we showed how you can chain multiple AVPF program together and we talked about it's the right choice for you only you can tell us if that's the right choice for you and we also talked about how to consider whether Ron Bumbaby in the UI was running in Kubernetes so you know what are the considerations why would you use which approaches with that I want to pause for a minute to see if you guys have any questions if you guys need any help alright I think we are getting close to the end of our session for a few minutes so the next lab is going to be you are going to do it if you are interested we are going to check out some of the new libbpf helpers just to help you learn them it's going to make transition to using libbpf libbpf program build on top of libbpf much easier so you are welcome to do the lab on your own the environments will be up running for you in the next two or three days and I do want to mention one thing so first of all we have a certification badge I know people love badges so if you are interested in getting this particular badge you can scan the QR code and it's going to challenge you on your libbpf knowledge from what you learned from this lab so there by the way which is going to bring you up the quiz so I would really appreciate if you are interested in getting the badge so go ahead and fill this out and we'll let you know if you pass we have your contact and if you do pass we are going to send you this badge so I encourage you to try to take the challenging and the last thing I want to mention is we are hanging out at solo.io there is a bumblebee channel if you are interested to learn a little more about bumblebee if you have questions we will be there to answer questions for you check out bumblebee.io and over github we certainly welcome you guys open issues we are very much appreciated with that I want to thank you so much for attending and I'll be here if you guys have any questions thank you so much and enjoy the rest of the conference yes we have people interested in the quiz shout out to you so I'm going to have the link here if you want to take a screenshot you will have the QR capture so yeah so the link you can type it I try to be easily remembered develop ebpfapps-quiz and you can scan the QR code do it on your phone yeah thank you any other questions I have to say I really love you guys you guys are very engaging I really appreciate that yesterday or the day before yesterday I gave another talk so many questions so I really appreciate you guys being very engaging so much appreciated you do have another question great question I do think we have some of these images already so you don't have to build it I think it might be on Google registry Christian let me know if you know the answer I'm going to try to find out I'm pretty sure I've seen those images I just normally build it myself yeah basically the image we have is in the Google container registry so we have our repository solo IO Bumblebee and then different programs with our B version is that what you're asking yeah great question alright I think we're right on time I will be here if you guys have more questions thank you so much for attending really appreciate your time and look forward to learn some ebpf together thank you yeah so Bumblebee actually doesn't work with the BCC styles yeah you can definitely use the ebpf trace I think we showed it in the lab to print out some of the debugging statement yeah so Bumblebee requires to use the ebpf Kory with the new libbpf loader