 Ah, there we go. All right, so get your stickers. Okay, nobody at the table anymore. You can get more stickers after the next talk. By the way, I'm Matthias, I'm on Prometheus team, just so you know, working there with Richie on Prometheus team and others. And next up, we have Kemal actually calling off mine, so next talk won't be that weird to introduce my co-worker, coming from a Prometheus perspective. But yeah, Kemal is talking about achieving zero instrumentation monitoring using EVPF, so yeah, please take that away. Okay, hello everyone. As Matthias mentioned, we are today going to talk about achieving zero instrumentation monitoring. So a bit of a background before, this is kind of relevant because I am talking on this. So I've been working on observability tooling since 2019. And if it's not obvious, I'm working for Polar Signals. We are relatively new startup and we are building continuous profiling tools. And I'm maintaining a couple of open source projects or I'm helping to maintain them. Thanos, Prometheus client, Golang, and for the relevance of it, Parker, which is an open source continuous profiling project, which is using EVPF. And as I already mentioned, I'm also a maintainer of client Golang's instrumentation. So that's my passion lies. You can check out the slides under these links afterwards. So why I am doing this? So I've been working on Prometheus-related observability tools for like four years back. And when I started to work on EVPF tooling, I suddenly realized that there are a lot of things that actually intersect here and we can achieve maybe some cool stuff just using EVPF and exposing some metrics, right? But of course, like as all my ideas, it's not original and people already discovered these areas and there are a lot of open source tooling around and today I'm going to talk about them. So first instrumentation or in other words, white box monitoring. If you are familiar with Goal Code and if you ever use Collide Golang, this is actually how you can instrument an HTTP service. You need to register a bunch of methods and you need to know about your histogram bucket distribution and maybe you can have some cool middlewares and add to your application. And then you can register to your Prometheus-HTP server and expose some metrics. But there's a lot of things involved to make it right and we actually, with Bartek, talk about these things, the best practices of instrumentation using Collide Golang a lot. You can check out on the internet. But when I first start working on EVPF, I ask the same question, maybe can we just get rid of this tooling and just have something generic and so that we don't need to get into the code for every time and add some code to just monitor or instrument our application. So for that, we should have some goals, right? So when I talk about zero instrumentation or black box monitoring, that means we wanna achieve this without modifying any code for our piece of application or not adding any proxies or sidecars, those type of facilities as well. So how we can do this? Today I will try to explain that. So first of all, I'm planning to just enlighten your way for about EVPF, how you can actually craft and compile EVPF applications and using these examples, maybe this could be a framework about how you can start a journey with your own instrumentation or observability for Prometheus ecosystem. And we will see a couple of real open source projects that you can just use today. So let's first talk about EVPF if you don't already know, maybe you are living under Iraq, I don't know because this has been a hype for the past couple of years in the CNCF environment ecosystem. So, EVPF actually stands for Backlit Package Filtering but it's like, and the E-Part is extended version of it, but this is not true anymore, like it's a concept that it's own and it doesn't represent anything anymore but first when they added this to the Linux kernel, it was for filtering some network packages. So right now, what EVPF is, it's a virtual machine embedded in the Linux kernel and you can write code and compile that for that targeting, that virtual machine and execute that in the kernel space. By using that, like you can do a lot of manipulation around networking, you can detect security breaches or like you can have a lot of observability tooling by using EVPF. There are already a lot of tools in the CNCF ecosystem and there are a lot of SDKs for different various programming languages. So how does it work? Let's like, this is just an example to actually an simple EVPF program that hooks into a Cisco. It's written in C. Right now you can use C and Clang to compile it or there's an alternative way to use Rust for that. Rust compiler actually partially supports it. It's not like covering all the use cases yet but there are some works ongoing. So this is actually, you just like, change your target with Clang, give it a C program. Of course, you need to be using special EVPF headers which we'll see next. And then you can load this byte code to the kernel EVPF virtual machine. You could be asking, is it actually safe, right? We are writing some C programs and it's running in the Linux kernel. It's actually, that's like why the Linux kernel developers like give a lot of thought on this and they come up with a way to actually verify the programs which is the EVPF verifier. This runs in the load time of your user space program and we pass this compiled byte code to the verifier and verifier verifier makes sure that the EVPF program is safe. There are certain restrictions like there's an instruction limit. It was a lot earlier but it's like right now you have a million instructions or so so you can write relatively larger programs. And the verifiers make sure that you are not doing anything harmful. It kind of makes sure that your program actually stops like this famous computer science program halting problem. It's actually unsolvable but with some heuristics that verifier actually make sure that it's kind of stops at a certain point so. And then from that byte code it just in time compiles and executes that. So how do you communicate with the kernel program and with your user space program? This is where like the EVPF maps comes in, right? And these are special data structures that you can manipulate in your EVPF programs. You can write data in them and using user space libraries which the most popular only BPF we're gonna see in that in upcoming slides and you can read this data back and do whatever you like with that data. So one of the things that we need to talk about is the hooks. So Linux churner already exposes couple of hooks that you can actually attach your EVPF program. These are syscalls, some kernel functions, some tracing points. Also you can have some custom hooks like K-Probes or U-Probes or Perf events and when you write your EVPF program with special syntax you can say that okay, this program whenever this event happens just run this function in the program that you define. So this is relatively new thing, compiling your EVPF program and running everywhere. Before this we were using BCC which stands for like BPF compiler collection like GCC. And what it does is like you just package that tooling with your application and in the runtime you actually compile your C programs to the byte code and then you load that. So you don't need to be targeting any cross-compilation issues. But as you can imagine it's like challenging because it's a lot of tooling that you need to pack with your host environment. If you are running in a containerized environment it's like it means large containers. So recently this is sold with this new paradigm called compiled ones run everywhere. Sometimes it is like pronounces query. The goal of that is just omit all the compiler facilities from your runtime and do everything in the compile time and make sure that you can access all the related kernel facilities with your program. How to do this is we are doing this thanks to the thing called BTF which is type definitions for the kernel. What we do is when we're compiling our program in the development environment we use our own BTF information and compile our byte code. But then when we are loading the program into the kernel we use the BTF information from the kernel and we do some relocations and make sure that the BPF program that we run in the kernel has the correct access for example as struct fields. So by this this actually makes our life a lot easier to actually compile and load programs to any Linux environment. Of course to be able to support this you need to be targeting certain kernel versions but this is kind of implementation detail. So how it does that like as I quickly mentioned like loader do some relocation magic and then compile the program so it makes sure that it actually supports your program. Thanks to Cori it makes our program super portable and because of that we can actually have something like that. We have a manifest, we have a container or a program we can just keep CTL apply and we can just run this eBPF program in our cluster and we can collect any cluster related metrics or any observability data that we need. One thing we should mention like to be able to run these programs you need to access to the kernel so these are privileged containers or pods so like you need to make sure that like what are you loading in your cluster. So before we continue to the metrics part we need to make this distinction between system metrics and application metrics. In Prometheus ecosystem with the help of like node exporter or C-advisor we can have a lot of system metrics available to us right and on top of that we can have application metrics which we get from instrumentation. So but like these tools are actually using Linux observability or monitoring facilities so they are mostly counters and you just get whatever you can collect from the Linux kernel itself. So for the application metrics as we've seen before you need to instrument your application. So but we can actually solve these problems using some eBPF tooling already available. The one of the most popular ones is eBPF exporter from Cloudflare and what eBPF exporter does to be able to solve certain problems that we have with node exporter or C-advisor it's using eBPF trace programs and collecting some data and like rather than collecting just counters with the help of these trace events you can have histograms and more granular metrics. We will see some examples about that but I mentioned some tracing tools so what are those? Again like I briefly mentioned about BCC about compiling programs and I mentioned that it's the old way but in the BCC also we have already enabled BCC tool BCC trace tooling so with the help of these tools you can reach out a lot of observability facilities from the Linux kernel. Most of it I don't even understand like this requires like a lot of in-depth knowledge so you need to individually check out the tools and if you wanna reach out like certain IO usage for example you can find the correct programs in this tool set. Again like the command is moving from the BCC to the BPF base tooling and again all the equivalent programs are written with eBPF APIs and they are readily available. Under this link you can check them out. So what eBPF exporter does like using the recently BPF tools it exposes like more granular metrics for example about our system. So how it does it like you have a configuration format with that YAML you can configure a metric and then you can have a corresponding eBPF program to actually populate that metric with all the necessary labels and you can load this to the eBPF exporter and you will get all the metrics that you need. Already in eBPF exporter there are some implemented examples and you can use these for certain system metrics. Like this is a socket example it's like trying to find out like the socket acceptance latency seconds and if I'm not it's in a histogram and it's collect this and it's expose this metrics and then you can visualize it in a Grafana dashboard using these metrics and you will have more granular information about your runtime system if you want to dig deeper. And other examples for example you can find out about the OOM kills because you can still like for each C group you can attach these programs and access to this information and you can have this metric as well. So there was a talk in 2018 if I'm not mistaken in the PromCon and the creators of the eBPF exporter they dive deep and they explain why did they do this in detail. One keyword about that is at this state they were using BCC and they were mentioning about the BCC challenges or all the programs were using BCC APIs. Right now the repo is updated to use the eBPF so some examples wouldn't apply right now so I suggest you to check the GitHub repo if you're interested. So one other talk I would like to mention is from Bartek. He also explored how we can use eBPF exporter and how we can actually create custom programs to be able to monitor some network application in the user space right. Because eBPF exporter also can help you to attach your custom programs. Again, this examples in this talk is also a bit outdated compared to what eBPF exporter has because these are also based on BCC but the tooling moved on to the eBPF so some of the challenges that mentioned in this talk is not valid anymore because thanks to the eBPF the programs are more portable. But just an example you can just add this example for example your cluster and all of a sudden you have all this red monitoring metrics for all online systems that you are actually looking for. One other example to achieve similar things is Hubble from Cilium. Cilium project, this is a network based eBPF tool and they already give you all the necessary metrics if you drop the Hubble in your clusters and just works magically. One other tool I would like to mention in the same area is the Pixi, it's another CNCF project. What Pixi does, they hook a lot of eBPF programs into your cluster and they collect lots of data and they store that for you and then you can actually query with their own language or you can write custom even scripts for that. And all of the out of the box Pixi can give you automatic red metrics and database query profiling already. They have implemented a lot of facilities around that. There is only one downside of it. To be able to expose some Prometheus metrics out of it you need to be using open telemetry because Pixi project doesn't like export any Prometheus formatted metrics out of the box. So there are also some more experimental tools. One of them is this is a little bit out of dated but the ePos and examples are there and there's a talk attached to that. What they do, they do is there's a project called Cube CTL Trace where you can actually execute all BCC tooling against your cluster. And in this repose they try to like discover using this tool and an operator what can we achieve and collect and expose them as a Prometheus formatted metrics. And there's actually a really nice talk about it. You can check it out if you wanna learn. But these projects are a little bit outdated that's why I'm saying this is experimental. It's been three or four years since they've committed anything and I don't think they are production ready but the idea was like really nice to discover. To just like expand the R horizon and going outside the Prometheus ecosystem there are lots of other things that you can do with the BPF. One of the things, this is actually a really fancy project, like they are calling this open telemetry goal instrumentation. What it does is it's actually attaching the BPF programs to the UReturn Props which you can do and this is actually how popular goal length debugger Dell actually works as well. And then you can actually manipulate what you have in the function code. So what they do, they just inject the tracing ID and so that you can have this. With the manipulated context you can have traces automatically. Using these type of methods you can actually have similar metrics as well. Yes, this is kind of, you can actually break how your Go Runtime program works because UReturn Props are actually important for the Go Runtime. Then you need to be super careful about it. But this is achievable. You can achieve similar things to expose metrics as well for auto-instrumenting. On top of that, those experimental stuff, you can be mentioned that one of the use cases is the security. You can use a tool like Tracy to actually check out what's going on in your cluster or in your running environment about security-wise and you can either do that. And shameless plug and like this is how all these things started. I'm working on this project called Park Agent which is an EBPF-based profiling tool. What we do is we attach all the profiling application to the processes that's running on an host and we just give you a granular picture of what your program is doing at the CPU. The first version of the agent was using Prometheus service discovery. So any service discovery we had in Prometheus itself, we were using that. What we were doing is we were attaching to the Kubernetes API, getting all the C-groups for the pods and we were attaching programs for that. By doing that, we were also be able to get all the metadata for that pod and we can attach those labels to our profiles. At a certain point actually, like one of our products is actually converted those profiles to the actual metrics and we just carry it over all those labels and we just store them in a ton of cluster and so that we were able to give a broader picture of what's going on in your cluster using metrics and you just can drill down from those metrics to the actual profiles. But right now after that, like we faced a couple of issues, so right now we built our own storage and we are moved on from that stage. But like while we were working on that, we realized that there are a lot of things that could have happened in the Prometheus ecosystem. So the gist of it, achieving zero instrumentation monitoring is possible but it's a lot of work. Thanks to EBPF, these are achievable but for each and every language runtime, you need to specifically implement certain programs and probably when the program runtime changes with the versions, you need to update that. If you want to automatically collect some runtime metrics, for example, it's not magical but it's like there are a lot of hard work. So maybe we can find a middle ground to actually automate this and there are a lot of startups out there or products that are actually doing this. There are some APM tools that are actually using EBPF and you just install that to your cluster and you get all these metrics. And EBPF is stable, like there are a lot of hype around it. LibBPF is like, it has a lot of developers. People are, community is converging on that and we are building a lot of tools around it and someday I'm hoping to have like not to be able to use client goaling but just have some magical container attached to our Kubernetes cluster and we got everything that we need. With that, thanks for listening. This is all I've got and if you have any questions, just go for it.