 Hey guys the mic works well, right? Good. So that to see you here the end of the day I'm a bit of jet lagged from Israel. My name is Val and I'm going to present today a tutorial session about eBPF and how can we use different open source tools in order to deal with day-to-day problems and Perform some interesting actions a little bit about me I'm the CTO and co-founder of Secret my in my past I was a vulnerability researcher and kernel enthusiast Have some hobbies of snowboarding chess player and football. I know it's not a big it's not a common mix Also founded at Secret eBPF IL community After that you can in the slides see a link to a slack workspace of the community and you can join it's a small one, but Very cozy and there are interesting topics going over there Just a short word about Secret. We are building API management platform. We are combining Real-time data and by analyzing traffic with API specifications We are utilizing heavily eBPF technology our agent is built on top of eBPF Soon going to be open-sourced still not open-sourced but it's going on the way just a Short disclaimer about this presentation as you will see I've built it entirely from Existing sources on the internet different blogs different tutorials and Existing open-source tools in the end of the presentation you will find the references slide with all the relevant links grouped by categories and then You can follow it and take a deeper look into particular topic if something catches your eye I will leave some time for questions in the end as well And I understood that there is also a virtual audience and there will be some virtual audience questions A little bit about the agenda So we will talk We'll do some small introduction to eBPF Not going to deep dive too much into theory theoretical, but we'll do cover up some basic stuff and then we will talk about DSL DSL stands for domain-specific language for a eBPF and There we will I'll show you a couple of live demos of using BPF trace tool and BCC to look at different tools that Some of you are maybe all of you are familiar with and We will discuss different scenarios after that will take some Specific use case of capturing HTTPS traffic Something that we are dealing with it also at secret where we need to catch a traffic from API's Communication and sometimes it's encrypted and we will see how using eBPF. It can be done We will dive a little bit into the some details, but again not too much. So don't worry We will talk about some management of eBPF programs and at the end the last topic we will talk about Cori which is an emerging Way of writing eBPF programs Cori stands for compile once run everywhere. I will cover it at the end That's it. So a bit history about the eBPF BPF stands for extended Barclay packet filtering BPF back at Barclay features invented a long long time ago also known as CBPF a classic BPF I Think most of us this time myself were familiar with it through a TCP dump tool where you can just create different filters and filter out some traffic based on Some specific language and some disclaimer like at the beginning when we started secret that we were using actually a Rapper TCP dump in Kubernetes environment in order to catch the traffic running as a sidecar but then eventually we switched to the eBPF based agent and Gain as a lot of different profits that we will discuss also Basically eBPF allows you to execute small programs inside the operating system kernel differently from like regular Kernel development like driver development. It secures the execution by running it in a Designated sandbox so you as a developer don't need to worry that you will Make a mistake or screw up the operating system your program is verified prior to execution and There are some limitations. We will touch them also during this topic during this call and After the successful application your program is running inside the kernel you are You're getting access to a lot of different kernel APIs, which is Very cool and was not possible before this technology and of course as you can imagine it allows a big gain in Performance because you don't you can execute some codes inside the kernel. You don't need to go back to the user space and Etc. So some basic diagram taking from the main website of eBPF eBPF.io There are different use cases that this technology can be used for Let's Talk about some very Basic ones. There is a networking things that can be done using eBPF and gain some performance and improve different use cases recently Cloudfair posted a very interesting blog post about how they improved socket binding using eBPF instead of relying on the traditional way there are security use cases where eBPF programs can be attached to LSM hooks and Can extend the functionality inside the Linux kernel in order to perform different security operations on On different Data such as traffic or other events that are happening inside the operating system There's a observability and tracing use case. This is one that we are doing it at secret basically utilizing eBPF in order to capture Traffic and analyze it parse application level protocols such as HTTP, HTTPS, HTTPS time, HTTPS 2, sorry and Getting some insights from live traffic and propagating it all the way to our SaaS platform There are also Some tools that are built on top of eBPF that are doing profiling and I think there are multiple different profilers open source nowadays that can profile and Perform different Tracing of your application based Regardless on the programming language that you are using and At the end we will touch like a very famous open source projects. I am not going to during this presentation. I'm not going to show the demos of like the biggest projects such as Cilium and Katra and NTC, but just for you to know there are Some of like very Famous and big broad projects with a lot of github stars that already built natively on top of eBPF There are different SDKs that are being more and more extended in order to provide the way and easy Entrance to different developers Into the eBPF world like it started with C language But nowadays there are different wrappers that allow you to develop your applications in Rust and There are also wrappers in Go and Python and some other languages as well So let's talk about the eBPF program life cycle very quickly It starts with a source code that we are writing in C language as I mentioned It's also possible to read to write it today in Rust. There is some open source project called red to BPF Which basically translates rust instructions into eBPF bytecode after that the source code is being compiled using Clang and the LLVM and it's being translated into eBPF bytecode And then at this point This bytecode is being loaded into kernel and it undergo Some crucial stages before it can actually be executed inside the kernel operating system So at the beginning there is a verifier Pretty complex portion not very well documented even so there is some Presentation Forgot whom who did that but talks about the internals of the verifier a little bit but basically the The purpose of the verifier is to make sure that we as the developers don't write something by mistake or maliciously that can Perform some unwanted operations inside the kernel It's at the operating system kernel. For example, we don't want to be able to modify some kernel memory directly That's just one or for example, we don't want to have Dead endless loops inside our eBPF program because it can stop and you can cause And logs and crashes of Linux kernel After the successful verification, there is a jit compiler basically there is a Sandbox VM that is running our eBPF code inside the Linux kernel and the jit compiler is in charge of translating the eBPF bytecode into the code that is going to run the eBPF VM Utilizing Very similar to a standard processor execution utilizing registers and Different opcodes in order to execute the programs on the kernel level Once the code is running there is Most of the times not always but most of the times we want to communicate with it somehow from a user space program and this is done via special specific communication options that are available for us in the eBPF The major one is maps. So basically max are some maps are shared objects that user space and kernel space Can communicate with each other to in exchange data It's also being used by different eBPF programs inside the kernel basically different programs can communicate with each other through shared maps And the entire Mechanism is event driven. So in order for your eBPF program to be executed There are some events that has to happen in kernel For example traffic arrives to a socket like socket receive message or for example There was a cisco call for a new process or etc etc once the event is being triggered there is a the eBPF program that was attached to this particular event is being executed and Then the user space can get information through the maps that explained earlier. Okay I think every time I saw some eBPF Presentation or video. There is always a slide that this picture Literally, I don't think I haven't seen one presentation without this slide. So I said I won't be an exception for that so BCC framework is BPF compiler collection At the before BCC In order to write eBPF program, you have to write it in the Intermediate language you have to write like kind of assembly opcodes, but BCC framework allowed to write the C Code which was translated into eBPF bytecode So but apart from the compiler it comes with a lot of different tools that were written by the community and still are being extended I Think even this picture is outdated already and there are much more tools in different Areas as described here Today during the presentation, I'm going to demonstrate. I'm going to run some of the tools and show like what we can get from it but Just for example There are some tools like for traffic sniffing there Some tools for file system reads There are some tools for TCP UDP protocol and different statistics about Connection about sockets and etc etc We will start with a BPF trace as I said before the easiest way into a BPF world is Using I I think and I think it's a shared opinion over the internet that's using BPF trace Basically, it's a DSL domain specific language that allows you to create very fast some Actually working applications that can perform different tasks Without the need to write any C code or any other programming language This The language of BPF trace is well documented in the public repo we will Go over some basic syntax, and then we will I will show you some of the Existing tools Again for in this diagram there is There's a diagram of the internals how it's going how it's working behind the scenes Not going to dive too much inside, but we will focus on this part on the program itself There's a very good presentation of Brent and Greg that explains each one of the parts with deep dive into code and how it works, but basically domain specifically in which is being Translated using parser into ST and then Some semantic analyzers and then code generation and then eventually we get a BPF bytecode So some basic stuff about BPF trace syntax BPF trace is a binary tool that accepts basically three major parameters there is a probe Probe is actually the event The kernel event as I mentioned before its event driven Mechanism so it's some event that should happen that will trigger some action So before the first parameter use you mentioned what is the which probe do you want to attach? Then there is optional parameter, which is a filter So you can filter out some of the events and you can execute your action on specific events for example if you want to Trace only a specific process you don't want to trace all of the process so you can put a filter saying like only for process ID that equals to Exquisite our process name that equals to ABC and then there is an action an action actually can be concatenated list of different actions that You can execute based on when the probe is triggered There are different types of probes and I think now it's a this slide It's a good time to extend a little bit of each one of them First there's a I'll actually will start with a trace point. The trace point is some predefined Entry inside the kernel so like the kernel was compiled with the trace points inside and It allows you to execute some handler when it reaches the trace point Very common trace points is upon entry of the function upon exit of the function function inside the kernel there are other types of probes which call the K probes or K red probes those are more dynamic. It allows you to insert Basically our handler in any kernel function as long as You you have a symbol for it And K probe goes stands for the beginning of the function in K red probe stands for the end of the function Red is return You probe we will touch actually actually during the example of HTTPS, but it's similar to K probe like but it's not in the kernel level but in the user level ebpf allows you to put Your actions to put your hooks also in the user mode in Some other processes and we will find it very useful when we will discuss how we can capture HTTPS traffic and Show it in plain mode and not encrypted Then there are variables that allow you There are specific as I said DSL that allow you to declare different variables global local variables variables with arrays we will see fine, we'll open one of the Programs of ppf trace tools and we'll go over the Code and discuss it There are some built-ins function that are our built-ins reserved words that are very useful for example Com Stands for a process name. So basically just can use inside your BPF chess program at com and then it will return a process name Count is a built-in function that counts different events. There is a he's 10 L. He's which Allows you to build histograms based on the events that are happening ret while Everyone knows that's a return value and there are much more. There's a lot of functions Basically there is a reference guide which explains each one one of the functions and Shows how to use them with small examples. It's inside the BPF Trace repo. I think now after some basic explanation of BPB pf trace. We will switch to a demo and I will show you some Basic stuff that we can do with BPF trace Second, I don't know if can you see it? So I'm inside the I've cloned the repo BPF trace and I'm inside the folder called tools Basically, I don't see what I'm showing but I'll try to So for example what we can show here like the common extension for BPF trace commands is dot BT and we will Use something like for example just a second. Okay, so execute Command that does a cease count Basically, it counts the number of syscalls calls called by each process I just run a BPF trace command it attached the probe as you can see and now it's counting once I stopped we can see Output and we can see like each process. How many syscalls does it call now? Let's open this Program and let's take a look how it looks like a second. I think I'll just switch to Duplicate because I can see anything so Yeah, can you still see it or is it too small? So sys count is as I said one of the tools that are provided part of the BPF trace repo and it starts So there's an option to execute some commands at the beginning of BPF trace a program Usually they are under the begin block so here we're just printing some information that it started and You need to hit control C to stop then This is actually The actual probe that is going to be executed. So As I mentioned before we have different types of probes in this case we are using trace book sorry, we are using trace point and We are attaching it for sys enter. So every time basically there is a syscall called Our probe is going to be the probe is going to be fired and then our action is going to be executed And what we are doing in the action? We have a global variable called syscall, which is an array We are using an ID of the process Sorry, no, we are using it. We are using the syscall number and then we are doing count. So basically Switch here you can see that for each syscall We have a number of times. It was executed in this program We didn't do the translation from a syscall number to a textual representation. It's possible to do But here we are Counting the syscals and we are storing it into the array the second array, which is also a global array called process the key is the process name as I mentioned before com is a built-in that provides as a process name inside the BPA program and We also using account in order to account How many syscals were executed? And you can see here the com returns a string representation of the process name And then at the end we are just printing the values that we received so As you can see very simple program written in a special language, not C language When we will discuss BPF Tool, we will see what's happening behind the scenes, but you can imagine Like we discussed previously that the Major the main communication way between user mode and the kernel mode that's through maps. So basically this program is being translated Into some maps that are being used by the user space and the kernel space and then those maps are being populated by a kernel program and The user space and just reading from the maps everything is encapsulated for us because of the Brilliance of BPF trace tool. Let's Do something? Which back about that? One more thing that I wanted to show Is actually instead of running Existing Programs there are also an option to run specific actions that's writing them as part of the code and not in the file So going back to the terminal so here I'm going to Execute the BPF trace program Minus E stands for that. We are actually writing a string for an action that is going to be executed And the string is here. We are attaching Again for a trace point this time the trace point is in the syscall called open at so basically when Some file is being opened And then what we are doing we are printing the process name and where we are trying to print the name of the file So that one of the args so Okay, there's a lot of files being opened in my computer But as you saw, it's a very simple one line one liners that I can just execute again, if you go to a BPF trace repo you will find a very Nice list that of those one liners that you can run and achieve different results What I want to show here. It's very very simple and There is a huge bridge being the huge gap that being bridged for you by Bbf trace that you don't need to write from scratch C program or even before like some assembly program and you can just run a high-level language in order to achieve this information from using utilizing the Linux kernel capabilities One more thing a funny demo that I Stumbled across and I will Show you Sorry for switching to this place. I have here Okay, so I found it also online and it really amused me some guy wrote a program that basically ebp a program that basically allows him to Lock in and lock out his computer based on the Bluetooth signal from his phone And It's it's really cool. That's how I I tried to run it and I had some problems there So I tried to fix it a little bit. So we'll try to run together here but in order to Explain you like what? What actually it happens? Okay, I'm going to show you So at the beginning What we are trying to do this I'm still not running any ebp a program that I'm trying to explain you like what the way that You need like I went I needed to go in order to understand it So basically I wanted to understand I needed to understand what are the kernel methods of Bluetooth You know that I want to put a hook on it. So basically I'm I'm Using a trace cmd in order to record the session Then okay, I'm giving a report there's a lot of stuff, but here we can find the stack traces the interesting function for us is a management device found which has been triggered when there is a signal from the device to to the laptop and the interesting part of it is that I Showed you where we can access the parameters of the kernel of the of the kernel function, not just the Cisco itself and I'm not going to show you the source code of Linux But basically the parameters allow you to see the Mac address of the device and also the signal strength So just in order to show you these Let's try to run Using bpf Okay, there's a lot of devices that are communicating and you can see it here and there is a second column is the signal strength so This is one the second one Function function called management device disconnected This is an event being triggered when a device that is already connected to my laptop is being disconnected from my laptop I will try to show you Okay, so now I'm connected into disconnect and Here like you can see that was printed. This is my Mac address of the of my phone So basically what we can do we can create some Very simple script that just using some log file we can try to detect when the device is In the range of the computer and then unlock the computer without needing to type the password And then the device goes a little bit away. The signal strength is going to be lower. We can lock it. So That's what We did here. I commented out the Unlock session lock session because I don't want to mess with the screen But We're just going to print locked and unlocked and let's let's hope that it will work We'll try to do a live demo now. So the only parameter that our script gets is the Mac address of my phone and I'm going to ask me to take my phone a Little bit away and let's hope that it will catch it. Can you can you try to disconnect from the Bluetooth? Okay, capture the disconnected device. Okay, can you connect? And come closer. Let's see if we are getting an unlocked. Okay, we got unlocked because now the phone now in the proximity Yeah, for some reason the locked event didn't we're not captured But my point is that It was very very simple to to do this stuff and instead of Writing some extensions for Bluetooth driver or whatever I just Examine to understood what are the functions that are being called or Linux is open source I saw the code. I saw the parameters that are being passed so I can see the values and then those Those are the actual Programs like a two three lines, so we are basically parsing a Mac address which is an array it's and then Here we are doing a Mac address and the signal strength and that's it So Okay, now we'll talk a little bit about the HTTPS traffic and here we will touch In a different zone of ebp app programs, which is in the user space So until now we saw attaching to trace points a similar way we can attach to k probes but Those are code this is code that is running inside the kernel Let's discuss a HTTPS traffic use case so basically, there is application and in a normal in there without the TLS the application just using skulls of the socket in order to connect Receives and messages When the TLS is involved and we want to encrypt the application we are actually going through another layer Which is a TLS library there are different Popular libraries that provide TLS implementation open SSL is one of them. There are also other options And basically their purpose is to encrypt the buffers before they're sent to the Linux kernel So if we are running if we put a K probe one receive a send message we will receive already encrypted buffer or Basically, we won't be able to present the plaintext so in this case K probes is not a good option But instead we can use you props the The mechanism is pretty Similar it's adding a hook inside some of the function But in this case the function resides in the user mode is some in some shared library. So what we need we need to Find the place inside the shell library where we want to put a hook as you can see from this diagram SSL write and SSL read are two very good candidates for the open SSL library and Those both both functions are exported symbols. So it's very easy to find And when we attach the hooks there, we can actually see the data just before it's being encrypted or just after it was decrypted so for this demonstration what I wanted to show is I Sent just the CURL CURL is using the this library I sent Yeah, I just sent CURL to github.go HTTPS github.com and well CURL does present the information because it's Client so it after the decryption, but the wire shark as you saw there it shows only encrypted information What I will show you here. We will use A tool called SSL SNF. We are running in this case. It's not a BPF trace program It's one of the tools that are coming with the BCC framework with a Python front end So Python front end by that I mean a user mode application that controls the eBPF program and We are going Getting the result of The of the response which is in a plain text As you can see here Message is chunked into different chunks because that's how we receive it inside the socket and we receive those buffers Sequentially to the user space But with some additional very basic logic we can combine it into a one Buffer and then we can have this message as a plain data We are Utilizing utilizing a very similar technique. We are not using the SSL SNF utility, but instead we have it in our Platform, but we do it also in secret in order to capture TLS encrypted traffic and they issue with you probes Thus that there need to be Specific for a user space shared library like so if you are in kernel Then there is only one place where you can put a hook like receive message or send message of a socket for user space libraries, there are different libraries different implementations and for some programming languages it becomes Very interesting and challenging specific especially for those that are That have its own VM like Java Because there is no exported symbol the code is a Java library, which is being compiled in runtime And it's much more difficult to attach To find who can attach to it E capture is another open-source tool that allows you to perform a similar things I think we look with short of time. So I want to present it to you, but it allows to capture HTTPS traffic and that they have They support today, I think three or four major open SSL libraries open SSL a few more SSL and a few others So like if an application is using different library you capture also can Capture that through a demo BPF tool that's a next thing that we are going to talk about actually this one is not To execute Specific commands it can it allows to load the BPF programs into account, but that allows to do much more It allows you to manage your ebp app programs and it provides some inspection and some Profiling and some useful information for you as the ebpf developer that you can gain about Yeah different dbpf applications And that are running that whether you develop them or they already running on your machine Let's start with Something very very simple This is a command a prog list of prog show it shows you all ebpf programs that are being that are loaded currently on the machine There's a lot of programs that Exists but what is interesting for us? Here we can see at the end there are three programs called the K probe is the weird should be a probe but On the function probably so Read write enter write exit and read exit. Let's do Let's send that let's run our previously trace point that we did and then let's do it again Okay, and we'll see here trace point and that was attached in the recent versions of vpf program We can also get some statistics of how many times it was heat and how many times in nanoseconds It was it used Then what we can also see we can dump the By code the of the program provided the ID then we can see the the actual code of our ebpf program again, it's not something that Interesting for us every time but sometimes when we need to troubleshoot we can see like what how the program program looks like We can pin a program. So like here every time I hit I stop the user mode And then if I show you now you will see that this program doesn't Exist anymore And when we are pinning a program even when we detach it from user mode it will still the ebpf program will still continue to be executed inside the kernel We have some Interesting things that Let's run it again Let's run the syscall and then instead of Program we can also show We can also show different maps that are being used as I said for maps are the mechanism in order to communicate between user space and kernel space And here we can see different maps that are used by our program I think one of the parameters is the actual program that is using the map We can also Show the content of the map Given the idea like let's take here. We can we have this map Remember when we explored the syscount BPF trace program Again, we have just two lines here and we discussed that those are being translated into maps behind the scenes so we can see here that those are the maps that are being created and And we can actually dump the values of the map in order to And We can we can see like in Live the values of the map it can help us to troubleshoot the programs Especially if the programs are a little bit more complex when different ebpf programs talking to each other through shared maps and some user space there are some Commands subcommands of the map command that allows you to actually change the values of the map dynamically and In some scenarios it can simulate and troubleshoot the different scenarios Sometimes you want to Pin a map to a specific program and then when you execute the program again You want to reuse the map from a previous run so you can execute Those programs with some maps pre-pinned instead of creating them every time the program spawns So there's a lot of useful commands and in general BPF tool is very useful. I wanted to show you actually again, you'll find it also in references, but There is a Series of tweets and a blog post by Quentin Monet that covers Most of the commands of BPF tool, which is a very powerful tool in the recent versions. They also have a pair of commands that allow you to Like profile your ebpf program execution and BPF tool has a Very excessive manual that explains and demonstrates how to use different commands Okay, very interesting thing sometimes We want to understand again, we will touch it in a In the last topic when we talk about Cori, but sometimes we need to understand which features are supported by Linux kernel and There is a lot of different features like ebpf is emerging and very trending technology. So every time in every new kernel There are new features are being added so a useful Useful command and there are also different filters that we can run that Basically prints out the information what is supported and which helper functions Present and also the configure kernel Which features are supported which features are not supported for example Okay, like here we can see that in my Linux here, which is Ubuntu 22 The BTF option which we'll talk in the next topic is supported My default in some older calendars. It was not possible So this command is usable to understand With what can be done or what not what can be done In other example here, we can see that bounded loop support is available When we were talked about the verifier and we discussed what is possible. What is impossible? We said that The verifier takes care that you want will not create infinite loops inside your ebpf program. So until recently sing until Don't remember the exact version, but until kernel in external five point something Loops were not allowed at all But then they added the support for bounded loops. So basically with a constant As a limit and then they just like unroll the loop into a series of commands There is also a limitation for each ebp a program. I think nowadays it's one million instructions Okay, so I demo Okay, we'll do it really quick. I want to live time for questions last topic I wanted to talk about is Corey which is a relatively new Even inside the ebpf domain way of developing ebpf applications So just a recap the traditional BCC approach writing ebpf code using Linux headers and compiler we are creating bpf by code which is being loaded and verified and then executed in the kernel in order to do that we need to take with us to the target machine the LLVM and clag compiler because we need to compile it on the machine Why because if we compile it on our machine no one can Assure us that it will be it can be executed on the other machine because they can be different kernel versions and kernel structs can have different Layout some parameters may be renamed and That's why we are compiling it on the fly on the target machine using the headers of that machine the issues with that is Clang and LLVM is a pretty fat binary that you can need to take with you to the target machine You are utilizing a significant amount of resources of the target machine if you need to compile BPF code in runtime for example if you're running on some huge Kubernetes cluster with Thousands of nodes and then you need to compile it on each one of them it can result in a lot of Unnecessary resources being consumed and also no one Assures you that kernel headers are presented on the time with target machines. Sometimes they are stripped and My last point which is also significant for the developers of ebpf programs is You discover even the trivial problems during the runtime when we are like for example compilation compilation program of your ebpf Compilation compilation error of your ebpf program will be discovered on the target machine so the solution that was Created to address those programs. It's called Cori is compile once run everywhere basically it creates It's utilizing a BTF files Which is like special files that contain symbols of Linux kernel, but in a very stripped way, so they are very relatively small and They allow you to get the crucial pieces of the information of the kernel specifically those that are required for your ebpf program and then They the your ebpf program program can run Pre-compiled on any target machine that is supporting BTF How it's done and there was a few additions in different areas of the entire Ecosystem so the clan compiler was modified in order to have a to create a special section for relocations of Binary and this section is allows the loader of Bpf program which we're talking the next bullet to calculate those Program dynamically understand how The ebpf bytecode should be created. So for example if in one kernel Some kernels truck contains a b and c and then another can kernel there was also included a b b 1 and c fields I'm talking about fields then the The loader will be able using this Relocation section to calculate dynamically and create a proper bytecode and then there's the sleep Bpf loader, which was created from scratch with which the competitor to BCC Chain that ties the BTF's the calendar BTF with the ebpf program and allows the load program dynamically So in a new way like if we take this diagram to Cori and there's ebpf code compiled into the bytecode There's a btf file my analogy to be btf file is like like a PDB files and then there is a lead bpf loader which takes both of them and then creates a relocated and Up-to-date bpf bytecode, which then been verified and jit compiled and run inside the kernel A few things about lead bpf. It's written in C language It's shipped as part of Linux kernel, but recently there's a separate repo that is being maintained It allows to build the core enabled applications and I think in the last year or year and a half all the BCC tools or most of the BCC tools were Transformed and there's also an equivalent version of Cori tools for those BCC tools based on lead bpf. There are a couple of wrappers and that were created one is lead bpf go which is a go wrapper that using like cgo bindings in order to execute methods of lead bpf and there is another lead bpf rs. There are a few other wrappers for for those languages Bpf tool that I showed you previously is also based on lead bpf and it has specific commands for btf-related operations Three more Mentions of existing open-source tools and I don't have time to show you a demo so btf-r bar archive is a repo created by Aqua security basically stores btf files for different kernel versions. So for example, even if you If you even if you are trying to execute your ebpf program on some machine that doesn't support btf or was not compiled with btf flag you still can do it by using the btf files from from this repo they support different distributions Fedora, CentOS, Ubuntu and I think few others and and then you can run your ebpf program on those kernels as well even if They were not compiled with btf support recent kernels. I think starting on for example in Ubuntu starting from Ubuntu 20 Don't remember the exact distribution number. They already compiled with btf enabled, but previous ones they were not The problem with it that the btf-r bar archive is pretty big one if you need to bundle all the btf files with you and take with it like and Ported with your application. It's almost a 1 gigabyte, which is not very practical So there was a tool created called btf gen guys from Kinvalik And basically you don't really need the entire btf file in order to execute because your ebpf program most probably is using just few structs of Linux kernel. So btf gen allows you to create minimal portable btf files and sync just Yeah, so like the compressed file that was created from all btf files for For Ubuntu is just weights a couple of kilobytes instead of almost a gigabyte Now it was like a very recently I think a few months ago. It was Emerged inside BPF tool. So basically BPF tool has a gen command and then you can run it with BPF tool We don't need to separate binary Last thing is btf hub online is some small SAS application that we built in secret Basically providing the capabilities of btf hub archive and btf gen Online so we are basically hosting Supplication we have an SDK return and go and rest API that you can just pull a relevant btf file So for example, you need to execute your ebpf program on some machine some Kubernetes cluster on some easy to machine and then The kernel is not supported. You can just send a live request get the btf file Which is a couple of kilobytes and then you can execute it the interesting part about btf hub online is that we have some scheduled lambda functions that Running like every day and then we provide the forward compatibility. So when there are new kernels are going out We are calculating. We are creating btf files for them as well That's it. So I think from here I wanted to mention like there's a okay, there's a Awesome ebpf which is a repo that contains most of the resources grouped by categories about ebpf. They're also tutorials videos blog posts examples, but also Tools, I think I mentioned some of them at the very beginning of the presentation like celium and Catron and also they have here BPF trace and etc. Etc So it's a nice area to find everything in one place Unless you know we'll get to Q&A just ten minutes. I know if it's okay. Yeah, um, I think you mentioned When like tracing a like a Java program You know it's much harder to To basically like trace when to capture the SSL data just because it's not gonna use like a you know Like an open SSL symbol. It's gonna use something different like you maybe go into how you would Like capture the SSL sent by a Java program Yeah, so what I mentioned is that in When we are capturing when we are trying to put a you problem SSL. It's a shared library in our user space which have a variety of different libraries in different versions and That creates a little bit complexity for this scenario Some libraries has exported symbols and written in compile language like open SSL in in this In this case the solution is simpler because you just find the exported method and then you put a hook on it And some other languages those libraries are not compiled. They are runtime from pile like Java they have a JVM and The Apache library is being compiled In runtime by a Java virtual machine, so you cannot find a Symbol so in order to achieve in order to hook that it's it's more difficult like for example, you can Inject some small agent inside application and find this Offset of the function dynamically after it's being compiled into the bytecode, but it's a it's a more complex scenario It's actually Python does he is Yeah, yeah, okay, there's barbecue