 Hello, everyone. Welcome to my presentation. My name is Furkan Önder. I am working as a Python developer at Bumper. In this talk, I will show you how you can instrument the C Python programs using eBPF. Before we begin, feel free to ask me any questions via Twitter or e-mail. You can find my presentation in my last video. So, what is eBPF? Let's start with the acronym. eBPF stands for Extended Berkeley Package Filter. From that name, you can see that its roots lay in network packages. In 1992, Stephen McCain and the one Jacobson wrote a paper called Berkeley Package Filter. This paper discussed a virtual machine that can run filters, which are programs written to determine whether to accept or reject a network packet. These programs were written in the BPF instruction set. A general set of 30-bit instruction that closely resembles the assemble language. On the slide, you can see an example of a eBPF program from that paper. In 2014, Alexei Stravoitov and Daniel Burkman started work on Extended BPF. That is how Extended BPF was born. Today, as the official website show, it's no longer an acronym for anything but a standalone name. This is because its capabilities span well beyond the package filtering that its previous name implied. eBPF is an internal machine used to extend the capabilities of the operating system kernel by loading and running programs that don't modify the kernel source code or add additional modules. eBPF was originally developed for Linux, but it now also has implementation of Windows. For more information about eBPF, you can visit the eBPF.io website. To use eBPF with CPython, we need to make some preparations. CPython can be built with the embedded markers, also known as props. These are the detrace props, which are not activated by default, so they have no impact on performance. So what is the detrace and the props? Detrace is a comprehensive dynamic tracing framework originally created by Sun Microsystem for troubleshooting the kernel and application problems on the production systems in real time. Detrace framework provides instrumentation points that are called prop. User can use a prop to record and display relevant information about the kernel or user space process. The prop activation is referred to as firing. To use the props, you need to have Python 3.6 or higher. Micros comes built in support for detrace. As a Linux user, I am building the CPython with the embedded props for SystemTap. SystemTap provides an infrastructure to simplify the gathering of information about running Linux system. It provides a scripting language for instrumentation of kernel or user space applications. Finally, CPython must be configured with the detrace option. Okay, let's display the available props. On the Linux, you can verify if the props are present in the built-in binary by seeing if it contains the .not-step-std section. By executing Redelf-Python command, you can see the .not-step-std section. In here, we are using the Redelf program. It is a program for displaying various information about object files on Linux system. On the slide, you can see the prop names and the .not-step-std section. So let's take a deeper look at the available props in the CPython. Function entry indicates that the execution of Python function has begun. Using this prop, you can get the file name, function name and the line number. Function return, the conversion of function entry indicates that execution of a Python function has ended. Line indicates a Python line is about to be executed. It is the equivalent of line-by-line tracing with a Python profiler. Gc starts fires when the Python interpreter starts a garbage collection cycle. First argument is generation-to-scan, like Gc-modules-collect function. Gc-done, this prop fires when the Python interpreter finishes a garbage collection cycle. First argument is the number of collected objects. Import-find-load starts. This prop fires before import-lib attempts to find and load the module. First argument is the module name. Import-find-load done. Fires after the import-find and load function is called. First argument is the module name. The second argument indicates if the module was successful loaded. The last prop is audit. Fires when size-modules-output function is called. First argument is the event name. The second argument points to a terrible object. I think now that we have a basic idea of EBPF and the CPython props. We can start instrumenting the CPython programs using the EBPF. Before instrumenting, I would like to mention the concept of the instrumentation. In a simple term, instrumentation refers to the measure of a product performance in order to diagnose errors and to write a trace information. We can also cover the profiling and tracing in the instrumentation concepts. I would like to briefly explain these two concepts. In the software engineering, tracing involves a specialized use of logging to record information about a program's execution. Profiling is a form of dynamic program analyzes that measures, for example, memory or time complexity of a program or the frequency and the duration of the function calls. I think we are ready to coding. Before doing that, I would like to introduce the BCC. I will write a simple tracing tool using the EBPF. To do this, I will use the BCC Python framework, which makes it very easy to write the EBPF programs. BCC, the BPC compiler collection, is a toolkit for creating efficient kernel tracing and manipulation programs built upon EBPF. BCC provides kernel instrumentation in C and allows the front-end user space application being written in Python or law. Let's start the coding. To start with, we are creating an argument parser that will get the process ID. We are adding the PID as an argument. The final arguments are being parsed. You are seeing an EBPF program. The program consists of a single function called TraceLine, which takes some pointer to struct PTRX as an argument. Inside the program, we are declaring the function name. After that, we are using the BPF used readARC function to read address to function name from the second argument. Because the second argument of C Python's lines prop is the function name, we need to get the function name. After that, via BPF prop, readUser function, we pass the function name from the address to variable function name. Similarly, we are reading the line number from the third argument. Finally, we are printing the function name and the line number using BPF TracePrintK function. In here, we are creating the USD object specifying the process ID. User land statically defining USDD is a mechanism that allows the EBPF programs to trace specific events related to the execution of C Python applications. After that, we are calling the enableProp function to enable the C Python's line prop and we are associating it with the EBPF program. In the next line, we are passing the EBPF program as a text and USDD object in the context parameter. This line associate the USDD context with the EBPF program. And finally, we are creating an infinite loop that continues to run the TracePrint function. TracePrint will print the function, this function, print the output whenever a trace event occurs. We keep looping until a keyboard exception. Before running the program, we need a test example program. I wrote a very simple program with four functions to make it easy to trace. You are seeing the nested calls. This program pins my name, Hello, Europe, Python, to entry and its own process ID at one second intervals. On the right side, as you can guess, you are seeing the output of the program. Now let's look at the output of tracing tool. In the first line, we see the process name, Python, and its PID value. Then we see the EBPF TracePrintK function and we are seeing the function name as a model because this line is not a function call. For example, this line could be a while loop or if condition. In our example, it's corresponding to while loop. We can see the function names in the lines just below because these are the function calls. Next to the function names, we are seeing the line numbers. Now we have an EBPF-based tracing tool that trace line by line the CPython programs. So how does EBPF program work? We wrote an EBPF program using the BCC Python framework. EBPF program compiled the EBPF bytecode. After the program is compiled the EBPF bytecode, it's loaded into kernel and verified. If everything is okay, EBPF bytecode is translated to native code by the JIT compiler and executed. EBPF programs are event-driven. It is attached to prop and executed when that prop is triggered. Program wrote the output and EBPF maps. The EBPF programs use the maps to collect, store and share data with user space via system call. There are the different of EBPF map, like htaple, array, ring buffer. The tracing tool that we have just written may seem long process for some of us. Actually, we could have written this program in a single line. To do this, I would like to introduce you to EBPF Trace. EBPF Trace is a high-level tracing language for EBPF. It's language inspired by EVC and C and predecessor tracers such as Dtrace and SystemTap. Btrace uses LLVM as a backend to compile scripts to EBPF bytecode and makes use of BCC as a library for interacting with the Linux EBPF system, as well as accessing the Linux tracing capabilities and attachment points. Let's look at the one-liner example. I want you to look at the top of the picture. We are seeing a one-liner comment. Using the shape parameter, we are passing the program. Programs begins with the USDD, pet of the CPyton and the probe name. It uses the printf function to print function name and the line number. The last thing is the process of ID, process of ID of program, sorry. As you can see, we can easily and quickly trace the CPyton programs line by line using a one-line program, which is very convenient. When we talk about instrumentation in CPyton, our topic may also be about profiling. There is a popular tool that call it PySpy. It's a sample profiler for Python programs. It can start profiling any running Python applications just by giving the process ID, which is very easy to use. PySpy works by directly reading the memory of the Python program using the process VM read system call on Linux or VM read call on OSUX. Figuring out the call stack of the Python program is done by looking at the global PyInterpreterState variable to get all Python threads running in the interpreter and then iterating over each PyFrame object in each thread to get the call stack. I have an example program written here. This program reads the Python files and converts them into abstract tree object. We have a function called file parser. It will read the Python files and converts them into abstract syntax tree object. In the for loop, in our current path we are finding the Python files and we are sending this file to file parser function. Let's profile the example program using the PySpy. Here you see a graph created after profiling by PySpy. It demonstrates one of the method of visualization PySpy output. It's called FrameGraph. FrameGraphs are visualization of theoretical data created to visualize stack trace of profile software so that the most frequent codpads to be identified quickly and accurately. Each rectangle in the visualization represents a stack frame. We are seeing the function names, file names and line numbers inside the rectangles. The x-axis on the flame chart represents a time while the y-axis shows the execution stack at that time. The most important thing to look at when interpreting flame charts are color and width of the rectangle. Width of each rectangle represents the time spent in that function. So what is the meaning of the color of the rectangles? The color can be based on various factors. In here the red rectangles indicate that they use the CPU more intensively during execution than the orange rectangles. I would like to mention another profiler based on EBPF, G-Profiler. G-Profiler is a system-wide profiler. It combines the multiple sampling profilers to produce unified visualization of what your CPU is spending time on. We have already created a flame graph with PySpy. This time we are looking at a flame graph created with the G-Profiler for the same example. We are seeing the different colors. In the top of the picture we are seeing the Python 3.9, which is the Python interpreter. The yellow rectangles are the Python frame. The native code is indicated by the purple rectangles. G-Profiler is a system-wide profiler. Unlike PySpy, which operates on per-process basis, it profiles all Python programs that are currently running on the system. If you wish to profile for Python process, you must use the four distinct PySpy, which brings more overhead. There is no such requirement in G-Profiler. With G-Profiler you only need to do this once and it profiles all Python applications using PyPurve. PyPurve is a part of G-Profiler. If you are familiar with the Perf tool, you can think of the PyPurve as a kind of souped-up Perf for Python. PyPurve is a sampling profiler. G-Profiler uses eBPF profiling based on PyPurve. With the PyPurve support, eBPF program can understand the C-Python stack trace and the profile each process without additional overhead. So, why is the eBPF is important? Performance. eBPF programs have fast-accessed resource such as memory. For example, they can access the memory of the Python applications much faster than PySpy. While PySpy doesn't introduce the overhead on the application itself, but it has some overhead for the system. It needs to access the application memory to extract the stack trace and uses lots of system calls for that. So, this takes time and resource. As a result, PySpy can perform efficient and fast profiling, especially in the system-wide. No context switches. eBPF programs are executed in the kernel space. As a result, they do not require context switching, so they work faster. Event drive-on. eBPF programs are event-based, so nothing runs without a specific trigger, and you never miss any event. Finally, the kernel-just-in-time compiler compiles the eBPF programs into machine code just before the execution, so the code is optimized for the specific hardware on which it runs. Thank you for your listening to me. Thank you for the great talk. I was aware of eBPF, but I didn't know you could do that. We will not take questions now, but you can find Furkan on the venue, and I'm sure he's happy to answer any questions there. Please give it up for Furkan.