 Hello, everyone. Good morning, especially to everyone who has traveled and who's jet lagged. Also, thanks for showing up day zero of KubeCon at, well, 9 in the morning, although it's 9.30 now and we're starting late. So today, we're going to talk about profiling. There is a bit of BPF and about how we're dwarves and elves coming on this journey. But before that, a bit of a story and an introduction. So I don't know if you're familiar with this. So this is a graph, metrics graph, what it shows the CPU course usage against time. So you can see a normal baseline there, but there are these spikes. And these random spikes, they're basically umkills. So process takes up too much memory. Programs crash. And you don't know why the program has crashed. You have theories. And then you can try some debugging, see what the system has gone through. But this is like a post incident. As the incident happens, you have nothing to stop it, really. So there are theories, but we don't really know. What we need is actually data. And so that brings me to my next slide. So we need data like this, exactly. So this offers some insights into what was happening. And it would be nice if we could have this data all the time as we are running the application, right? So sorry, yeah. I'll explain the visual here later. But it would be nice to see how much each function is basically consuming how much CPU. And that should give us an idea of what's actually causing these memory spikes and causing programs to crash or take up too much time. So this is basically what my talk's going to be about. How do we go from taking a program to reaching this point and getting this kind of data? So an introduction. I am Sumera. My day job is staring at those icicle graphs that I just showed you. And on the process, we discover very cool compiler things and very weird kernel-adjacent low-level bugs, maybe, but interesting processes, data structures. And I'm also a Parca agent maintainer. Parca is the software I'll be talking about. It's an open-source project. And I contribute to it. And as for my job, I work as a junior software engineer at Polar Signals. We do profiling and observability in general. So I'm sorry about the video not being able to load. So let's talk profiling. And what's profiling? It's as old as programming. And what we consider programming right now is dynamic program analysis. It can also be technically statically program analysis, but that, I think, happens post-incident. Dynamic, we mean. As the process is running, we can measure resources that consume either memory or CPU time and how often we're calling functions and how much space that's taking up. So there is something called as sampling profilers. What sampling profilers do is they, so every 10 or 100 seconds per second, they try to get a snapshot of the state and they do it for some duration and send a snapshot. But it doesn't happen continuously. So why we need profiling is basically we want to improve performance. We save money because we optimize things, then we use less CPU time. And then so basically we can use more of our resources in the same time. And it's great, but the problem with sampling profilers is that it is only momentary. We don't get a continuous one. And we only have to start profiling once we detect the problem. So we don't know as we are leading up to the problem, we don't really know what's going on. And it's not really automated. We might have to instrument the code. We can do this without instrumenting the code as well. But often it's a very tedious process. We don't have a one-click solution right now. So that brings me to continuous profiling. What's continuous profiling? Continuous profiling is basically where we, like it says, we continuously profile as the program changes, as a new function is called, as your low-level kernel functions change. We profile that. And the profiles that you see, they change accordingly. So you know in real time what's going on. So this is a visual just saying over time you can see how much CPU is used, how much memory is allocated, how much heap, what's going on with the heap. And now let's move on to high-resolution dynamic continuous profiling. What I mean by high-resolution dynamic profiling here is that we get data with a very high level of granularity. This is something I mentioned in my abstract. And this is something that is very important when you want to see data. You can't just see high-level data that, oh, this is the core of the CPU that crashed, or getting an idea of that. This is the program that crashed. So maybe this is the part. Maybe it's the garbage collector. Maybe there is a memory leak somewhere. But we want data that's more refined. So we want to be able to see all the processes on our machine. We want to see the process name, the process IIT, the name of the binary, not just the binary as you name it, but the binary as per your proc API. That's what's going to help us, ultimately. So then we have, like I said, if you look at this visual, this is the parka server. This is showing us basically all the processes that are running currently in the system. And each trace that you see, or each line that you see represents one process. And so I was hovering on one of the process. And I think this is the, OK, I cannot see the name of the process. But you can see some of the labels that we see there. They have a binary name of the binary. And OK, this is the beam process that's in Erlang. I think I was running an Erlang program at the time. So you can see the name of the binary. There is the com label. And then there is a build IIT. And we'll get to details of what this means later. But what I wanted to showcase here is you can also see the functions and the stack trace for the process. And we can also, now going on to the next one, this is parka itself, actually. So in the previous one, we were seeing a lot of the processes. I think there's a system-wide view. Like you have a route, and then there are different functions. But then you can zoom in on one frame, and this is for parka. So you can see there is the route, there is the runtime, the go runtime being called. And here, if you hover over the frames or the stack trace, you can see some more information. So what's the cumulative amount of CPU usage? Cumulative means, like, suppose we were talking about function A. Function A's cumulative CPU usage means the amount of CPU A itself uses, but also the amount of CPU used by everything that A calls. So, and you can also see the file, exactly which file, the function is in, and which line number. And then you can see the address it takes in memory. There's something called the build IIT, and there's the name of the binary that's parka. And this is just basically another view we have in parka where you can see things, the same information, but in a table. Should you want a bit more like a formatted data if you want to search, you can do some of that also using this. It's just a bit more, less visualization and more, I think, structure to put it in one way. Then this is another visual, like what are the labels that we have in parka, right? We keep trying to add labels as per end user requirements and as we require. So like here you can say, you can query the CPU samples to get like, because when you have like all the processes, it becomes very difficult to find the processes. Some processes use very, very little memory, like, or very less CPU, like if you want something to find something about like your, say, graphics subsystem. A lot of that as compared to a program that you're actively running, like say, a Fibonacci sequence that is taking up a lot of time, CPU time. So it's very difficult to like pinpoint on like your display. So that's why we have labels. So we can just search by binary name. We can search by the compilers that the program uses. So if you know that you run a lot of your programs, they're compiled using say LLVM, but there is this one thing that's compiled using Musil. So that is something that would help. And if you want to see the kernel release versions, so these are the program binaries with a certain kernel release version, that's something you can also look up. And so next we have target discovery for high resolution profiling. Again, these are all the things that we want when I say we want data with a very high level of granularity. But how do we get here from like, we just have a program, right? We just have access to a process. So how do we get here? So all we really need is the binaries and the process ID or access to the root access to the proc map system. And so this is something I call this target discovery, discovering all the processes that are running in your system. We profile system-wide. So we basically go through the proc API and we see, is this a process? Well, let's profile it and find out about it. And just have the data in case, you know, this crashes so we can know what's causing all of this errors. This is a few. So Parker has this agent that runs on your machine. And the agent is actually what we use for target discovery and getting this stack information. Parker is mostly the backend server that makes sense of it and visualizes it. So in the Parker agent, we get the labels sort of like this. There's a compiler name. There's the C group name. There is a build ID. I'll get to what build IDs are in a bit. And there is a process, there's the PID. There's the architecture of the binary. The binary is meant for there's a kernel release, the name of the executable. And if there is debug information in the binary or if it is stripped, you can see one of the labels says, I hope it's visible, but one of the labels says it's stripped. So we want to know if the debug information is present in the binary, which makes it very easy for us to extract that. Or if it's not, and we have to go with a very convoluted and difficult way to do it. So next thing, what I had in the title is dwarfs and elves. So what are dwarfs and elves? They're very related to how to, they're not technically part of eBPF, just to give you an idea, but there is something that we use eBPF to work through. And eBPF makes it very easy for us to parse what we call dwarf and elves. But what's dwarf and elf? So, okay, I think I missed a slide there. So dwarf and elf. So binaries, I said all we need is a binary, right? Now, binaries are generally very complicated things, I guess. And there is this very popular command in Linux, it's called the file command. And so let's see what is there in a binary. If we use the file command on, I'm using it on Parc IH at binary. So we'll notice something, it gives us an output of something that says elf that tells us the endianness. Then it shows us the architecture, we can see it's ARM64. Is it statically linked? Is it dynamically linked? Then what's the build ID? And does it have debug info? And it's not stripped. So what do we mean by these terms? So binary has several sections. So there is a section known as dwarf, there is a section known as elf. There's the executable code. If it's a go binary, you also have a go PC line to have a part. And so what's elf? So it's a file format, basically. So binaries on Linux have, there are different formats, but Linux binaries mostly are standardized and use the elf format. It's executable, linkable format. And there are a lot of things, a lot of ways you can interpret the elf format. But one of the really good ways to do it is using the dwarf specification. What's dwarf? It's a debugging format. And we can use it to basically parse a lot of the sections of the elf files. And a talk on dwarf, or explaining details of our dwarf would be five separate talks in itself because it's a 400 page specification which we do not have time for right now. What's debug info? So debug info is what it says. It's a debug information that's usually stored in a binary, but it makes the binary a bit, what, because longer or larger, yeah, larger. And so sometimes a lot of people just ship it. From the binary. And that makes life a bit harder for us, I guess. But there are debug info servers where for every distro usually, that's strip debug info. So there's a binary and there's the debug info with that. But it would be nicer if they came in the same package, but okay, we'll find a way around it. Now debug info's, binaries have something called a build ID, which you can see when we are using the file command, you can see there is a build ID. And debug info's are associated with those build IDs. That's why even when they're stripped, we have something to link them together, right? And so what, actually, and there are several other tools also to read the binary with the elf and the dwarf information, but I'll be talking a bit about read elf that's one I use. So if we use, read elf is something like I said, it can take your binary read what's going on. So this is when we use read elf on park agent and this is on executable code. You can see it says contents of the EH frame section. EH actually here stands for exception handling and frame is, okay, so it seems like I have one more minute to wrap up. That's sad, but okay, we have read elf by information I'll zip through and we have to basically parse this and we tried to put it in tables, which shows us the start of a function and the end of address of a function and then there is the program counter for each. We have that for different architectures, looks slightly different. Then we use, basically we're reading the registers and their offsets. So we read them using EVPF, which makes it very fast and that helps us build a stack trace and a stack trace is essentially really just memory addresses and then we send them in a very compressed format to the server, Parker server. In the Parker server, we fetch the debug info as I talked about earlier. We link it to the build IDs, we already have the build IDs here in the compressed format. So we take the memory addresses essentially and then turn them into functions. So we have our stack trace now and we have how do we do this and how does Parker agent do this? It, we run a v profile using EVPF with a perf event hook. So this helps us read the registers and 19 times every second it sends data and to the server and that's how we use we go from a cycle of binaries. Yeah, that's how we have the binaries. Then we look underneath them. We have dwarf information and there are frame points also but like we use that to fetch memory addresses, compress it, send it in a format and then we use the server to symbolize the memory addresses and voila, we have our stack traces. So I think there's more lovely stories I'd like to share. We have support for all these languages and actually some more right now are in the works like Luachit and some more languages but I would have loved to talk about the stories about these languages but we're really out of time and I am sorry for all the time it took to set things up but I hope you liked the talk and thank you again for your patience and everything and if you're interested and they're on the score reach out to us and I'll be there in the hallway track or here. So feel free to reach out to anybody from puller signals and we would really like a lot of feedback on this because it's like almost two years old now or three but we still want to have the best or state of the art open source profiling. This is the first of its kind, it's zero instrumentation so we really want feedback on how to improve this and make it best so try it out, let us know. Thank you so much.