 Hi, very happy to be here. I'm Ori, Founding Engineer at Ground Cover. I've been implementing observability solutions for quite some time for many different use cases and I fell in love with the magic that is also known as EBPF. I wish to share with you today the most important insight that I've found throughout my experiments that with EBPF we can reach what I call full observability. When I say full observability, I'm talking about our ability as developers and DevOps to see everything that's going on in our cluster. This includes all the interactions between our services, all the metadata that accompanies them, various events such as program exceptions and protocol errors and determining exactly when all of this occurred and to which processes it belongs. So let's see how EBPF will revolutionize the world of modern observability. The best way to understand that is with examples, and this is where GRPC comes into the picture. GRPC is quickly becoming the preferred tool to connect microservices, replacing a lot of the modern uses of HTTP and it is also quite hard to observe and this is exactly why it makes the best example for our revolution. So let's implement observability for GRPC. When monitoring something, let's first ask ourselves, what do we really wish to observe? Let's take a simple example of a client and a server communicating over GRPC where the client periodically every 10 seconds requests resource only from the server. It's quite a common use case where the client needs to know when the resource is updated. And in that simple example, we need to see two things. The first is the GRPC data, which is the most basic part of the protocol. In our case, it's the resource being downloaded. We also need to see the headers, especially the path header, which is the resource name only. EBPF is a magic, as I said, that allows us to stop program execution at the beginning and at the end of every function we'd like inside the kernel space or in the user space and there we have access to the arguments to these functions. For example, we may attach kernel probes to the send and receive system calls and there we can see the arguments to these functions, which in this case is just all the data coming in or out of the container. And that is really powerful and it looks kind of promising, right? The headers and the data we seek have to be there. Well, if we were trying to observe the HTTP protocol, this would have worked phenomenally. We would have seen everything we would like to in just using K probes, but GRPC makes things harder. Let's understand why. GRPC headers, like HTTP headers, usually repeat themselves. In our example, they repeat themselves every 10 seconds. So to minimize the total amount of data sent over the connection, GRPC uses a header compression mechanism called HPEC. It works just as if the client in the first request says to the server fetch resource Oli for me. And from now on, I will call resource Oli resource seven. And from that moment onwards, the resource name is not transmitted anymore. Only the ID that represents it. And what's the problem with this for us? Well, GRPC connections, like the one we are talking about, can live for a very long time. These requests every 10 seconds can go on forever. And this means that we are most likely going to start observing the connection when it's already alive. And in that point, the resource name, alongside other headers, is not transmitted anymore. To fully observe, we need to know the resource name. And it would also be nice if we could not harm the connection in any way. We would like to probe it with zero downtime. So what can we do? Oh, EBPF also introduces Uprobes, user mode probes, which can be attached to user mode functions, including library functions. Uprobes can be used to get data directly from the GRPC library's memory. The library obviously knows that the resource ID 7 represents resource Oli, otherwise it just wouldn't have worked. The server, if it didn't know what 7 was, would not know what to respond to a request for resource number 7. So using Uprobes, we can get this data from the memory of the library. If there was a GRPC receive function inside the library that received both the headers and the data as arguments, we would probe it and see all the incoming information we would like. We would only need two probes, one for this function and one for the complementary GRPC send function. And we would be done. When you think about it, the same concept works for all types of stateful connections, including encrypted traffic. The encryption library leaves a cell, for instance, obviously knows what the unencrypted data is. After all, the library is used to translate encrypted to unencrypted data, so you probing inside the library, the encryption library, will provide full observability into encrypted traffic. EBPF allows us to see information from the library that the library developers did not make visible to us, because EBPF is pure magic. Monitoring with Uprobes can sometimes be a long and hard process. We don't have time to see all this process right now, but the gist of it is shown in the image. You have to map the flows inside the library, the data receiving and data sending sequences inside the library, and find functions that receive the information you seek as arguments. And then you Uprobe them. In our case, five Uprobes are what we needed. One to see outgoing data, three more to see the incoming data, and the last one to know when a gRPC connection has been closed, and everything works. And this is really exciting seeing all that work put to action. So please join me and let's watch now how Uprobes are used to fully observe gRPC connections. In this demonstration, in the bottom right, you can see the EBPF code. In the top right is the gRPC server now starting, and the first thing that's going to happen is the EBPF code attaches five probes to the server. The client at the left will then start periodically requesting resources from the server and printing all the information in the connection. As you can see, with zero code changes to the server or to the client, the EBPF code has complete access and full observability into gRPC connections. EBPF allows us to monitor everything in our cluster. And this is the real magic that will stand behind this revolution. The work you heard of here depicts the first steps of this revolution that will result in full and effortless on-demand observability. I invite you to stay tuned in the coming years and see how it all turns out. Or even join me and be a part of these two. If you want to hear more or even to contribute in some way, I invite you please message me. I would love to see you here the rest of the week and to be of assistance. So thank you very much for having me today. If you have any questions, then catch me now or later or message me. Thank you.