 So, happy to be here. Welcome to this presentation. And Mauricio, I work as a software engineer for Microsoft. I'm part of the team there. And in today's presentation, I will show you how to debug a Kubernetes cluster by using eBPF-based tools. Okay. So, in the agenda of today, I want to show you, to describe a little bit what is the problem about debugging a Kubernetes cluster. I will explain which are some of the traditional tools that we use to do that task. I will give a demo of that. And then I will introduce this inspector gadget project. I will show how it works. And then I will present also a demo of this project. So, yeah, let's get it started. We all know that debugging a Kubernetes cluster is not that easy. So, when there is something that is not working on a Kubernetes cluster, the first thing that we do is usually we check if the pods are running to see if there is any pod that is crashing or something like that. We also check the logs of the pods to see if there is something going on there. And in some cases, we don't have all the information that we need there. And we have to use other tools to get some more information about what is going on on our cluster. But what happens with Kubernetes is that, you know, Kubernetes is this distributed operating system. So, our services are usually running on different nodes. So, you don't know where they are running or better. Even if you don't know, if you know this is not easy to trace the microservice because that's not running locally on your machine. So, you have to connect to a host, you have to connect to other one. And also, in this microservices era, you have all those small services that are communicating to each other. So, when there is something that is not working fine, there are so many things that you have to understand, that you have to look at to see what's wrong. Maybe there is a networking problem. Maybe there is a storage problem with a given host. So, there are a ton of things that could be going wrong in a given cluster. There are some traditional tools I call them traditional because those are tools that are designed to trace what is going on on a single node. So, when you have a process, a monolithic application, you can use those tools to understand what is going on. But those tools are designed to work in a single node. So, they are tools that are not designed to work in Kubernetes. And one of the limitations of those tools is that they are not container aware. What it means is that if you want to filter what is going on, if you want to understand what is going on on a given container, you cannot do that by using those tools. Maybe you are able to filter by a given PID, by a given process, but there is no easy way to understand what a given container is doing by using those tools. Also, they are not integrated with Kubernetes at all. What it means is that they don't provide any information about Kubernetes. So, they are able to capture the activity on the host, but they don't enrich the information with Kubernetes-related information. So, yeah, let me show you how those tools work and some of the limitations that they have. Okay. So, before going into the tools, I want to show you the application that I want to deploy here. This is a very simple HTTP server that is composed of three different replicas, three different pods. And in this case, I have configured something related to security. So, I have CTL there, configured. I also have a second profile. And I also have limited the list of capabilities that this application can use. Also, I have limited the resources that my application can use. And, yeah, finally, this is just exposed as a node port service on the cluster. Let me deploy the application and check if that's working. So, I just want to send a lot of requests to be sure that this is working fine. As you see, this is a very, very simple HTTP server there. So, what about if we want to understand what is going on on that application? If we want to say the activity that this application is performing on the cluster. So, let me open a different terminal. I will be using that to send some requests to the cluster. And in the terminal on the top, I'm going to run some of those traditional tools that I was talking before. In this case, this is a very simple cluster by Minicube. I have two different nodes there. And in order to trace what is going on, I have two SSH in one of the nodes. And I have to install those traditional tools there. I'm not going to talk about how I did install those tools there. I think this is not important for this presentation, but in a real case, you will have to find a way to install the tools in the different nodes. So, I look into the Minicube N02 node. I have different tools there to install. And let me execute some of them. For instance, let me start with TCP Tracer. This is going to show you the different TCP connections that are being processed by this node. And if you see in the below window, I'm sending some requests to that, but not all the requests are being print in the window there. The reason is that there are three different replicas, two nodes, so only some of the requests are going on to that node. So, what happens here is that we have a partial view of what is happening on the cluster. We are only having the events on a single node, and we don't have like a global view of all the activity that is going on on the cluster. And, yeah, also in the events that we have there, there is no information about Kubernetes. We have the PID. We have the name of the process. And, yeah, in this specific case, we have the source address. So, by having the source address, we will be able to see what is the pod that was processing that request. But, you know, this is something that is not easy to do. And if you are trying to debug a problem, this is something that you will not like to do manually in all the cases. Let me show you a different tool. This is OpenSnob. This is to show the different files that are being open on a given system. As you can see, there is a simple node, only two pods running there, and there is a lot of activity going there. So, if this was a real one with 100 pods, that would be even more activity there. I sent some requests to that, and what I'm trying to look there is to see if I'm able to understand the files that my application is trying to access. But, as you can see there, there is a lot of information, a lot of events that I really don't care about, because those are not related to my application. But, yeah, there is no uneasy way to understand what is going on there. In this case, there is a possibility. I could try to check what is the name of the process of my application to get the PID. And then I could try to filter only by the events that are generated by that given PID. So, yeah, while preparing this demo, I realized that I have to use the PID of this NGS worker process. So, let's pass that to OpenSnob. So, this is only going to print the open files by that given process. Again, I sent some requests, but, you know, not all the times this is printed, because this is only going to print the events when the requests are right to that specific part. So, yeah, in this case, we are able to see what is the file, what is the process, and so on. So, let me go back to the presentation here. So, let me introduce the inspector gadget project. So, the idea of this project is that we want to fill the gap that is in those traditional tools, or better, those traditional tools are great. There is a lot of work on implementing those tools, but those tools are not a good fit for Kubernetes. So, inspector gadget is a project that is trying to close that gap. The idea here is to make those tools easier to use in Kubernetes. This is heavily inspired by a similar project that is called Kube CTL Trace, and we reduce many of the tools from the VCC project. Of course, we use as our building technology, we inject eBP apps programs into the kernel to understand what is going on. And, yeah, we are Kubernetes aware. What it means is that we are able to filter the events for a given Kubernetes, let's say, name, space, pod, whatever, and we also provide Kubernetes information for the events that we capture on the cluster. And additionally to that, we provide a Kube CTL plugin. What it means is that a user that is used to use Kube CTL doesn't have to learn how to use a completely new tool. So, we provide an interface that is as aligned as possible to Kube CTL. So, for instance, how to filter by a given name space, how to configure the Kube config of the cluster that you want to handle, and so on, that's done directly or better in the same way as it is done by Kube CTL. So, what are the features of InspectorGadget? We are able to filter by Kubernetes, as I mentioned before, if you want to get the events of a given name space, if you want to get the events of a given pods with some labels, this is possible. And we enrich those events with Kubernetes information. One characteristic that I say is very important about InspectorGadget is that this is very simple to install. I mean, when you are trying to fit something in your cluster, you don't want to worry about installing something that is very difficult to set up. So, the idea here is to have something that is very, very easy to do, that you only have to run like one command, and that's it. Because the idea is that we are providing a tool to fix the problem, so you don't want to get into trouble installing that tool. And this is, I would say, more like a detail, but this is an important one. So, there are some cases when you have a pod, and when the pod tries to run, there is a crash just at the beginning, and you don't know what is going on. Maybe you don't even have the logs of that pod because this is just crashing at the beginning. So, we have a bunch of logic, I would say, complex mechanism in InspectorGadget to be able to capture the first events of a container. So, we have a mechanism that when a container is starting, we stop the creation of the container for a bit, then we set up all the infrastructure that we need, then we let the creation of the container go. And what it allows us is that we are able to trace the very first events that the container is doing. So, we have the complete information about what the container is doing from the very beginning. And this is something that is like an ongoing work. So, we are trying to implement some APIs for different users. So, so far we have some goal and packages that can be used by third-party applications. So, if you are implementing a goal and application and you want to understand what are the processes that are created in a host, the TCP connections that are created, you can reuse our goal and packages and then you don't have to implement the EVP logic by yourself. This is the InspectorGadget architecture. We deploy InspectorGadget as a demon set on the cluster. So, there is an InspectorGadget pod node, then there we use, we inject those EVP programs into the kernel. As you know, the kernel is shared between all the pods running on the same node. So, by injecting those programs, we are able to capture, we are able to trace the information from those different pods. In this case, the user interacts with this QCTL gadget plugin and, yeah, we implement like a controller approach where we use the API server to restore those custom resource and the InspectorGadget which is that to understand what is the operation that has to be done. We have different categories, different kind of gadgets. We have advisors. So, those are gadgets that help you to recommend configuration for the cluster. For instance, we have a setConavizer. This is a gadget that captures the different system calls that are secured by a given application. And then this gadget tells you what is the setCon profile that you should be using for that specific application. We have audit. So, those are gadgets to check if a security profile is blocking some activity from your application. Again, if you have a second profile, this tool will show you when that second profile is blocking some system calls from your application. Profile, those are for checking the performance of the systems. One example is for checking the latency of the block IO disk on the cluster. Snapshooters are forgetting the status of a given subsystem. For instance, for printing the different processes that are running on the cluster. Top, those are, I will say, one of my favorites. So, those gadgets allow you to check what is the resource that is the most used on the cluster. We have, for instance, file top. So, this will show you what are the files that are the most used on the cluster. So, if you have any problems like high IO utilization by using this gadget, you will be able to understand what is the file that is creating that high IO activity and who is doing that. So, what is the pod? What is the container that is trying to read or write that file? And finally, we have tracers. So, these are like the most traditional ones. If you are familiar with VCC, those are like the snoopers from VCC that allow you to understand different events on the cluster. So, when a new process is created, a file is open, a TCP connection is created, and so on. Okay. This is time for a demo of InspectorGadget. Okay. So, the first thing is how to install InspectorGadget. Well, we can go to the InspectorGadget repository. There, we have the different releases. And for each release, we have the Qubectl gadget compiled for different operating systems and architectures. In this case, I'm running a Linux and this is for machine. So, I'm going to take that binary. And, yeah, I only have to download the binary. Actually, yeah, there's no direct to a binary. There is a compressed file. So, I have to uncompress that file. There is the binary there, the license. And to make this binary available as a Qubectl gadget plugin, I have to copy that into a path that is available to Qubectl. And then I can use this just as Qubectl gadget. So, yeah. So, so far, we have installed the client part of InspectorGadget, but we still have to deploy that into the cluster. So, if we check the different pods that are running on this gadget, name space, we can see that there is not any pod there. So, let's use this Qubectl gadget deploy. Sorry. So, this is going to install to create all the resources that are needed on the cluster. It creates some service accounts and roles. It also pulls and creates the demon set and the different pods in the node. So, after that command is done, we are ready to use InspectorGadget. So, what I'm doing here is creating a different window to run some commands. So, yeah, let's start by using one of the categories that I showed you before. This is just to print the different processes that are on my cluster. In this case, I'm printing all the processes in the default name space. If I want to print the processes in a different name space, I can use this dash n option. So, those are the processes in the Qubectl system name space. Actually, the terminal is a little bit small here. So, yeah, here we are. Also, if I want to print all the processes in the cluster, I can use this dash n option to print everything. So, there, we have the processes for all name spaces on the cluster. We also have the option to print the sockets. So, this is useful to get all the sockets that we have on our cluster. So, in this case, we see that the three different replicas of our application that are listening on that given part. Let me show you some of the tracers. So, this is to show when a file is open on the cluster. So, if I send some request to my application, you can see that in the top we get the file that is being accessed there. So, that should be in this .html. So, unfortunately, there is not enough space on this screen. And on the left part of the terminal, we see all the information related to coordinates. We also get all the information on the specific null, like PID, common, for this specific tool, file descriptor, and so on. Let me try to use a different tool. This is for tracing the TCP connections. But, yeah, in this case, that tool prints a lot of information. And the screen is so small here. So, I'm going to configure that to only print some of the information that I'm really interested in. So, yeah, in this case, I only want to print the null, the port, the type of event. This is a set or close process and information related to the IP addresses and ports. Actually, yeah, I made a typo there. It was the port. Yeah, that's right. So, if I try to send requests to my application again, we can see how this is printing the different requests. And what is interesting here is that I get all the requests. So, this is what is happening on all the cluster. I mean, not only the requests that are sent to a specific backend, but I get, like, the global view of what is happening in my application. Okay, but this is just to understand what is going on with my application. What about if I have an application that is broken? What about if there is something that is not working right with my application? So, what I'm going to do here is that I will break my application. For sure, I will tell you how I broke that. But, yeah, let's see if we are able to fix that by using inspector gadgets. So, I broke my application. I'm trying to send some requests again. But, yeah, the application is not responding. We are getting a connection refuse. And in this case, we can see that the different pods are in this crash look back off state. So, if we try to see the logs, let me check the logs of the application. I'm going just to use the label to avoid typing the name of the pod here. We can see there in this line that, yeah, there is something going on with the bind call. For some reason, we are getting a permission denied error. So, yeah, what could it be? If you remember, I was configuring some things related to security. So, yeah, maybe we are missing a capability on the pod security context. So, we have a tracer that is called trace capabilities that will print the different capabilities that our application is trying to use and will tell us if the kernel is allowing that capability or not. So, I'm going to, yeah, to scale down and scale up just to force the recreation of the pods. And on the top, we can see all the capabilities that application is trying to use. If you look at the last line there, we can see that the application trying to use this net buying service and it was denied by the kernel. So, what it means is that the kernel is not allowing the application to use this capability. Right. So, what is happening here in this case is that we don't have that capability on the list. We were dropping all the capabilities. So, yeah, I have to add that capability there. Actually, while I was recording the demo, I messed up a little bit with being. So, it will take a little bit longer until I update that. Okay. And almost done. Not yet. Actually, I have to close and start from the beginning because I wasn't able to update that there. Here we go. So, I'm going to add the capability there. And, yeah, I have to reapply that. And, yeah, let's try to see if this thing is working. Again, we are getting a connection review. So, maybe there is another problem. We tried to get information about the pods. Pods are running. So, it seems to be different problem in this case. If we get the logs, there is no any useful information there. Everything appears to be working fine. So, yeah, let's check the specification of the service that we are using to access this service. Actually, what I want to do here is to print the Jamal specification to get the full information about the service. So, yeah, what I'm looking here is for the target pod to see what is the pod that the service should be exposed in the pod. So, yeah, the target pod is 3.8 in this case. So, how can we know if this is the pod that our application is using? Well, we have a different gadget that is called trace bind. This is going to show us an event when application opens a pod on a given pod. Again, what I'm going to do is to scale down, scale up, and we can see there what is the pod that our application is using. So, what will happen in this case is that I have configured the service in the wrong way. I was using the wrong pod on the service. So, let me update that. Yeah, I only have to redeploy that. Yeah, let's try again. Okay, this time a different error, but, yeah, let me try some more times. Yeah, so sometimes it's working, sometimes it's not working. This is one of those difficult problems to debug. So, maybe, yeah, we can use this TCP gadget again to see what is happening to those connections. So, I sent a request. It went to Minicube node, and I can see that everything is working fine for that one. But, yeah, this one was sent to the Minicube and 02 node, but in this case it wasn't working. So, I'm just trying to send some more requests to see if I am able to get more information. So, yeah, if you consider, it seems that all the requests that are going to that Minicube and 02 node are not working fine. So, what could be going on there? Maybe there is a networking problem with that node, but actually the request is arriving there. Well, in this case, let me check the second profile. If you remember correctly, the second profiles are installed on the host. So, maybe there is something wrong with the second profile on that specific node. So, let me use this audit, set con gadget to understand if the second profile is blocking some activity from my application. Yeah, again, in this case, I have so many information. Let me only print the things that I'm interested in. Okay, let me send a request to my application. And, yeah, as you can see there, we have that there is this same file system call that was performed by those bots and the code means the action that the kernel took in this case was to kill the process, actually the threat of my application. So, what is happening in this case is that probably we are missing that system call in our second profile. So, let me SSH into the node, let me update the security profile, the second profile, sorry. So, I'm going to check if the same file is there is not there. So, let me add that just to the beginning. I'm going to save and to restart the bots to be sure that they pick up the right security profile. Okay, so if I try to send more requests again, okay, no, actually I'm just going to start the audit set con again just to be sure that there is no any more violations of the second profile. So, I sent some more requests there and everything is working fine there. Of course, this is a demo in a real-world scenario it will be a lot more complicated because there are some more things to look at but this is just to show you some of the things that you will be able to debug by using this inspector gadget tool. So, what is the future ahead of this tool? Well, we are trying to make this a more a community project. So, we are trying to get more involved with the community there. So, we are preparing for some meeting these as some bots to the CNCF. That could happen today, this week or next week. We are almost ready to do that. And from the technical point of view, we are implementing these APIs. So, if different third-party applications wants to consume the data that we are capturing with the cluster, we are going to provide APIs for that. And yeah, just to mention two different talks that are related to this project. So, today in the same track in the afternoon there is going to be a talk for my colleague Jose. This is to show a similar tool that we develop but this is a tool that doesn't require the API server to work. And on Wednesday I will be explaining how to generate security profiles by using eBPF-based tools. One of those tools is inspector gadget. And yeah, that's it. Thank you very much and happy to take any question that I can have. Thank you so much, Mauricio.