 Hello, and welcome to this CNCF webinar. Today, we will be talking about how EBPF can be leveraged to improve the efficiency of Kubernetes security use cases. My name is Oshra Nir, and I am the developer advocate for Cubescape and Armo. With me today is Chris Kuhl, Principal Technical Product Manager at Microsoft, and an Inspector Gadget Maintainer. Take it away, Chris. Okay, thanks. So, my name is Chris Kuhl. I am here to present Inspector Gadget, and give you the introduction to EBPF. So, let's get started with the EBPF part. EBPF is basically small programs that you can load into the kernel as bytecode, and they run inside of a virtual machine or a sandbox. And so, this allows for one to easily make modifications to the behavior of the kernel without having to recompile it. So previously, to do something like this, you would have to build a kernel module. The kernel module would then be loaded into the kernel, and the kernel module would have full access to the kernel. This means that it could, if it were not programmed correctly or maliciously, it could crash your kernel or do other things. EBPF runs in the sandbox, so it's isolated from those kind of issues, and we'll look at that in just a second. Main use cases for this are tracing and observability, security, networking, and we're gonna look at some examples of these in just a little bit. So, EBPF really brings a lot of flexibility to the kernel. There's really no need to wait for certain kernels when you wanna make certain changes, and you can just create a BPF program, load it in, and as long as that kernel supports the functionality that you're using with that BPF program, which nowadays there's very good support for EBPF. We don't really need to worry so much as we did a few years ago. So, as long as there's support for that, then that program can run in pretty much any kernel if you have the right permissions. So, it's really efficient. It's run as a just-in-time compiler. It makes performance overhead very low, but another aspect of the performance is that often to do things previously that you can now do with EBPF, you would have to have like a back and forth between the kernel space and the user space, and what that causes is a context switch because every time you pass data or pass operation between kernel space and user space, you get a context switch, and that's very costly. And so, with EBPF, we can often do those kind of things fully in kernel so you get really maximal performance and just basically pass up the results or the information you're looking for. It's also safe, and so when we say safe, what this means is how, as we hinted before, you cannot corrupt in memory with a EBPF program. It's actually proven safe. You get the same sense of the mathematics can be proven by folks who do that. I'm not one of them. But basically, it's safe in the sense that the kernel is safe. This doesn't mean that you still can't and you should not be wary of which EBPF programs you're loading because those can change the behavior of the system, for example, attached to a network socket. It could block all the network activity there. So, yeah, it's safe in that it's not gonna crash your kernel. And finally, covering some of the use cases more in depth. Tracing and observability, it just really fits what the functionality that EBPF provides because often you want to be able to dynamically look into something, extract the information you want, and to do that in an obviously a performant way that doesn't really affect the overall system performance. And you wanna be able to get information at different levels of the system. And EBPF allows you to get that at a very low level. And so it's super performant. And it allows you to attach to different things here. We're listing K probes, trace points, and U probes. These are things that points where you can attach and are designed to be attached inside of the kernel. Well, in the case of U probes, it's actually in user space. And you can attach to sockets, syscalls, et cetera. And so on the security basis, when we mentioned the things you can attach to, these things are being expanded quite often. You have in kernel 5.7, we had the new possibility of attaching to Linux security modules. This allows EBPF programs to basically make decisions about whether an operation should be allowed or denied. And so, as I mentioned before, now this policy can be loaded directly into the kernel. And so the final one we wanna cover here is networking. You know, BPF programs can attach to different places in the network sub system, including all the way into the hardware. XDP is a functionality that allows you to basically do hardware routing using a UPF programs, which is very powerful and very performant. You know, you have companies like Facebook who use EBPF for their load balancing. And they have, you know, if you look, they have really great results in using this. You know, it basically allows you to draw, modify, and forward network packets. In the Kubernetes space, I think the most popular thing is, you'll find Assyllium, which uses EBPF extensively for its network functionality, which is basically what it does. Okay, now we're gonna look at Inspector Gadget. Inspector Gadget is a tool that we have built basically to collect data from the system and make it very easy and make it a kind of a common platform that anybody can use and also share individual BPF programs. What we packaged, what's called Gadgets. And so let's take a look at it. So yeah, it's a framework. It's basically designed for building, packaging, deploying, and running EBPF programs. And we package these into what we call Gadgets, which we'll look at in just a second. And it's primarily for debugging and inspecting Linux systems. So this is, you know, doing that means you're collecting data. So you can think of it as a data collection tool for EBPF. It's kind of a container-like runtime. And we say container-like because it supports a lot of the, you know, verbs you would know from a container system, build, push, pull, and run. And these are what we call image-based gadgets because they are packaged within OCI images. So it supports and understands Kubernetes. What this means is that when you take the data from the EBPF program, it comes from the kernel. It doesn't know anything about Kubernetes. And so we enrich that data with high-level information about Kubernetes, the container runtime, et cetera. And you can also filter the data that you get using these higher-level primitives. For example, if you say, just show me things from this single pod, you know, you can do that with Inspector Gadget. And the way that actually works is that you, we, Inspector Gadget takes that information, it translates it into what the kernel understands. For example, if you say, only show the information from this container, it needs to translate that to a mountain name space, a C group, various things. And so Inspector Gadget takes care of that. You don't have to do that yourself. It's very easily deployed inside of Kubernetes. You just basically do QCTL gadget deploy. And here you have basically Gadget as the sub-command for QCTL. This is actually a plugin which you can install with crew and you just run deploy and it'll put it in your cluster. It also supports running directly on a Linux host. So you don't have to have Kubernetes with this. And this is actually a very powerful feature because sometimes you might be running Kubernetes, but you wanna debug Kubernetes. And if you're running with full Kubernetes support, that means this relying on some of the things like the API server. And sometimes you wanna get underneath that. And this direct Linux support allows you to do that. It also supports many different export targets. CLI, of course, for interactive sessions. GRPC when you're dealing with the data programmatically. Prometheus or just the logs, et cetera. We also have a mechanism to do some user space processing and package that inside of a gadget so that you can deliver it along with the gadget itself and along with the BPF program. And this is done in WebAssembly modules and you can do this then in any language that WebAssembly supports. Go, Rust, JavaScript, et cetera. So for deploying the sign side of Kubernetes, it looks something like this. You have Qubectl gadget or whatever third-party tool you create to interact with this. The API calls for Qubectl go through the API server and they interact with the Inspector Gadget pod. This Inspector Gadget pod is actually deployed with the command that I showed you earlier, the Qubectl gadget deploy. And this will go happen on all the nodes. And then when you, once you deploy that, you can do, you know, run a particular gadget and this pod will take care of pulling that down, loading it, et cetera. So now we support a lot of different types of gadgets. And if you look at this diagram, you'll see, for example, that there are gadgets that attach to sockets, gadgets that attach to syscalls, gadgets that attack to certain block devices. And so the way BPF works is it attaches these things and when an event happens, for example, with sockets, when a packet comes in, it'll run, right? And if it determines that that's something of interest, it'll put it into the eBPF map, which is then consumed by the user space, but we'll take a look at that in just a second. In fact, let's do that right now. And so I'm gonna, here's a diagram of basically how Inspector Gadget works. So at the top left, you'll see the OCI registry. Now that's the part that generally does not, is not on the local machine or not on in your local infrastructure. That might be something like GitHub packages, et cetera. And then you have the local gadget store. This is where gadgets are stored when they're pulled down. And then you have here the ICE IG process. This is the Inspector Gadget process. This can be in Kubernetes or it can be on the local host. And then you see this yellow line, that delineates the user space on top and the kernel space underneath. Because what we'll do in the end, we'll load those BPF programs into kernel space and we'll pull out the data on the other side using Inspector Gadget. So let's take a look at this. So the first thing that happens is when you run, do an IG run a gadget or a Cube CDL gadget run a particular gadget, then what it'll do, it'll first check in the local gadget store. If it's not there to go to the OCI registry, it'll fetch that and put it down into the local store. Once it's in the local store, the gadget manager can then load this into the kernel. And in this case, we are attaching to a socket. And so when it loads this BPF program, it also creates a BPF map and this is represented by this yellow circle in this diagram. And so once that's loaded, it's waiting for an event. When an event does happen, it then loads that into the map and Inspector Gadget is ready to do its job on the user space side. So it pulls out that information, it sends it to the enrichment processes and this will enrich it with information about the Kubernetes. It'll enrich it with the container runtime information and this supports Docker, container D, et cetera. And we're also working on system D support. So even when you're not running inside of a container infrastructure, you can still see what service it's associated with. So the next step is an optional step and this is the post-processing step. Now there are times, for example, we have a very simple example for this and that's our DNS gadget. The DNS gadget, the DNS actual information is formatted in a way that is not so easy for BPF to process because it needs some loops and things like that which are, you know, kind of BPF is a little bit restrictive with what it does inside the kernel. And so we actually have a Wasm module that simply formats the string the way we want it. And so that's a very simple example but you can do a lot more with that. And then the next step is we pass that on to the data output element. And this can be if you're running on the CLI in an interactive process, it might just be a tabular information or it could be in JSON format. If you're, you know, if you want to send this to, you know, an API, you can use GRPC but you can also send this to Prometheus or even logs. So that gives you kind of a good overview of how this works. One thing to note here is that a lot of what I'm talking about today is currently behind an experimental flag. That experimental flag should be removed in the next couple of releases. And we're kind of excited about getting it out there in its full form because we believe that this is a extremely powerful system that takes a lot of the boilerplate out but not only a lot of the boilerplate out of people who are developing things such as the CubeScape folks, but also it will allow us to share EBPF programs more easily and for people to package them and just to have this off-the-shelf experience that we understand from containers. And so it's really about making this easier but also creating a community around this and enabling people to really share what they create. And so with that, I'm gonna hand it back over. Thanks. Thanks, Chris. In this part of the webinar, we will be examining a security use case that uses EBPF by way of a gadget from Inspector Gadget. In a talk in KubeCon North America in 2020, Shane Lawrence of Shopify said, no matter how good a job we do on the left, there is always going to be an issue that prevention didn't catch. When protecting critical services, we need effective monitoring. This seems like it adds toil to already overwhelmed cloud security and DevOps practitioners. In the next few slides, I will be talking about the source of overwhelm and how using EBPF and applying additional logic can actually reduce that load a bit. This image depicts what I have started calling naive scanning. It shows the basic method of most every vulnerability scanner you have probably worked with to date. Essentially what we can see here is a scanner testing the packages in an image and overlaying information from one or more CVE databases like NVD for instance. The result is typically a long list of vulnerabilities with their CVSS score. That is a source of the overwhelm I mentioned earlier. Cubescape took vulnerability scanning to the next level by enriching the original scan with runtime data. The way Cubescape works is by installing a node agent. This node agent uses Inspector Gadget to hook into different EBPF events. It uses file activity information to understand which files are opened in each of the workloads that are running in our Kubernetes cluster. This information enables it to cross reference the SBOM together with this information and find which software packages were touched during runtime and if they are loaded into memory or not. So if a Python package was opened, Cubescape knows from EBPF that it was opened and it'll take the Python file to see if it's listed in the SBOM and belongs to one of the packages. If it belongs to one of the packages, then it'll be marked as in use and reachable. Cubescape then feeds the vulnerability scanner with the original SBOM that naive scanners use and with a filtered SBOM it created. The filtered SBOM only lists the packages that were touched during runtime, giving the scanner that precious runtime information mentioned earlier. In the proof of concept for reachability, Armo researchers ran a scan with and without reachability on a Redis image. We can see that running a vulnerability scan on the Redis image tested identified 166 vulnerabilities. But we're looking through the lens of reachability, things aren't that bleak. The number shrinks to 36, which represents a 78% reduction. More current example shows that scanning Redis naively, that is with a static scanner, we get one critical vulnerability and 11 high. The common practice is that these are those that you need to fix as soon as possible. Using the trace open gadget from Inspector Gadget, Cubescape was able to identify which files are touched by the container during runtime. This scan gave us zero criticals and one high. This exercise was able to highlight that less packages are actually used in runtime. It created a modified S-bomb without the files that are not accessed. The result being a list of the vulnerabilities that can hurt us today. In this example, it's just one, thus guiding us to a prioritized list of vulnerabilities. Now, let's take a more detailed look at how Cubescape utilizes Inspector Gadget for this. Cubescape uses the trace open gadget, which gives it information about the file opened by the system. That includes file name, file path, user ID and group ID. On the right, you can see a merged PR from Ben Hershberg, lead Cubescape maintainer, co-founder and CTO of ARMA. This highlights the beauty of using open source projects. If something is not giving you everything you need, you can change it upstream to benefit everyone. This is what the implementation looks like on ARMA platform, which is powered by Cubescape. Users get the results of the static scan. However, they can then filter for reachability and for every vulnerability found, users can see if it is reachable or not. CVE reachability was the first Cubescape implementation of EBPF, which enriches runtime information for the purposes of securing Kubernetes clusters. But it is definitely not the last. ARMA is now working on adding the following features to ARMA platform. Seccomp profile in order to automate the creation and management of Seccomp profiles, providing fine-grained control over system calls that containers can make. Network policy generates network policies based on in-cluster data. Network policy in order to generate network policies based on in-cluster data. And runtime anomaly detection, which is pretty self-explanatory. In conclusion, EBPF is a useful tool in the cloud-native world, as it enables visibility into runtime with a relatively low cost in comparison to running scripts of traces and probes on the Linux kernel. There are many areas in which EBPF can help gain important insights. One of them is security. Using EBPF, security use cases can be enriched with runtime data, which ultimately saves practitioners a lot of toil and helps them focus on what actually matters. I'd like to thank Chris from Inspector Gadget for joining me today, and thank you for watching. If you'd like to keep in touch with ARMA, you can follow us on LinkedIn and Twitter. If you would like to follow or join the Cubescape community, you can check it out on Twitter or GitHub. There are also Cubescape channels on the CNCF Slack. Stop by there and have a chat with me. See you next time.