 Hi and welcome to our talk. My name is Guillaume Fournier. I'm a security engineer at DataLog and today, Sylvain and I are going to present the rootkits that we implemented using eBPF. If you don't know what eBPF is, don't worry. We are going to present this technology and tell you everything you need to know in order to understand the talk. So let's start with a few words about us. We are the cloud workload security team. We usually use eBPF for good and our goal is to detect threats at front-time. Everything we do is added to the DataLog agent, which is an open-source project, so feel free to check it out if you are interested. That being said, for DEF CON, we decided to use everything we knew about eBPF to build the ultimate rootkits. So as I said before, we are going to start the talk with a brief introduction to eBPF. Then Sylvain will take it over to talk about how we implemented obfuscation and persistent access in our rootkits. After that, I will come back to present the command and control feature along with some data exfiltration examples. And then I will talk about the network discovery and RAS bypass features of the rootkits. And finally, Sylvain will present a few detection and mitigation strategies that you can follow to detect rootkits such as ours. All right, so let's start with eBPF. eBPF stands for extended Berkeley packet filter. It is a set of technologies that can run sandbox programs in the Linux kernel without changing the kernel source code or having to load kernel modules. It was initially designed for network packet processing, but many new use cases were progressively added. So for example, we can now use eBPF to do kernel performance tracing along with network security and runtime security in general. So how does it work? So eBPF is simply a two-step process. First, you have to load your eBPF programs in the Linux kernel, and then you need to tell the kernel how to trigger your programs. So let's have a look at the first step. eBPF programs are reached in C. So it's not exactly C. It's more like a subset of C because of many restrictions that eBPF has to follow. But I'm going to talk about this later. So once you have your C program, you can use LLVM to generate eBPF bytecode, which you can then load into the kernel using the BPF C score. eBPF programs are really made of two different things, eBPF maps and the actual program. So there are a lot of different types of eBPF maps, but all you need to know is that they are the only way to persist data generated by your eBPF programs. Similarly, there are a lot of different program types, and each program type has its own use case. However, regardless of the program type, each program has to go through the same following two phases. So the first one is the verifier step. So I will talk about this later, but for now, just know that this ensures that your program is valid. And second, your eBPF bytecode will be converted into machine code by a just-in-time compiler. And when those two phases succeed, your program is ready to be executed. Step two is attaching eBPF programs. So in other words, this is when you tell the kernel how to trigger your program. So there are many different program types, and I can't present them all, but I'm just going to talk about four of them. So for example, you can use a K probe to trigger an eBPF program whenever a specific symbol in the kernel is called. Trace points are similar to query probes, but the hook points on which they can be attached have to be declared manually by the kernel developers. Those two programs require another syscall in order to be attached, and this syscall is the perf event open syscall. So the other two program types I wanted to talk about are TC classifiers, so it's SCAD-CLS, and XDP programs. So those program types can be used to do packet processing, so whenever some network traffic is detected at the host level or at a specific network interface level. Those two require a net click command to be attached. And the only thing to remember here is that each program type has its own setup, and thus map require a different level of access. Another very important fact about eBPF is that eBPF maps can be shared between different programs regardless of their program types. Alright, so the eBPF verifier, so the verifier is used to ensure that eBPF programs will finish and won't crash. To do so, it's really just a list of roles that the verifier checks, and your program has to comply with those roles. So for example, your program has to finish, it cannot be an infinite loop, so your program has to be a directed acyclic graph. You can have unreachable code, you can have unchecked differences, your stack size is limited, and your overall program size is also limited. And finally, one of the most infamous features of the verifier is its very cryptic outputs. So basically, if your program doesn't pass the verifier step, you will have a huge log of everything that the verifier looked into, and eventually some kind of, you know, error telling you what happens. But yeah, basically you are in for a very painful debugging session. Last but not least, eBPF comes with a list of helpers that will help you access data or execute operations that you wouldn't be able to write natively. So for example, you have context helpers, you have map helpers, a lot of things that you wouldn't be able to write in C, and that you would need external instrumentation to do. In short, you have about 160 helpers, and most of the heavy lifting of your eBPF programs will be based on those helpers. So that concludes this introduction to eBPF, and I will hand it over to you Silvan, so that you can kick off the presentation of the break case. Thank you Guillaume. Before we get into the details, let's see why eBPF is an interesting technology to write a rootkit. First, the safety guarantee brought by eBPF means that a bug in our rootkit cannot crash the host. An error in the execution will not cause any log message to be emitted. The user therefore has no way to know that something actually went wrong, and notice the presence of the rootkit. As we saw earlier, the eBPF by code is converted to native code, and the number of instructions is limited, which limits by extension the performance impact that our rootkit can have on the machine that could otherwise be detected by the user. On the commercial side, eBPF is used by an increasing number of vendors, in various use cases, network monitoring, security for instance. With eBPF becoming widespread, the chance of one product being abused to load malicious programs also increases. The safety guarantee we just talked about should not give the security administrators the false feeling of security. There is a lot of activity around eBPF, and each new version of the Linux kernel comes with a new set of eBPF helpers, bringing new capabilities. As we wanted our rootkit to run on widely used distribution Linux such as Red Hat Enterprise Linux or the latest Ubuntu LTS, we used a limited number of helpers. Using recent helpers or features like KLSI would have probably made the development of the rootkit easier. One of the primary tasks of a rootkit is to hide itself. What does it mean in our case? eBPF programs are bound to a running process. If this process gets killed, all the attached eBPF programs will be unloaded. For that reason it is essential that we both hide our program and protect it from being killed. The eBPF programs and maps used by the rootkit should also be hidden, and we should forbid other programs to gain access to them through their file descriptors. So let's see our rootkit in action. So let's start the rootkit, which gives us its PIE. Then we can try your ps command in order to see if we can detect it from the output. We can try using its procfs entry and nothing. We can even try using a sub file or even a relative pass, and we still have the same issue, no such file directory. And finally we can try to send a signal to see what happened. And we get no such process error. The obfuscation capabilities of our rootkit mainly rely on the use of two eBPF helpers. The eBPF program right user helper allows our eBPF program to write into memory of the process that issues a syscall. This can be used for instance to alter the data that is returned by a syscall. It's also possible to alter the syscall arguments. There is one caveat with this helper. The memory to be modified has to be mapped into the kernel allyspace. Otherwise, a minor or a major page fault will be triggered, causing the bpf property user call to fail. The other bpf helper is to ease the bpf override return. This one allows you to change the return value of a syscall and has an interesting property. If this helper is used at the syscall exit, it will simply change the return value of the executed syscall. But if we use it at the entry of the syscall, the execution of the syscall will be completely skipped. It is important to note that this helper can only be used at the entry of the syscall or at the exit. So let's see how the obfuscation of a file actually works. At startup, the rootkit will populate a map with the pass of its pre-prog-pid folder. Now the user space issues a fragmented syscall such as state. This syscall usually comes in two forms. One that accepts the pass to the file as a string, another one that accepts the file you scripted for the file that the user space program must have previously retrieved using an open syscall. So let's consider the former. To properly identify the targeted file, the rootkit needs to do an accurate resolution of the pass as the pass-specific file could be a relative pass. At the entry of the syscall, there is not enough context to do the resolution. So we need to loop deeper in the kernel in our case in dbfs code. So we have the resolution, but at this point, we cannot block the syscall as we are outside of the allowed hookpoints for the ebpf override return helper. So the only thing that we can do is to change the return value so that the user space believes that the syscall failed. We also need to scrub the content of the structure that could have been filled by the kernel. Now let's consider the later, the version that accepts the file you scripted. We do the same pass resolution as before, but instead of just pretending to the user space that the syscall failed, we store the file descriptor that the kernel allocated into an ebpf map. If the same processes issues the syscall with the same file descriptor, we can, at the syscall entry, return an error and block the syscall. In this situation, the user has no way to know that the file descriptor exists. Anyway, as we control the read the syscall, we can also hide all the references to the file descriptor in procfs. Blocking the syscall that accepts the PID as an argument is trivial using bpfr return, same for loading kernel modules. Now let's see a demo, let's demo the obfuscation of our bpf programs and maps. So we still have the rootkit started, and if we list the maps and the programs, thanks to the bpf tool command line, we can't see anything related to the rootkit. Now if we start a binary loading some kprobs and maps, and we list again the programs, we can't see the programs related to the binary but still nothing related to the rootkit. Then we can even try checking the kprobs and still nothing related to the rootkit. The idea is to hook into the ebpf syscall itself. Two things are required here. We want to avoid the user space to iterate through the IDs of our entities using the bpf proget next ID command, and we need to prevent the user space from getting a file descriptor to the program or map from a DID using bpf proget fd by fd, basically to prevent ID guessing. Once these operations are endowed by the rootkit, the user space is not able to view or modify our programs and maps. bpf provide user is a crucial part of our rootkit, but it comes with a pitfall. When an ebpf program that makes use of it is loaded, the kernel prints a pretty scary but legitimate message in the kernel ring buffer. So let's see how the rootkit endowed this. First, let's check what is the current status of the kernel ring buffer using the command line dmissage. So nothing special here. And we can even start a cat on the device used by the ring buffer. Then we start the rootkit. So we can see that the cat terminated with a pretty legit looking message. We can recheck the kernel ring buffer thanks to the dmissage command line. We can only see legit looking messages, but a bit more. And then we can stop the rootkit and we can run again the dmissage command line in order to see what was overwritten by the rootkit. So let's see how it works. In this diagram, journal D is waiting for a new message and is blocked on a readc skull. Two bpf programs are loaded sequentially by the rootkit. The first one, which doesn't use the bpf provide user, will hook on a readc skull and will make it return 0 and will write a legit looking message. This will guarantee that the warning message is related to the bpf provide user won't be read. Then the second program using bpf provide user is loaded. At this point, the readc skull can be unblocked and we can override the content of the warning messages with legit looking messages. Another important task for the rootkit is to set up a persistent access across reboots for instance. The rootkit can copy itself to a dedicated place and hide its binary file with the same mechanism that we already saw. For the persistent access to the system, we can use a generic method close to what we described in the obfuscation part. We can replace the content of some critical files read by the root demands such as currently cronzy or even sshd. Let's see an example targeting sshd and using the read override approach. So the approach here is to happen an ssh key to the authorize key files. Only sshd should be impacted, meaning the file will remain the same for the user point of view. And we want to have it available through the command and controls. So let's see this in action. So let's check the authorize keys content first. So we can see that only one key is present. So let's start the connection and it seems that the password is required. So now we are going to start the rootkit and we are going to specify that we want to inject an ssh key to the authorize keys file but only for ssh. So we can try your connection again and it seems to be successful. And now we can check what is the content of the authorize key from the user point of view and nothing changed apparently. Persistent access to an application database can also be set up using an azalt type of evpf program. Uprobs evpf programs attach to user space permission. In addition to being safer and easier to use than ptrace, they offer a valuable advantage. The kernel will automatically set up for us the hooks on every instance of the program. Let's see a uprob demonstration using prosgres sql. So first let's try to connect to prosgres sql using the word bonsoir as password. This one seems to be the good one. Then trying hello and this one is rejected. Now we start the rootkit and we will get the opposite result. Now the valid password is hello. So the idea here is to hook on the md5 crypt verify function of the prosgres sql that check whether the user provided the right md5 for its forward passwords and the challenge sent by the server. Overwriting the expected hash contained in shadow pass with a known value makes the comparison succeed and gives persistent access to the database to the attacker. Now I will hand over to Guillaume that will show you the command and control capabilities of the rootkit. Thank you Sylvain. Let's talk about the command and control feature of the rootkit. So what exactly do we want to do? We want to be able to send commands to the rootkit to extract data and to get remote access to the invoket hosts. Unfortunately there are a few evpf-related challenges that we need to face in order to implement those features. First you can't initiate a connection with evpf. Second you can't open a port. However evpf can hijack an existing connection. So in order to show up this feature we have set up a very simple infrastructure on AWS. A web app was installed on any c2 instance and we used a classic load balancer to redirect HTTPS traffic to our instance over HTTP. In other words the TRS termination is done at the load balancer level and HTTPS requests are sent to our instance unencrypted. So our goal is to implement cnc by hijacking the network traffic to our web app. First we need to figure out which evpf program types we're going to use in order to implement this feature. Although evpf provides a lot of options to choose from we decided to go with two evpf program types xdp programs and tc classifier programs. So both those programs are usually used to do deep back gate inspection use cases and while xdp only works for ingress, tc works on both ingress and egress traffic. Another difference between the two program types is that xdp programs can be offloaded to the network interface controller which essentially means that your program will be run before the packet enters any subsystem into the network stack. On the other hand tc programs have to be attached to a network interface but like much later in the network stack which means that they are triggered later in the kernel. With both programs you can drop allow and modify your packets and with an xdp program you can also retransmit a packet. This option is actually super interesting for us because it means that you can you can essentially receive an answer to a packet even before it reaches the network stack which in other words means that you can do this even before it reaches any kind of network firewall or monitoring on the host. Skipping the network stack also explains why xdp programs are mainly used for DDoS mitigation and tc programs are usually used to monitor and secure network access at the pod or container level. So what you need to remember about this slide is that first xdp programs can be used to hide network traffic from the kernel entirely and tc programs can be used to extra trade data on its way out. Okay first let's see how we used xdp programs to receive commands with the rootkit. So we implemented a client for the rootkit and this client communicates with the rootkit by sending simple https requests and with a custom route in custom user agent. So after going through the load balancer the request eventually reaches the host and triggers our xdp programs. Then our program parsed the requests, the http routes and understand that this request is not meant for the web app but is meant for us. So after reading the user agent the rootkit executes the request that can end and moves on to the final step. So this final step is probably the most important one. It overrides the entire request with a simple health check request and we do this for two different reasons. First we don't want the malicious request to reach the web app or any kind of user space monitoring tool that might be running and that might detect the unusual traffic. And second we want the client to receive an answer in order to know if the request was successful. So as I said before we could also have dropped the the packet entirely but since we're using TCP the load balancer would have retransmitted the packet over and over again until the request times out. And this would have generated noise and increase our chances of getting discovered. That said if you have I mean if you were working with a UDP server this would be a totally valid strategy. So let's have a look at how we can send progress credentials remotely. All right so on the left of the screen you can see two different shells. Those shells are connected to the remote infected hosts on AWS and on the right this is my local shell and this is the attacker machine. Okay so let's start with trying to log into the progress database using the normal password and again the rootkit is not running yet. So as you can see the Bonsoir password works fine and then let's start the rootkit and restart to login again and as expected and as you've seen before during a Sylvain's demo it doesn't work so you have to change into hello and this time it will work. Here you go. Okay so we're going to try to do the same thing but instead of hard coding the new password with the rootkit we're going to define remotely through cnc what the new password should be. So as you can see we have a custom client that will make a request to HTTPS, an HTTPS request to defconn.demo.dog and then we will provide both the rule and the secret to override the normal secret with. So the request that we'll go through is a very simple one with a custom route and the user agent will contain the new password that will be used at runtime. So as expected we get the 200 okay from the health check which essentially means that we know that the new password now is defconn and not hello anymore. So as you can see hello doesn't work but if I change it to defconn here we go this time it will work. Okay so this is how we send a command to the rootkits. Now let's see how we can exfiltrate data. So to exfiltrate data the client has to send an initial request to specify what kind of data and what kind of resource we want to exfiltrate. So the xdp part of this process is basically the same as before but this time the xdp program stores the network flow that made the request along with the request resource the requested resource sorry in an ebpf map and the reason why we do so is because when the web app answers the health check we want to be able to detect the packets that are meant to be sent back to the client. So when the HTTP answers answer reaches the tcegress classifier our ebpf program looks up the network flow and overrides the answer with the requested data. Now the question is what kind of data can you exfiltrate with the rootkits right and the answer is well pretty much anything that is accessible to ebpf and the reason for that is as I said before multiple program types can share data through ebpf maps regardless of you know what those programs are supposed to do. So basically you can exfiltrate things like file content, environment variables, database dumps, in-marry data if you start looking at the stacks of the programs anyway you can pretty much exfiltrate whatever you want. So let's have a look at a simple demo that is we can exfiltrate postgres credentials along with the file content of edc password d. All right so and again the two shells on the left are the ones connected to the infected host in on the ws and on the right this is my local shell. So the first request that I make here is to list the I mean to do is progress list which basically means place list all the credentials that you have detected so far since the rootkit has started and as you can see the answer was sort of the hashtag answer was overridden with the content of a map that we used to store the passwords that we've collected at runtime and again remember that with progress you don't need the clear password to log in you just need the hash password that is stored in the database. Here you go so now we're going to try to do the same thing to dump the content of edc password d. So to do so this is a two-step process the first thing you want to do is tell the rootkit to start looking for this specific file and as soon as a user space process tries to open the file and read the content of the file our rootkit will actually copy the data as it is sent to the user space application and save it into the nubbf map so that it can be retrieved later. So this first request will tell the the the rootkit to start looking for edc password d and now let's go back to the host and you know do trigger some kind of sudo operation so that we can actually I mean so that a user space process tries to open the file. Here you go and then this time instead of saying add we're going to say get and this will dump the content of the edc password d file. Here you go. All right so the cool thing about this technique is that it applies to any unencrypted network protocol so for example we also implemented it for dns which means that you can actually use it to do dns poofing. So the only difference between the the normal way of doing this and and the dns poofing is that instead of using a tc program to override the answer of the request you will actually switch tc and xdp programs because dns requests are made from the host instead of received by the host. All right so let's move on to our network discovery feature. So I know everybody knows what it is but I have to say it anyway network discovery is the ability to discover machines and services on the network that so that you know where you want to go next in the infrastructure and also discovering services is a super important step when you are trying to pivot between hosts because it will tell you what kind of attacks you might want to try. So the word kit has two different network discovery features one of them is passive the other one is active and you can control both of them through command and control. So I'm going to get into more details later but basically the only difference between the two is the kind of network scanning you're looking for and also the level of traffic that you are willing to generate on the network. So first let's have a look at the passive option. So the passive option is simply a basic network monitoring tool so it will do pretty much the same thing as any other EVPF based network monitoring tool which means that it will listen for any ingress or egress traffic and then generate a graph from all the collected network flows. It will also show you the amount of data that was sent per network flow and to implement this feature we used our TC and XDP programs so the TC programs were used to monitor the egress traffic and the XDP programs were used to monitor the ingress traffic. So for this version of the word kit we are limited to IPv4 and TCP UDP packets that said support for IPv6 and other protocols could have been added easily. So the reason why the passive option is pretty cool is that it will not generate any traffic on the network in other words it is basically impossible to detect that someone is tapping into your network and more specifically the network that is you know that reaches this specific infected host. However this doesn't work for services that do not communicate with the infected host and so in other words the graph will definitely not be complete and that's also why we implemented the active method. So the active method is a simple ARP scanner along with a SYN scanner. So we implemented it using only our XDP programs which means that only I mean that the entire process is done without involving the kernel stack and although this will be a slower process you can use this method to discover hosts and services that are that are reachable by the infected host but that are not communicating usually with the infected host. And again the rootkit client will generate a nice network graph for you once the scan is complete. So on a technical level this feature of the rootkit is actually quite interesting because as I said before eBPF cannot create a connection from scratch so in other words we had to figure out a way to generate hundreds of SYN requests while dealing with this limitation of eBPF. So let's see how we solve this problem. So in order to send a SYN request you first need to know the MAC address of the IP that you want to scan. To do so we used the same trick that we've been using so far which is to override the request from the rootkit client. So when our XDP program receives a scan request for a specific IP in a specific port range it will override the entire request with an ARP request for the target IP. And then instead of returning XDP pass which is what we've done so far and also which would send the packets to the network stack our eBPF program returns XDP TX. So what XDP TX does is send the packet out to the network interface controller it came in from. In other words our HTTP packet was transformed into an ARP request and was sent back to broadcasted back to the entire local network. So eventually the target IP will answer the ARP request and we will be able to store the MAC address of this specific IP in any eBPF map. However during this entire process the TCP packet that was used to send the HTTP request was never acknowledged by the kernel and that is simply because it never made its way to the kernel in the first place which means that the load balancer or the client itself will eventually try to retransmit the packet and when this packet is retransmitted and when it eventually reaches our XDP program we will do the exact same thing but this time instead of because we know the MAC address instead of overriding the request with an ARP request we're going to override the request with a send request and more specifically a send request with the first port of the provided port range the target IP and the MAC address of the target IP. And assuming that the remote IP or the remote host doesn't have any kind of protection against send requests sorry, send scanning it will answer either resets or send plus act to this first request. So reset would mean that the port is open and send plus act would indicate that there might be a service running on the host. And this is where the basically the network loop happens and the reason why we were able to generate hundreds of packets without having to I mean while dealing with the limitation of EBPF that is the inability to create packets. So whenever we get an answer from a send request we override the received packet with another send request on the next port and we also switch the IPs, switch the MAC addresses and send it back again to the target IP and we do so in a loop until we go through the entire port range. So eventually the clients will try one last time to retransmit the initial HTTP requests because once again during the network loop we never answered the second retransmit. So eventually when this third retransmit reaches our XDP program we will override the request with the usual health check requests so that the 200 okay answer will make its way back to the client after the request was handled by the web app in user space. All right so let's see it in action. So on the right of the screen this is a shell to the infected host on the AWS and at the bottom here it's another one and at the top this is my local shell on my machine. So the first thing you want to do is to start the rootkit then second is to start damping the logs of the rootkit. So in EBPF you can actually generate logs using a trace pipe obviously you would not want to do this in a real use case for a rootkit but this is a great way of visualizing the scan as it goes through. Yeah so that's why I'm doing this and that's why you will see what the rootkit does at runtime. And then let's make the scan requests. So what I'm saying here is please scan the IP 10.0.2.3 from port 790 and for the next 20 ports after this one. So the first thing you can see is the request is immediately changed into an ARP request and we already got the answer for this ARP request. So next up when we get a retransmit we will change this into a scene request. There you go so the scene request went through and then you can see the loop that happened and we you know like increased port one by one until we reached the final port requested by the port branch. And then now we are waiting for the third retransmit and this retransmit will be the one that we override with the health check requests which means that we will eventually get here go the 200k and in other words you know the answer from the user space web app. All right so now what you want to do is retrieve the output of the scan and extract all the network flows that were detected at runtime. And here you go so you would say natural discovery get and eventually so it actually requires a lot of different requests because there is a lot of data to extract but eventually you will get the entire list of network flows that were captured by the work kit. Here you go so you have all the different individual flows and then more importantly you will have a graph generated for you so this one is the passive sorry active graph so as you can see in a range there you can see the arp request and replies from I mean between those different hosts and then in gray those are the scene requests and the reset answers and in red is the only scene plus act answer from the remote host. All right and then you have also the passive graph which is the one that we saw before. Okay so now let's move on to our rasp bypass. So rasp stands for runtime application self-protection. So in a few words a rasp is a new generation of security tools that uses runtime instrumentation to detect and block application-level attacks and more importantly it leverages its insight into the application in order to make more intelligent decisions. So simply put it is some kind of advanced input monitoring tool that can detect malicious parameters and can understand if a malicious input will successfully exploit a weakness from one of your apps. So the textbook example of a rasp is usually a SQL injection. So the rasp would implement multiple functions instruments sorry multiple functions in the libraries that you use and such as for example the HTTP server library or the SQL library and it will check at runtime that the user controlled parameters in your queries are properly sanitized. If not the rasp will stop the query before it reaches the database and redirect the client to an error page or some kind of error message. In other words a rasp relies on the assumption that the application runtime has not been compromised which is exactly what we can do with the BPF. So just a little disclaimer before I move forward. I want to stress the fact that we are playing outside of the boundaries of what a rasp can protect you from and more importantly this bypass does not apply to one specific rasp but to all of them because this is one of the core principles of how a rasp works. So let's have a look at how a rasp protects a go web app from a SQL injection. So let's say that you have a web app with a simple products page and a get parameter to specify the category of the products that you want to see. Chances are your web app uses the default go database SQL interface. So this is a generic interface that you can use to query your database without having to worry about the underlying driver and the database type that you're using. More importantly for us since it is such a generic interface this is usually where the rasp tools instrument your code because simply it's much easier to hook at this layer rather than having to hook onto all the underlying drivers. So when your request is handled by the web server the query will be formatted with the provided category parameter and eventually the web app will call the query context function of this SQL interface. This is when the rasp checks the the the query and makes sure that everything is normal and if it is the execution will resume its normal flow and the underlying query driver will be called. So in our example we use SQLite so the SQLite driver is called. Eventually the query makes its way to the database and the answer is sent back to the client. However if the rasp detects that's you know something is wrong or detects some kind of SQL injection it will block the query and redirect the client to an error page. Alright so now let's see what we did to bypass this protection. Well the answer is actually pretty simple. We added a Uprobe on both the database SQL interface and the SQLite driver interface. What this allows us to do is call one of our BPF programs right before the rasp checks the SQL query and trigger another one right before the the the SQL query is executed by the database itself. And thanks to the BPF pro bright user helper we can override the input parameters of the hooked functions so that the the rasp sees a benign query and the the database executes our SQL injection. And the cool thing about this is that we can even do it conditionally which means that we can bypass the rasp only if one specific SQLite password was added to the beginning of the query. Perfect so let's move on to the demo. As you can see we have a very simple web app so it's a shoes retailer so you have a lot of different products and you can filter by category. So let's try to do a simple injection using the get parameter. So the the injection will simply be union select star from user and because the rasp is not running right now the SQL injection should work here you go and if you scroll down this time you will see the users along with the passwords. Perfect so now let's restart the web app with the rasp which is what i've just done and try this again. So let's go to the shop and then override the category parameter. Perfect and this time the rasp blocked the request because it detected that you know someone tried to do a SQL injection and the SQL injection would actually have succeeded. Great so now let's start the root kit by providing the path to the web app and then try to refresh the page. So again this should also be blocked by the root kit by the the rasp because we haven't provided the the secret password for the bypass to work and the secret password is of course defcon and when we say defcon the entire process that I described before will be triggered and as you can see the rasp did not detect it. So that's all for our rasp bypass I hope you had fun just before I hand it over to Sylvain for the detection and mitigation strategies I wanted to say that unfortunately we won't have time to talk about the container breakouts that are implemented into the root kit however they have been presented during our black hat talk this year so if you are interested feel free to check it out that being said Sylvain take it away. So let's talk about detection and mitigation. How could we detect and protect ourselves from this type of root kit? We could do this at different levels. First if a vendor provided you ebpf programs you should go through an audit an an assessment phase of their programs some changes that that the code has to be gpl it probably uses some internal kernel symbols so you can ask for it. What should we be looking for? The program types that are used but also the ebpf helpers used the communication so maps between programs may indicate a potential risk in the case of that the vendor program is compromised. We developed a tool to assist in this auditing phase by inspecting the health files containing the ebpf programs it is able to list the used entities programs and maps and complete a graph of the interactions between them. The tool was run on our root kit with the following result we can identify on the graph that the xdp program has storing information into maps that are also used by some kprobs which correspond to the command and control capabilities of the root kits. It is also possible to mitigate at runtime the loading of such programs by monitoring calls to the bpfc's call and logging the usage of it. It will even be possible to protect the bpfc's call itself by either restricting the call to it to only some trusted processes or have the programs inspected before loading and rejected if they contain suspicious patterns or make use of some dangerous helpers. We could also compute and validate the signature of the programs before loading them. An initiative exists to add this verification logic to the kernel itself. Using tls everywhere for network traffic also helps mitigating the risk of a raw bpf program that intersect network data. Now if we were not able to block the loading of such a root kit how difficult would it be to detect its presence? Even if it's possible though very challenging to write an almost perfect bpf root kit we should concentrate on the action that the root kit would have to block and lie about the result of such actions. For instance our root kit disables the loading of the kernel modules because such a module would have the ability to list the bpf programs and the active kprobs. Now let's imagine that we insert a module that executes a specific action only known to us. The blocking of the module by the root kit would then be easy to detect. Monitoring the network traffic at the infrastructure level could help detecting hijacked connections or strange package returns mission. Our root kit being far from complete and far from perfect it should be relatively easy to detect it. That being said we hope it will bring to light the potential and the risk of such a new bpf based root kit while presenting some interesting technique. The code of both the root kit and the monitor is available at this addresses. Please have a look. Thanks for your attention and have a great conference.