 I am KP Singh. I work in Google Zurich on security. We are part of an organization called Detection and Response. And this is a project to improve the stuff we're doing there in the upstream community. So we're going to talk about why we are doing this. We're going to like how does all of this work. We're going to talk a few detections that we build using that. What are the alternatives that are there in the community that exist. And we're going to do a little demo. Because I'm aware of Murphy's law, this is going to be a screencast and not a live demo. So, sorry about that. So motivation, right? Why do we want to do this? So there are two things that we want to, like there is signals. These are not unique signals. These are like things that correspond to malicious activity. And you can get this data from the kernel using things like audit. I executed a process. I opened a socket. These are signals or data that you can get from the kernel. You can use perf events for that. They could, a subset of these signals could correlate to malicious activity, but they don't mean that something bad is happening on the system. And then you have mitigation. This is the classic LSM space where you notice something bad happening and you deny the action from happening. And what this is, you would typically use your SELinux, app armor, LSM. So you could use sec comp. You could say that I want to reduce my attack surface. I don't want these system calls to be executed on the system. So security is essentially signals plus mitigation. That's the premise I'm going to build on. Currently, these two are disjoint in the Linux kernel. So if I want to build a new signal, I need to change audit. And that's hard. Like you have to update the space components. Let's say I want to change audit to start logging environment variables. And this means that this is a disjoint system. So you update audit or you add something to perf. And if I want to, if I realize something bad is happening based on that signal, I need to update the LSMs as well, which is more work that you have to do. And by that time, the adversary has already moved forward. So you were trying to audit, you were trying to audit LDP load environment variables. Now you've updated your LSM. You need to update the LSM to deny a particular fingerprint of an attack. So that's how signals and mitigation, we need a joint story for signals and mitigation in the Linux kernel is the motivation behind this. So again, what are some examples of signals? Like a process that executes and deletes its own executable. It shouldn't happen. It could happen, but it doesn't mean you need to always deny it. You may want to deny it. A kernel module that loads and then hides itself in slash proc modules. Why would it do that? Maybe. Suspicious environment variables. If you see LD preload being set twice on the environment vector, why is that happening? Maybe it's an honest mistake. Maybe it is something somebody tried to fool the lip-synter doing something. So these are all signals of malicious activity. Examples of mitigations. Like you, somebody mentioned that you don't want to load any kernel modules after the system boots. Great. You may want to whitelist certain modules that are allowed to be loaded into the kernel based on known hashes. You want to prevent known vulnerable binaries from running. The reason why you want to do such things is that when you build a binary, you want to release and qualify it. And typically, that takes more time than it would take you to update the policy that would prevent that binary from running in the system. And for a large deployment of binaries, this is actually very important to stop a known vulnerable binary or known vulnerable version from running. So that's there. You can also change these signals that I talked about. Not only those signals, but any signals that you think are relevant into a configurable MAC policy or an audit policy, but at the same place in the kernel. So that's essentially how signals and mitigation would go hand in hand. And this is why we want to do kernel runtime security instrumentation. So, and the way we want to do this is in our production systems and our endpoints. It's not just our data centers, but work stations and Linux work, developer work stations as well. So this is a great segue, actually, what Steve was talking about in the previous presentation. Unfortunately, I don't have the anonymous mask with me. I wish I did. But we won't use eBPF. And is everyone aware of eBPF? I saw you people laugh a little bit when Steve talked about it. But it's essentially a virtual machine that lives in the kernel. You can attach these programs to specific points. They are written in a C-like syntax. It's non-duary complete, so it can be verified. And eBPF is extended eBPF. It has maps. It has, like, more stack size, more registers. You have, like, more complicated lookups. And it has these things called helpers, which has more complicated logic that lives within the Linux kernel. And this is essentially what we want to use to build those detection helpers and use them in the LSM. So, essentially, extend the LSM using eBPF. We'll go into the details of all of this as we walk through the presentation. So, yes, we want to implement the LSM hooks as eBPF programs. And we don't want to do that with just the BPF tracing programs. You don't want to trace the LSM. You want to effectively have a new BPF program type that is geared for the security use case. And you would use those BPF programs to configure your Mac and audit policies. So, people ask me that why are you using an LSM? Why are you being an LSM? And the answers are, like, the answers are basically what LSM was created for. They map not to the API. So, we earlier used to trace execs' calls to log process execution events. And execution execs' call is the API and not the behavior that represents process execution. So, when I have a security analyst come to me and tell me that we want to log this when a process gets executed, that's what they tell me. They don't tell me that I want to log something on an exec V event. And this is very key where LSM fits in. It doesn't hook at the API level, but it hooks at the behavior level. So, that's one thing it is very good at. It has also good mapping to kernel data structures. So, when you access a particular kernel data structure, that's where the LSM hook comes into play. And it gives you access to these things called security blobs, which the LSM can maintain some sort of bookkeeping or state. And this helps you build more complicated detections without having, like, complicated user space called to deal with the same thing. You also benefit, like, LSM is a proven mechanism to do security in the kernel. So, you benefit from the years of research that has gone into this space, and verification of the LSM system. So, you want to have that extensible with something very flexible that's EVPF. But you want it to be an LSM so that it's a proven mechanism as well, not just a random tracing thing. You also give back to the LSM in a sense, right? You add flexibility and extensibility. Right now, there are very specific LSMs. I saw there is an LSM for memory protection that is coming up, that is SARA. And you essentially can extend the LSM framework and not create these tiny LSMs and you make them flexible. This is actually a very good point as well that you get the feedback. The LSM, we're all engineers. We write great code, this is great. But there are these security analysts, they're intelligence teams who work on, who get, like, want to really write these logics and write mitigation logic. And we want to connect the LSM community with these people. So, if you have these people writing detections and make it easier for them to write detections, audit policies and MAC policies, you get a feedback loop that goes from the security analysts is how I try to refer to them and to the LSM developers, the maintainers, or the Linux security community in a sense. So, here it is. Here's what it looks like. You have the each LSM hook is represented by a file in security FS. Security FS is a fancy name for Cisco security. You have your PPF program, which is this object file that you compile, that is the run, this is the byte code I talked about. And this is, you're essentially saying to the LSM, please run this piece of logic when a process is executed as a part of the LSM hook. You get to, you have, you open the file, you have a program after using a PPF syscall and you attach these two together and you can attach multiple programs to each LSM hook. And it works along with the other LSMs. So, you can have AC Linux, App Armor, blah, blah, and then KRSI would run all the PPF programs there. So, that's like a easy high-level overview of what it looks like. So, what happens when you load that program? It can do two things. It can return an error, like, hey, you're not allowed to do this stuff. Or you're like, this sounds fishy, I'm going to log that. So, you can do audit logs and the audit logging happens. Currently available mechanism in BPF is using the Perf Events Buffer. So, what you essentially do is you get data from the kernel using those helpers I talked about. You assemble a logging format and then you dump it into the Perf Events Buffer using this helper called PPF Perf Event Output. So, that's how you do audit policies in Mac at the same place. So, again, what's there in the program? You have helpers, right? The one that we talked about is Perf Event Output, but you can also have helpers that correspond to detections in the kernel. And there's this new thing that, I think Steve alluded to in his talk as well, is the BPF parts or like when they're trying to make trace points more generic. And we want to use that part where there is type information available. So, essentially, I can say that I want to access this field of the entry, and I don't want the program to break when the offsets change in the kernel. So, we want to use that part of what the BPF maintainers have developed in our thing so that the program is more flexible. I don't want to write a small helper each time I need to access a field of a struct. And this is only read access, by the way. It's not a write access. So, if you want to do a trivial scalar read, like say, what is the start time of this task? You could go about it with writing like a helper, an API functionality in the kernel, or you could just read it using this stuff. But then you risk breaking, if you don't have the correct offset you're reading from the structure, you risk breaking the BPF program. So, essentially, behave like the kernel module. And this is solved with BTF. What is BTF? So, these people who developed BTF, they took like the hundreds of megabytes of dwarf information and they distilled into this compact file that gets exported into, into like, say, kernel BTF VM Linux. So, it's like 125 megs, translates to about 1.5 megabytes. And it, like, describes all the various structures that are there in the kernel, what are the offsets of different fields, and how does it relate to various source codes. There's a LibBPF library or a user space component that looks at your BPF program that says, hey, you're accessing the DI node part of the I node. And I know that this is there in the kernel. So, let me update your program to that, before you run the program. And this makes your BPF program corey is what the people who developed it call, but it's essentially compile once and run everywhere. So, it's where we want to have flexibility, but backward compatibility go hand in hand. This comes as something new before I gave the last talk because this was not there when we absolutely disallowed any structure reads in the GRSI program. And it makes it more flexible. So, it has actually enabled us. Last time I gave the talk, we just had one execution hook, the process execution part. We can, now you can attach eBPF programs to any LSM hook. So, I don't have a 53,000 number like Steve had, because there are not 53,000 LSM hooks, but I just quickly added what we had there and we have like 198 hooks that you can attach eBPF programs to. Which is not bad. I think it's a good balance between tracing and security. And again, I'm making this very clear. This does not use trace points or fancy trampolines or all of that stuff. We want this to be a proper LSM. It's just using eBPF as a mechanism to extend the LSM framework, but these are functions that are there in the current. These are like, there is a hook called KLSI BPRM check security, which takes the argument, it runs the eBPF programs and it is registered as a proper LSM hook, so that you benefit from all the things that are about the LSM framework we talked about. Okay. We're going to talk about some detections. This is going to show some code on slides, which is fine. So, the story here is security analyst walks to me and he says, and they say that we want to deny any process that tries to unlink its own executable. Like, okay, we're engineers. We can find that out. What do we, what information do we need to have that sort of detection build? We need, on the inode object, we need to see what are the processes executing that. And on the process object, the task struct, we need to see what is the executable for that particular process. And how do you track that? You use security blocks. And that's what it looks like. The, all the stuff we've been talking about is essentially this. You say that I am going to hook at the inode unlink LSM hook in the kernel. KRSI did some bunch of work internally and has provided you an API for that detection, which gives you three forms of responses. It says that you are a process that is unlinking itself, unlinking its own executable. You're a process that is unlinking your parents' executable, or you're a process that is unlinking any other running executable on the system. And based on how you perceive that as a, as a, and you can link that with other signals, and you can deny that action from happening. So in this case, like I am saying, only if you unlink your own executable are you not supposed to do that. Otherwise, sure, like it's a, it's probably an event that somebody's not aware of on the system and they're doing that. So it kind of adds that flexibility angle to that Mac. You may just even want this to happen. And you're like, okay, you're doing this. I'm going to log it. I'm going to make a note of that in my, in my long list of audit events. And if you do something bad after this, I'm going to deny everything you do. So that's how, that's how this works. Again, the other example that we alluded to is the LD preload being set twice. You hook at the BPRM check security, which is essentially the LSM hook for process execution. You say like, give me the value of LD preload. And this helper is like smart. This looks at all, all the environment variables. It gives you the value back. Even if the attacker here tries to like overflow the environment variable buffer, like it's, it's like giving you, you know, these environment variables can be really large. They can be two thirds of your, I think three fourths of the maximum stack size, which is roughly translating to six MB. So they could just set like a big environment variable. It also counts how many times this is set. And the, and the detection is based on like you setting and leave preload twice. And this is where you want to, you can get the value. You can get notified of an overflow here. And then you can say what you do when these two things happen or when one of these two things happen. So again, use that as a building block to assemble your Mac and audit policy. You want to, this is again the file open LSM hook. There is Keras says again, providing you a helper for, is this a process operation on the, on the procFS file? There is, there is some special hooks in the LSM that allow you to do and track such things. So whenever the process starts, you get an inode object pack and you can say that this is, this is a file that this process created in procFS. And when someone tries to read a right to proc grid mem or like open proc grid mem in this case, you can, you can log this. It's probably okay. Like you may be writing that, but we want, you, in this case, we want to log that. This happens a lot for a different, if a different process tries to write to a proc grid mem, it is very rare that a process tries to write it to its own. There are some cases where this is genuinely required. That's where the auditing comes in handy. Okay. So we, we have explained some detections there. We, the plan is to build more detections and use the flexibility of the BTF stuff so that at the very least by exposing all the hooks, you can create an LSM. If you need to access security blobs, this is again something that KSI will have to build internally. You don't expose that. But you can create these, it already exposes the LSM ecosystem to some, something that you can build using eBPF. This is a question that I was asked last time I talked about. There was a new patch set that was sent a couple of days back. I didn't get a chance to look at it because I was traveling. But what we found is that line lock is primarily targeting discretionary access control and it needs unprivileged eBPF. KSI is system wide mac. So it, it, we don't rely on unprivileged eBPF. You are, you have the, you have all the capabilities of route. So we don't want unprivileged eBPF for now. It has a, it has a second like attachment semantic where processes are like, they're attached to essentially task credentials or like a second data structures on the, on the task. It's theoretically possible to do system wide mac using these attachment semantics, but it gets complicated because that's not what it was designed to do here. Right. You can, you'll probably need to change in it. You have no trivial way to replace the attached eBPF programs. Kernel threads, like I, we, we noticed that KSI hooks get executed for user mode helpers. Every now and then you will have a user mode helper come in, execute a binary, and this won't be logged by landlock if I, if I, if I understand that correctly. So there are, there are some things, I think landlock has its place as if unprivileged eBPF is a thing for, for unprivileged sandboxing, but it's not the right design for like system wide mac here. Why not audit? Well, like the obvious answer is that mandatory access control needs to have like, be handled separately. And we want to do that together in one place. There is considerable, with all due respect to audit, there is like, it's, it does give you a lot of information, but there is performance overhead. So like it, when you, even when you enable configure it on the, on the, on the kernel, like it causes a performance penalty. And it also imposes like in, in KSI, when you get information out of the kernel, you can choose the data structure in, in, or the format you want to assemble and write to the performance buffer. In this case, like you get strings back from the kernel. So that is, that there are some formatic constraints that we run into when we like essentially build detection pipeline on top of audit. I did some preliminary performance comparison with audit. We have some grains of salt at the very corner. So please take it with the grain of salt. I brought, I didn't get enough for everyone, but I think there are enough grains. KRSI is the blue part. So the, the, not the exact numbers, by the way, the, the Y axis is like, it's clipped at 500. So not doing any marketing trickery here. But the, the essence is that audit has a very fat distribution. So it has like high tail latency. It can have bigger impacts on the system. And even when you enable audit, you get performance penalties there. So that, that's the, that's the kind of, that's the kind of preliminary analysis that we found for process execution events. And we couldn't log environment variables without it because we have to build that functionality anyways. So that's, that's how it compares right now. And this is a question I got asked as well. Like, why don't you just go and trace the LSM hooks? Like you have BPF probe type trace. It's, it can do, or like you have trace points. Just create a trace point and trace all the LSM hooks. Seems like an interesting proposition. But you can, first of all, you lose all the formal verification stuff that we, the LSM tries to bring you. Leaving that aside, you are, you are, it's not trivial to override the return value using current BPF tracing programs to signal a MAC decision. You have no access to security blobs because you, you want to write into security blobs. This is not something that you're allowed to do from BPF programs. At least we don't want KLSI programs to be writing arbitrary data into the CUN. And it's not really an LSM. If you go through all the security underscore functions in security.c, the hook, these security functions are wrappers around hooks registered by LSMs. And they are, they do, they, sometimes they do some processing on the data before they pass it to the actual registered callback. And you miss out on that if you're, if you're just tracing LSM hooks there. So that's one of the concerns we, we, we ran into. I'm going to quickly like do a demo. I'm going to pause a bit while we do that. So hope this. So I'm, oh, great. This is how you can mess up a screencast demo as well. Anyways, so we're starting a VM that is running KLSI as the LSM enabled along with the other LSMs. And, and then I'm listing the hooks to show you that they're all these LSM hooks available. And we will expect to see more because there are places where the LSM hooks, like we, we just not got the feedback. I need a hook for that particular point. And then what you have is I'm running the, this is the user space component that is essentially loading your eBPF program. So it's loading like four eBPF programs into, into the various hooks. So there's one for environment process execution. There's one for inode unlink. There is one for like the procfs file op file open. And, and we, oops, just a second, my laptop is, I did spill coffee on it. So that could be one reason I'm going to go to the webcast here. You cannot beat Murphy's law. This is like, this is amazing. So, yes. So I, whenever, when I, I, I assess to that machine and I go to the other, other side of the terminal, I see like a bunch of execution information getting audited there. I run this thing called mem, which is essentially a program that reads opens proc, but mem and tries to write, write to that. And what you see is an event that says proc blah, blah, blah was accessed, right? You also have another one. I run another program that's called fault, which is basically, you cannot set these environment variables on bash, like twice, bash actually filters them out. But you can maliciously in the execs is called like past the environment variable vector twice. And that's what this program does. I have excellent values for them. But it gives you a warning that this is set twice. You can actually deny the program from being executed as well. And, and then there is the famous, the executable unlink where this is basically removing argv zero from, like calling unlink on argv zero, and you get like an unlink event on the, on the system. So that's essentially, believe me, it was there. It's again detecting one type of unlink event. This is unlinking itself. You could have other types of events as well that you can detect on. So, yes. This slide has appeared multiple times as a cause of my failure demos. But again, thank you so much for all the people who have contributed to this project and who will in the future as well. It does, it is not possible to build these things without all the contributions. And thank you for listening to me. And if you have questions, I'm here to answer them. It's almost inevitable when you introduce something that is sophisticated as this mechanism, you're going to have an exploit. Where would you see that happening? I mean, the answer, the answer has been already brought up in the, in the, in the room is EBPF needs to be, it's the, it's the place where you could have potential. But that's where it's an opportunity to like work with the maintainers and make it more secure and safe. And that's actually something we're looking at. The answer is, the room knows the answer. But yes, EBPF could be the place where you could introduce exploits. Questions? If not, let's thank the speaker again. Thank you.