 As Lea said, I'm here to talk about IOU ring, but first, you know, just a thank you to Elena. She does a lot of work coordinating things and, you know, keeping the program committee in line. So a big thanks to her, as well as the Linux Foundation, all of our sponsors that make this happen, you know, recording the room, the stuff isn't free. So a big thank you to all those people that help make this work, too. But anyway, IOU ring, so fast it's scary. We're going to talk about a lot of things, but before we do that, I think it's important to give a quick overview of IOU ring. Hopefully, you're not hearing about this for the first time, because I've only got a couple slides on it. But if you haven't, and this is something that interests you, I would encourage you to go out. There's been a lot of talks and presentations, blog articles, and whatnot written about IOU ring. So just type IOU ring into your favorite search engine. You'll get plenty of information. But the basic idea of IOU ring is it's a form of asynchronous IO. And asynchronous IO is pretty cool, because it allows an application to submit multiple IO requests without having to wait for them to complete. So it can say, hey, go off, write this buffer to disk. And if you do that with a write, just call it. You've got to wait for that to complete. You get the error code, and then you can move on. With this one, you say, hey, here's the buffer. Here's the vial descriptor. Go write it to disk. I'm going to go off and do other things. And then I'm going to come back later and check to see if that worked. So it's kind of cool. You can do other things while you're waiting for your disk or network adapter to catch up. And this is pretty nice, because traditionally, if you wanted to do a lot of IOU, you would have a pool of threads or other processes that would go and do this so that you wouldn't block. You could send it off to this pool. The pool would do all the servicing, and you could keep going on. The problem is if you've got a bunch of processes, you've got overhead of context switching and synchronization. If you've got threads a little bit less, but you still have issues around synchronization and still bouncing back between the user space and the kernel. So asynchronous IOU allows you to do a lot more IOU with a lot less resources and a lot less synchronization and overhead. So it's pretty nice. And like I said, if you Google around, you can see some pretty impressive numbers through this stuff. Now, IOU ring isn't the first form of asynchronous IOU that the kernel has seen, but it's undoubtedly the most popular, and it's likely going to be far more successful than what we've seen. In fact, I'm pretty sure at this point it already is far more successful than all the previous attempts. And it does this basically because it's got a pretty simple ECDU interface. It's basically composed of two ring buffers. These are shared memory between the kernel and user space. One of the ring buffers is a submission queue. So you basically form up your IOU requests, you put on the submission queue. Kernel takes that, does whatever you've requested of it, and then it puts the result back on a completion queue, which then you can go back and read whenever you like. And there's also an optional kernel thread that exists. Basically, it's a kernel thread that basically pulls the submission queue. And whenever it sees a new submission on there, pulls it off, processes it, dumps down the completion queue. What's kind of neat about that is you can do IOU without any sys calls once you've established the IOU rings. So you just basically write your request into a structure, put that on the submission queue. You don't have to do a system call. That kernel thread will pull the submission queue, see it, and do your work. So it's kind of neat and allows for some really impressive performance numbers. And I guess the last thing I wanted to mention about IOU ring is this is really a pretty thin layer designed to kind of replace the sys call layer for these IOU, for these IOU operations. It's not intended to replace all of the existing kernel mechanisms for IOU. So for example, IOU ring, it gets these requests for IOU. And if you're doing file IOU, it leverages all of the existing VFS and file and device infrastructure that the kernel has. It doesn't duplicate any of that. So it's more or less the same code path of, you know, open read write. It's just it really kind of replaces that sys call layer and all of the overhead there. So that's also kind of nice. It's a lot more manageable and you're going to get a lot of the same behaviors that you're used to seeing with the traditional synchronous calls. Now that's pretty much all I'm going to say about IOU ring for background. So I'm just going to pause for a minute if anybody's got any questions about that. You can just raise your hand. No? Okay. Do we have any on the virtual? All right. We're either still asleep or I've explained it exceedingly well. So as the saying goes, you move fast and you break things, right? It's important to know it's not move fast or break things. It's move fast and break things. And IOU ring is definitely very fast, which means it's definitely going to break things. First thing is it broke audit. IOU ring operations for quite a while bypassed all of the audit mechanisms in the kernel. So you could submit IOU and you would have no idea that it was happening if you were looking at just at the audit logs. Now for some of you that don't use audit, probably not a big deal. In fact, you'd be like, hey, less performance overhead. But there's a lot of users out there that do require auditing either for legal reasons or security requirements or they just like to know what's going on with their system. So this turned out to be actually a pretty serious problem. The reason why is, as we talked about, IOU ring is intended to kind of bypass the syscall framework. And audit has traditionally always relied on the syscall framework being there and part of the code path that applications took to perform IOU because before IOU ring, that was pretty much true. And it uses syscall framework to basically do all the setup, initialize, you know, the VFS logging and whatnot and doing all that. So without that, auditing didn't happen. The other thing, and perhaps even more serious, was it broke all of the LSM access controls. So think if you've got SE Linux, App Armor, SMAC, any of the number of LSM access controls, and you had it restricted so that only certain processes, only certain security contacts or domains would be able to do certain sorts of IOU operations, well, you could bypass all of that with IOU ring. The idea is basically this concept around credential sharing or what IOU ring calls personalities. The idea is that with IOU ring, you have the ability to store your credentials in the IOU rings itself. So imagine you were a privileged user, root user, whatnot. You could store your credentials in an IOU ring and then hand that IOU ring off to an unprivileged or normal user. And that normal user could then go ahead and perform any IOU ring operation as that privileged user. So, oh, and there would be no LSM oversight of this. There would be no LSM access controls imposed on, you know, the transfer of credentials, the delegation of privilege across this. So that was pretty scary. That was actually kind of terrifying when we first figured that out. And like I said, this was called personalities, which we're going to get to this a little bit later. But I think that kind of masked this for a while, because those of you that play with the Linux kernel may already recognize the fact that the Linux kernel has a concept of personalities already. So if you were reading any documentation at the time, you'd see like, oh, okay, IOU rings, you know, allows people to, you know, store personalities. You might think that's a little odd, but okay, whatever. Well, turns out it was credentials. But that was not it, actually. And this was really kind of funny because after we had a lot of, and once we realized there were these issues around IOU ring, we had some pretty long discussions with the IOU ring developers. And for the most part, they were very cooperative. And they kind of understood that, okay, these were serious problems that needed to be addressed. But, you know, as is usual, it takes a while to get there. And it took a little bit longer to get the holes fixed. And so we got those fixed. And that was good. And then as happens in the Linux kernel, functionality comes along. There was a new IOU ring operation that was proposed. You know, some people, you know, notify the LSM group and said, hey, you know, look, they've, you know, the IOU ring folks are implementing something called a command pass through, you know, what do you guys think about this? This is, you know, can we put an LSM hook here? And we looked at it and we provided them some feedback. And, you know, we said, hey, the API doesn't have enough context for us to be able to make a good access control decision on this. We kind of need some more information. And then things kind of went quiet, you know, for a couple months. And can anybody guess what maybe happened a couple months later? I mean, somebody's got to have an idea, right? Let me just randomly guess something. Otherwise, it's really quiet and boring. Richard, close. Yeah, very good. But, you know, what's the implication of that? Right? You know, if you had to guess, did it go well? Yeah, they broke it all again. So, yeah, evidently, evidently. And this is why it's never, never a dull day being a security developer in the kernel. So basically, yes, they put in this command pass through which, without going into details, I think we're all familiar with the IACTL, CISCALL, right? And kind of envision this as IACTL, but maybe worse, if you can think of that. It basically allows arbitrary users to submit arbitrary commands directly to hardware or to kernel modules in the kernel. Once again, without any LSM oversight or any access control points. So that was fun. We did have some auditing, though, so I guess they didn't really break auditing too badly this time. So, baby steps, I guess. Anyway, so those were our problems. Those are some of the things we had to solve. And even though they broke them relatively quickly, it took us a little while to go ahead and fix those. So, here's how we went about doing it. First thing we wanted to do was we wanted to restore audit visibility. Simply because once we have visibility with audit, it makes it easier to fix the LSM access controls and do a lot of other things. So, first off, we had to kind of teach audit about IOU ring and say, you know, okay, these are, it's like a CISCALL, but it's not exactly a CISCALL. So we had to create a new audit record, which we call U-ring-up. I know, we're not creative people. Then we had to support all the different multiple calling contexts that you can get IOU ring in, because as we talked about earlier, it's asynchronous, which means you're going to have multiple different ways that IOU operations can get serviced inside the kernel. Sometimes you actually get it, you know, say, after this is called, finishes executing, but before the kernel returns up to user space context, other times you could get it through that dedicated kernel thread, and then there's another thing, IOU workers, there's a lot of different places. So, we had to do some work in audit to support all those different contexts. And basically, we had to kind of pretend that IOU ring was basically a CISCALL. So all those hooks that we had around the CISCALL framework for when you first enter the kernel and then exit back out, we basically just put very similar hooks with a little less overhead into the IOU ring code before and after it serviced an operation. So we, from audit perspective, IOU ring is very similar to a CISCALL, which allows us to basically reuse most of our mechanisms in audit. And for the most part, that seems to be working well. Then once we got that sorted out, we could actually see what was going on inside the kernel with IOU ring. We moved around to actually putting the LSM hooks in there and kind of restoring access control points to IOU ring operations. So the biggest thing, like we talked about earlier, we had the personalities of the credential sharing. So we wanted to make sure that we had a control point in there so that whenever there was a task that was going to execute an IOU ring operation with the credential other than its own, we wanted to make sure that went through an LSM hook so that the LSMs could weigh in on whether that should be allowed or not. And thankfully enough, I guess if there is any kind of positive here, if you're going to transfer an IOU ring from one process to another, you basically just transfer it as a file descriptor and you can already transfer file descriptors between processes. There's several ways to do that. We already had hooks for that. So that was covered. So we were all set there. We just needed the LSM hook for the credential delegation. And the only other hook we added in is we added a hook for if you were using a configuration that spawned a kernel thread, we put a hook on that. And there's a variety of reasons, one of which, at least from a credential point of view, was that when you create one of these kernel threads to pull for new IOS submissions, it inherits the credentials of the task which creates that kernel thread. And also because of if you have this kernel polling thread, you can do IOU without ever having to issue a syscall. And as some of you know, there's things like Sec Comp and other things that work, that provide some sort of security mechanisms at the syscall layer. We wanted to make sure that there was an access control point so that, you know, if you wanted to block certain applications, certain users from using this dedicated kernel thread to do this and potentially bypassing some of these other security mechanisms, we wanted to make sure we had an LSM access control point to gate that so that you could define your security policy to either allow or restrict that. And as I mentioned, there was the command pass through, the little surprise addition at the end. We basically, we had to add in another LSM access control point for this because there was just no other way to control this. And it's particularly crude right now. It just went into the kernel not too long ago. We're going to backport it into 5.19, it looks like. Like I said, very crude. There's not much granularity to it. It's either you can do it or you can't. We don't have the ability, because as I mentioned before, there's not enough context there to make a good access control decision to say, we want to allow this particular command to be passed through, but we want to block this one. We don't have that ability right now. However, you know, and this is the benefit of, you know, being able to come back and have these conferences, was able to get together with the developers and the maintainer that put together the IOU ring command pass through earlier this week. We had a lot of good discussions that above. I think we've kind of, we've sorted things out a little bit more. I think they understand our concerns a little bit better, and they're going to start looking at ways to improve that in the future. So that's a positive thing. It's crude right now, but I think there's hope that in future kernel releases, we'll have a better solution here. So probably on to the next thing. Does that, how's everybody doing? Does this all make sense so far? Zero questions. Nothing. Wow. Guys awake? Laugh a little so I know you're awake, something. Okay. Richard's got one. Hi. Richard Briggs. I have worked with Paul in the past. He was at Red Hat a little while ago. When the IOU ring stuff first came in, we approached them and said, well, we need some insight as to what's going on here. And they said, but that'll make it slower. And they were pretty reluctant to work with us at first. And we basically said, well, the auditing stuff is fairly important for a number of our customers and for other users of the Linux kernel. And if we're not able to get your cooperation and putting stuff in here, then we may need to submit a patch upstream to make audit and IOU ring mutually exclusive kernel configuration options. And at this point, they kind of went, well, all right, we'll work with you. Yeah. And it's a good point. And it's something that comes up fairly often is, you know, especially around performance oriented features like IOU ring, is somebody will say, well, we can do this as long as we, you know, we don't care what you do from a security perspective. We'll just make it optional. And so you can turn it off. And that's always going to be useful because there are certain use cases, certain users that they really they need every, I mean, every microsecond, every now second of performance they can get out of their systems. And that's fine. I'm not going to criticize those people. But I would say most users, and there's definitely a class of users where it's a strong requirement. If they don't have these security features, they can't deploy these Linux systems. I mean, it's a hard requirement for them. But we still want to be able to offer more these new performance functionality too. And so it's good if we can make security features optional, but at the same point in time, it's very important for us to find a way to develop the security functionality and the performance functionality in such a way that they can coexist. And they need to coexist well, because if it's awkward, like for example, with the command pass through right now, where it's a very coarse control, there's going to be a lot of situations where people are going to have to basically allow a command pass through for more security domains than they want, and for perhaps more commands than they want, because they simply don't have the flexibility and granularity in the security functionality that they need. So it puts users in an awkward situation, where they're either going to have to disable something, or where they're going to have to basically enable far more access than they want. So if there's anything you take away from this talk, it's please go out, find a way to work with each other. That's the key here. And when we don't work well together, we end up in situations like this, where we could have multiple kernel releases spanning several years, where we have huge security holes in the system. And that's not good for anybody. And that's kind of a good little segue. I don't know if, well, I know Richard hasn't seen the slides, but it's a good segue into the next session. As an LSM community, we're doing a pretty poor job at identifying security critical kernel changes outside of the LSM. And this is not a new problem. This has been going on for quite a while. And we do spot some changes. So I mean, it's not like we miss everything, but the kernel is a big space. There's a lot of stuff going on. And I think everybody in the LSM space is kind of heads down, you know, we just we don't have enough time in the day. There's not enough people looking at code. Priorities are always in the state of flux. And we just can't keep up with it. Making things worse is that a lot of these other security or a lot of the other kernel subsystems, especially those that are focused more on priority or on performance, generally don't engage very well with the LSM community. Because why would they? We slow things down, right? You know, it's not in the front of their mind when they're developing new functionality. And I don't mean to single anyone's subsystem out. I'm using IOU ring today because it was a recent example and I thought makes for a good use case. But there's definitely other subsystems that we've had problems with in the past. And it could be for a number of reasons. I think it changes depending on the individual and perhaps, you know, time and whatnot. But much like we're doing a bad job with this from an LSM side, I think they're doing a bad job of it from the other side. And these are both not new problems they've been going on for as long as I've been playing in the Linux kernel space, which is quite a while at this point. So I don't really see either one of these things improving in the near future. And it gets even worse. This is a real downer section here. Even when we do cooperate, there are still some challenges. I talked about some of the terminology differences earlier where we've got in the LSM world, we typically talk about credentials and, you know, even the kernel, it's struct cred, right? But when the IOU ring folks were implementing this, when they were caching the credentials, they kept referring to them as personalities, which like I said, is perhaps doubly confusing because the Linux kernel already has a concept of personalities. And so funny thing was when IOU ring first came out, and there was an LWN article about it, and there was some other information about it, kept referring to personalities. And so I'm, you know, I'm quickly reading through the article because, hey, new feature, I should look at this, see if there's some security stuff we should look at. Didn't even register with me that there was credential sharing because I would see personalities. And like I said, I thought, this is weird. I don't know why they're caching personalities. But okay, there's probably some reason that I'm not aware of. Had I know, had they used the term credentials, when I was reading this information, it would have jumped out right away and was like, oh, okay, we need to go talk to the IOU ring guys. This is, this could be an issue if we don't catch this soon enough. Another problem we have is that, you know, especially for new functionality, you know, those of us in the LSM space, we know security pretty well. We know the LSM's pretty well, but I didn't know anything about IOU ring till I first read about it. And on the same or on the same concept, but from their point of view, they knew IOU ring very well, but they didn't really know what an LSM would need. They didn't even know that an LSM would care about IOU ring because, well, it still uses all of the same VFS code paths, and you've got access controls for VFS. It should just work, right? So I think everyone has good intentions, but it doesn't always work out the way we need it to work out. So how can we do this a little bit better? Well, my thought was maybe we need to start doing some automated code inspection. Coordination is still the best. And please, these next few slides, don't take this as an excuse to not go out and talk to other people about what they're doing and figure out, you know, okay, how can we work with you to make sure that, you know, the kernel security remains intact and things are good going forward? So please, continue to do that. But I think we also need to be realistic that it's not going to work as well as we need it to work. And we've got several years of proof that that's the case. So my hope is that perhaps if we can come up with some automated code scanning, can help focus what limited review resources we have on to commits or changes or new functionality, new subsystems that are security relevant, that do potentially need some extra scrutiny. And so this is kind of well outside my wheelhouse and area of expertise. But I was thinking like, well, okay, what, having done this for a while now, what are, what are some of the things that usually stand out when we have these problems? What, what could we use as potential flags or patterns to see that, okay, maybe we need to have some extra LSM access control points in there? Well, one thing, and I've actually been doing this first point for a couple of years now since I started maintaining some kernel code is I have a small little script that runs, you know, fairly regularly a couple of times a day. And it looks for changes in LSM subsystems that happen outside of the LSM tree. Now, there's plenty of reasons why this is okay. And this happens every kernel cycle. There's a few kind of innocuous little changes that go in. And that's all right. But occasionally, you'll have somebody that'll come in and, you know, they look at it and they're like, I know a little bit about the LSM. I mean, I know a little bit about SC Linux. I can just go ahead and drop some code in there and it's fine. And of course, they don't CC the SC Linux list because that's all right. They've been doing this for a long time. They know what they're doing. And for the most part is okay. I mean, it's only a couple of people that do this. It usually works out pretty well. But there have been the occasional case where it's like, hey, I see what you're trying to do. That's fine. But we need to kind of change this a little bit. So that's a good thing to catch. Another thing is if you've ever looked at the kernel source code, you've probably run across an LSM hook at some point. They all start security underscore blah. So basically wanted to look at all of the LSM call sites scattered throughout the kernel and just keep track of those and try to see like, when do those change? Do people move them around? Does somebody add a new one? Is somebody maybe taking a hook and said like, wow, I'm adding some new functionality. And it looks like maybe this LSM hook could also be used in my new functionality. And maybe it can maybe it can't, but it's something that, you know, maybe we should take a look at. Also, we talked a lot about credentials. So I'm not going to hash over that again. But basically anytime there's a credential override in the kernel with there's actually a specific kernel function for that. So we want to watch that carefully. And if somebody's doing that, you bring that to our attention. Also anytime the kernel makes a capability check, while most Linux kernel developers, you know, they might not necessarily be super familiar with the LSM. And when there's a hook that needs to be placed there, they are pretty good about figuring out when you need a capability check because that's the traditional Linux, you know, permissions model. And so they might put in a capability check but not do anything with the LSM. And if we had had this in place before IOU ring went in, we probably would have caught the IOU ring stuff because they did have the capability checks in there, even though the LSM story was lacking. And finally, I want to put a check in there. Whenever there's a new IOU ring operation that's added to the kernel, you know, let's watch that because as we've seen, IOU ring is a bit of a special case because it doesn't have that syscall layer. So this might be something that we need either for auditing or for LSM access controls as we saw with the command pass through. So we got a list of things. I had an afternoon. I was like, you know, okay, let's let's see what I can do with a couple hours. So I'm going to there's a link at the end of the slide. You can go look at my scripts. I put them up there so you can grab them and take a look. Here's just kind of a quick example of some things that I'm doing. And this is all running out of bash script. This is not fancy. Like I said, this is an afternoon's worth of work. That first they get log command line. If you're not too familiar with get this basically just compares two branches, two trees and basically says, okay, what's in, you know, compare the LSM next branch with what's in Linus's master tree and basically check to see, you know, Hey, is somebody putting code in the security directory that hasn't gone through the LSM next tree? You know, for example, are the VFS guys, did they add something in did they add a new LSM hook or did they change one of our existing hooks? So just kind of gives heads up if something's going on there. And the next thing is this is perhaps it's a little more interesting. So the first thing I'm using a tool called C grab, which I don't know if you're familiar with that. I spent when I first started this, I had never really heard of secret before, but it popped up when I was searching because grep is nice, but it doesn't always necessarily work extremely well with source code sometimes, you know, lines wrap. There's other oddities that won't go into but I started off using grab and it was kind of a little awkward. So I looked around found C grab and it works beautifully. The parameters here dash dash code that basically tells C grab, you know, Hey, ignore all the comments, because oftentimes people put function names and comments, which is good, but I don't want any false positives there. No column and no numbers that basically, you know, in the grep output, you know, you can see a row and a column for the file. That doesn't display those because, you know, if I was doing that in a snapshot, that's going to change quite a bit. So I didn't want that. And the dash dash word is just basically like I said, C grab is kind of intended for going through source code. This basically says, Hey, look for an identifier override cred. So, you know, function names, you know, variables, that sort of thing. And as you'll see, I'm not blindly searching through the entire kernel. When I was first doing this, I was a little concerned that I was going to get a lot of noise. A lot of change that was going to be continuously flagging a lot of false positives. So I wanted to kind of limit this to particular areas. You'll notice like drivers are noticeably missing in that subters list, which in general is okay because from an LSM, from an auditing perspective, there's usually not a whole lot we do with individual drivers. We try to stay up a layer because otherwise the amount of work adding hooks in would be tremendous maintaining those. But you can see it is a pretty, pretty full list of things I was looking at there. And so once I had these commands, it was just a matter of, you know, throwing them into a quick little bash script and this runs multiple times a day. It's pretty simple. I execute those commands and a few others. It basically generates kind of a snapshot of the kernel at a point in time where it calls out, you know, all those different call sites, all those different, you know, git history. And like I said, there's no function or there's no line numbers and there's no column numbers. So if there is a little bit of, you know, patch merging and whatnot, if they move up and down a little bit, that's okay. It won't necessarily catch all those. But it still catches kind of the horizontal white space. So if the level of indentation changes, so like let's say somebody puts a LSM hook, it was inside an if statement or if it was inside a loop and now they bring it out, that'll show up, which one would think that might actually be kind of helpful. So we'll see. So anyway, it creates a snapshot. It compares the snapshot to the previously run snapshot. And when I say compare, it just does a diff. And then, you know, if there are any differences, it sends me a report via email. And then the last thing it does is it replaces the previous snapshot with the snapshot just generated and then, you know, wash, rinse, repeat, just keeps doing this multiple times a day. Like I said, this is a very simple first attempt afternoon worth of work. I was like, okay, let's see what this gets me. I mean, is this worthwhile? Is it just going to be all noise all the time? Is it even going to find anything interesting? Well, so far so good. So I did this probably about in the middle of the 519 RC cycle, I think it was around RC4, RC5 that I first turned this on. And like I said, my biggest concern was the amount of noise that would come out of this. And it actually turns out the noise is very minimal to almost non-existent thus far. It's been a limited number of times, but it seems to be working pretty well. Turns out that the changes that it catches outside the merge window have been very small. There's not been a whole lot there. Most of the things that do pop up there are fixes for code that went in during the merge window. And, you know, people realize, oh, this isn't quite working the way we want. So we need to change this a little bit. So and that that's to be expected. And so that was that was pretty good. So what actually what are some of the positive things that it did catch? Well, during the 519 to the 6.0 merge window, IOU ring kind of reorganized their code and decomposed a lot of things into multiple files. And when they did this, this caught all of the LSM call site changes and the capability changes. So that was good capability wise, so not necessarily specific to the LSM. It did catch a number of debugging mechanisms that were added during the last merge window and that, you know, were gated with Capsis admin. And but perhaps I think one of the more interesting things, and this kind of surprised me, both the good and the bad way, good that this tool actually caught it bad in the sense that this was kind of one of those like, oh, man, how happened does or how often does this happen sort of thing? I'm not going to list the specific commits. I want to shame anybody here. But there was a PROC FS tunable that like a lot of PROC FS tunables was gated with Capsis admin. And this commit went in and they changed some things with the tunable, and they removed the capability check. And so I sent a private email off to the patch author and the maintainer and said, Hey, I know very little about your subsystem. I don't know if this was intentional. It wasn't mentioned anywhere in the commit description, but this commit removed a capability check from this PROC FS tunable. Was this was this intentional? Is this okay? And had a little discussion and neither the patch author or the maintainer had realized that Capsis admin had been removed. Now the good news is we had a bit of a discussion some back and forth. And it turns out that maybe that was okay. Perhaps it didn't really need to be there in the first place. So that was good. But, and this highlights once again how important it is to go out and talk to people. When we were having this discussion, they realized that, okay, maybe there are some other parts of their code that do need a capability check for Capsis admin. So this was a little bit of a surprise. I wasn't expecting to see this, but it was kind of nice to see that at least we're on the right track with some of this, that maybe this could be useful. And so next steps. Once again, I keep harping on this, but don't take away from this talk that, you know, we can do all of this through automation. Keep talking with each other. Also, while this little automation toy has had some limited success for a small period of time, it's pretty crude. It's really, it was just aiming at low hanging fruit to see if this is something that could have some value. And I think there is a lot of potential here. I think, you know, one of the simple next steps would be to identify some more complex code patterns. You know, there's a lot of tools, Kachanel, which I'm sure I'm pronouncing incorrectly, but that has a lot of power to do, you know, some code analysis. It's already used in the kernel source tree for doing other things. I think that might be something we could leverage here or similar tools. I'm also, once again, this is well outside my area of expertise, but I know there's been a lot of focus recently on machine learning and other forms of pattern matching and whatnot. That might be an area that could yield some fruit here as well. Basically, my hope here in giving this talk is that somebody either in the room or on the virtual session who has a bit more experience in here that could potentially find this interesting and run with this a little bit, because I really think this is perhaps our best hope at preventing the next IOU ring problem. And with that, my last slide, I promised you a link to the scripts that I'm using. That's it right up there. These slides I will put on the scheduled thing later today. My apologies. I got back, sorry. I got back late last night from an event and didn't get these slides up there, but I will do that today so that you'll have this URL. Otherwise, you can take a picture of it. But that's my email address if you want to be in touch. It's also my Twitter handle. If you want to be in touch that way, whatever works for you. I guess the only catch I will say about the scripts is those scripts are intended to be run from a little kind of home grown job scheduler thing. So there's some stuff in the scripts that might not make a whole lot of sense. So you can just ignore that and just look for the C grab and to get commands and make sense with that way. So that's it. Are there any questions now? Someone other than Richard? He's a nice guy and all, but come on. All right. Oh, do you have one, Richard? So the automated stuff is good to have in there because it can catch certain things, but then you have to constantly update it to look for certain keywords. And so that's going to be a bit of a challenge, but it's it's part of the equation of trying to get at some of this stuff. So you're talking about the, you know, keep on talking idea. I mean, I guess the one comment I will make. So keyword searches, I mean, the keywords I'm searching for, you know, the credential overrides, the capability checks, those are fairly static. I mean, those don't really change. So keeping on top of that is easy enough. The LSM hooks that we look for in those call sites, that's generated dynamically from the kernel source and the script. So the overhead for that is very minimal. Sure. But then there's entire new subsystems and things like looking for, looking in the entire IOEuring thing. Now that you know that it exists, you know, to look there for that kind of change. When, so it, it turned out that Paul and I had actually been at the same IP sec bakeoff many years ago and didn't realize it until I pulled up a bunch of photos of it and Paul's kind of like, yeah, there I am in that picture. Oh my God. Okay. We went through this stuff when looking at IPv4 and bolting on IP sec onto that, it was, it was a security bolt onto the side. And so when IPv6 was being developed, there was active work to try and integrate security in it from the get go so that we weren't trying to play catch up. In 2013, when Edward Snowden blew the lid off the NSA, at that point, the IETF had, I guess they added a bit more formality to their process. And if in every RFC after that, there was a section called security considerations, and it had to be reviewed by people to be able to make sure that what was being added was going to not cause some new problems. So the Linux kernel development process is a bit more organic. And I mean, there's some quote from Linux about you guys, you security guys are crazy, or insane, I think was the name of the word that he used. And so there is some reluctance from the rest of the Linux community to be able to acknowledge the security concerns that we do raise. But that's the kind of thing that I would like to see is in any new features or patch sets of some bit more formal security review being added to the process. I agree. I mean, I think it would be wonderful if any new, you know, kernel subsystem or significant bit of functionality did have to go through a security review and at least, you know, get a quick thumbs up or thumbs down. But the hard reality is that's not going to happen any time soon. I mean, we've, there's been several cases that I can point to where we've explicitly requested, you know, new functionalities come in, we said like, Hey, hey, can you, you know, like, you're going to release, it's going to be in the next kernel release in like two or three weeks. Can you, can you hold this for a second? Because we need to add some extra hooks. And they're like, Well, no, we're not going to gate new features on the LSM. And we've heard that time and time again earlier this week at plumbers. There's a new IOU ring functionality for spawning new executables basically replacing the whole fork and exec process with IOU ring. And there's a couple of us that were in that talk. And, you know, we asked, you know, Hey, you know, can you please share the code with us? Can you please CC the LSM mailing list? And they said, Oh, sure, we haven't thought about it yet. But yeah, we'll do that. And we're happy to work with you as long as you don't impact performance. So I mean, this is the sort of thing that we're we're up against. And so, you know, I think if we're careful and we're persistent enough, you know, we can make inroads, but I'm not expecting this to change substantially anytime soon. The LSM is it is it is and the LSMs are becoming more and more common. In fact, I'm pretty sure every major enterprise all links distribution has at least one LSM enabled these days. So it's not it's not a fringe technology anymore. Case? I just pulled your scripts really quickly to take a look at just how are you gripping please don't I'm not a script person. Oh, no, I'm just trying to look at it and yeah, see your approach to it. And I'm in looking at it, I realized like, Oh, wait, I'm sort of already doing this a little bit for some of the hardening things because I ran over the same problem of like, Oh my God, why did you touch that? How did didn't you CC me like what? Yeah. And while it's not a perfect solution, listing a couple of the like file or or identifier regular expressions in the maintainers file will at least trigger adding someone to CC when certain things come up. So it might make sense for I mean, it's a pretty limited set of words. Yeah, you know, it's like, are did you change capable? Or did you touch capable? Did you touch NS capable? There's a short list of them. And again, it's not 100% because not everyone runs get maintainer. Yeah, but it has worked pretty well for for me when I added those types of things for things and like sec comp and like, I get, I get CC when an architecture ad sec comp support and I can go did you add a self test stuff like that. So it might be nice to like do it on two fronts, like you're proactive trying to find them. And then this more sort of passive add a bunch of keywords, like, Hey, if you touch any of these CC the like LSE maintainers subsystem. It's a very good point. I hadn't really thought of that. But yeah, I mean, it's because and that would be much more preferable. That would get us closer to the let's have a conversation beforehand. So yeah, that's definitely something to look at. It's fun. My laptop just beeped at me. So find out what that means. So I'm just curious, when there's been a release where there has been a missing LSM hook or something like that, does that generate a CVE? Because arguably there's now a bypass for security control. So there's a fun topic around Linux kernel CVEs that we could probably spend a couple hours on. I'm personally of the mindset that CVEs aren't anything to be ashamed of and we should have CVEs for these things. But there's some prominent kernel maintainers that have a very different view of those things. So, you know, they're also as an individual kernel maintainer. Well, as an individual open source maintainer, the path to getting a CVE is fraught with a number of difficulties. So I usually try to make people very aware when there are issues. And we work hard to get things back ported whenever we can. There once again, there are other issues we face with getting security fixes back ported specifically around IOU ring. I think we are going to get it back ported. So that's good. But yeah, I think we, or at least my approach thus far has been to make people very aware of this. And I've seen that a number of Linux distributions have then created CVEs for them. But unfortunately, my results have been very mixed trying to do it as an individual. So I just rely on getting the word out and hope that the Linux distributions will pick it up and do the right thing in terms of CVEs. So, okay, thanks. Yeah, I'm sorry. It's not very satisfying. And I'll be the first to admit that. But there's a number of process problems here as I think you found from this talk. So we're just trying to do the best we can. One more question. So I think we, well, I don't know. Am I over time? I'm at time. Okay. Well, thank you very much. I hope you enjoy the rest. Sorry. Last question. Sorry. Would it be possible instead of opening a CVE to have like an issue in GitHub or like for example, for hardening, there are tickets. And maybe. So, yeah, yeah, there's the kernel bugzilla. So we do have at least speaking point my SELinux hat on for a minute. So we do maintain a mirror of the SELinux kernel tree on GitHub for people that prefer to get it that way. And we do have some issues that we allow people to submit issues that way. It's a kernel development. So we prefer to use the mailing list, but I know some people prefer GitHub. So that is a possibility. We don't use that for any formal, you know, GitHub has some really cool CVE and security functional. I use that for Libsec comp, which maintain. That's all on GitHub. But for the kernel stuff, we still really kind of steer people towards the mailing list whenever possible, just because trying to synchronize up to main forms of communication, it's difficult. So all right, I think we were at time. I'm not seeing any other hand. So I'm going to get down, but I'm going to be here both today and tomorrow. Feel free to catch me in the hallway. I'm more than happy to talk to you guys. And if you're virtual or if you think about it next week, you've got my email. There's my Twitter too. But anyway, thank you all for sitting there patiently with my presentation at the beginning and enjoy the rest of the conference.