 Hi, I'm Greg. Hopefully it's all working. I'm going to give a talk about Linux kernel security, the process of the Linux kernel security team mostly, how we do our work, how the security process even works and all that stuff. This is also based on talk I gave last week at the kernel recipes conference in Paris. So if you've seen that already, it's much the same material, hopefully different questions. And it's also based a little bit on a talk I gave for this very mentorship series a year ago, talking about how the kernels developed and some issues around security things there. Here I'm going to dive more into the specifics of how the security team works because it seems like a lot of people don't really understand. And first off, big disclaimer, this is all just me. My personal opinion, I've been part of this kernel security team since we started it in 2005. And I'll go into how it started later. And of course, nothing reflects the opinion of anyone else except me. And hopefully they agree with me. Usually they do. Sometimes they don't. But that's okay. We're all individuals. Big question is why talk about this now. The CRA is very, very important. It's coming up a European resident. It's coming in for Europe, be it a good version or a bad version. It's very, very relevant covers secured cover software and devices, which is a very good thing. Software and devices should be regulated just like paint and toys is regulated software and toys be regulated as well. Their goal of it was to carve out and say open source software is very important and rely on it in the EU here. So let's make sure it covers in the same regulations as close or software. That's good. I want a level playing field. However, the way it's written that really is going to be harmful for open source software so much so that as written and has originally reproved, it might mean that nobody can use open source software and devices which is going to be very good at all. There's two posts from Linux Foundation there that give you some background. Linux Weekly News has done a great article. Apache is posted about this. The Eclipse Foundation is posted about this. ML.org is posted about this. Lots and lots of companies post about this. The German automotive manufacturers said this is a bad idea and here's how to change it as well as the French telecoms. The German auto manufacturers agree on something that's very important. And to be fair, I'm creating laws is tough. We're trying to work through it, working with a bunch of really good people in Brussels to try and educate our representatives on how open source software works, but it also covers a lot of security issues. A lot of security things are caught out in there such as no devices can ship with any known vulnerabilities. My answer to that question is that is known by whom, known by me or known by you. If you as a company don't have to know about any vulnerabilities then you're not even going to look for them. There's lots of other things like porting problems and fixing them within 24 hours of reporting and other unrealistic things, but hopefully it'll work out. Another big reason is every month or so companies keep asking me to join the Linux Kernel security team. I'll go and talk about it why there is no companies on the Linux Kernel security team. That's not how we work at all. Along those lines, people keep asking for early security notices. They want to know the bugs that we know about ahead of time because we're going to fix a lot of open source projects have these. I'll talk about it later why this isn't going to work and why it's probably just illegal. You can't really have these things. And also going to hardware embargoes. We've had a lot of issues with hardware for the past couple years. Spectrum meltdown is called this out dealing with hardware is hard. The job of a kernel is to paper over all the bugs and hardware to make it look like a unified model, you know, care with CPU you have underneath you should just work. Unfortunately, hardware leading times and fixes are very, very long, which is actually going to tie into the CRA because the CRA directly affects processors because processors run microcode which is firmware which is covered under the CRA. So if you're going to have a very short time to fix a bug under the CRA, you're not going to be able to have any of these 15 12 month long embargoes that the hardware manufacturers seem to keep imposing upon us. That's going to be a big deal. Hopefully AMD and Intel know about this now and they're working to help resolve that but that's going to be a big issue with that. So please interrupt or ask questions until you can pick them out of the chat and talk to me. So real quick, here's the size of the kernel. As of the last release we did six of five, 36 and a half million lines of code. It's huge. It's very, very large. However, you do not run this all you run about five to 10% of this. Everybody runs a different five to 10% about that. So about 2 million lines of code, two and a half, three. Your phone runs around three and a half million. Your laptop runs about 2 million. Your server runs about one and a half million server runs less code than your phone by almost double. Everybody runs a different part of the kernel. So we don't really know who runs what and that'll come back later. So when you see this huge number all the time that 37 million lines of code, you're not using that. Again, you're only about using 2 million lines of code. And this is our rate of change nine changes an hour very, very fast very quick to keep on top of this we've been running this rate for the past five or six years keeps going up and down a little bit but we're averaging nine changes an hour for a very long period of time. In the years this is a huge rate of change this is a lot of a lot of people we have over 4,000 maybe 5000 different developers from 400 to 500 different companies every year, huge huge rate of change. And talk about this, the new release model is still comes up every once in a while as of 2004 we changed our development model to be, we do a new release every two to three months and all releases are stable. This is in contrast to the old. You have a stable tree and you have a development tree and then we switch them every three or two years. Again, every two or three months new stable release. That's all we have. People still are confused about this. And back in 2007, we got together at the kernel summit and we made this rule, we said we will not break user space because we want you to be able to always update your kernel to a newer one. And this is important for security because we want you to be able to take all our latest versions we will not break user space. This will just work. And of course we have a caveat on purpose. We're human. We make bugs we make mistakes. You tell us about it we'll fix it. So I get reports are saying hey this broke over here. Great we'll revert it and we'll move on and keep on going on purpose. That's the only really big rule we have in the kernel development process is we will not break user space on purpose. People get yelled at when they do it on purpose. We fix it and roll it back. I saw some report that somebody booted an old red hat. Five or something not real red hat five all user space on the latest kernel that we just released and everything just works so it's a good indication that we have kept this promise. Version numbers also of course mean nothing there's no odd and even and development are stable just we just increment them every two or three months. New version comes out I'll talk a little bit briefly about that and we move the bigger number. Every couple years when it just gets too big so don't think of like oh the five series or the six series or the four series they mean nothing it's just a new version after one after another just means this one was older than that one. And here's how we do releases as I've said many times Linus has released six dot four and then we do the development releases every first one is every two weeks and every week after that does a new release candidate. And then after two and a half to three months he does a new one and we go on this is how we do our development is how everything works. And at the same time, I'm taking the latest release from Linus and doing a stable release. And we take all the bug fixes that landed Linus's tree and into the stable tree as needed we pick them out we do we do AI we've been doing AI for about five years it's just pattern matching to find the bug fixes and throw them over there and put them in a stable tree they're tested really really well and go out shoe helps out a lot with the testing here on the call other people have a lot for the testing. And we go on with that so people run the stable releases as that goes. After a new release from Linus comes out I throw away a stable release and go on to the next one, you upgrade because you can always upgrade to a newer kernel, all your user space code will work. All is good. Um, the stable current rules are pretty much this. The unique thing was made must be in Linus's tree. That was unique I've seen other development and models where that doesn't happen and that causes a lot of regressions of time because you don't know where you need to back where it fixes and whatnot. Fix must be in Linus's tree less than 100 lines usually sometimes bigger. Copy them over there and away you go. These are the rules. Pretty simple. Um, one kernel a year I pick for a long term kernel. We maintain it for at least two years recently there's been some news that this is somehow changed the ruling hasn't changed we always say we'll do it for two years. And then we bump up and we'll say we'll do it longer if we talk if companies say it's okay. For some of these 414 and 419. They're running for six years it turns out nobody's really using them. Nobody uses the 414 kernel for six years. Nobody's picking them up. So we're not going to maintain these for six years anymore. Look on the current or website on the release page it'll show you how long things are going to be maintained for if you have questions about it just email me. I've seen lots of really odd speculation and strangeness, but I work really closely with the distros work really closely with the embedded developers. Chrome OS Android to pick up to pick a good window of how long to maintain stuff. Sasha helps me out a lot with the stable kernels and long term kernels so we do a really good job there is a great job there. I picked the last kernel of the year. It's just a time based kernel so it looks like 65 if you look at how when we do our releases, maybe 66 depending on. Oh sorry, it'll be 66 or maybe 67 whatever comes out by the end of the year looks like it's going to be close in January for 67. And as long as it works well 61 we had to make sure that worked really well. It did it didn't have a major bugs or that we knew of. So that was good to go if 66 looks bad and maybe we'll go to 67 things like that. Anyway, that's it. Nothing new here. This is, I don't know why this suddenly got in the news, but it did. But nothing has changed here. So we're ready to change. Thanks for the link. Longer kernels, of course, slow down because it's harder to apply bug fixes from the latest kernel to an older kernel. I'll call it a number of the ret-ledon's spectrum meltdown problems that we've had are not backported into the very, very old long term kernels because they just didn't fit. So 414 is missing a lot of those. You really wanted to use a newer kernel. Hence the number rate of changes. And this is kind of interesting. So like we're running 36. So we're running nine changes an hour in latest history, we're picking out about 36 changes a day so it's an order of magnitude smaller. These are the ones that are going to this history. And these are sorry, these are the ones that go into the stable tree and the long term kernels. These are what you run your distro your device on. So it's a smaller subset, but they all have to be in latest history. And over time older kernels are actually harder to maintain the newer kernels you need a senior engineer on that. So it's more changes to and it's more work. It usually is just not worth the effort. A lot of problem with longer kernels is older kernels is the threat model has changed. The world has changed. We now treat USB devices as insecure. So newer kernels can handle this older kernels can't. So if you're trying to still run an older kernel in a modern day, maybe the threat model doesn't match up so well it's not a good idea. Always go the newest kernel you have the latest security features and a lot of latest other issues as well. And kernel releases. Like I said every release we do is considered stable, meaning you should always be able to update to it. We guarantee we will not break anything and we do not ever want to to fear upgrading so you should always be able to update kernel should just work. When you have other problems with out of tree code out of tree drivers that's on your own companies know about that they are paying their own developers to maintain that. And they need to pay them to drag them forward. You'll see a lot of SSC trees that have a lot of out of tree code. The Pixel 6 has a lot of out of tree code it has 300 out of tree drivers. They have successfully run every single RC release from Linus for the past two and a half three years with their tree. So you can drag it forward every single release works just fine upgrade works. So it can be done. It is not very difficult. So when people say oh we have too much code out of tree we can never update to a new kernel. I think that you just don't want to do the work. It is possible. It isn't that hard at all. We have a couple of questions in the in the chat. So, yeah, no older kernels might not be compatible with newer ones are compilable. Well, older kernels may not be compatible with newer compilers. So yes, so older kernels are harder to compile with newer versions of GCC and clang. That's very interesting as well. Darren said the core kernel team LTS duplicate work code. CIP is different. There are different projects. CIP piggybacks on top of the LTS work. Talk to the CIP people. I talked to them pretty frequently. So it's a different project. CIP is saying we're going to only support the kernel in this tiny little subset of these pretty much three million lines of code and we're going to maintain these based on these configurations. So they're taking a subset of the kernel maintaining longer. They're piggybacking and relying working on the work, building on the work that I do and Sasha does. So we work together, not duplicate effort. So how does the security team, all right, I'll talk more about security bug fixes later. So I'll talk about that. Oh, and actually, how do we, the security team identify bugs that only exist in the stable version. That's very, very rare. We do get bug reports saying this isn't a bug in an older kernel. The fact the odds of it not being a bug in the newer kernel is usually very slim. Sometimes it's because we fixed in a newer kernel, and we just backport the fix and away you go. That's the majority of it very, very rarely will we see that something is only applicable to an older kernel version, just by the way code works. It's just not very common. You'll see there is a bug. I'll call out there's a bug fixed for butterfs file systems on the stable mailing list saying hey this only applies to older ones. Butterfests maintainers. Can you please review this to make sure it's okay. We just ask the developers. And why Android not find it difficult with shorter LTS versions. No. Another EU fund rule is the EU rule about keeping devices alive and maintainable. So there's a new rule that came out I think it starts the beginning of next year that any new device that shipped must be supported for five years after the last device was shipped. It means that really a lifespan of software in these devices for seven years. So to do that, you couldn't even do a six year LTS. It just doesn't work. It doesn't work at all. So Android is going to have to update their kernel and their devices over the lifespan. And it's going to work. They agree. I was talking to Samsung, and Samsung was like we update 2000 packages every year and Samsung phones because of the new Android version. What's the kernel? It's just one more package. We'll do it. So new device devices will be updating the kernel, be it every year every two years talking to different companies to a new major version in their devices over the lifespan of the device because they know it needs to stay alive. It's just a good maintenance of devices. So Android is actually going to be fine. So and I'm working closely with Android team. So it's not an issue. Cool. And how'd I get involved in stable kernel I started it in the beginning of Christchurch. There's another talk all about stable in the kernel development process. I think I caught everything. Good. One last just came in. I'm learning about debugging on the kernel dummies.org. I'd say go there. And we have also a lot of webinars from various developers on the, this webinar site too. So there is several of them. Go ahead and check the check those out. Okay. Oh, and a lot of these questions are already answered. I gotta have a whole blog post about this and how to pick a good kernel what the different kernel versions are. And look at that. There's a whole bunch of information. If you have questions after reading that, please let me know. It should explain a lot about here and also talks a little bit about the security stuff, which I'll go into some more. But also here's something I want to real people to realize and a lot of people don't. The majority of the world does not run commercial distributions anymore. In the beginning Linux kernel, Linux distributions were a very big deal. Companies wanted support. They wanted to do it that way. And they grew and Red Hat and Sousa who I used to work for are great. And still are the most economical. There are paid oracle paid distributions that work very well. Turns out, the majority of the world realize that they can run community distros instead. They can run their stuff there on. Very large cloud providers very large ad providers all run debbie and they run their own kernels. based on kernel.org, so much so that over 80% of the world's servers at least are running not commercial systems. They're not running commercial Linux. They're not running a supported one. They're usually running Debian. Debian runs the world. And then even more so, Android. Android is in billions of devices. All the other Linux systems are a rounding error in there. They're all running kernel.org kernel with a little bit of Android on top of it, some patches that are all open source. Android developers work upstream very, very well. And it's all the latest LTS releases. Android requires the latest LTS releases in their devices. So these are not companies that are happening. It's the communities. Debian is developers, the kernel developer community, and it is not companies. So when hardware companies or other companies want to try and report problems or issues, they try and do like corporate NDAs or corporate companies talking. It just doesn't work. They do not achieve anything. And we, community, of course, cannot sign NDAs. It's just as simple as that. So you have to work with the community. Companies need to work with that and realize that. So again, 80% at least of the servers are out there are running Debian or kernel.org servers in the world. They're not running Red Hat. They're not running RHEL. They're not running Oracle. They're not running Rocky. They're running other things. Now, not to say there isn't a huge quantity that are running Red Hat, Suza, Rocky, all these other ones. There is, but it's just not the majority. So please always remember that. So let's talk about the Kernel security team. This is a bunch of individuals. And I'm going to talk about reactive security. We do not do proactive security. There's a whole other, the Lennox Harding Project run by Case Cook and other developers. Gustavo does a really good job. He's sponsored by Lennox Foundation. They do proactive security. They work in the kernel to add proactive security features, find whole classes of patterns of bugs that could happen and knock those away. They change the code to work so the compiler can prevent things from ever happening in the future. That's what they do. We just do reactive security. We are reacting to bugs that are reported to us. Again, two different groups. Oh, there's also the LSM, the security model people. They're doing security models like SE Lennox and App Armor and such like that, SMAC. Those are other types of things. This is just talk about reactive security. Again, other groups do the proactive. So this started back in 2005. Developer asked us, hey, where do we report bugs to? And they wanted to know what was going on. And we realized we didn't really have an official way to do this. So Chris Wright, who was working for OSDL at the time and now he's the CTO of Red Hat, got together with a bunch of us and came up with some rules and posted a patch and said, here's the security contact. And we started with the initial version of a couple of people and we went and so that was 2005. 2005 was an interesting year. It's the year the stable kernels started and it's the year they get started and it's the year that we had security team. 2005 was like the year we grew up. It's kind of interesting. One really good thing that Chris Wright and others, the lawyers that helped us and Lennox, from the very, very beginning, we said we are individuals and we are not a formal body and we cannot sign NDAs. And that has come back to save us so many times. It's a very, very important thing that people need to realize. We work and we say we will not disclose anything and we will not talk about anything and we have never broken that promise. A long time ago, we had a few emails that got leaked from some broken email systems but nothing ever leaked from us on purpose or accidentally, whatever it was about. So we did not do NDAs and we're not a formal body. We're there by individuals. So the people on the Linux kernel security team are a bunch of individuals. They were there on their own recognition. We're bunch of current developers. We cannot represent any companies. Sometimes there was somebody on there once that was representing a company or was trying to take the information that they learned there and told their company about it but that person's not on the list anymore. Alias, this isn't even a mailing list. It's just an alias. You send an email to there and it explodes out. It's not stored anywhere else. It's just an email explorer. And that's it. We're just a bunch of people and we work to, we get a bug report. We create it and then we work to fix it. All our job is you report a bug to us. If it isn't in one of our subsystems or areas, we will drag in the responsible developer. We'll work to fix as soon as possible and we will get it merged into Linux history and then we're done. And then we don't say anything else. So we move on to the next issue. We get about one or two reports, real reports a week. Some times more, sometimes less. We say we work to create a fix as soon as possible. Some of them take longer. One is taken, I know, six months. There was a networking randomization bug that took I think over a year, year and a half working with a researcher. We do try and get them done as soon as possible but we sometimes also things happen and some bugs are harder to fix than others. That's it. And here's the interesting thing. People ask, do we wanna join the security team? You're asked to join the security team if your subsystem has enough problems that we keep having to drag you in. So that's how the group has grown over time. We're like, hey, we keep copying you. Let's just drag you in now and that way we save the round trip up another email and you're on the list. You handle all the bugs that are associated with your subsystem. So it's not really a good idea to be added but people do a good job. So, and then every once in a while we'll go through and ask that for everybody. Still wants to be on the list or not and herds the list and keep things up today. People move, they change jobs, people go on maternity leave, things like that. So it changes over the years but that's it, we're just a bunch of kernel developers. Our rules are we fix the issue as soon as possible and we do not, once we have a fix we do not want to ever embargo it. We don't ever want to sit on a fix that we know. Our rule says seven days. I think we did one for 14 days once after the fix happened just because we needed the vendor needed some time to roll it out to their fleet and they worked with us. They were the ones that reported it up to us so they knew about the issue ahead of time. And of course we do not do any kind of announcements. We don't tell anybody anything. If the reporter wants to say something afterwards that's up to them but we do not do anything on our own. And is working on a security question, paid work for most of us or private fund. Some of us might be able to get away with doing it on our company time but the majority of us are doing it as an individual and as our private. So private emails, personal emails and we do it on our own. So we are not able to do this as part of a job unless your manager or your company allows you to do this but again you do not represent a company. And that's it, we don't do any kind of announcements and that has caused people confusion over the years and people keep wondering why and people don't really understand the issues involved and so much so that in 2008, right after the team got started for a couple of years people were complaining that we were fixing bugs and not telling anybody what the bugs were. And Linus wrote back, here's the email and I'll have a link to the full thread in a minute. The problem with the kernel is bugs are just bugs. We don't know if their security bugs are not usually. Sometimes we do, most of the time we don't. So we don't want to ever explicitly mark a bug as a security bug because then that takes away from the bugs that we don't mark as security bugs. And Linus here calls us out saying people might think this is a trustworthy thing if we say this is or this is not, it is not true. We do not know how to classify bugs and Linus goes on and on all bugs are fixed all fixes are important. You should take them all. You made the guarantee that we're not gonna break user space so there's nothing wrong with taking all the fixes. So take them all. And here's the whole links for the whole thing. And you just want to take them all. And this has come back to be proven over and over again. Many, many years ago, there was a bug fixed in my code in the T2I layer that I wrote the bug originally and I fixed a couple of years later. I thought it was just a normal bug fix. Turns out it was able to be cause you could get root on all the rel systems in the world because they never had updated to it. I didn't realize it was a security fix and I mean, I think I know what a security fix is at times but I didn't, but it was a bug fix. And any bugs that the kernel layer have the potential to be a security issue because we're operating in a privileged environment and they could be exploited. And people have amazed us with what they've been able to exploit. We've had just an off by one count people can actually exploit, things like that. So people can exploit things that are very, very good. I'll keep on going. Here's that link to where to report bugs and things like that. I do also want to call out, remember we're going 36 patches fixed in the LTS kernel today. One company a couple of years ago said we were going to evaluate every single patch that goes into a stable kernel and declare whether it is or is not a security fix. I said, wonderful, that's great, please do that work. They came back to me in a year and they said there's no way they couldn't keep up. They took them too long. They didn't know how the use cases were everything. They didn't know how things ever worked for other people versus how they're, even in their system sometimes they couldn't figure it out. And so they said, we're just going to take them all. And that's the best thing to do, just take all the bugs. The Android kernel security team did a study and they proved that 90% of all their bugs, security bugs that were reported to them were already fixed in the stable kernel tree before they knew about it. So, and if you look at CVE, CVEs don't matter, but CVEs were showing up an average report date of a hundred days old. That's because Red Hat abuses the CVE system, but the way that bugs are fixed in the kernel, we fix them, push out the fix and we don't report anything. We need people to update them and then update their systems with them and then they have the fix before it's ever publicly known, be it by us or by somebody else or just by people figuring it out that that was a fix that this is a security issue. You should always take the bug fixes. And again, almost all bugs can be a security issue. I think there was some comments, questions, questions, questions. Yeah, there are a few questions here. Questions, okay, cool. It starts at, is Android considered? Is Android considered a digital and sorcery? Android is, I mean, all devices, everybody has a fork, right? And I have a fork for my systems, so patches that are applied on top of Linux's are a release base. I have a sys call that I use that nobody else has and upstream rejected it. Same thing with Android. There's a handful of patches. They give a report every year showing how many have been accepted, how many have been merged, how many are remaining. It's just a couple of patches on top of the latest tree and then they go. So it's not very different from the LTS version. Android tree is public. You can actually boot a device, an Android device without the Android specific patches, but there's some things that they add that a few user space interactions to some things that some Android tools like. So it's not very different from the LTS version at all. Again, go download it, you can see. Android patches not present in LTS Cursion. They are, Android common kernel is maintained a different way. That's up to Google, talk to them. It's just stuff that isn't, most of the stuff in there is upstream. Some stuff isn't, some stuff comes vendor, some stuff is backported, some stuff is an LTS is someone, it's up to them. And user space never breaks a new kernel, sometimes it breaks a third party kernel modules. We don't care about third party kernel modules. Literally, we don't know about them. We don't care about them. It's up to you what you do with that. That's out of tree, you're paying. It costs time and money to keep code out of the kernel. You're doing that for a reason that's on you. But again, I showed pixel 300 out of tree kernel modules update every single RC release is possible. Maybe I changes, API changes are very small, usually. Easy to keep up to date. You can do it if they want to do it. Luckily, Google's finally trying to move those 300 patches or modules upstream, but that's up to you. We do not have an internal kernel API that's stable. Oh, we have a document called stable API nonsense. Talks about why you don't want a stable API. You can have it and you can do that, where you go from there. We do change the kernel structures and APIs without notice, whether the code is the notice itself. You do it all the time. Again, out of tree drivers, we do not care. And that is the cost of out of tree drivers, just merge your code upstream. There's no reason not to get your code upstream. We will take drivers for if there's the one piece of hardware out there. You're maintaining code out of the tree for some reason, usually because you like it and you need it. Submit it upstream, we'll take it. If it's a driver, wonderful, we'll take it. I think I caught everything, did I, Shira? Yes, I think you have. I was wondering if, yes, I think there is one question I thought might not have been answered. Let me take a look. Okay, at the Linux foundation, there is a type of RFC where changes, failures, new implementations, et cetera are recorded. I, do you have any comment on that? Oh, I missed that one. There is, oh, so Linux foundation just happens to host three developers, you know, Linus, Shua and me. There's 4,000 kernel developers, 5,000 kernel developers last year from 500 different companies. The kernel development process is independent. The Linux foundation does help with kernel development by sponsoring some of us. Obstensibly, Oracle and Google and Intel have more kernel developers than us, so they provide more to support. There is no RFC changes. The only way we accept changes is if you have working code, you submit the code because it's working for you. That proves that it works and we accept it as part of our development process. We have whole talks on how the development process works. So there's no real RFC other than working code. Internal kernel ABI changes so often works like a security feature. No, there's no obscurity in the kernel. I mean, with all the source code is there. So we're not fixing by the fact that we changed so many bugs and we fixed so many bugs. Maybe you can say that, yes, we're sneaking in known security bugs along with the other bug fixes, but also realize some subsystems refuse to be part of the kernel security process. Infamously networking for decades did not want to be part of it. They wanted all their bugs reported publicly, would deal with them, get emerged quickly and pushed out the door. They didn't want to deal with anything privately. That way they could have more developers working on it and work better. So some subsystems still work this way. So it's not a matter of visible or not. All our fixes are visible. So just take them all. And kernel doesn't really have viruses. So there's no such thing. And risk five, kernel security policy that we don't care about processor types. We support 12, 15 different architectures. Risk five is just another one, no difference there at all. Okay, I think I cut them all, right? All right, that's the tip. Okay, so again, all bugs can be a security issue. So people always complain, well, I don't want to take them all because they could cause a problem. Yes, we're human, bugs happen. So here they'll say this, a fix for a known bug is better than the potential of a fix causing a future problem. A future problem is when the found will be fixed then. And we have documented proof of this. A very big cloud provider, edge cloud provider, thought they would just take a specific fix. They weren't going to take all the fixes. They thought, oh, this is the one fix I want to fix this security issue. That was unknown CBE had to do with a spectra issue, speculation issue. So they took that and then they went off and then they reported a bug to us and the kernel security team said, hey, the mitigation doesn't really work. And we said, yeah, we know because we fixed it three weeks ago. It didn't, because, you know, some of these fixes are fixes on fixes and fixes on fixes on fixes and 20, 40, 60 long fixes because it turns out that some things don't always work right when they hit the real world, get better testing, so we fixed them. And we will fix them. And so that's the thing. We test these as a unit. We don't test these as cherry picks. We test all the fixes together and then there's a problem with that. We'll add more fixes and do a next release and add more fixes and do a next release. So always take them all. And a potential you always want, your system is always more secure if you have known bugs fixed. Otherwise you're running a system that is known to be insecure. Otherwise that's always much more riskier than the potential of future problems. And future problems are always gonna be there. So test, test the system, test your devices, test the kernel. Again, you're running a different portion of the kernel than I am. So test that all the time. Because the most important thing is we don't know how you use the kernel. So a lot of these bug fixes and otherwise there's no way I can tell if it's a security issue or even a bug fix because I don't know your use case. And like I said before, I don't know what percentage of the kernel you use. I don't know what lines of code you use, what lines of codes you don't use. I don't know any of that. And I really, really don't want to know that. Open source, the license of open source, one of the rules is that we do not dictate use. You can't tell you how to use our software or how it can be used. So we can tell you the license of the copyright and things like that. But we can't tell you how to use it. So I mean, Linux is used from what? Satellites from helicopters on Mars, automatic cow milking machines, routers, all your Android phones billions in the world, servers, keyboard, my keyboard here, runs Linux, all sorts of crazy cameras, run Linux, different types of use cases for things. I don't want to know this and I don't need to know this. Just take all the fixes that we give you and you're good and you're happy. There's some cool tools out there. I know the Android team has a patch and a script that matches up all the, they know what code they build. They know what 10% they build and what they rely on. So they look at the stable patches that come in and they say, oh, look, it happens. The three of these happen to be things that we build. So let's just review them to make sure they look good. They review those three instead of the 50 and they're good, they're happy. It makes it easier in the way they go. Much, much easier. But they take the whole patch because they don't know what other people might be using the Android kernel trees on top of the way they go from there. So we don't need to know any of this. And Ben Hawks, as a security researcher, said this really, really well. I'd really recommend going reading this blog post because, and this is also, governments have a problem with this. People don't realize that a bug can be very, very serious on one deployment, somewhat important than others and no big deal and not even a bug at all in others. It all depends on what's being used, how it's being used, how the user space is considered around it, configured around it. It's just all is this, all at the same time. So when the EU says rules like all bugs must be fixed, all vulnerabilities must be fixed. Well, how about all vulnerabilities for your device or all vulnerabilities for how it's used in the network and how it's used in this configuration? Those are more and more realistic things. Vulnerability remediation is very, very hard. So please just take them all. And about the updates. Do I have one idea why some users opt to cherry pick over updating the version? They think they know better than they don't. It's sad to say, I've worked with many, many companies of very big phone manufacturers. I walk in and say, hey, stop cherry picking patches. Look, here's your latest kernel and your latest device. And here's all those security bugs you missed because they're cherry picking things. Just take them all. There's no reason for you not to take them all. There's no reason for you to think that you should only be able to pick certain ones out. I mean, a one line fix should be tested just as much as a thousand line fix. It just, your testing infrastructure should be the same. If you don't have a way to test and update your devices, do you have bigger problems than security issues? Fix that first. And then just take them all and away you go. There should be no reason we test them as a unit. We ship them as a unit. We support them as a unit. You should never try and just cherry pick. People think that you should, I don't know why they do that. It's just not a good idea. And it's very anti-good software engineering. How much is kernel CI helping us catch bugs early? Kernel CI is awesome with testing the infrastructure for all the stable kernels and they test and see how that works. Kernel CI and Lunaro is a lot of really good testing as well, sponsored by Google. And other test systems out there, Chrome OS, Shua does it and other people do this. Kernel CI does a lot of work to catch the trees as they're going into Linux's tree and Linux's tree is up. Kernel CI tests individual tree. Kernel CI does a lot of great work. I strongly recommend Kernel CI and everybody should help. Note, I'm on the Kernel CI board and I'm a member of it and we want Kernel CI to be the unified way that we do testing for the kernel. We can't handle too many different ways of doing testing. So Kernel CI is great. Architecture specific issues. Yeah, we have architecture specific issues. Lots of different ones. That just depends on the chip. But again, I think CPU is no different than a mouse. Some mice have bugs, some CPUs have bugs. Just some people happen to run more CPUs than others. Anyway, just a hardware, we fix them and we move on. And I'll talk about CVs in a second. And my thoughts on CIS killer. CIS killer is awesome. It's great tool. It's wonderful, but it sprays a lot of noise. And I wish that the companies that run the CIS killer would provide more resources to fix the bugs that they find. They are not doing that. They've been caught off for that. They know that, they don't feel like funding it. But CIS killer is a great way to get involved in the process. She was mentor program and I help her out a little bit but she does all the work there and all her mentees start with CIS killer bugs. They're great. Great place for her to start with bug fixing. Sometimes it's a very artificial but sometimes it's actually fine. It's good stuff. You want to learn more about security vulnerabilities. There's lots of resources out there. I'm not going to talk about different security vulnerabilities and things like that. Cool, I think I caught everything, right, Shua? Nice, now. One, there's a couple of them payment artists now. Yeah. Okay, so what's the tip you would give to a company that is using really outdated kernel, say 3.18 apps? Update, update to the new kernel. That's it. Just update. There's no reason why you shouldn't be able to update. If you have out-of-tree code, drag it forward and your 3.18 kernel is known to be insecure and vulnerable. So here's the thing. When we ship a kernel, it's known and good for everything that we know at this moment in time and the world at this time. But the world changes, vulnerabilities change, ideas change, how to exploit things, networking changes. If your device is hermetically sealed and not interacting with the world, great. From 3.18, you're fine. Stuff that's off on a moon or in Mars, wonderful. You're not interacting with the world. Stuff that is interacting with the world, you have to update it. Otherwise, it's known as secure. So just update it. And there's no reason you shouldn't. I'll call out Siemens, huge giant company. Tons of industrial devices. They support devices for 20 years. I think one of their large air conditioning systems. They said, there runs Linux, and they said we update the kernel every year for the lifetime of 20 years because it's part of maintenance. You fix mechanical parts, you fix electrical parts, you update the software, keep it up to date. You're keeping a living and breathing system up to date and secure with the changes that happen in its environment, be it physical, electrical or software, same thing. And then at the very end of the lifespan, maybe the last two years, then they'll stop updating the kernel to newer versions because they know it's on its way out. Same thing with the maintenance of the device and they're gonna replace it with a new system anyway. So if Siemens, giant, giant, huge industrial company knows how to do this very well, any embedded system company can do this. It's just not. Rocket science is just being a good developer. And that being said, Linux is in Rockets too. So, cool. All right, I got a little bit more. Maybe. Okay. Again, here's our rules. We fix all the known bugs to as possible. We get the releases out to users very quickly. And but this doesn't work for hardware bugs. And it doesn't work for hardware bugs right now because hardware bugs are slow. We get notified of bugs. I got notified a while ago saying, hey, we found a bug in our CPU. We're gonna fix it in 15 months. Oh my God, I don't really wanna know that. But just fix it faster. It takes a while for them to roll up microcode and things like that. And hardware's in companies, bugs. So, we have rules about this. So, we handle these separately. We have an encrypted restricted mailing list that we spin up for each individual project. We do not have NDAs. We have an implied NDA that we will not tell anybody. Everybody as on the list cannot tell anybody else outside the list, even for the same company. You can't test it on infrastructure that is not, that has access to other people. Very, very restricted. And we cross company, where you talk to different operating systems, we drag and zen, sometimes the BSDs. You can talk to Microsoft for Windows for some issues. So, we coordinate companies and we coordinate the operating systems. And sometimes we drag in the distros as well to get the releases out on the same day. For this, we will tolerate embargoes. That being said, this is a very, very difficult thing to do. For one point, a couple of months ago, we had five of these running at once. And it was caused a problem because one developer turned out was not on one of the lists. And he got mad when it became public and all these things were fixed in his area. The colonel, we thought he was on that list because he was on the four other lists for other four issues. This is a difficult process. Linus has talked about this. And I was, these hardware companies, we are gonna try and make these embargoes much, much smaller. These long, long embargoes just don't work. In the end, we do all the work to fix the bug in the last two weeks anyway. So this, maybe we just make it a two-week embargo. Just don't tell me about the bug. So until two weeks before you're gonna make a public. Things like that. So we're trying to work through this and try and figure out how to do this very well. It'll work out a little bit better. And here's the whole description of how this whole process works for the hardware embargo list. If your company wants to be involved with this because you make hardware like chips and you're not on this list already, on that link, it has a list of companies and representatives, talk to me and we can get you out of it as needed. And again, I wanna talk about this again because this gets very important. We do not do any kind of announcements. We don't pre-announce, we don't assign CVEs. We cannot assign CVEs. And we cannot assign CVEs on purpose. I think I talk about this later. No, I don't. I gave a whole talk about CVEs. I think a link will come up later about how CVEs are broken for the kernel. Mitra knows they're broken. If you really want CVE, just go ask Mitra for one. They'll give you one. They refuse to give them to me anymore. They just don't work for the kernel. Red Hat, as an engineering department, abuses the CVE process. They admit it because it allows their engineers to add patches through their engineering process easier. So CVEs for the kernel don't mean anything. Another fun example, CVEs can't reflect the fact that they took 50 pouches to fix one issue. Things like that, I have a whole talk about this. And there's no early announcement list for anything. And let me make this more and more clear. I view any early notice list that you have that is a leak and it should be considered public. Any early notice that you have where you think you're only notifying a small subset of your users is just giving people that information to the world sooner. Unless your project isn't used by anyone. If your project's not used by anyone, then it doesn't really matter that much. These early notice lists are very nice and they don't really useful or anything. But otherwise, these should be considered public. And the reason I'll say this publicly, why would your government allow this to exist if they weren't public already? If your government didn't already get the information from these early notice lists because your government runs Linux, your government runs open source. My government runs open source, the government where I live runs open source. France, all of France runs Linux and relies on Linux, Netherlands runs Linux, Germany, all these German train systems, all these things run Linux and rely on Linux. If you have an early notice list and the government is not on it, then it's, I mean, it's just really not useful. Look at who uses Linux the most. China, Korea, North Korea uses Linux a lot. All these major, major governmental groups use Linux. So the fact that we don't have an early notice list has actually saved us. When Spectrum Meltdown happened, the US government called many corporations to task interviews and had public hearing for some. I had to talk to the Senate subcommittee about this and they asked me why we, as kernel developers, Linux did not tell the US government about this problem at a time. And we said, because we told nobody, we don't tell anybody this, so we told, we didn't tell anybody and we just fixed the problem and pushed it out as soon as we were allowed to. And they said, wonderful, that's about the only answer you could have given us. And that goes to say that either we tell all our governments or we tell nobody. So the Linux kernel, the fact that we aren't a corporation, in fact, we're individuals, in fact, we don't have NDAs, in fact, we don't do early announcements, has proven that this is the way open source software that the world relies on has to run. So just take all our fixes, because again, we don't know if they're all security fixes or not, just take them all and away you go. So call that out. So along those lines, they happen at least once a week for things we know, mostly more times because lots of public bug fixes are really security fixes anyway. We don't do all our security fixes in private. Look on the mailing list, look up, see all the fixes that go into networking stack and to scheduler, all these different subsystems are happening. They are actually security fixes and they happen in public. So there's nothing new about that. It looks like any other big fixes, sometimes we don't know them until later. And again, we don't give any CVEs anything and we don't differentiate. It's a bug, a fix, a bug is a bug is a bug as I said many years ago and that's what we mean. Here's my talk about CVEs, why they're not working for the kernel, why they don't work and you guys can go see that link to find one and how do you go for that. It's a couple of questions. Oh, sorry, get a couple of questions. All right, I'll go to school back. Oh, not many. Good, automotive market, specific changes, Harvard security model be added or extended. My car runs Linux, just fine. So Tesla, I mean, I don't have a Tesla, but Tesla runs Linux and most cars run Linux. Automotive is not anything more special than anything else. Linux runs in rockets, runs in satellites. If you want to add special security modules for automotive, wonderful, great. Well, we'll take it. Cars aren't, cars actually are very low margin, but they're also very low quantity as far as devices that run Linux. There's not a lot of cars out there in the world compared to how many phones there are. I don't know, there's nothing different. OSV from Google GitHub, including one for the kernel. There is a group that I can't remember their name right now that we do interact with and we ask for identifiers. And I gave a talk, if you look at my talk, I talk about how we can identify bugs and security fixes in the kernel and how they're tracked over LTS releases and how they can be tracked into your device. We do work with them and we do pretty much get a new identifier for every single fix we have, but we have identifiers. Identifiers are unique for Git, so we just use the same identifier. And we're working with them to create these stuff. And that project is run by some people that used to work for Red Hat and a few other companies and they used to be CNAs and they know Mitra and they know how the whole system is currently broken. And I think, Shua, did I get everything? Yeah, I'll set. Cool, all right, my last talk. My last slide, I mean, sorry. So I will say this, I've been saying this for a few years. If you're not using the latest, I need to put the latest in there. I was called out on that on my last talk. Latest stable long-term kernel year system is insecure because it's full of known bugs. Let's call it out right there. So I will publicly say that the enterprise kernels for most companies are insecure because they are not taking all the bug fixes. Oracle, I'll call it out, is doing a very good job. Oracle takes the LTS kernels. They do very well all the stable kernels. SUSE does a lot of the cherry picking and most of the backporting and SUSE does a good job. Red Hat is kernels off in the weeds over there. Talk to Red Hat if you wanna get their kernels. Most people that run Red Hat can run their own kernels too as well. So yeah, if you're not running the latest one, I mean, I'll call it Android does a good job. Android, their kernels are maybe six months out of date but I'll take a six months out of date kernel then a three year out of date kernel any day. Trying to narrow that window. Some devices, some vendors are very good and there's some very like fair phone stays up to date very quickly. Other smaller phone vendors stay up to date very quickly. So you can do it. Sony, Sony does a great job of keeping their devices up to date with newer kernel versions all the time. So that's it. If your kernel is not using, if you're not using the latest one, it's known and secure and that's it. I'll save this off slide. All right, any questions? Oh, one last question. What takes the most time? So we have scripts to pick out the patches and identifying the patches is easy. We have tools that do this, although my scripts are public. We've been doing this for a very long time, applying it to old trees. I'm seeing if they apply, seeing if they're relevant. It's not really long. It's up to you. It depends on the developer. It depends on the patch on how long it takes, how much part takes most time. It's just, each patch is special and unique just like every other one. So there's no other logistic reasons. We're usually good. Sasha has caught up a bunch of patches. I was traveling for the past two weeks for conferences. I need to catch up on the rest of them tomorrow and then we'll be up to date. So it's really not very fast. How do I deal with conflicts if the script fails? The LTS are very big. If it fails, you get an email that says, hey, this failed. Do you want to have this applied? People take those failed patches and either developer can update them and send me the updated ones or people I will call out. There's some very good developers that help us out and they'll fix them up and send us updates and do that as well. So people do this all the time and you get an email. If the patch that you say needs to go to a stable kernel doesn't apply and it fails, we will let you know. What tools do we find vulnerabilities in the kernel? That is a different talk. I think we had a whole talk show about from Gustavo and whatnot on how they find vulnerabilities and how they fix things. That's it. I'm doing reactive when you have to refine the problem. Other people find problems, otherwise. Yeah, we have a few talks, several on proactive, like you're saying, KMSAN, KSAN. So we have several talks on how to grow, proactively find things and fix them. Take a look at that static analysis, dynamic analysis and Sysbot. What Sysbot does is falls into fuzzing and dynamic bug finding. Yeah, go ahead and take a look at those. Yeah, I call this off. If you're interested in doing that, there's tons of, I mean, we have just look at the Sysbot output and pick up bugs from there, do that. Finally, people do try and report Sysbot bugs to the security team. We almost do not accept those anymore because they're all been publicly found anyway. I just go and fix them. It's not an issue. Is being a developer mandatory to contribute? If you contribute, you are a developer. That's it. There's whole talks on how to get involved in the current old development process. I think she was given a great one about that. We have tutorials online. There's processes for getting involved. As you contribute, you are a developer. Oh wow, it took that long for us to come up. All right, Rust. Rust is cool. Rust is a nice language. We are doing fun things with it. That being said, Rust only solves problems where you have some type of reference counting or some type of variable going out of scope bugs. It does not solve logic bugs. If you look at the majority of our security bugs or majority of bugs that are going in the kernel, we have lots of logic bugs. Like oops, we didn't check for that or we didn't know to check for that or the range that we checked for that was the wrong range to check for. Rust won't solve that at all. That being said, Rust will take out a huge swatch of some types of bugs. And if you look at the work the case and Gustavo are doing, they're taking out a huge swatch of those same type of bugs in C already. We have ways to handle strings better. We have bounded check. We have bounded arrays. Lots of those things that Rust ostensibly solves for us is already solved in C. It'll be nice to see if somebody gets a Rust driver in the tree that actually does something. The problem with Rust in the kernel is people are like, we'll just do a driver. Drivers consume from all of the kernel. So you need to have bindings from Rust to pretty much all the kernel in order to get a driver to work properly. It'd be easier to write a poorer portion of the kernel in Rust today than it would be to write a driver. That being said, there are patches out there for lots of cool stuff on the Rust mailing list. People are working on bindings. I know the Android team has a binder written in Rust. There's some five driver patches in Rust on the mailing list right now that people are reviewing that actually look pretty good. You can write a nice tiny little five driver. Five drivers are pretty well safe, self-contained in Rust. Now that being, if it actually solves any bugs or not, it's actually really easy to write a five driver and see that isn't buggy. Yeah, this is interesting. Let's see how it works. So it might take a few classes of it, but people aren't standing still. They're fixing those types of classes in C to keep them us from ever having them happen. They're working with the compiler developers as well. And there's also open talks on here on this process about Rust. There's training for Rust and there's a whole bunch of cool stuff by the Rust developers in this presentation series. I go recommend watching those. Whereas the entire kernel move to Rust in the near future, that's near future is not very realistic. You can't just rewrite code. It doesn't work that way. Oh, and there's the, of course, the Apple, the RM driver is written in Rust. But if you look at when that driver was submitted for inclusion, it turns out there was bugs in it because it was making some assumptions about how the RM driver should work that were not correctly. So these were just pure logic bugs and logics with how it interacted with the C code that Rust, the language didn't solve but these were still problems in Rust, the driver. So that's a good example of how Rust doesn't solve logic bugs, but it does allow you to write nice drivers in Rust if you want to. Anyway, I do not know what Falco is between Kubernetes and the kernel, sorry. And I think I answered all the questions, right Shua? There is one looks like about, you might have answered this one. How long, oh, no, that, oh, modern dive. It's a little bit different question about the modern. Yeah, I just want. Modern driver production. Again, you can write a driver today in Rust if you want to. This step interaction with the kernel, with the C code and the driver code, that's the interesting part. Whether you like, for Fi, you can write a driver based on the bindings that are being posted right now. Whether that's accepted or not, let's find some issues involved in how we do bindings. If you want to try, but I will say a DRM graphics driver has been written in Rust for the Apple hardware and works wonderfully. So you can go use that today. It isn't merged yet. Do all the bindings aren't all merged up but they're working on it. Developer did that, did a wonderful job. They're great developers. So there's a real world example of a modern driver in Rust today, but it's not merged yet. It'll get there. A shot, Alex, yeah, those developers. And everything in the kernels that are continuous development, that doesn't stop. I think it's, did I run out? No, I saw time, but anyway. Linux security versus commercial players like Q and X security. Q and X is a different type of operating system. It's a microkernel, it's slow, doesn't run on all the hardware we run, doesn't have the use cases that we have, it's just a different model. So Q and X is nice. If it works for your hardware and your use case, great. Just like SC4, SLE4. Great, wonderful types of things. Different types of security models. That's, again, the security model argument like for the LSM or the SC Linux or the Alparmar or things like that, different types of things. And there is not going to be a new version of the Linux device drivers book, as I say every time. A Rally doesn't want to do one, so that's up to a Rally. They all have the copyright, so we are not gonna do it. Sorry. When there's an API change, we must fix all the drivers in the kernel. The maintainer doesn't do the work. No, if you submit a patch to me that breaks the API without fixing all the internal users of the API, I can't take your fix. It's that simple. Making cross tree API changes is hard on purpose in a way. We do it all the time though, so it can be done. There's tools and procedures for doing it, but it's up to the person who wants the API change in the kernel to make the API change for everybody. That's one of the advantages of getting your code upstream. You get your code upstream, it's merged, any API changes that would break it on the build and hopefully as it runs are fixed by the developer who made the change that would have broken it. That's why you want to get your code upstream. You don't have to adopt staging for your API. I would disagree. Staging is actually pretty simple, but yeah, maybe you can ignore staging. Staging there for kernel developers to get involved in the kernel, it's full of lots of very old, crafty code, and it's a great place to get involved to learning how to fix up code in a very safe and self-contained way. We use it for mentor programs, for interim programs, and outreach uses it, and people can fix it up there. And I think I got it all. Russ and the kernel were replaced C in the future. This has already been answered, sorry. I don't think so. Again, look at the other presentations in the series that talk about Russ. They answered that question. There's nothing wrong with C. C actually works really well. I don't know why people hate on it, but you can write good code and see. Should anything else I need to do or something? Yeah, I don't think so. And there are no more questions. I think all of the questions are answered. If you have a question, you can either put it in the Q&A or chat or raise your hand. I think we have, we don't have any questions. Could I get any of the Q&A channels? Linux Foundation do succession planning. That's plenty. We know what, we've talked about it internally. The kernel developers know what we're doing. Don't worry about it. It's not the Linux Foundation, it's a project. Projects, we have to let ourselves. Yeah, there seemed to be a lot of confusion about what Linux Foundation does and what is the difference between Linux and Linux Foundation? So it's a Linux project is a separate project. Happens to be under Linux Foundation's umbrella. Yeah, I mean, Linux Foundation is a, it holds the copy, it holds the, doesn't hold the copyright. It holds the trademark, actually a sub-corporation holds the trademark of Linux. And it acts as a help for us to do foundational issues. If the Linux Foundation wasn't there, we would have created something like the Linux Foundation to do this. All big open source projects need this, they need to be able to do marketing, they need to do conferences, do things like that. Linux Foundation helps out huge, it does our conferences for us, it does stuff like this, it sponsors three of our developers, but sensibly it's there as a trade organization for a legal place for companies who wish to contribute on projects together to talk in a way that's allowed by law. So to deal with anti-monopolistic rules, you have to be able to do this in public in a trade organization. That's why it has to be a trade organization. There's just the law in different countries. There's now the Linux Foundation EU, there's the Linux Foundation China, it's been Linux Foundation Japan for forever. Linux Foundation Korea, I think might not still be around, but that was around for a while. It's just a law, it's like the milk board, milk corporations have to talk to each other through that as well. So, Linux Foundation doesn't control development, doesn't tell us what to do. In fact, like our contracts say, I say all the time, I can't tell Linux Foundation what to do and they can't tell me what to do. So they just, we do our own work as far as it. And can me and Linus travel on the same plane? Since we live in different parts of the world, we used to live in the same city. It's actually never come up. Oh, no, we've traveled on the same plane before. Yeah, it's not a big deal. It's not a big issue. Anyway. That is one about, oh, sorry. Oh, that is one about static. Is all kernel code statically tested? Yes. So yeah, so we have tools. Coconel is a wonderful tool by Julia Luol and company or her graduate students. She has done a great job. I think she's given a mentorship series, talk about it as well. She's ostensibly fixed more security bugs on the kernel than anybody else and preventing them from coming back in. So we have tons of good static analysis tool. Dan's... Carpenter. Smash. Smash, Dan's carpenter. Smash, smack is a security model. Smash is a great static analysis tool from Dan Carpenter. He does a lot of great work. We have a very good static analysis tool. We even run co-varity. You're gonna see the co-varity bugs. Co-varity is a little interesting, but that's a close source corporate tool. We do run that as well. You can fix those bugs if you want to. But yeah, we run a lot of static analysis tools on our code body all the time. And we run it on every patch that's ever submitted. So every patch is submitted. We have bots that go out and run all the Coconel tools and Dan runs his tools on them and we report bugs if there's anything wrong with it. We won't take the patch. So that's actually it. And all of them did webinars. So go check those out in this static analysis series that we have on the same site that you'll find Greg Stark as well. Great, yeah, they do great work. You should see that. If you're interested, go check their workout. Very important for us. Standard checklist of tools. Yeah, I wish we could run to make sure new series is not breaking something. Test your workload. So I don't know what your workload is. My, your workload is different from my workload. My workload involves my server or other things to that. So my phone, phones have a different workload. So test and exercise the code in a way that solves your problem. Linux is a tool to solve your problem. That's all it is. It's a utility for solving a problem that you need to solve somewhere else to do something else. So test it to see if it solves what you need. Let's check this. Kernel documentation is updated. Most of the time, all the time in the code itself we have self-documenting code in that we write the code comments that get pulled into documentation. The website's updated every RC release. The entry kernel is documented. I think Jonathan's given a whole talk in the mentorship series about kernel documentation, right, Shua? If not, I should. Oh yeah, I should have him come in and do the talk. I haven't yet. Yes. That would be good. So documentation is always up to date. How to envision an evolution of external security measures in coming years. Talk to the kernel, the people that are doing that. There's the Linux hardening project. They know they have a whole long list of things that they want to do to the kernel to make it better for handle hardening in different types of situations. Work with them. They have a whole list of cases, a great list for people to just start picking up a going from there. So that's the evolution right there. That's a good start. Otherwise, if you have other ideas, submit them and go for that. CVEs, again, don't matter for the kernel. So don't even think about that. Anybody can ask for a CVE, it doesn't matter. Again, if anybody takes about CVEs, mean nothing for Linux, doesn't matter at all. I think that's probably good enough. Thank you, everybody. Thanks for coming. And thanks, Shua, for running. This is a very, very good resource for everybody. Really appreciate it. Thank you. Thank you, Greg and Shua, for your time today and thank you, everyone, for joining us. As a reminder, this recording will be on the Linux Foundation's YouTube page later today and a copy of the presentation slides will be added to the Linux Foundation website. We hope you are able to join us for future mentorship sessions. Have a wonderful day.