 Good morning everyone, or afternoon depending on where you're located. My name is Brad Spengler. I've been involved in Linux kernel security for close to 20 years. I'm the president of Open Source Security Inc. and a developer of GR Security. I'd like to start off by thanking James Morris and the rest of the CFP team for the opportunity to speak to you all today, while I'll be giving a talk entitled 10 years of Linux security. So obviously discussing 10 years of Linux security is a huge topic that could easily take three hours or longer, but I only have about 45 minutes. So what I've done to get a little more content in is pre-record this talk, both to give 10 more minutes that would normally be reserved for Q&A, but also to allow more interaction during the talk. Feel free to ask any questions in the chat during the talk itself. I'm there live and can hopefully answer anything that comes up. If the system allows me, I can also share some additional source links there for more information. As an outline for what I'll be discussing today, first I'll quickly run through some of the key changes relevant to security over the past 10 years. Then I'll dive more deeply into three larger developments that have had an outsize impact on Linux security over the same time period. Then I'll discuss some current exploitation trends and what's on the horizon there. And finally, I'll conclude with some recommendations for improving the Linux kernel security and some of the processes involved with its security. So as a disclaimer, this talk is about Linux security as a whole. So it's not concerning the security subsystem comprised of SELinux, AppArmor, et cetera. Also, there are some known and fixed bugs that are mentioned in this presentation. The intent here isn't to harp on individual bugs. And the names of the people involved in the patches that introduce the bugs aren't being mentioned. The point of me mentioning some individual bugs is as an example for larger process issues. My perspective in writing this talk comes from working outside the upstream development processes, looking at areas no one else is looking at and finding ways in which we can do better. So I don't necessarily come at the problems with the same level of insider information, but I do provide my perspective and the information I've collected sincerely. So with that out of the way, let's get started. As I'm pressed for time, I'm going to move very quickly through the timeline. The slides here will be published online. So all the information will still be here for reference. The timeline that I'll be providing isn't exhaustive and focuses mainly on changes that rose to the level of being mentioned on kernelnewbies.org. It doesn't cover changes that aren't specifically security related, but still can result in a security benefit. Like, for instance, a welcome increase in K-unit and self-test being submitted recently in the past few years. The timeline is also x86 specific. That's not that no non-x86 work is being done. In fact, the PowerPC and ARM64 maintainers are landing significant security functionality recently. And I'll talk more about that later. So 10 years ago at the first Linux Security Summit in Boston, I gave a talk entitled, Linux Security in 10 Years, around the time of a lot of kernel exploits being published against Linux. Many of them written by me with techniques still in use today. For instance, the commit creds prepare kernel cred technique that's now covered in some books on kernel exploitation. I gave some suggestions at the time based on things we had already done in GR security for several years were planned to do, like removing read-write execute from the kernel, eliminating information leaks, protecting global function pointers, IDT, GDT, et cetera, dealing with invalid user memory accesses in general, reference counter overflows, overflows of allocation sizes, implementing CFI, and preparing for the shift to data-only attacks. Kernel self-protection was a mostly unheard of concept upstream at the time. Though on our end, it was a main focus of ours since 2003. But 2020 being the conclusion of the 10 years mentioned in that talk, I decided to give this follow-up to reflect on the past 10 years of Linux security. So in June 2011, the kernel hardening mailing list was created. This came out of an initial interest after my 2010 talk and more specifically was created to track the progress of a Google Summer Code project for Vasily Kulikov, who I hopefully haven't mangled the name of. The main thing that came out of it was the hype-pid restriction on slash proc, but after the Google project ended, the list kind of fizzled out with only 12 posts total in 2014. As we'll talk about later, the same list was resurrected in late 2015 with the announcement of the KSPP, the Kernel self-protection project. In 2011, SMEP support was added, preventing returning to user land and kernel context for CPUs supporting the feature. The YAMA LSM was added, which re-implemented an old GR security feature related to P-trace restrictions. BPF-based Set Comp was added, which saw early adoption by Chromium and SystemD. And similar in hard lake restrictions for dealing with some temp-race vulnerabilities were added, and those were adapted from a patch first written in 1996 by Andrew Tragell, who I also hopefully am not screwing up the name of. In 2012, SMEP support was added, which aimed to prevent direct user land data access in kernel context, but it wasn't usable until late 2014 when Broadwell first shipped with it. Unproved user namespaces were enabled in 2013, which while enabling some new forms of computing, it's been very important for the recent containerization. Unproved user namespaces have been a net negative on the kernel security and a good example of how not to introduce new functionality that exposes a huge amount of kernel attack surface. Specifically, it seemed to have been enabled without any deep analysis of the attack surface it was exposing. This color didn't exist at the time, and I believe Trinity was also kind of in its infancy. Today, Unproved user namespace use is a key component of many kernel exploits. In 2014, a year after publication of our blog that called it out as an effective cargo cult security, KSLR was merged into the kernel. It's proved to be a large developer time sink and generally has never been able to provide the benefits it claims. Just recently, upstream discovered a vulnerability we discovered in 2018 that even core dumps were leaking large amounts of uninitialized kernel data leading to a KSLR bypass. We'll quantify in a later section how much of a waste of time KSLR has been for kernel developers. Also in 2014, the BPF syscall was added for direct extended BPF use being allowed for unprivileged access in 2016. There's a lot of innovation happening there. You may have seen the presentation yesterday about KRSI and using eBPF there for security monitoring purposes, but with it is a lot of churn. And so that too has been a frequent source of vulnerabilities with generally disastrous results given the function eBPF performs in the kernel. In 2015, coincidentally on the same day a Washington Post article was published on the security problems of the Linux kernel, the kernel self-protection project was announced. The announcement mail mentioned that it was a community of people working on kernel self-protection technologies, most of which were developed via our work in GR Security. It also mentioned the Linux Foundation's core infrastructure initiative or CII was funding the work and that it enabled the project to make a dent. We'll talk more about that later. Skipping this one in the interest of time. So in the two years after the KSPP announcement, we see a handful of features based off of ones in GR Security being merged, like the GCC plugin infrastructure done by MSHA, a port of user copy, part of user copy, VMAF stack, and an option for dealing with execution of malicious user mode helper binaries that unfortunately hasn't really seen much adoption that I've seen and requires some user land adjustments to make use of it. In 2017, ref count full and fortify source were merged. Fortify source came from initial work in 2005 by RJ and Vendeven. The ref count code was a manual re-implementation of something present in GR Security for many years. It did receive an update in very recent kernels this year to improve its performance closer to what we have in GR Security, but due to its design, the coverage will always remain much lower. In 2018, the security world changed forever after the publication of the meltdown inspector vulnerabilities. So there are a large number of changes developed to help address those. Initially, PTI and Retplanes, PTI is page table isolation. In 2019, after nearly 20 years of missed innovations because the functionality wasn't present, the kernel finally got LSM stacking. Ironically, it happened around the same time as KRSI was coming onto the scene, which may help restore some of that innovation via commercial security vendors, at least within the confines of what's permitted through the LSM interface. Also in 2019, some fixes for the Intel MDS vulnerabilities were merged and the lockdown mode was added. A few weeks ago, an exploit was published for lockdown, disabling the lockdown mode via loading a malicious ACPI table. Though there was some hesitation in getting a CVE, given the number of kernel subsystems that don't view root to kernel privilege escalation as a vulnerability, it seems like some will be allocated purely because of how heavily lockdown was marketed as a standalone pull-proof security measure and not just a collection of best effort tweets for a subset of a more general problem. And that's it for the timeline. Next, I'll be going more into depth on the KSPP. So since the previous slides mostly covered the output of the KSPP during its close to five years of existence, and you've probably otherwise heard plenty about it via conferences like this, I decided to do something a little different here and present another view into the KSPP via an analysis of its mailing list contents. Solar Designer was kind enough to supply me with a mail spool from the end of April of the list, and I've provided a link to that spool along with the Python script I wrote for the analysis in a later slide. An ALSA script deals only with the KSPP's use of the mailing list, so specifically involving emails from 2015 onward. So I mentioned earlier about the Google Summer Code project that started in 2011 that used the same mailing list. It doesn't include the emails from that, so just keep that in mind. The raw results are here to look over in this in the next slide, and then we'll discuss more about what they might mean. So we can see from the count of emails here per year, the KSPP list peaked in terms of number of posts in 2017 and has been on a significant decline since then, with about half as many posts submitted to it in 2019 as in 2017. And the last number here is an important one that I'll talk about, and it's something that we'll talk about in another slide. So as far as who's contributing to the list, the data shows Case Cook sending nearly five times as many mails as the next top contributor, whose contributions to the list involved a large KSLR Pi patch set with many revisions. I'll note that for this analysis, no effort was made to merge results for multiple email addresses for contributors. For instance, Greg Cage has contributed to the list with multiple email addresses, but it doesn't significantly affect the results to merge them. What's shown in this slide is all contributors by email address, making up 1% or more of the total email count. So based off the stat from the first slide, that 8.5% other category that I mentioned is where we would expect to see the most interesting discussion. Though this does sometimes happen in early reviews of certain patches, or sometimes RFC submissions can not provide a patch. Sometimes it's just general discussion. The 8.5 doesn't include things like that. So it's purely things that don't involve patches or reviews of patches and that sort of thing. But this is where we would expect to see proposing new security ideas, discussing threat models and brainstorming prior to deciding on an implementation and solidifying it in patch form. 8.5% means at least very little of that discussion is happening in public on the mailing list, or at least in the case of much of the KSPP's work to date. Much of that work was already done by others whose work was re-implemented. Patches themselves make up over half of all emails to the KSPP list. One third of emails are reviews of those patches. We talked earlier about KSLR being a drain on developer resources. Males specifically mentioning KSLR made up nearly 9% of all emails to the KSPP list. So even more than the entirety of all other security discussion happening on the KSPP list. Yeah. A sign of this being a corporate funded effort, only 9% of the emails to the list were sent on the weekend. For it to be uniform, let alone a community effort, we'd expect that to be above 30%. Finally, there's a very long tail of people sending emails to the list. Out of 614 total contributors, only 23, those were the ones on the previous slide, sent 1% or more of the emails to the list, and only 17 sent between 0.5% and 1%. And I've included the links here to the mail school and the analysis script in case you want to check my work or perform your own analysis. So the KSPP is focused only on merging new security functionality into the latest upstream kernel. The approach taken for several features involve large rewrites of kernel code, like the unnecessary VLA removal because of fear of a bug on, the ref count conversion and the scripted KMALAC to KMALAC array conversions. While this improves security of the latest upstream kernel and eventually improves security for many or most Linux users years from now, it doesn't benefit the majority today. The majority of distributions are running older Linux kernels. These large rewrites don't meet stable inclusion criteria and don't get backported similar to other KSPP work. There's no technical reason why it couldn't be backported to increase the audience for the work, but I haven't seen any significant interest in doing so. One trend that's appeared since 2018 with the publication of Meltdown Inspector is that a lot of security work is bypassing the KSPP. When page table isolation and rep planes were developed, as well as virtually all other later Intel vulnerability fixes, it was done in private and without the involvement of the KSPP lists. We're likewise seeing that some architecture maintainers are pushing through security changes on their own, especially PowerPC and ARM64. Special shout out to Will Deacon and Michael Ellerman, who are doing good work for their respective architectures. So while this signals waning relevance of the KSPP and matches the decline in mail count seen since 2018 on a previous slide, it's good that certain maintainers at least have a high regard for security and don't need the extra nudges that the KSPP list was designed for. Next I'll be talking about another large change with the impact on kernel security, the introduction of extended long-term support kernels or XLTS. In September of 2017, Google announced a project treble presentation at a project treble presentation that Greg Cage was increasing the support of LTS kernels from two years to a whopping six years. I haven't been able to find any prior public discussion of that move, which is unusual considering how much the stable process depends on public participation. More on that later. This is a 3X increase in the support period with the backports being increasingly difficult due to code churn and other changes over time. For the first time that I'd ever observed in a year after the initial announcement, Greg published a blog entitled, What stable kernel should I use? Several pages down in that blog when discussing the older LTS kernels listed on kernel.org that Greg maintains and mentioned nowhere else conspicuously, it said this, which I'll repeat in full. And I've included the quote here on the slide. There is one huge caveat when using a kernel like this. The number of security fixes that get backported are not as great as with the latest LTS release because the traditional model of the devices that use these older LTS kernels is a much more reduced user model. These kernels are not to be used in any type of general computing model where you have untrusted users or virtual machines as the ability to do some of the recent spectrotype fixes for older releases is greatly reduced if present at all in some branches. Now I hope we can all agree that important acknowledgments like these shouldn't be hidden multiple pages down in a developer's blog given the number of systems in the world using these kernels that are claimed to be supported. So the question is raised, why aren't known security issues being fixed in kernels marketed as being supported for six years? To answer that, we need to look at the process on the stable mailing list. Even if a backport is trivial, any get cherry pick failure results in a mail like the one you see here on the slide being sent to the list. It's an automated mail that gets sent out. For the security fix to be applied, someone needs to mainly backport the fix and then submit it to the mailing list for it to get included in the future stable release. So what happens if no one volunteers to manually backport the security fix? The vulnerabilities simply don't get fixed. And thanks to the upstream policy of not recognizing security vulnerabilities, unless you're following every commit like we are, you have no idea what's being fixed, what's not fixed or if something is missing, when or if it will ever be fixed. Craig's passing mention in his blog a year after the fact was likely the first time the public knew some Intel vulnerability fixes simply weren't being applied to the 4.4 kernel that was claimed to be supported. We've also noticed issues regarding review for patches going into the stable kernels. For instance, there seems to be lacking a process for ensuring when a backport is applied that all current commits listing it in their fixes tags are backported as well. We've seen numerous instances where known broken fixes are applied with the correct fix already readily available. But it can take several additional stable updates for those to get applied. In one instance, we observed an entirely new and unused 8,000 line KVM file being backported all the way to the 4.4 kernel with no one noticing it for two weeks. An additional stable kernel was even released in the meantime, backporting another fix to the entirely unused file. Originally the result of a bad backport. From the dip stat graphic alone, which appears when you do a get pull to that latest release or even in the announcement email that announces the new stable kernel, it stuck out like a sore thumb and it should have raised some red flags for people but apparently it didn't since it took two weeks over two weeks to resolve. As examples of some of these larger issues with the stable process, this slide and the next slide referenced two of our blog posts that go into full detail of the path an upstream change took on its way through the stable process. Either introducing new vulnerabilities through basic lack of review or improperly fixing a vulnerability that was claimed to be fixed. In the interest of time, I'll skip these but if you're curious about the details you can just reference these slides later because it's got the full information in the blog posts already. So I'm skipping this one also. So based on the previous information and public admission from the stable maintainer we know needed security patches are missing from stable kernels. Kernel.org it's mentioned that the 4.4 XLTS will be supported for two more years until February of 2022. One basic way of seeing some of the issues that remain would be to look at Syscaller results for the 4.4 kernel. So let's see what syscaller.appspot.com shows for it. And I've included a screenshot from that website here in the slide. As you can see from the screenshot it's listed as broken with no bugs listed. The Android 4.4 kernel page won't help us either as that hasn't been updated in nearly two years. And it's not just 4.4 either, the 4.9 kernel is listed as supported for three more years and it too is reporting no information on the syscaller dashboard. These kernels are actually also missing functionality important for the full use of syscaller. So what the public thinks is happening in terms of support for these kernels simply doesn't match reality. It's possible that after this talk someone will step forward to do something about the situation. But it shouldn't require multiple years and a public presentation for this to be fixed. People involved in kernel development know this is the current state of things but it seems like complacency is set in. I know that the syscaller developer is aware of it. He's posted about it publicly on Twitter but this is still the way things are currently. So since we have our own independent stable backport process that follows the upstream kernel commits and other sources regardless of any fixes or stable tags which may or may not be present we have some unique observations here. As of our final 4.4 patch we applied at least 1,250 security relevant fixes missing in the upstream 4.4 XLTS. This is over a period of about four years and three months we just retired the 4.4 kernel the beginning of April this year. In recent years we've noticed AutoCell that's AI based tool developed by Sasha Levin that works in conjunction with some manual review by him and I assume some other people. We've noticed it getting much better at spotting security fixes that previously we were the only ones backporting. And so these were ones where there was no fixes or stable CC tag. AutoCell is starting to pick a lot more of those up. However, without an associated increase in people performing the necessary manual backports especially for older kernels some of these additional patches just get dropped at the stable mailing list level. As Dan Carpenter noted in the LWN comment which I've linked in the slide here everyone is overworked. We also noticed that AutoCell could use some more manual review and deciding whether it's appropriate to apply a change to an older kernel and what changes are necessary rather than just being satisfied with a change cherry picking cleanly. We provide here an example of a BPF change backported thanks to AutoCell to 5.4 from the 5.5 kernel that was exploited as part of ZDI as prone to own contests. As far as prior things said I don't want to understate the fact that the stable process has improved significantly in recent years as far as what changes are proposed for backporting. This is primarily due to AutoCell but also the increase in use of fixes tag. It used to be much more rarely used and now it's seeing much more use. It's still not used across the board but it's getting there. When the stable kernel is recent enough and the changes are correct and cherry picked cleanly the process works reasonably well. If I were to have given this talk even three years ago this section would have been much more critical. There are still large areas for improvement but the current state is at least significantly improved over the previous status quo of mostly ignoring anything without a stable CC. Next I'll be talking about something with a bit more positive history and that's CIScaller. In the past 10 years the adoption of CIScaller has had possibly the largest impact on upstream kernel development. It's integrated nicely into existing development processes at least for the latest upstream kernel as we saw in a previous slide and is able to notify developers when an issue in their code is found. CIScaller is constantly improving and integrating new debugging and testing features. Some of those features are listed here like K-Cov Enable Comparisons which helps reach code paths more quickly and the various sanitizers which while not part of CIScaller themselves were developed with the broader goal of improving fuzzing and testing with CIScaller being a major vehicle for the technologies. K-SAN detects use after freeze and other out of bounds memory accesses. K-M-SAN detects uninitialized memory accesses and the most recent addition K-SISAN detects concurrency issues. But CIScaller is a bit of a double-edged sword. It can be used for good or bad. When it's used to its fullest extent and keeping up on the reports it generates it's a huge benefit for the security of the kernel. But when those reports sit around for a long time or when the configurations used for the public CIScaller instances don't reflect what's being used by the public it's useful to an attacker to find some potentially exploitable vulnerabilities. The rapid pace of kernel development means there are always a large number of unfixed issues particularly when CIScaller reaches new areas of the kernel like with USB or more recently in Netlink. More of a philosophical question as I don't have data on this but over the long term is CIScaller improving code quality or making people lazier and more reliant on a tool to review their code? It's something I think about and I've mentioned on Twitter a few times. Next I'll be diving into some exploitation trends. Data-only attacks are on the rise as public attacks continue to focus on the path of least resistance. Some years ago an option called Static User Mode Helper that I mentioned earlier in the talk was merged but it doesn't seem to have caught on at all. Attacking user mode helpers are a prime target in today's world of containers because the kernel itself performs the task of breaking out of any container and executing the attacker's binary with full privileges. As mentioned earlier, public unfixed CIScaller reports are a good source of potentially exploitable vulnerabilities. With tweaks to CIScaller or changes to the default config used you can uncover vulnerabilities that won't be fixed for some time. For instance, our internal CIScaller instances instantly found a memory leak in AFS triggerable just by creating and destroying net namespaces that wasn't rediscovered for six months after we tweeted about it and the default CIScaller config changed. The attack surface exposed by unprivileged user namespaces isn't decreasing anytime soon. In fact, it continues to be added to. I've provided three commits here where additional functionality was exposed through unprivileged user namespaces. When exposing this additional functionality the question is usually, do we want this functionality and does this newly allowed code pass existing fuzzing tests? It would be good if that became a strict requirement for exposing new functionality like this to unprivileged users. Project Zero exploits for the past years have mainly abused views and user fault FD for exploit reliability. In a continuing trend, attackers still don't care about whether a vulnerability has a CVE assigned or not. As we know with the upstream policies it's much more likely that it won't have one unless it gets publicly exploited, is highly visible or is discovered by a vendor interested in assigning CVEs. In the slide here there's a link to a Project Zero blog talking about a vulnerability being exploited with an in the wild O-Day allegedly developed by NSO Group. In this instance, the fix did get applied to the upstream stable kernels. The commit message plainly mentioned the use after free, mentioned that it was found by CIScaller and CC the stable mailing list. I had read a complaint that it didn't have a fixes tag, but the fact that it was applied properly to the upstream stable kernels seems good enough for me, but perhaps that was part of the issue as to why it wasn't applied to the Android kernel that needed it. It may not have been clear that it needed to be applied there. Yeah, so still it wasn't a fix and some affected Android kernels illustrating that downstream security gaps will continue to exist, even if upstream process works as it should. Finally, interpreter security will become increasingly important. The work from Microsoft, Red Hat and others are necessary steps that ideally should have started long ago. It was always an important topic, but the typical focus before was avoiding read, write, execute on disk, binary signing and other techniques. So the importance of protecting interpreter inputs wasn't top of mind. But now, in the days of all the rest of the low hanging fruit being locked down, now this is becoming much more of an important topic. Reactionary security doesn't work anymore. Outside of project zero, few exploits are published publicly. And I'd argue from a defense perspective, project zero's exploits have been of little use as they frequently reuse old techniques or reuse some of their same techniques from previous exploits of theirs. Today's security professionals, especially the younger generation, expect to be paid for their work. Companies like Zerodium will pay them for their Linux kernel exploits. As we saw on our earlier slide, companies like ZDI will run contests with prizes in order to get early protection filters out to their customers. And still other security companies hire up these individuals to develop and sell such exploits. There's been some movement recently from Google to pay out for some of these exploits, but currently at least, it appears to be well below market rates, at least as far as what Zerodium claims to be offering. It's always been the case for us, but as upstream moves further along the path of self-protection, techniques will be increasingly more important than individual vulnerabilities. You can't wait for a public exploit to start thinking about how to improve security. As someone mentioned to me recently, the mouse in the cat in the mouse game has slipped into a hole in the wall. Finally, something that was published late last year, but got little public attention, it seems, is automatic exploit generation. The idea has been around for some time, but I really feel like the work that was presented in December of last year was a big improvement in the state of the art. It's become rather capable, able to generate a couple dozen unique exploits for an individual who has been a part of the business of vulnerability and works reliably as long as CFI isn't in place. If AEG takes off in public use, it's going to make what are called end day or patch gapping exploits and the protection against them even more important. And if you're interested in learning more about that, the link is available here in the slides. It's an interesting presentation that touched on one of the same points from this talk that the harsh reality is that we cannot patch all blogs immediately. Their idea was neat actually. Going from that base position, they used AEG to help prioritize the fixing of CISCOLOR reported vulnerabilities based on whether they were able to automatically generate exploits for them. Their presentation mentioned that CISCOLOR reports increased by 200 bugs each month. On the date they checked November 24th, 2019, there were 459 reports not fixed. Next, I'll be giving my recommendations for improving Linux security and its processes in the future. Rather than give you a list of what all we've been working on the past couple of years that we think will be important in the upcoming decade, we're going to leave it at defenses for data-only attacks and instead offer suggestions for how upstream can refocus and put its existing limited security resources to the most effective use, while perhaps also attracting the newer generation of security minds. So the number of bypasses that existed and continue to exist for KSOLOR still haven't got the message across, but the fact remains that KSOLOR has been repeatedly proven to be security theater. I don't think there's been a single time where an exploit needed a KSOLOR bypass where one or multiple weren't readily available. Operating under the assumption of no information leaks for the Linux kernel is a losing bet. Across multiple architectures, KSOLOR continues to consume a lot of developer time. It's resulted in the confused percent P hashing effort that's hurt bug reporting and has applied inconsistently across the board. As we mentioned earlier, it takes up an outsize focus on the KSPP list. For example, high support to extend KSOLOR for an extra two bits of equally leakable entropy is currently on version 11 of a patch series that started in August of 2017, nearly three years ago. Another point, consider modern and realistic threat models and implement security that can be effective and performant under those models. For example, why is yet another secrecy based kernel mitigation being proposed to deal with ROP when CFI has already been demonstrated for years? Those discussions need to happen. Seemingly simple mitigations can create sunk cost fallacies. This happens when they cannot offer the security they purport under realistic threat models. Each workaround for an attack can add performance hit. And if you knew about all the workarounds that would be necessary ahead of time, you likely would have never merged the feature in the first place. But after you've sunk years of work into a particular indefensible mitigation, will you remove it or keep adding to the hacks and performance hit? This kind of thinking hasn't played out yet for fine grain KSLR, but I believe it'll prove to be another example of this. We have to resist the thinking of, we have to do something, this is something, let's do this. I've linked here to a blog post by Halvar Flake, which nicely summarizes the steps that should be taken before introducing a new mitigation. Keep in mind that with the dearth of public exploits, developers only have to convince each other to ship a mitigation. There's little to no independent or expert security evaluation happening outside of Jan Horne. So shout out to him as he's doing a lot of the heavy lifting actually. The quality of benchmarks for security features proposed for inclusion into the kernel needs to improve. If you recall from the academic paper for Kaiser, the feature which later became PTI and actually what ended up getting merged in the 4.4 kernel because they didn't backport the actual PTI implementation, it once claimed a 0.1% performance set, which we now know is not true at all. A 1% performance set was claimed for stack leak. Replenes were marketed as low impact but ended up having a large impact on network throughput requiring a lot of manual changes to eliminate indirect calls in the code paths there. The benchmarks I see performed mostly involve kernel compilation and a task that only spends a tiny percentage of its time in the kernel. Most changes to the kernel on a benchmark that only spends 1% of its time in the kernel aren't going to see a huge performance hit. It doesn't mean the hit doesn't exist just that you haven't adequately profiled the hit or have diluted it through the choice of benchmark. Ideally, a suite of benchmarks should be performed on a farm of consumer server CPUs of at least the past five years, especially with all the intel vulnerabilities lately and how quickly the performance characteristics of certain instructions changes or keep waiting for pharaonics to share the bad news with the public. Development organization is still focused on patches and mailing lists. There have been lots of calls to improve that but it doesn't seem like there's been any traction yet. In many instances, some of the examples we discussed earlier, including some of the examples we discussed earlier, information was available online pointing to a bug or a proper fix. The current process involves essentially ignoring that external information until it falls in line with the standard Linux development process. Maintainers exist to corral patches into the upstream kernel. What about corraling information into a centralized location to improve the quality of the kernel? Linux has a group of people for handling private vulnerability reports. What about a group that ensures stable kernels actually get the security fixes they need? The problem, of course, is that the general public wasn't involved in the decision to bump LTS support from two years to six years to the primary benefit of phone vendors but that help will need to be found from somewhere. Fail fast, iterate faster. A lot of the review for security patches is trivial nitpicking and canned kicking. Almost as a kind of hazing to see if the person will just go away or whether they'll put in the time to address perceived problems. With security code often touching a number of different subsystems, there end up being too many cooks in the kitchen with different priorities and requirements and this produces wildly different standards and timelines compared to work submitted for an individual subsystem. In the speaker notes for this slide, I've included some links on the history of constify fixes that were submitted upstream around 2009. Much of the reason constify fixes like the ones that were proposed back then have been trickling in for the past decade is because of this broken process of it just being so difficult to introduce some security changes that affect multiple subsystems. Is having the KSLR Pi or SARA code dragged out over three years in effective use of developers time? When an invisible patch is presented that has no chance of being merged in any form, is it really helpful to leave the submitter on by suggesting trivial nitpicks and typo fixes and asking them to put out a version two, a version three, a version 20? For example, when the HKSP patch was submitted to the KSPP list, the first question wasn't, what is your threat model? Or, what are you trying to accomplish? Let's think about what the most effective way of accomplishing this would be. But instead, the response was, interesting, are you all going to split this up into submittable pieces so that they can get proper review and merged upstream? The time to decide on an unprivileged CCTL for an easily abused new feature is when the features first introduce. Not years after the necessity is obvious via multiple exploits. With examples provided here of unprivileged user namespaces, user fault FD, and Perf. It must be possible to do better here. And I noted that it is because we've done this already. Finally, find a way to find security work again. No obituary was posted for it, but the CII is long dead. At least as far as operations outside of awarding a security badge to the Linux kernel, which happened last month for some reason. I have included a link to one of the members of its advisory board mentioning the death of the CII on Twitter last year, 2019. The last announcement from the CII was in 2017. The last mail on the CII discuss mailing list was from a non-member in December 2017 regarding the CII, regarding CII support for OpenSSH having ended in 2016, but still being listed on the CII website. According to OpenSSL's website, its funding from the CII also ended in 2016. Yet on CII's website, it shows it as a currently supported project. This is simply dishonest. You can't solve problems, you won't admit exists, and you can't get funding to solve those problems if companies think the funding already exists. Finally, directly from the still active CII website is a quote I'd like to read in part because it's a good mission statement if what we're acted on with real intention. The stakes have never been higher for open source software security. With millions of people around the world relying on open source software and vulnerabilities like Heartbleed putting everyone at risk, it's time to change the way we support, protect, and fortify open software. And I'll leave you with that to think about. So thank you everyone for your time this morning. I hope that like my talk from 10 years ago, this talk helps provide the public visibility to push for some of the changes that I mentioned, with the result being a more secure Linux kernel for everyone. With the remaining time I have, I can answer any additional questions you may have in the chat. Thanks again.