 Okay, this apparently starts now, I thought it started at 35, but I'll go ahead and get started. Welcome, everyone. This is Monsters, Ghosts, and Bugs. So quick poll of the room. Who here has built their own kernel before? Who here has crashed their own kernel before? Okay. Excellent. Okay. For a slightly more getting us into the mood of this talk, think of a system you have, any system, maybe your laptop, maybe a machine, and ask yourself what kernel is running on your system. Do you know what kernel is running on your system? And then ask yourself, if you know, why are you running that kernel? Did you make a deliberate choice to run that kernel? Or because perhaps you couldn't choose what kernel you are? Or did you just kind of use it because it came with your system? And I titled this talk Monsters, Ghosts, and Bugs because, okay, it was pretty catchy. But the idea behind this talk is to get you thinking about how different kernels are maintained and what impact that can have for you and what drives how these kernels are maintained. Kernels are always going to have bugs, obviously, like all software. And ultimately, you already need to have a plan about how to deal those as a consumer of kernels. Sometimes you're going to get spooky ghosts like Spectre and Meltdown. And you'll need to deal with those in a different way. And then I'm also going to spend some time talking about monster kernels that sometimes get a bad reputation, but do also have some good reasons for existing. The first thing I'm going to talk about is how the upstream kernel community releases a kernel, simply because it turns out that how the upstream kernel releases a kernel has an impact on how distributions may choose to release their own kernel. You've probably seen this in other talks, if you've seen Greg talk about it before. But this is my take on the entire thing. If you've ever seen a report from the Linux foundation or LWM, the usual line is, wow, there are so many people working on the kernel every version. And this is true. There are lots of new contributors and new patches each time. And the kernel is always released on a fairly regular schedule. This is the extent of my graphic skills. This is a rough diagram representing the kernel cycle that's happening right now. All the way in the left, you have the 5.2 kernel, which came out early July. And all the way in the right is 5.3. And each of these blocks represents a week of time. We're somewhere around RC5 right about now. And if all goes according to plan, 5.3 will be released after RC7, somewhere around the middle of September. How do we know this is a schedule? Mostly because this is what has happened to previous kernels, and there's no reason to believe this is going to change. Probably Linus Torvald still sets the release schedule. And this is the schedule he's chosen to pick. And the schedule is so predictable, you can probably guess when the next kernels are going to be released. This is a website that someone has put together that is a little bit tongue-in-cheek with the name. But it's really handy because it shows you when the past kernels release and also gives an estimate about when future kernels are going to be released. I always say rough idea when using schedule like this because things can change. The diagram I gave only had seven release candidates. These days we also seem to be getting a lot of more kernels that have an eighth release candidate. And then around the time Spectre and Meltdown happened, we ended up with a ninth release candidate. If you're trying to plan kernel releases down to the day, you're probably going to have a bad time. The kernel community isn't interested in guaranteeing releases on a particular date, but the releases do happen on a remarkably cyclic basis. So the first part of the kernel release cycle is the merge window. After one kernel is released, the merge window for the next kernel opens up. As the name implies, this is when everything is getting git merged together. This time period usually lasts about two weeks. This is a busy time for kernel maintainers. Kernel maintainers are going to be doing preparation of a git tree to be pulled by Linus and maybe also accepting pull requests themselves. Many maintainers don't actually submit to Linus directly. Their pull request will end up in his tree indirectly through someone else. But all the trees will eventually make it to Linus's branch even if indirectly. But this is an important point for two reasons. One that a kernel patch is not considered fully accepted until it's actually in Linus's tree. And two that all kernel releases are also happening off of Linus's master branch. Using this merge window, a lot of maintainers will not actually be reviewing patches. If you try and email Greg K.H. to patch this time, for example, he'll have an auto respond that says, please contact me after the merge window, simply because it's too much volume otherwise. Before a feature lands in Linus's tree, it's usually expected that developers have done some degree of testing, sometimes more than others. But once it gets pulled into the Linus's tree, it's going to get testing from a much wider audience simply because it's got a whole new set of hardware and new people to test on. And you're going to find that the merge window is where most of the bugs are going to occur or introduce simply because this is where most of the commits are coming in. The kernel is still mostly usable at this point. And it's still very important to continue to be testing and using it this time simply to continue to try and find and possibly fix bugs. Eventually all good things come to an end. And after two weeks, the merge window closes and Linus releases RC1 off of his master branch. The good news is at this point, things usually start slowing down a lot. New features are going to stop coming in except when maybe a feature does try and come in. Sometimes features can come in after RC1, but in general, the bigger the feature, the more unhappy Linus is going to be and the more likely is to reject a pull request. There's a schedule to the kernel and the schedule is that new features need to come in during the merge window. But in general, by this time, most of the new features are in because it's time to continue testing. I mentioned that bugs are often introduced and perhaps fixed during the merge window. These may be the simpler bugs because other bugs may have gone unsolved or just unnoticed simply because they hadn't gotten enough testing. And this is the point of the RC cycle, is to try and find and fix bugs. If you see patches that are being sent now, they are oftentimes patches for the current kernel. But the RC, especially later in the RC cycle, this is all for the time to start looking towards the next kernel. Maintainers will start to do more patch review after the merge window. But for patches that are sent now, unless they're fixing new or major regressions, patches will be queued up for the next kernel. So for example, a patch that's sent now will be queued up for 5.4. I've mentioned several times that Linus will often reject pull requests and maintainers will reject patches for the current window if you try and send them at the wrong time. And this is because trying to get a maintainer to take patches off cycle doesn't end well for anyone. Simply because if you try and get a maintainer to take something that's not ready, it's going to introduce possible stability regressions across the entire kernel. Greg gave a talk a few years ago called, I don't want your code, kind of a harsh tame. But he tells a good story about a time that he tried to take a patch before it was ready and the consequences of the calls. So there's a good reason for maintainers being pretty strict about when they take patches. Eventually a week passes and another RC happens. The procedure for each RC is the same for each week. But the expectation is that the number of patches going into each RC should be getting smaller since only bug fixes should be coming in. Sometimes it goes faster than others. Sometimes RCs get bigger, which nobody likes. Sometimes the time passes. And eventually Linus decides the kernel is stable enough and declares a release. And everyone is super excited once again. Okay, so a brand new kernel has come out. This means it's immediately time to go and start putting it on all your systems, right? Maybe not. Just because a major kernel has been released doesn't mean it's bug free. After a kernel has been released, it's going to get bug fixes for a period of time, typically until shortly after the next kernel version is released. So 5.3 is going to be released in the middle of September. And it's going to get stable updates 5.3.1, 5.3.2, 5.3.3. But they won't continue indefinitely. They'll eventually stop about the time 5.4 is released. I mentioned stable updates as receiving bug fixes. When I'm using the phrase stable update, I'm using a very particular term here that's defined for the kernel. The bug fixes that are going to stable updates are a very specific set of bug fixes. They are typically small self-contained fixes that are less than 100 lines. The issues that are fixed must be really issues that bother someone, no theoretical race conditions. And the fix must be in Linux's tree. All of these requirements are designed to hopefully improve the stability of the stable fixes. Because not all the fixes that are proposed out there are actually appropriate for stable, the way patches end up getting into stable is that the patches tag with cc stable in the commit message. Sometimes the submitter will add this, sometimes the maintainer will add this. But if a patch doesn't get added at commit time, it's okay. If you find that a patch is missing, you can email commit hash to the stable mailing list and it will be picked up. And when I say you here, I mean anyone, any kernel contributor can do this. If you found a patch and perhaps done a bisect and you notice that it's in Linux's tree but not in stables, you can email the stable maintainers and they'll pick it up. And then this is a page that gives a short description of the entire stable process if you want to know the entire details. So this is a screenshot I took of kernel.org. If you notice, there are multiple versions here that are marked long-term stable or LTS. These are stable versions that some kernel developer often Greg has decided to maintain for a period of time. Greg usually picks one kernel version a year to be the LTS version. Sometimes he announces it beforehand. Sometimes he doesn't want to because he's found the path that if he announces a particular version is going to be an LTS version, people will try and get patches in before they're actually ready and again causes more stability problems. But usually without fail, once a year you'll get a new LTS. And I actually had to update this slide right as I was working on these because Greg announced that 5.4 is going to be the next LTS version. So as you can see here, Greg maintains a lot of the stable versions. Ben Hutchings is a maintainer for Debian, great person, and he also maintains the 3.16 stable versions. But all these stable trees are not actually created equal. The way these are intended to be used is that you take the most recent LTS, use it for one year, and then update to the next unless you have other reasons. Older LTS releases are typically designed for perhaps hardware enablement or other very specific use cases where you can't update for logistical reasons. And in general what you're going to find is that the older LTS releases are getting bug fixes, but they're also going to trail off after a point so they're going to be getting just a core set of bug fixes. Okay, so that's the story of upstream and stables, but obviously people need to consume these trees in some form. And the question is, is that then your distribution, what kernel is your distribution actually running? I did some look online to see exactly what various distributions were running. This is where you can find information about the Ubuntu kernels. And it's kind of hard to capture in one slide, but from the very detailed schedule they have, they're using a mixture of LTS kernels and their own kernel versions. And this is a screenshot of Arch, and if you look here is that they're running things that are fairly close to the stable mainline kernels, updating on a regular cadence with the monthly stuff. Okay, and then when I say distributions, it's also important to think of distributions as things like Android, which also runs a kernel. And this is a screenshot from Wikipedia showing the kernel versions that were used all across the time with some of these slightly older versions up there. And I'll talk more about Android in a bit and their journey in figuring out the best way to maintain a kernel. And so naturally the question is, okay, we have a bunch of distributions, they're all running things differently. Why are they running these kernels the way they are? And it kind of ends up running down to a trade-off of stability versus features. If you're going to run in a single LTS version for the lifespan of the release, it's probably going to be fairly stable, but you aren't going to see a whole lot of new work since the only commits that are going into stable updates are supposed to be bug fixes, not new features. Okay. One of my full-time jobs is Fedora kernel maintainers. So I'm going to talk about how the Fedora kernel is maintained, just as an example about why a distribution would choose to run things in a particular way. And the Fedora kernel model is to rebase to new kernels as soon as they're released, similar to the arch model, so you get a release schedule that looks like that. And this ends up being the easiest for the Fedora workflow and also well-aligned, but the Fedora philosophy to be a leading edge distribution. With each new kernel version, you're going to be getting a lot of new things, new features. Okay. Everyone's, I need to drop some buzzwords here because they're mandatory containers. Each new kernel version, you're going to be getting new extensions to containers to help them run more securely and hopefully be better running. Another favorite buzzword, EBPF. EBPF is a feature that's been extended over the years to be able to do new tracing. Each kernel version, there's more ways to trace. You're going to get new hardware with each release. I can go on and on about the different things, but the point is that you're really getting the latest with each new kernel version. But this also comes at a cost. All these new features I mentioned sometimes don't work or even worse things that used to work stop working. A real life story from last December, it turns out there was a major regression in the block layer that was people reporting it as EXT-4 file system correction. And there was a lot of back and forth in terms of trying to figure out what the bug was. And eventually, the maintainer did eventually find the commit that fixed the problem. But of course, this was very frustrating for the users who were trying to narrow down the problem and figure out when their file systems were going to stop getting corrupted. Most issues you see aren't going to be that dramatic, but this is kind of the point about updating kernels, is that how much risk do you want to put on the users in terms of giving them no features but perhaps have them potentially have an issue. For something project like Fedora, our users expect that they may occasionally see a bug, hopefully not file system corruption, but some bugs and they'll have to work with us to report bugs to get them fixed. So again, this means that if you choose to run an LTS kernel and not update to a full new version, you're not going to see new graphics drivers or container features, but hopefully you're not going to see file system corruption either. And again, if your goal is deployment on a known set of hardware and you're not really touching it a whole lot, this is probably what you need. I spent some time talking about the upstream LTS kernels, but it turns out there are a lot of distributions out there that aren't going to use the LTS kernels and tend to do something that kind of looks like an LTS kernel but follows a different schedule. Hey, I am obligated to talk about my employer here, Red Hat. A lot of people sometimes like to talk about this is how Red Hat maintains the enterprise kernels for the Red Hat Enterprise Linux. And they've kind of given them this nickname of Franken kernels and because especially from the upstream community perspective, they really do look pretty scary. These kernels tend to pick up individual patches and backport specific features rather than, say, taking stable updates. And the reason they get a bad rep and get these bad reputations is because from the upstream perspective, you don't know exactly what's in these. The kernel version may be, say, 4.4, but you have no idea what's in them. There may be some bug fixes but not others. Yeah. Sometime last year, Red Hat announced that EBPF was going to be backported into its kernel which brought a lot of chuckles as an example of a feature being brought into a very older kernel. And another big objection to these kernels is the argument is that there's a security risk. And the argument goes by not taking all known fixes, you potentially have a security hole since all issues may be security issues. And I can't really dispute that if you don't have a particular patch or bug, you have an issue. But I'm flipping this around to get the suggestion I got from some people to not call these Franken kernels but well-curated kernels. The point here is that these kernels are maintained because there are enough people looking at them that they can usually figure out if something should or shouldn't be included and if it turns out maybe something was missed, they can add it fairly quickly. And this philosophy is that it's better to only take fixes that are known from a known specific set of commits versus taking everything simply because that's one possible way to reduce risk. It also turns out these kernels may offer other guarantees for say binary compatibility and taking stable updates that have other commits that haven't been validated can help possibly make that harder. So when you run one of these enterprise kernels, you should know exactly what you're getting because the people have selected exactly what goes into them. And part of the idea of only taking certain fixes is that sometimes regressions slip into the staple trees. There's a famous quote out there related to Linus' law which is that, given enough eyes, all bugs are shallow. And this is what the kernel community has focused on for many years. The focus has always been on code review to try and find these bugs before they get committed. But I think the stable trees are a good example of where we've seen the limits of this process. Patches for the stable trees get CC'd to the mailing list, as I explained before. And the patch is presumed to be accepted unless someone objects. And then one of the issues we've seen is that developers don't notice that they're supposed to, that they don't want the patch included, or that their dependencies have been missing and that there have been problems in the past with this. Obviously, nobody wants regressions. And I don't want anyone to think that this doesn't mean the stable trees aren't usable. They are a fantastic resource and everyone does a great job of working on them. But it goes back to figuring out how exactly you're maintaining these, and the stable community has had to figure out what are good ways to be able to maintain these trees to get the fixes while also reducing regressions. And the stable trees are, I'm not going to say a newer model, but if we go back to our big scary monster of an enterprise kernel, the people working on that have just decided that for their particular purposes, it's easier just to pick up individual fixes than to try and say, guess what's in the stable tree. Excuse me. And this does come down to needs. The people trying to ship one of these kernels have needs that don't quite match with the upstream LTS kernel, so they do their own thing. I want to talk about embedded and Android as a lesson learned in dealing with kernels and shipping features. The embedded Linux had kind of a reputation for many years of shipping very out of date kernels with patches that had never been reviewed upstream. And the reasons they did this weren't always because they say we're making the deliberate choice about this is how we want to deliver, but maybe they didn't see a better way to do this. And part of this had to do with what the embedded system was trying to do for shipping a product. I just presented the timeline about how the kernel is released and when patches are going to be sent. If you think about it from the other perspective, this means that if you want to get things included, you need to be thinking backwards. So let's say you want to release a product on date X. If you go back to the handy dandy chart I showed about when kernels are going to be released, that means something on date X is probably going to have a kernel Y, which means you need to be thinking about submitting it at kernel Y minus one or Y minus two. Kernel development takes time. Again, I talked about how maintainers don't want to take things off cycle because it turns out it will potentially cause regressions. But if you're a company who needs to ship a product, you can't always work with this. You can't really go and say, sorry, we're not meeting our holiday deadline this year because somebody didn't like our changes to the DNA layer. So ultimately this means that if you're a company who needs to ship something, you're just going to put whatever you have and ship it out. And that was what a lot of the embedded world looked like for many years, some out of tree patches on top of a very old kernel. Obviously in many respects it was successful because products were getting out, but it wasn't the best thing to work with as an engineer or as a consumer. And this is what Android did in the early days. The first versions of Android shipped a bunch of features which had never really been reviewed upstream. Google wakelock controversy if you ever want to see a very long thread about maintainers trying to figure out what they want to do with some of these Android features. And what Android chose to do in terms of shipping a lot of AdFu code was success, but it wasn't very sustainable from a kernel maintenance point of view. But fast forward to today and things look better for Android. Even to that a high level Android is still doing the same thing of shipping a kernel out of tree patches. But it turns out this is the secret of all distributions. Most distributions are maybe shipping a few out of tree patches, minor tweaks to be able to do what they want, even ones that are tracking LTS. But what's changed for the Android community is that they are much better at engaging with the upstream community. They think about their features and they try and propose them. Not every feature they want is necessarily accepted, but they continue to work with the communities and try and find solutions. I mentioned Android here as also a heavy consumer of the LTS trees. They've learned to use that as the basis for their trees so they can benefit from pulling the stable updates that come from upstream. And internally they're doing all their testing on the LTS trees and they're reporting that back to the upstream community so everybody is benefiting from their increased testing. And I mentioned Android as a success, but really today the embedded picture looks much better and embedded distributors have gotten better about working with the community and there are a lot of board support packages that are in fact running the mainline tree. And this is thanks to a lot of hard work and education by kernel developers explaining companies how exactly to contribute to the community. And this is really a key point to a successful kernel strategy. How are you engaging with the community? Maybe the community doesn't want your patch, but you ultimately need to figure out how to handle that. And this kind of leads to the next topic. People sometimes seem to want to run their own kernels off of kernel.org instead of running whatever the distribution provides. And this is certainly something you can do. And when I talk about this, I never wanted to discourage anyone from doing things for the sake of learning. The best way to learn about the kernel, I think, is to try something and break it. You'll get a good chance to learn about what hardware you have if you try and compile your own kernel and then say, miss something or try and apply new patches. You can learn about how things work by testing patches. If you want to try and run your own Raspberry Pi cluster on your own, go for it. But if we're thinking about projects or deployments and, say, that have real users, you really don't want to be running your own kernel. And I don't say this to try and keep myself in a job as a kernel maintainer. Being a kernel maintainer is not always the glamorous life that everyone thinks it is. It's oftentimes pretty tedious. But a big part of what kernel maintainers are doing is thinking not about just what today's kernel is, but tomorrow's kernel. Let's say you found this amazing patch set out there that improves your workload by 1,000%, but it's not going to be accepted upstream for any reason you can think of. So you decide you're going to keep running your own kernel with this patch set. You build, you test, you deploy, you're done, right? This is your periodic reminder that when you deploy your kernel, you are not done. Please make sure you are giving kernel updates. You are getting updates, especially for security fixes. So if you're running your own kernel, you are responsible for getting updates. And this is the piece of advice that is given out there. If you decide you want to maintain your own kernel, please make sure you track one of the LTS branches from kernel.org and just take those updates. I spent a lot of time talking about why the enterprise kernels may choose to not use an LTS bake and pick and choose individual patches, but this really only works because they hire enough kernel engineers who have enough experience to know exactly what they can and cannot pick. You do not want to be trying to maintain one of these kernels on your own or even as a small team. Android is an interesting case of this as well. I used to work on Android phones, and at the time, the kernels we used didn't track stable updates. This meant we were regularly hitting bugs that had been fixed upstream, and then we were scrambling to bring the patches in, usually at inconvenient times. And really, you will forever be playing catch-up if you're already running an LTS and trying to figure out what fixes you need to bring in. Upstream is doing a lot of work to try and maintain the stable trees upstream. Please take advantage of that. But even if you decide to keep using LTS like everyone recommends, there's a good chance you're still going to have to do some work to maintain your out-of-tree patch simply because you're eventually going to run into merge conflicts. And merge conflicts are why kernel maintainers will never be fully out of a job. There are certainly some tools out there to help maintain your out-of-tree patch sets, but ultimately, it's usually going to be humans who need to figure out how the code works. APIs get changed and removed, structures get added and removed, and things get moved around, and you need to know the code pretty well in order to try and do backports correctly. Another story for my Android days. I did a merge incorrectly when I was helping to work on a kernel update, and this introduced a subtle memory accounting bug that went unnoticed for months until there were fine enough reports of negative numbers of pages that I could help narrow down the problem. And if I had to go back and give myself some advice then, I really would have tried to get more people reviewing the backport I was doing to help maybe try and catch the bug. And again, the reason why a lot of these enterprise kernels actually work for doing backports is because they have enough people looking and reviewing the patches and who know the code well enough to know exactly why these problems occur. And again, I never wanted to discourage anyone from doing this work from learning because it turns out you can learn a lot from trying to do backports, but if you're thinking about kernels that are running on production systems, you really want to set yourself up for success. Then we have our favorite bugs exposed last year, Spectre and Meltdown, the ghosts from the title of this talk. I think there's been enough talk explained by people like Greg about how the disclosure of this was handled pretty terribly, and we've certainly learned since then about how later variants were handled. But we're never going to be completely done with security. And we'd like to believe everything is going to be handled perfectly next time, but I'm pessimistic here. So if you're running your own kernel and maintaining your own out-of-street patch set, what are your plans for when you wake up and read the headline about the latest zero-day or issue, you are the one responsible for applying the security fix your tree and getting it out. If you have a backport and the security fix doesn't apply to your tree, you're the one who's going to be responsible for trying to resolve that merge conflict. And how confident are you that you have actually fixed the bug and not accidentally reintroduced it? This is a real problem people have done in the past that major distributions have tried to apply security fixes, done something correctly, such that the security fix bug was still there. It's not only pretty embarrassing, it's not a good experience for anyone. So circling back again, I make it sound like if I'm trying to make it sound like distributions always get it right, they don't. That's a good example. They may give a lot of bugs. But the point here is that distributions are already making these mistakes. So try and take advantage of these mistakes before making your own. Okay. So wrapping things up here, the question of what kernel you pick ends up coming down to what's your focus. If you don't care about the newest features, it might make sense to run a kernel that just gets bugged versus fixes, say, on an LTS kernel. If you have particular stability requirements, binary requirements, maybe you want to run an enterprise kernel. Maybe you want to try and run your own kernel, but you really might want to think twice about it and have a good plan for dealing with things. But more than anything, I think I use the word think a lot, think about exactly what you want for a kernel and try and make your life easier. Okay. Thank you. I think we have a few minutes for questions here. Yes? Yes, that's a good point. The joke about fragment kernels is because they're stitched together, and that is a good metaphor about why they're why they're got together. Okay. Thank you for your questions. I'll be hanging out at the red hat booth after this, so you're welcome to come by and ask me more questions there. Thank you very much.