 OK, can everyone hear me OK? OK, I'm going to get started. This is Fresh Kernels with Fedora. First of all, about me, I'm Laura Abbott. I'm a full-time redhead employee on Fedora engineering under Paul Freels. I'm one of two full-time kernel maintainers, the other being Justin Forbes. The two of us are ultimately responsible for anything that goes into the Fedora kernel. We don't work alone. There are people both inside and outside red hat that contribute to make the Fedora kernel successful. But on a day-to-day basis, we're the ones looking at bugzilla, making builds, and generally keeping the kernel flowing. The kernel is running on every Fedora system, so it tends to generate a lot of interest. My goal with this talk is to give an overview of why we as maintainers make some of the choices we do and how this overall works with what Fedora is trying to accomplish. So this is what I'm going to be covering in this time slot. I'm going to be talking about how the upstream kernel does releases, how this forces how Fedora chooses to do releases, and how this all ends up working really good together for Fedora. So first some introductory terminology to make sure we're all on the same page. Open source projects have the concept of upstream and downstream. Upstream just refers to the entire community working on the projects. You'll often hear us say this patch needs to go upstream first. And all this means is that the patch needs to be accepted by the community first and then later taken to the Fedora, which makes the Fedora kernel downstream. Having as much code upstream in the kernel as possible generally makes things much easier for maintainers. But there are some patches that we will keep out of tree and will never go upstream. So when we do a new kernel release, we typically say we are going to be rebasing the kernel. And all that means is taking patches we have out of tree and making them work with the new kernel. Finally, you may hear me make references to RCS. This is for short for release candidate and refers to a kernel version that is still actively being developed by the community. OK, let's get to the good stuff. I'm going to talk about how the upstream kernel does releases. This may have been covered in other talks, so you may have heard it elsewhere. But this is my take on the entire process. So whenever a new kernel comes out, the Linux foundation or LWN usually gets a statement like, wow, this was the busiest kernel release ever. And this is usually very true. There are people from an amazing number of companies and places while working to attribute. And every release, there are always new people contributing their first patch. And this is all, again, a pretty busy process. But despite this, the kernel is released on a very regular schedule. OK, so this is the extent of my graphic skills. This is a rough diagram representing the kernel cycle that's happening right now. To the left, you have the 4.9 release, which came out at the beginning of December. And all the way on the right is 4.10. And each block represents about a week of time. And we're somewhere around RC6 right about now. And so if all goes according to plan, there will be RC7 the week afterwards, and then 4.10 release the week afterwards. So how do we know this is the kernel release cycle? Mostly because this is what happens in the past. Linus Torvalds, the creator of Linux, still ultimately sets the kernel release cycle. And this is the process he's done for the past very long time. And he's very unlikely to change it without a good reason. So it's so predictable. You can probably guess when future kernel versions are going to be available. If you've ever been curious about the schedule, this is a handy website that will give information about what kernel version might be available on a particular date. The name is a little bit silly, but this is really useful for getting a rough idea of what kernel version might be available, which can often be helpful for things like project planning. But I always say rough idea just because things can change. The diagram I gave earlier had seven release candidates. Sometimes there's an eighth release candidates. Sometimes things get delayed by a few days. Trying to plan kernel releases down to the day is not really going to end well. Kernel developers are not interested in guaranteeing specific days. But the kernel releases are going to happen on a very cyclic basis. So let's talk about some parts of the kernel release cycle. The first part is the merge window. So right after one kernel is released, the merge window for the next kernel opens up. As the name implies, this is when all the kernel patches and trees are going to be getting merged together. This time period usually lasts about two weeks. This is a very busy time for kernel maintainers. Kernel maintainers are going to be prepping branches of patches people have sent to them, and they may be accepting pull requests themselves. There are a lot of kernel developers these days, and many of them do not actually submit to Linus Torvalds directly. They will submit to another maintainer who will eventually submit to Linus. And this is also an important point. Kernel releases are happening off of Linus' master branch. So patches may go through several maintainers tree before they finally make it to Linus' master branch, at which point you can say a patch has finally hit its final destination. A lot of maintainers will not be reviewing patches during the merge window. Greg KH, one of the top kernel maintainers, has an autoresponder for some mailing list that says, I will not look at this patch until after the merge window just because they're busy with other things. If you choose to try and communicate with the kernel community, please remember to be a little bit patient during the merge window because maintainers are busy. But it's not just busy for maintainers. Anyone who's trying to use the windows is going to use the kernel during the merge window is going to find this to be a busy time as well. Before a patch lands in Linus' tree, developers have usually made some sort of effort to make sure it gets some level of testing. But once a patch lands in Linus' master branch, it's going to get testing from a much wider audience, either use cases that developers didn't think about, or hardware they didn't have access to. So as you might expect, this is going to find bugs. And the merge window is going to be where most of the bugs are going to be introduced. The kernel is still fairly usable during the merge window. Fixes can come in fairly quickly if a specific commit can be identified. And this is why it's always important to have a wide variety of developers testing and reporting bugs for the kernel. But eventually, all good things must come to an end. And after two weeks, the merge window closes, and Linus releases RC1 off of his master branch. So the good news is at this point, things will start slowing down a lot. There will typically not be any more major features coming in. But I say typically just because some features may make their way to the merge window post, may make their way to the kernel after the merge window. But the larger the feature is, the more unhappy Linus is likely to be, and the more likely he is to reject a pull request. There's a schedule to the kernel. And the schedule is that new features need to come in during the merge window. And this is because once we hit the RC time frame, it's really time to start testing and making the next kernel stable. I mentioned that bugs are often found and fixed during the merge window. But that may not be all the bugs. Trickier bugs may be still not fully fixed or just hadn't been noticed. Some kernel developers may choose not to start testing until actually the first release candidate comes out. And so you're going to see a lot of bug fixes for the kernel set during this time. But the RC time is actually when developers are starting to look towards the subsequent kernel as well. Once the merge window closes, maintainers will often start to do more patch review. So for developers who are sending patches now, the patches will typically be queued up for the subsequent kernel. So for example, patches set now will not be queued up for 4.10. They will be queued up for 4.11. Because unless the patches are fixing newer major regressions, they really should not be going in. Because again, there's a time frame to the kernel. And requesting a maintainer take a patch in at the wrong time is not going to do anyone any good. Linux will probably reject the pull request. GreatKH has a talk on YouTube titled, I don't want your code. The title is really harsh. But it is a story about what happens when maintainers try to take code at the wrong time and why it's important to be following the schedule for the kernel. But eventually, another week passes and another release candidate happens. Each release candidate is going to follow roughly the same process. But the hope is that the number of patches going into each release candidate should be getting smaller and smaller. Sometimes this goes faster than others. But time goes on. And eventually, things start getting more stable. And Linux decides that things are stable enough and declares a release. And everyone is super excited to get a brand new kernel. But just because a major kernel version has been released doesn't actually mean it's bug free. After a kernel is released, it's going to get bug fixes for a period of time, typically until the next kernel is released. So as an example, when 4.10 is released, it's going to get kernel releases 4.10.1, 4.10.2. And these are called stable updates. And when I say the word stable update here, it has a very particular meaning. The only items in stable updates are small bug fixes that have been accepted into Linux's master branch. And this is fairly strictly enforced about what goes into a stable update. But the result of a stable update is that it's much less work for a maintainer to push out, and it's going to be much less risky for users. OK, so that's the rough story about how upstream does releases. But what about Fedora? Fedora has kernels. How are those getting kernel versions? And that kind of depends on what Fedora version you're looking at. RawHeat is the development version of Fedora that will eventually turn into a stable release. The rawHeat kernel is getting daily snapshots from Linux's master branch. So whoever is just in their eyes maintaining rawHeat is doing daily rebases of the rawHeat kernel, although most of the conflicts in rebase work is going to happen during the merge window. There can be stability problems, like you might expect. But overall, rawHeat is very usable. The laptop giving this presentation is running rawHeat right now. And as always, today's rawHeat is tomorrow's stable release. So when you're running rawHeat and reporting bugs, you're helping to make future Fedora releases stable. And speaking of stable releases, how do those get kernel updates? There are a lot of different options Fedora could take to give kernel updates. Some distros may choose a single kernel version to stick with that for the entire lifecycle. The only updates for that will typically be security updates and bug fixes. This isn't what Fedora chooses to do for a number of reasons. A kernel update is only supported by the community for so long. I described stable updates previously. Eventually, a kernel version becomes end of lights and will get no more updates from the community. When that happens, there are roughly two options. You can either move to a new kernel version that will be getting updates or try and do the stable updates for the kernel version yourself. Supporting a kernel version yourself can be a valid option, but it takes a large engineering effort. To support this, this means you need to be doing the work of doing stable updates, identifying fixes, and backporting them yourself. Backporting is not a simple task. It's very easy to make mistakes if you aren't careful, and it takes a lot of work. Confession, I've done backports incorrectly before that I've introduced subtle bugs that took months to find. And when we think about security updates, this gets even more troublesome. So when you think about something like bad sheep, bad cow, whatever they're calling it these days, you really want to make sure that backport is done correctly. Otherwise, you may still have a security vulnerability. And really, for what the Fedora project is trying to accomplish, there's really not a lot of value in trying to do this work. It makes more sense to move to a new version versus trying to stay in the old one. But other projects may choose to do this work because they have different requirements than Fedora. There are some maintainers out there who will publicly maintain kernel versions for a period of time, a much longer period of time. These are called long-term stable releases. Generally, from one of these to come about, someone has decided it's in their project or company's best interest to support a single kernel version for a long period of time. And they found a maintainer willing to do the work for a long period of time. This is great. These are very valuable for industries like the embedded world where the lifespan of devices may be years upon years. The problem is that these releases aren't related to the Fedora release cycle at all. We know that there's going to be a kernel version release, but we don't know exactly when it's going to happen. And trying to make this up with Fedora release schedules just doesn't end up working. So the end result is that it's best to try and move to the newest kernel possible. So this is a screenshot of kernel.org while I was working on this talk. You can see some of the long-term kernels, 4.8, 4.4, 4.1 I was talking about, as well as the newer kernels. So when the screenshot was taken, raw height was on a 4.10 release candidate. F24, F25 were on 4.8.15, even though 4.9 had been released. And this is a pretty common situation for a lot of reasons. When a new kernel is released, people always want to know if they will be getting the new kernel. And the answer is almost always, yes, you will be getting new kernel. And the subsequent kernel is always, OK, but when? And can we have it right now? And the answer to that is, when we think it's ready, just because a kernel is released doesn't mean it's bug-free. I remember I talked about the stable updates, and there is a reason why they are delivered. Josh Boyer, former kernel maintainer, wrote a really good blog post talking about the fact that a newly released kernel may not be that much more stable than an RC. And rebasing a kernel and making it available before it's had some time to soak and find bugs isn't going to be a good experience for anyone. So as maintainers, we tend to want to wait at least one or two stable updates for a release kernel. From experience, this is usually good for finding most of the common bugs that people are likely to hit. And like the regular kernel itself, stable updates will happen, but they're not guaranteed to happen on a regular schedule. Stable maintainers may be in vacation or be at conferences, so they're going to be delayed. Ultimately, a lot of when Fedora does new kernels, ends up being dictated when the community releases. But rest assured that, yes, there will be new kernels eventually. So I've spent some time talking about why Fedora chooses to give full kernel updates as often as it does, but I'd like to talk about some of the advantages you also get by moving to brand new kernels. A big one is hardware enablement. In Fedora, we try and enable as much hardware support as possible. So if the code is available in the kernel, it should just work on your hardware. The result is that moving to a new kernel version may make hardware go from not working at all to lights on. This often affects things like displays and graphics. And this can also mean hardware may go from working poorly to working a lot better. And this is also a lot of times, while you'll see us make requests you to test on not the newest kernel version on Bugzilla, there are so many patches coming in that it can be hard to tell if a particular issue may have been fixed by a version. There's also always new features coming in. Everyone's favorite buzzword these days is containers. And the kernel is always getting support for containers, for new container features. All the work that's been done on adding various names, dates, supports, just fairly frequently we get requests for things to add patches like SE Linux support. And by running the latest kernel version, you get a chance to develop against these new features and make your applications more secure. So I've hopefully spent a lot of time convincing you that how we do kernel updates is generally good. But part of the problem with getting new kernel updates is that there may be bugs. And updates are never going to be bug-free. This is the trade-off for getting new features and new hardware support is that there may be bugs. And as maintainers, we need you to work with us and report these bugs. We sometimes get nasty grand bugzilla reports that imply we really like breaking the kernel with each update. This is the opposite from the truth. We know it's very frustrating when you update the kernel and something breaks. I don't want bugs. And if I could, I would try and fix everything. I like debugging a lot. But the reality is that even when working with the upstream community, we don't know every part of the kernel. The kernel is huge. And sometimes we just don't know why something was broken. But I really don't want to be too pessimistic here just because the kernel is a constantly evolving project. It's never going to be finished. There's always going to be new hardware and new features that are going to be added. And sometimes fixing a bug is just bringing it to the attention of the right person at the right time. Great example of this is that I reported a bug to the EFI maintainer probably back in October. And then just a few days ago, I got an email from someone saying, hey, we think we have a fix for this. Just because someone else managed to take the time to actually track down what they think the issue was. So there's always hope for getting bugs fixed. So wrapping things up here, you may have seen this reference in Matthew Miller's keynote. The fourth foundations of the Fedora community are freedom friends features first. Our kernel philosophy really captures that. We try to bring you the latest kernels with the newest features as soon as they're stable. And as maintainers, we're not just IRC bots sitting there running on a server somewhere. We're people who are trying to listen what's being told to us because our overall goal is to make the Fedora project the best it can be. And upgrading regularly to newer kernels is the best way to make that happen. So questions? OK, so the question is that if you find a problem on the Fedora kernel, would you rather have that reported on the Fedora bugzilla or the kernel upstream bugzilla? I think the answer is my preference is that because Fedora is often on new enough kernels, it's generally worth it to try and report the bug with us first and let us check to see if it's something we already know about. But in general, you should always feel free to report bugs on the kernel bugzilla. I will say that the kernel bugzilla can sometimes be hit or miss in terms of maintainers responding. Some maintainers are much better than others. Power management I know is very active on the kernel.org bugzilla, so if you have suspenders in problems, I highly encourage you to report them on the kernel.org bugzilla. And the red hack and what sort of things those were. OK, so the question is, what sort of things do we have in the Fedora kernel that don't get pushed upstream? A lot of these are what I consider like little nicety patches that just make Fedora a little bit more usable. But the upstream may just tend to quibble about. A good example is maybe disabling the floppy drive just because that's there. If a maintainer isn't able to give an output response for something, we may take a patch say to remove a warning. Perhaps drivers that are only features. We're also currently carrying the secure boot patch set just because there's ongoing work to try and get that upstream. So the hope is that everything major feature should go upstream. But there's probably a handful of just general little cleanup patches that we haven't submitted. I should probably go through one of these days and see if any of those might actually be valuable to be upstream, but yes. I guess I'm not quite sure what the question is there. It sounds like you're asking there. The question is, is that do we build third-party drivers? The answer is no, we don't build third-party drivers. A lot of this is because manpower thing. There are only two of us full-time on Fedor Engineering, so we just don't have time to be able to support all the third-party drivers, which is really unfortunate from a usability perspective, just because in ideal world, everything would be entry. But for various reasons, Nvidia, for example, you just can't, you may not be able to bring them entry. So we will oftentimes say, no, we can't support this. But we're trying to work to possibly improve this, either get testing so that you will be able to perhaps be aware if your third-party driver might break. So that's a good point. My colleague Justin Forbes has been working on a test harness to be able to test things like the Nvidia driver. So the hope is that even if we can't officially go and fix the Nvidia driver, people will be aware that the Nvidia driver may not be working. So users can go nag the Nvidia developers, or perhaps maybe wait a little bit longer to upgrade their kernel. Yes. Right now, that's really the area of Justin Forbes. He has a small kernel test harness that runs a series of small tests to verify that some things are working out. We don't really have a huge continuous integration effort, for example. It's mostly just a matter of making sure it boots up and then watching for karma on Bode, which is a reminder that if you make sure you're testing on Bode, excuse me. Any other questions? OK. Thank you very much.