 All right, I think we can get started. Welcome, everyone. I wanted to talk a little bit about how we can change the way we manage stable and LTS trees to kind of fit better with how we tell users to use those trees. Before we get started, I'll just go over some history to give context. We'll talk a little bit about how the kernel used to do releases, how it does releases down, and sort of the evolution of how the kernel does versioning of kernel releases. The first kind of way to work, the first step, was Linus basically sending his initial, this is just a hobby, can you guys help me, Mel? It started with version 0.01, and the versioning numbers worked like you would expect them to work in a small software project. We got a small feature, Linus bumped the version a little bit. We got a bigger feature, Linus bumped it a lot. This is how most projects do versioning, and then when he had this critical mass of code that he felt was enough to pretend to be an operating system, he said, great, this is version 1, and go ahead and use it. Now this whole process took a few years, and when we got to version 1, the kernel got some users who are not necessarily kernel developers. Users who want to develop applications on top. Users who want to run their workloads. Users who don't want to run on a development branch. Development branch is bad for stability, and this is where the concept of stable kernel is born. So the kernel continued with its versioning scheme where the size of features and the scale of features means by just how much we're going to bump kernel versions. But the kernel also introduced the concept of a stable branch. So development would happen on one branch, and the stable side of things would happen on a different branch. So 2.4 would be, for example, the development branch, and 2.3 would be the stable branch where things would get backported to. And this concept was introduced where development happens on a dev branch, and then we backport the fixes onto the stable branch. The tricky part here is that the kernel continued with the mindset of features means kernel versions. But the lack of central management meant that we don't know when to cut the release and when it's a good time to kind of say, this is when we release a thing. So throughout the lifetime between 2.0 and 2.6, the kernel release process kind of grind to a halt. We're going from 2.4 to 2.6 was an adventure that took three years. And when the kernel got to the later stages of 2.6, the community kind of realized that this process isn't going to work, and if they continue on this trajectory, it will take an infinite amount of time to go to the next version. So the kernel community stopped and said that this current release model doesn't work all too well, and we need to change it a little bit. And that happened when we were on 2.6. In 2.6 there was this kernel summit, and there was a decision around changing workflows of how the kernel does releases. So instead of feature-based releases, the kernel said, OK, we're going to release a kernel every 10 days, every 10 weeks, sorry, no matter what. We started saying version numbers don't matter, because at this point they really don't. We just release every 10 weeks. But this is really engineers doing marketing, because we never changed how our versioning scheme looks like or how we sell it to customers. This is something we kind of mumbled to ourselves and kept going in hopes that customers hear it, but they didn't. We also said that every release is stable, so there isn't a separate stable tree anymore. Every release is a merge window that takes two weeks, followed by a stabilization period of weekly release candidates for about another seven, eight weeks. And at the end, we can say that the kernel release is stable, good enough for customers to run on. We obviously had customers who want more stability, and this is where the concept of LTS trees was born, where LTS versioning was kind of the fourth number at the end of everything. And this is basically how the kernel works these days. We do a release every 10 weeks, and then there's some LTS kernels that keep surviving for longer. But this is the modern release method for the kernel. This worked incredibly well. The kernel was on a 2.6. That's something versioned for eight years. And things seem to be going great. We, however, messed that up a little bit, because we changed how we do releases. We stepped away from this feature-based release. But we didn't change our version in Scheme, and we didn't sell it well to our users and our customers. We told them to ignore versions, but the version number is still there. And customers still assume they mean something. They assume that going from 3.0 to 4.0 means something that is not a trivial move. A change in the major version is big and scary. When we also stayed on 2.6, instead of addressing this concerning time, we allowed customers to get used to this versioning scheme up to the point that when the kernel wanted to move from 2.6 to 3.0, we realized that some users baked in the assumption that 2.6 will be around forever. They made the assumption that the kernel always started with 2.6. And then we had to bring some hacks on top to work around that. We also found that customers like the idea of minor versions changing. They felt comfortable with it. So when you go from 2.6.70 to 2.6.71, it's not as scary as going from 3.0 to 4.0. It feels like a minor version, and it's not scary. It's pretty safe. The downside of it is when those numbers start getting big, users started saying that, well, we'll just wait a little bit more, right? Going from 80 to 81, it feels like a small change. So we'll just wait until .85. And then we'll do the transition. Well, from the viewpoint of the kernel development community, it's not the case. Every version number is as significant as any other version number. So we just saw them with false assumption that the higher version numbers are largely irrelevant. So to address that problem, we said, hey, that's great. We're going to go to 3.0. High, minor version numbers prevent the customers from upgrading. We're going to insert the sense that every version is important into them. So we started bumping a major version number. I called this the fingers and toes part of the release process because Linus claims he can do 20 releases every major version because he can count them on his fingers and toes. It's completely meaningless. It goes up to 20 just because of an arbitrary decision. And then this is kind of furthers the disconnect between how we say we do releases and how users and customers perceive those releases. When we went from 2.6 to 3.0, we found that the DBI issues around customers, assuming we're going to stay on 2.6 forever, we had config options to make the kernel kind of produce a fake version, saying it's 2.6. That's something. And it took a while to go past that. But after that initial hurdle, things have been working mostly well. We do about 20 minor releases for every major one. So we go from 3.0 to 3.20, and then to 4.0, and then for 21 for some other reason. And then we're at 5.0, and ideally we'll go to 6.0 when we reach 5.20 or so. Version numbers still don't mean anything for developers. We still say it's just a counter where we are on the timeline. It's completely irrelevant. We don't know which features got in when. You should always just use the latest version. And this is where we kind of cemented the whole we're in right now. We started moving major versions, and I think it's scary. If you go to your average user and tell him we need to move from 3.20 to 4.0, they will consider it a multi-year effort that needs a whole bunch of planning around it, paperwork, and in general, they only do it when they have time, which is never. The move in major versions, it's just scary, right? 4.20 to 5.0 is just one version. 5.0 to 5.2 is two versions, although the perception is that going from 5.0 to 5.2 is much easier than going from 4.20 to 5.0, which is just confusing, and it doesn't match what the kernel does with regards to its release process. So even though we got better testing, we got better about not breaking user space, we're good at catching performance regressions, everything. All the pieces are there to allow customers to migrate faster. We're still not selling them what we're doing right. And this is sort of what I want to talk about here. Right now, the kernel does a 10-week release schedule. Every 10 weeks, we've got a release. Many of our users listened to us. It wasn't all bad. So distros like Debian or Arch Linux bought into their idea. They keep using the latest stable or latest LTS version. Android tries to be on the latest LTS. Everyone can understand that it's OK upgrading because it's just a time-based schedule. It's fine. The version number doesn't really matter. We also find that there's a shift right now where users move away from enterprise kernels. I'll talk about that a bit more later. But in general, users want shiny features. They want them right now. They want BPF. They want a new IOS schedule. They want all the performance they can get. They don't want to be in an old kernel. They have new products they have to sell, and they don't have time for backports. Or they just want to be on the latest release. So we're kind of aligned with them with what we're doing and what they want. Users also want frequent security updates. They want them to happen quickly. They can't stay in an old version and hope that no one exploits it. They really want to be able to upgrade faster. So our motives are in line in the sense that they want to upgrade fast and we want to upgrade fast. The disaffected DLTS world, so enterprisey distributions like Reddit, SUSE, et cetera, became less popular than they were before. Maintaining an ancient kernel becomes exponentially difficult the longer time passes. And the speed the kernel is growing right now means that there's so much backport it needs to be done. Making the maintenance so difficult on those old kernels that it just doesn't make sense to keep with that older model where we're stuck on an old kernel release because customers are afraid to upgrade. So in practice, all those enterprise kernels basically turned to be Franken kernels where they backport most of the new kernel. So you might have a distro claim and it's 3.0, but in practice it's running like 5.0 under the hood. There is no reason for users not to upgrade, right? We're telling them we're never gonna break user space. We're playing it safe. If we ever break, you come to us and complain and we'll go ahead and fix it. We're trying to make that movement forward as easy as we can to those users. So they have no reason to sit back. All they have to do is to report it on the mailing list and people will jump and fix it. Linus is, everyone knows how cranky he gets when user space gets broken. There's really no reason to hide those breakages. We also have a lot of cloud natives which is basically the crowd at once the latest features. They want containers, they want new platforms, they want to risk five, they want all the new shiny things. They don't want to run on an enterprise kernel that doesn't support half of the things they want. So there's been also a slow decline of older LTS kernels. This is my crappy diagram of how versioning scheme used to work in that LTS world. So the users will basically pick a kernel version, let's say 5.10 and they would stay on that until forever until 5.10 kind of becomes impossible for them and then they would think of a new branch they want to go to, they will go to that branch and then they would do the same thing there, listen and repeat. What we're trying to sell them is this idea of saying just wait until the next version is out and then migrate to it, just switch your trees, right? So you start with Linux 5.10.Y and then when Linux 5.11.Y is there, you should just switch your branches, start using that one. This creates some difficulty because it means that there needs to be a conscious decision by the user to decide when he wants to switch. First, there's never a good time for that. There's other projects that are always busier and there's never a good time to say we're changing our kernel, especially this often. And it also means there are some technical difficulties in changing that, right? Usually software, we have like infrastructure software, it assumes that we're using a single branch of Git to do our things and when we change the branch in the middle it causes some failures, we need to adjust a lot of systems to use the new kernel branch, we might have to forward port patches on it, it becomes less smooth, we'll have a lot of more hurdles to jump. So what can we do to improve this whole process? We know a few things. We know first that the kernel provides the same guarantees it does going between major versions and minor versions and stable versions. The kernel promises the same thing and it's only really one thing, it promises it's not gonna break user space. It doesn't promise any more than that. The kernel might break user space by accident and it will go and fix it, but those breakages can happen at any stage during the release process. It's possible that you've upgraded major version numbers when something broke and it's possible that you've upgraded your stable version and things broke. There's no guarantees that stable release versions won't break you unless you really test your workloads before it's released, right? So version numbers still really truly don't matter. What matters here is how customers see this move. We need to sell the idea that moving from one major version to another major version is really as safe as going between stable versions and in practice, it should be. In practice, customers should be testing their code. Every time we release a stable kernel just as well as they test it every time a major release by Linux is out, right? There is really no difference in what the customer should do at any step. So what we really want to do is just eliminate this friction point at the end where we have the down arrow. We really don't want customers to kind of face that decision. We're saying that it's perfectly safe for you to move on so you should just take your things that keep using the latest stable or the latest LTS kernel you can. So how can we do that? The way the stable and the LTS trees are structured right now is that we have one branch for every upstream release, which makes sense because the stable trees are basically a fork or derivative from the upstream trees where we only bring fixes instead of bringing all the patches. So what we're basically doing is forking the stable tree at every time Linux releases a kernel. We say we're forking it, we're gonna call it a stable tree and we're gonna bring just fixes on top of it. So the structure of branch per kernel release makes sense there. What doesn't make sense there is that while Linux's tree keeps moving forward quite easily, so within in the same branch you will go for 5.10 to 5.11, 5.12, we don't have that mechanism in a stable tree. We still have separate branches that aren't, they're not cohesive. You have to jump between different kernel branches and that's not a good user experience. We want to create a different structure. We want to create stable trees and LTS trees where those problems don't exist. We want to change it in a way that we keep providing the same guarantees as we did before, that we're not gonna break user space. We just want to change how we present it to the user. We want to stay on the same schedule as before. We will release a stable kernel every time Linux's does this release and we're gonna have an LTS kernels that live for longer that are sort of more stable in theory. But we do want the movement to happen forward by default. We don't want a conscious decision going from 5.10 to 5.11 to 5.12. That decision is very difficult to make and there's no reason for it to happen. It should be just as safe as doing anything else. So looking back at this diagram, this is sort of how we started with long-term LTS trees. And this is what we're trying to sell to our users now that I spoke before. What I'm proposing here is changing this diagram a little bit to something more linear. We want to have multiple stable branches and multiple LTS branches in the same git branch. We don't want different git branch for different version. We want to make it appear just like Linux's tree, honestly. Just like Linux goes from 5.10 to 5.11. We want to do the exact same thing in a stable and LTS world. This is what I'm proposing for those two branches. I added a visual example because it's easier to kind of see how it looks like. And the screenshot on the right side is an example of the very same git branch going from 5.7 to 5.8 within the exact same branch. This is done by git magic, where we manipulate git to make it work. We don't cherry pick patches between the branches. We just do a hack on git merges. This means that the git tags survive as is. The shall ones don't change between the trees. They're identical to the ones on the classic stable LTS trees, which means that it's a bit easier for all the supply and chain folks to verify that it's all the same. It's easy for by section to keep working in that context. We don't break things like git blame by changing the author between the trees. We just manipulate git a little bit to give the appearance that all this development happens within a single branch rather than multiple branches. The idea there is that, yet as flexible enough to allow us to do this, the kernel has a problem of how we present it to the user and we already have the mechanism to fix that. And this is basically a manipulation of how we present kernel trees to the end user, nothing more than that. What does this mean in practice when we go to users? Right now, we're asking users, are you on a stable branch or you're on a stable kernel tree? Are you on an LTS kernel tree? And then that's confusing because we have scenarios where Android, for example, who claim they are on a LTS branch. Their LTS release is about two years old and they just sit on that and they don't move. We have a lot of companies that take the kernel version and they kind of freeze it and they stay with it until the product becomes irrelevant or until the company goes bankrupt or whatever. They claim they are running an LTS kernel but in practice it's not being updated, it's an ancient version and it doesn't get all the fixes it need. That's not good. What we want to change that perception a little bit where users are not on a stable LTS or a stable branch or ALTS branch, we want to say that they're on the stable branch of the kernel, they're using the LTS branch of the kernel. They're not using some random choice, they're using the branch that kernel developers ask them to use for their best experience so they could get all the latest features, so they could get all the latest security fixes. We're kind of saying that you don't need to worry about versions anymore, you just need to say you want to use an LTS version, that's great, this is the branch you're gonna use from now on and forever. You don't need to worry about changing branches anymore, you don't need to worry about deciding what's a good time, you just follow this new Git branch and you'll be happy as long as you have your product running on top of it. There are concerns here, this all came from what I heard as feedback from companies and users. The first one is that we will need to do extra testing at some point, that there is this additional requirement that the kernel should be tested more often, should be tested more thoroughly at some point, that there is this additional increase in test load that we're putting on users. And I very much, I disagree with that and I think it's just a misconception of how this proposal is parsed. In practice, we're not changing any requirements, we're not adding or removing any guarantees, we're just masking them behind Git. So any testing that should have happened before should happen now. Users think that if you move between stable versions, so if you go from 5.10.50 to 5.10.51, you don't really need to retest the whole thing, it's safe, it's only just a few fixes. And it's just a few fixes until it bites you back because a bug snuck in and you didn't test that it's there. You really have to test every time you move in a kernel version and you can complain to us if we regressed anything and people will jump in and fix that. But the testing must happen either way, now or with this proposal. This new proposal also masks the merge window inside of it, right? It means that you might be on 5.7.15 and you do a Git pull and you add up with 5.8.1 and within that pull, you actually got quite a load of patches that are all the merge window stuff that landed and that's a big massive jump. That is also very much correct. This goes back, this goes to two things. One is that we still need to test no matter what. It is possible that the merge window will introduce a lot of bugs. It's also possible that a single stable kernel update will introduce a lot of bugs. Both of them still need the same amount of testing. Other than that, I also ran the statistics of how likely commit is to introduce a bug and you might find the commit in the later RC stages like RC six and later has three times the potential to introduce a bug or introduce a visible regression into the kernel tree. We had that conversation on, in case somebody discussed a few years ago, Linus's claim was that it's by design for two reasons. One is that later RC releases tend to become more complex. It's all mostly bugs that it took a while to figure out that they are and the fix to those bugs also tends to be complex and complex patches tend to introduce bugs. This is just the way it is. Another one is that there's not enough testing that happens during those late RC releases. People kind of rush to get their fixes in so they can be in time for the release. This is all by design. Linus's claim was this is how the process is supposed to be working. He understands why it happens and that could possibly be improved but the meta process behind it is that's how the kernel wants to be working. That's a different topic I disagree on but all I'm trying to say here is that in practice, merge windows aren't the scary part of the release. It's the RC7 which is the really terrifying part. When folks are concerned that we're pulling in patches too fast for merge windows but aren't concerned when we're taking them just as fast from late RC releases that's confusing to me. So really bigger patch drops that come during merge windows are indeed riskier. They indeed might lead to more bugs. In practice they're not as risky as folks think. With regards to the schedule around all of this, these two branches, the stable and the LTS branches aren't branches on their own. In practice they're just derivatives of the stable branch and the derivatives of the LTS branches. We're not changing schedules to anyone. There will still be a new LTS release at the end of every year and there will still be a stable release every 10 weeks. We're just changing the way it appears to customers. And all this rolling tree forces them to do is to use the latest version of that. It doesn't force them to move faster or sooner or later. All it does is saying what is making them behave how we want them to behave. So we're telling users there is a new shiny release. Please move and use that. The old one went ahead of life so you can't use it anymore. Instead waiting for users to kind of accept that and do the move on their own and we're just helping them move forward because there's really no reason not to move forward. This isn't the big change in how we're gonna do LTS and stable releases. This already happens with a bunch of users and a bunch of distros. If you're using Debian, that's kind of how it works. There is a, if you're on Debian's testing branch, you're basically using the equivalent of these for stable kernels. You will always have the latest stable kernel about two weeks or three weeks after it's released. This isn't inventing anything new. It's just a way to align how the kernel does releases with how we're asking projects to use it. And I feel safer proposing it in general because I know that we already have users who use it this way and they like how it works. This is how Chrome OS does. They had a talk in LPC last week about how they upgrade every two years. They're doing basically this. They're just setting the year in between but this is the workflow they're already using. We have a bunch of distros, a bunch of devices that are using this concept of being on the latest LTS tree or being on the latest stable tree. So what we're just trying is to make it easier for them to do that and we're trying to make it easier for new customers and existing customers to do the same thing because this is how we want our users to consume the kernel. This is not introducing anything. This is not introducing the change into the rest of the process. It kind of builds on top of it. We're just trying to address some of the mistakes we did in the past with how we version the kernel and what we tell our users around what they should be using. This isn't removing any functionality. We're just gonna have two new offerings basically and we will ask users to use them instead when we can. This isn't changing anything around the traditional stable trees. They're still gonna keep getting released and it will be the base for this work. So the next slide presents the two new branches but those two branches are just derivatives of the existing stable and LTS branches. There aren't new branches on their own. There isn't anything in those two branches that won't be in the actual stable and LTS trees. This is just our way to tell users that if you wanna follow the development philosophy of the Linux kernel, these branches are gonna make it super easy for you. You don't need to worry about anything else if you start using them. And I think I have a lot of questions. You said this is why we're confusing users with version numbers. In theory, what you say is confusing because the same thing applies if you move between two different stable releases, right? Would you need to recertify it if you go from five 10.1 to five 10.2? You could argue that yes, right? You can argue that not. My argument is that we're confusing by having major version numbers that suggest that those changes are big and they should be recertified. And we have the minor version that kinda suggests that those are fine. You can sneak those in. Well, in practice, it's not the case, right? You can have bigger changes. You could have changes in stable versions that impact you more or impact your security story more than changes in merger version numbers. So it's true. You might need to recertify. It depends on who you're providing your services to. But if you do need to recertify it at that point, you also need to recertify between minor stable releases. So again, I'm not imposing new challenges on you. I'm just saying that it's something that you should have probably done before if you think you need to do it now. Yes. So the movement between stable branches isn't immediate. So one, five, something's out, you're not moving in it. There is a few. We understand that the development process isn't perfect. We still have bugs and we still need to iron them out. So usually the movement there happens two to three weeks after the kernel is released. So if 5.15 goes out now, we're gonna wait until past the merge window 5.16 and then we're gonna move those rolling branches over. So we do give some time for the kernel to stabilize before we move forward. Ideally, we wouldn't need to do it in the future, but for now, we're not perfect. That's what I'm hoping for, right? LTS is kind of our admission that our process is bad because on one hand we're saying to users we should just upgrade to a latest LTS. On the other hand, we keep LTS as a lie for five years. There's this contradiction between what we're telling customers and what we're doing. My personal hope is yes, that we'll get rid of LTS releases. I was really hoping we could stay on two-year LTS periods and then try and remove them slowly to the point that customers can run stable kernels. I honestly don't see it happening in the near term. We just don't have enough resources around testing validation qualification. We're still fighting that battle. I'm really hopeful that 20 years from now we would get rid of LTS releases. No more questions, I'll be around if anyone wants to chat, but otherwise, thank you.