 Hi folks, thanks for coming first thing in the morning Apologies in advance. I woke up at 2 a.m. Today. The jet leg is real And I'll start with a little bit of background about the kernels are stable and LTS story So the kernel about 20 years ago declared that every release the Tunis does is stable That means that we see a two week to Week merge window and then we say about seven to eight weeks of release candidates where we try to fix all the issues introduced by the merge window What does it mean to be stable? It means that after a Linux cuts its release We're taking that kernel and we maintain it until the next releases out So for about 10 months we take all the bug fixes that go into the next release and We put them into the older one and we're hoping it It's something that helps users gain confidence in upgrading to that Previously released kernel version Now an LTS is a bit different with LTS We take usually the last release of the year would be made as LTS and at that point We do the same process of back porting bug fixes from Linux's tree back onto this LTS kernel And we do this over a period of two to about six years depends on how much interest there is in that specific kernel Sometimes it's two sometimes it's four something and six depends on the community and its interests Why are we doing LTS? Right? It's not obvious because we just said that every Every release thing is does is stable, right? The single most important thing to the kernel community is its users it goes above everything no new shiny features No new shiny device support the the Really the only thing the kernel community cares about is that users have a great experience using the kernel and LTS trees or LTS kernels are really this admission that the process we have with upstream is not perfect We still have bugs taken in we still have issues going into the upstream tree and some users find it difficult to use the upstream tree and so to accommodate for those users we take this Project which is called the LTS trees where we try and make a more stable Kernel and we pay the price of doing that whole work of back boarding and those users We do not see new features come up until they update to the next LTS tree Really what we want to get to the point is that users are happy using this kernel that they're confident from Up they're confident to upgrade from one version to another. They're not afraid of being stuck on this ancient kernel Just because something might break We want that the kernel that those users use to have all the fixes that ended upstream. There's Literally thousands of fixes land in every release cycle And we want those fixes to get to users kernels as soon as we could possibly lend them there And we also want those users to not just be confident not just to regress When they're upgrading but also to feel to feel confident to upgrade to every single release that we do So we release about once or twice a week There's a new LTS release and we really encourage all users to pick up that release run it through their tests run it through their Qualification environment and then roll that kernel out to the production environment Why is doing LTS hard right the kernels fairly unique it's the single biggest open source project in the world We see over 10,000 release. We see over 10,000 new patches In the period of 10 weeks Linus in his keynote this morning mentioned that 6.7 was the biggest Release he ever saw commit wise and that means that the LTS folks have that many more commits to review So they could chip into their users Even though the kernel is over 30 years old the kernel is still fast evolving the systems and the architecture within the kernel are Changing rapidly we see a production of new Features new ways of doing things which are so complicates backports and I'll talk about all of it in the next couple of slides We also have fairly limited resources doing this work Many of us are working on the kernel for their companies individually and so on very few of us are Paid or encouraged to work for the good of the community and so we're trying to make a lot of this effort work with a very limited set of resources and people who do this work and As far as testing goes it's improving in the past couple of years We've seen tremendous improvement in how we test and release kernels up to five years ago. It was Not the best shape and even now it still could use some more loving attention and we were always trying to improve that process So let's go over the issues. I've talked about previous slide in depth a little bit The first one is is those easy conflicts. We all saw so many of us try to take a patch from upstream Put it on some older kernels and we might have saw some conflicts here and there probably simple stuff their context issues Maybe some spelling changes somewhere that we need to take account of and and those changes seem to be simple Right and we could say it's fairly easy to take a bunch of these commits and kind of backport them to all trees The tricky part is that this doesn't work at scale. There's only a few of us who can do this Manually and if we see 10,000 commits every release cycle and let's say only few thousands of them make it into the stable trees And let's say only 10% of them have conflicts. It means we have to resolve In the hundreds of conflicts for every single LTS tree that we have Time-wise, this is not something sustainable. We simply can't do this work. There's only so many hours in a day It's also the case that sometimes we look at something and was when we think well This looks like a very trivial conflict and I'll just fix this up and I'll ship it No one knows what's happening in the kernel and sometimes things that seem like a very easy trivial conflict are really not Sometimes we miss context Sometimes we're not familiar enough with the sub system and it's really easy to mess up something It looks like an easy conflict and break user systems and we definitely don't want to do that It's also the case that the older kernel trees get so if you look at kernel trees that are five six seven years old There are drastically more Tribal simple conflicts that happened as we backport patches to them just because of how much they diverged from upstream These divergence cost us in from having like very simple Cherry pigs they just work into having to resolve tens or hundreds of conflicts every time we try it backward a batch of patches Back to LTS trees This is an interesting one, so the kernel is about 32 years old and We still see major fundamental changes in the kernels In the past year or two we saw the introduction of folios, which is a new way to manage sets of memory pages We saw a major refactoring in the block layer which deals with decades of technical debt We saw the introduction of x-ray, which is there to improve performance and scalability Which are wonderful features to have upstream. It's great to see how the upstream kernel can Improve so drastically even though the project is 30 years old The concern for the LTS trees here is that every time this new feature is introduced We diverge from upstream in the in the older LTS trees So for example in the 550 and LTS tree it will not have support for folios It will not have support for x-ray, right? And what this means is that we now have this permanent divergence between two trees and it causes Two big issues one is that further fixes that end upstream for example a fix in the memory management system that touches Something that works with folios is now not easy to backport to an older LTS tree So any new fixes that land that touch those features will not land in an LTS tree It also means that sometimes we have issues in an older LTS tree and those issues are Inadvertedly fixed by this change so for example a bug that was Buried in the memory management code and then the introduction of folios kind of solved it and we say it a lot that Big refactoring brings new bugs, but it also fixes Many old bugs just because it's a new code base And so what this means is that if you have code that's broken in 515 or an older LTS tree It will just not get fixed because upstream has moved away from using that code whatsoever So you're forever stuck with having those bugs I Previously talked about users and this is kind of an umbrella for what I'll talk about next is that we're doing this to Provide our users infrastructure Which is critical for their users where it's whether it's running workloads running your media player your cars and whatnot The the big thing we care about is users and what we're trying to do is find a way to balance our Limited set of resources with trying to support as many users as we can who use the kernel And what we see is that with newer LTS trees We have many users right because users upgrade and they adapt new trees and they end up being mostly on the Last like release or two or three and then the number of users tape tapers off rapidly as we get to old releases So currently our oldest one is 414 and there's really just a handful of users We still care about it and I can't wait to get rid of that release Having said that it's not the case that we don't care about the users who are on 414 will care about them deeply. It's just that when we start when we look at What we can do to support them our options are fairly limited The first one is kind of the priorities of what maintainers and what patch authors see as important to them We often see complex fixes that we want to backport the old trees and We try them ourselves and we fail miserably and then we try to find expertise in the field We try to find people who whether it's the author of the patch or whether it's the maintainer of the subsystem to help us and try and Both backport the patch and show us ways to test that backward Which is a problem because as Linus has mentioned in the keynote mechanics are already overloaded and overburdened and Mainteners will respond favorably when we ask them to do this work for a kernel. It has many users So the recent LTS kernel trees Will probably get help to get it done So if I were to backport something to six six from six seven maintainers would Try and help us there because they know that doing this backport will benefit a huge amount of users However, if I were to ask them to backport a fix from six seven to four fourteen and that fix isn't trivial You'll get queued in there very long queue and it's very likely that they will never get to it because to them They want to have the biggest impact. They can't have with their time And when when both developers and maintainers look at what they can do in the kernel world The really the best use of their time is doing development on the upstream tree So fixing issues in Linus's tree Developing new features improving systems any work that can do on the upstream tree It's probably the best way to send the resources to After I've done this work if we try and get from the development tree Which is Linus's tree to the tree that users actually use which is one of the LTS trees Then those developers also see the value there. They recognize that Creating a fix in in the vacuum is not enough So if a fix so if a fix only landed in Linus's tree But did it actually reach the users who need that fix then we didn't really fix the issue, right? The fix is only still theoretical at that point and so Developers do appreciate that and they do help us with doing this backwards to recent kernel trees What we've seen really going backwards is that the older the tree gets The last maintainers care about it both both because the decrease in user count But also because the delta between the old kernel trees and upstream has grown so much That doing this backport is very difficult and I think we those of us who try doing it Occasionally backporting a kernel patch from one version to the version prior is fairly easy There's usually not a good chunk of changes there And usually we see small context changes and it's doable if we're trying to do a backport to a kernel. It's five years old We have seen fundamental changes in the kernel some patches become nearly impossible to backport And developers also know that and they sometimes might not even try and just ask users to upgrade to the latest tree The kernel just like any other project has been around for a while and just like any other project the kernel community is composed of people and While the kernel tree has stayed with us for 30 something years some folks have Done other things the folks have started working for example on a particular file system And then they may have moved on to work in a different file system And they may have moved to work on the different subsystem altogether Maybe they're moved from file systems to memory management or maybe they move to work on something else Which is not the kernel. Maybe they're retired. Maybe they're working on Kubernetes and now it's really hard to get to those developers And so we see within the kernel trees that we have subsystems and code Code chunks within the tree that are barely maintained and when we see issues in those areas of the kernel We find it very difficult to find that the required expertise both to create patches for those Issues but also to back for them Because by the time we found someone who knows enough about the upstream code to help us No one really knows what the how the code looks like in the kernel It's five or ten years old and it's very hard to find folks who would spend the time and effort To help an orphaned subsystem or a completed deprecate subsystem to go backwards An example here will be stuff like the Italian deprecation, right? We just decided to kill it upstream Because there's no expertise to maintain it. There's no machines to maintain it And so if you're still using that on your older kernel trees, then you really have to upgrade your tree You really have to move to something. That's not it and you would say Another issue is testing which I mentioned in the past the kernel at this point has a good set of synthetic tests with many tests that That goes all the way from unit test to like small functional testing We have multiple test suites that run and give us results and they find real bugs It's something that we didn't have five or ten years ago And this effort is growing very quickly things like kernel CI things like the zero debut are really valuable in the This effort, it's wonderful to see so many tests being run every time I push a commit to the stable queue For example, and it's really great to it's a great way for us to gain confidence in the kernel before we release it What is very difficult for us to get is real-world testing, right? The things we do in practice in our companies in our home To run our workloads or products is very different from synthetic tests like Unix Bench doesn't Represent any real workload. It's great at finding the regressions in the scheduler. It's not great at catching issues that will mess up your production systems And we found it very hard to get feedback from real users on whether we're breaking the kernel or not No one really wants to throw a health-baked kernel into their subs into their production environment just to give us feedback And so what it means is that we don't have great confidence before release a kernel until unless we get those signals We don't know what each and every one of you run or use the kernel for right some of it is cars Some of it is spaceships and some of it is just no Media servers in your home and we really can't test for all those scenarios We really don't know what people do with a kernel and unless we hear back from folks. We can't help there We also see a case that People kind of hope that someone else will come and fix the problem for them or find the problem for them even right with Linus does seven or eight release candidates and There's there is definite participation in those but there but most folks basically just wait until the kernel is released And then I kind of start testing it because I think at that point the kernel is in a good shape But if you haven't tested that during those release candidate cycles Then it's very possible that the bug you care about wasn't fixed and then you're installing The kernel on your system and you will break it because you haven't tested it during the release candidate Cycles and then we see something very similar with those table and LTS releases because when we cut LTS release candidates We definitely get feedback of Users testing it both on large and small systems But that feedback is very limited and there's this hope that somehow the bug that you care about would be fixed by someone else This problem becomes worse the older the kernel gets because as I said before users really mostly run on the latest trees. There's Diminishing the amount of tests happening the older the kernel gets we seek a lot of feedback for the six six Tree current in the six one tree if you look at the 414 tree or the 419 tree only very few users actually test and report issues before they Before we are able to release it which is very unfortunate because it means that we had bugs that we don't want to add into those Old kernels and we don't know that we added them until it breaks a real system So maybe thinking about what we can do to improve this situation and I think that When when I started this talk or when I just chat with people about this about how we can make Stable kernels more stable or how we can make LTS kernels more stable Usually I hear feedback that when we need to find a way to make those trees last for longer, right? So how do we make kernel trees? Stay active for two years then for six years ten years or how do we make them work for decades? Because that's what makes them stable, right? But like we think that if we don't change the underlying kernel for decade Then suddenly things will improve and all we have to do is just find a way to make it happen And to me that that's that's the opposite of what we want to do. We want to be able to Support the trees where we have the most users and we want to make sure that users are able to upgrade The next one is that we want I mentioned before that the more with the verge from stream the difficult The more difficult the process becomes the harder it is to backward patches to it The harder this to test on it and harder this to upgrade from that kernel because the bigger the Delta is The harder the upgrade processes because you will be hitting so many more issues that weren't tested before and that's why when We do the backpours to the LTS trees we try and take as much as we can into them in the form of dependencies So if we have a fix and it depends on two other patches We will take the fix and the two other patches which might not be stable material But we do that so we will diverge less from upstream the less we diverge the easier the maintenance effort becomes and the same Thing goes to our users if you are using the LTS trees Take the whole LTS tree even if you might not care about half of it, right? If you don't care about half of it, there's no harm in taking that half and in the case that you are mistaken And you actually do care about a patch or two in that half, then you got a free fix for it, right? There's no reason to cherry-pick out of the LTS tree whatsoever and another thing is to be part of the effort and kind of come in and help there is For a project that is the biggest open source project in the world We have so few folks who are actively participating in this release workflow And it will be great to see more folks and more help and more support there And this is for your own benefit right this happens for the benefit of the users at the end and Most of you here are the users and if we and if you don't test what we released And we can't promise that we won't break you because this is really that though I Believe that this is the only thing that the kernel community promises to its users that it will never regress user space and It's unfortunate to see that that only so few users actually take advantage of that promise because this is quite a strong promise and This is probably the most important thing to users, right? We want our workloads to keep working We want our products to keep doing what they're doing. We work the cars to keep driving And if I can take advantage of this promise, I would definitely do that And so that this is a bit tricky because We're not trying to support more kernels for longer. We're not trying to get to the point Where we can support 20 kernels over 20 years This will be the opposite of where we're going for what we want to do is to be able to support as few kernels So we want to take a step back from from the six year kernel Supported and try to cut it down to four and then see where it goes next We want our users to be confident that they can upgrade we want to use this to run the latest LTS tree And we want them to upgrade to the next LTS tree when it's being released at the end of the year. So if we're on 6x6 which will just released we really wish that most of our users would be running on this 6x6 tree the trick for Getting away from very long kernel trees is just upgrading very often so This is all nice and great, but some of us can't just take her products and upgrade them immediately Sometimes would depend on the vendor sometimes would depend on a bunch of other circumstances that prevent us from being able just to upgrade to the latest LTS tree And so while we say that that It's it's something they should be doing you should be upgrading to a last LTS tree And you should be maybe even trying to use the latest table tree as far as you can It's it's really hard and so some of the ways I would suggest it to improve that process for for like yourselves is The first one is invest in your testing story. This is probably the single most important thing that the project can do for itself The Linux kernel has realized that over the past 10 years and there's this massive improvement in a testing story The kernel has and we definitely saw issues going away even going back to about 20 years ago Systems like lock depth for example when S&P was introduced helped eliminate classes of bugs That would have otherwise bitten us for forever This investment into testing infrastructure into tests and the testing story overall is very important Both to the Linux kernel as a project and to you all as users. This is critical And I kind of split it up into three parts the first one is quick small tests. It would be really great if you can Be in sync with the kernel community releases So if when when Linus does is release candidates It would be great if you can take them and test them out on your products to make sure it doesn't regress What we do the LTS release candidates It's wonderful if you can take your product and make sure that that it still runs on that latest released release candidate Next one is whole system test and it's something that's been very difficult To get if there is a way where you can run your product your system Whatever it is and find a way to give feedback both to the upstream community as well as to the LTS community If things still work or things still break and it's it's fine to say hey I saw you released this release candidate. They tried my thing and it works This is an awesome feedback things are working great. Perfect. You don't have to wait until issues come up Even just to hear that things are still working fine. It's really really valuable and the Wider those tests are the better the result is both for the kernel community as well as for the users the more code paths We can exercise the better it is in the long term to keep the stability of the kernel tree both for ourselves and our users The last one is by section and then we saw it a lot with the both in the upstream community and the LTS tree is that users sometimes see an issue when they upgrade they might have upgraded to the latest kernel tree and something broke for them and And just because we don't know what everyone are doing with the kernel It's very hard for upstream developers to dig into What broke sometimes we see a red and foul system corruption or crashes null pointer the references sometimes they're easier to debug but sometimes they're very very tricky and From my observation many of the issues that are not easy to reproduce Are getting lost just because we don't know what everyone are doing with the kernels So developing the ability to bisect within the kernel tree and run those tests on your products is really valuable You will get a much better response if you are able to go to the upstream community and say hey My my workload broke and this is the committed broken. Can you please help me fix it? This is much better than just saying hey my workload broke So developing the ability to do bisections is really really important If we can't upgrade let's say we have this limitation where we can't go from one LTS to another But we still want to make sure that we have that path forward One thing we can do is and I've been doing it in my capacity working in Google Is being able to run a set of our workloads on the latest LTS tree So maybe we don't run the whole The the entirety of the workloads on the list LTS tree we might run 1% or 2% of 5% of our workloads on that LTS tree It's a great way for us to weed out issues In that tree to make sure they don't affect your product, but also for you as users It's a great way to make sure that you have the path of upgrading from one tree to the other open Because if you know that your workload can run on the latest LTS tree Then you know that you have the option of upgrading that you're not blocked on something unknown Because the worst thing to do here is to say hey, I'm stuck on 5-4 for whatever reason and I'm not sure why but I can't upgrade This is really a bad state to be in so doing this proactively Just to shake out the bugs the choppy the list LTS tree is a wonderful way to make sure that you have a path forward to upgrade another thing I've observed is that sometimes we're forced to do an upgrade sometimes the The stars align and maybe the upstream LTS tree goes end of line and of life Or sometimes we need a new feature that exists only in the latest LTS tree Sometimes we it might be a new product that we design and we decide to use whatever kernel version and Then at that point we invest the efforts to move into that kernel and to stabilize it And what I'm suggesting is is that once we've made this jump Let's say we've upgraded from 5 10 to 6 1 and let's say we're happy in 6 1 Don't take your foot of the gas pedal if you're at 6 1 great. You've accomplished that upgrade How do I get to the next kernel version? How do I go to 6x? I already have I already know how to do this I already did this right now Jumping up one kernel version. Let's keep doing it to get all the way to the end. This is a really great way to Kind of Create this culture of being able to upgrade on an annual cadence or maybe even more often It also becomes very easy to upgrade if the Delta is small So so going from a really ancient kernel to really new kernel is difficult because there's so much more Content there's so much more new code that wasn't tested on your systems It's really complicated You will be hitting so many issues and the more issues you hit at once the more difficult it is to fix them If you hit 10 10 separate issues all at once You will have a hard time finding which issues are those really are and splitted them into individual ones and then fixing them where if you go from Where if the Delta is small and you jump from one kernel to the next and you might hit only two issues and Fixing those two issues is fairly straightforward because just how small they are because the number is so small The the the more the number grows the effort needed grows Exponentially to address them The last one is to report stuff We have seen so many cases of users who are Stuck with an old kernel and they have never told anyone they've never said anything Upgrading like some users stuck on like 510 and when they try to upgrade it broke their graphics card So they never did and they just decided to stay on 510 No one knows and no one could help them and so it's very important for us to be able to Hear from you to get feedback and this is really going back to the first slide. This is all about the users This is all about you. You need to tell us if things are working or they're not working and we're really trying To get as much feedback back as we can even if you just tell us say things are working great for us That's wonderful, but just please test and report it back Yeah, thank you for questions comments Thank you for your great talk so I'm working for the embedded products and What prevents us to use the upstream kernel is so-called vendor kernels Which is made by the SOC suppliers So I think it makes really huge fragmentation and if there any kind of activities on that program or maybe some suggestions to deal it with it Yep, this is definitely an issue. We've been trying to tackle. I saw it myself many times I think that that Greg has been putting up an effort in the past decade or so with working with vendors and trying to get them to do the Right thing has been running around between vendors countries Entities trying to get them to be aligned with the LTS tree and I think we're slowly getting there I think that if you have a certain vendor that is not playing well You can try and contact in us directly to try and in all seriousness We'll be trying to get everyone on the same page and and I I'll tell you about a case earlier this year where I felt really good with myself We're working with a certain vendor who is actually ahead of us of the product I manage and they were complaining to me that that hey, why are you using the sentient 6-1 kernel? You need to upgrade So we slowly get in there But unfortunately this all comes back to the relationship you have with that vendor if you can push that vendor to Be better aligned with upstream. That's great. We've seen Cases where companies do it on a contractual basis Where part of your contract is that you have to make you have to upgrade within certain amount of weeks after the release of the upstream LTS and also pressure in vendors to upstream their code because you're stuck with the vendor tree because it has so much Out of tree code if you get that vendor to upstream all that code Your life will be much easier Thank you so much. I think it's quite interesting that you mentioned that when we upgrade our kernel the GPU breaks So is that I think that's one of the things that stopping us because we are an R&D company and then we try to test the Sometimes you try to use a newer kernel and then with a different QDA version and then things always break So that's what's stopping us from always upgrading to the latest LTS So is there anything working in the on your side or in the back-end side that is like providing an archive Saying that this new kernel supports this version of the GPU things like that or because these are I think very prominent Challenges that is stopping us from always upgrading our our kernel. So What do you suggest? Yeah, so we recently have something called the regression tracker upstream Which is a way for us to say that hey a certain functionality broke at a certain point and we try and keep track of that So I would say that if you try to upgrade and you saw that a certain piece of hardware break You can report it. There's a regression mailing list and there's a very great community of people who try and help there And they help us to try and track down those breakages, right? Because it shouldn't be the case that some GPUs work with some kernel versions It should be the case that all GPUs work with all kernel versions, right? And so if at some point a certain GPU worked with a kernel and then it stopped working then we want to find out why and we Want to fix it So we have the regression mailing list Think it's regressions that I'll find it I can find it for you later and it's a great just report issues there And we will try and fix and it really helps if you can pinpoint the specific kernel where things broke Or a specific commit where things broke because it sounds like you would be able to bisect the issue, right? And just those reports are really valuable Thanks, Sasha for your important points share most of them Another question. How do you measure your users of the older kernels also given that some of them are indirect users? The challenge always is with users. They don't need to speak up when something is broken at most sometimes even not then And I guess the LTS kernel is also indirectly used quite heavily Downstreams by you mentioned them vendor kernels, but also in-house kernels, which are based on LTS at least So do you have some numbers on that? We don't we have not like kernel communities and like we don't know who does what Our Guestimates of how many users we have are based both on what you said of how many users come and say that we use it But also on our kind of knowledge of the world So we know for example Android uses this kernel version No, Chrome uses that kernel version. A lot of it is guestimates that we have to know where we are But that's why we're asking for feedback from users We want folks to come back and tell us say I'm using this kernel version. It works for me. This is an awesome feedback It's so appreciated and and I don't think as many people realize just how good it is Hi, thanks for a good talk So I think one of the indirect motivation of using our old kernel is that people want to use old distro Because they don't want incompetent incompletability inside user land For example, they want to use our GCC package that is supported on old Debian So they use old kernel without knowing Do you have any insights for that? Yep. So about 12 years ago. There was a kernel summit in Cambridge, Massachusetts And the one thing that came out from that summit was what's called the Cambridge promise and Linux Linus has stated very clearly The kernel will never break user space The single thing that the kernel community promises to use a space it will not break it If you see cases we're upgrading to a new kernel version breaks your user space and it's a real Scenario right you have a real user space package that stops working as you upgraded your kernel Please report to it. Linus has gone through above and beyond to fix these issues You reverted entire patches of code. He did crazy things to make use of space keep working This is the number one priority of the kernel So if you don't feel comfortable upgrading because you're afraid to use a space will break Go complete if you identify such an issue, please please please report it. Yeah, sorry I know about the promise, but People if they want can upgrade a kernel that is used by all distro But the only way they can upgrade a kernel is up against the whole distro Because they don't have an ability to build a kernel by themselves That point it depends on your distro vendor Then point you should pressure your distro vendor to help you or upgrade to a new version of the distro That's kind of the relationship between you and your distro. Yeah, thank you Excuse me. I'm a devian developer. So I want to say about the TPM They may use the only LTS kernels Not to not to all of the kernel versions so and we provide the Backport kernel packages so you can use the Newer kernel versions without upgrading whole distros. Yes, in general, I would recommend using Distros is similar as Debbie and rather than more enterprise of distros that tend to be stuck on the kernel That's 10 years old And it's kind of frank and colonel it barely kids jogging along Debbie and there's been a great part of the community and I know which that more distros will be We're doing it as well as Debbie and us Hi Sasha. Thanks for the talk. So what do you think is the sweet spots for the LTS tenure support four years or? Depends on the community. It's really not up to us, right? It depends on how much the community the community is there or not. It's not like something that If the people are there will support it but but so far it doesn't look like enough people care about old kernels, which is fine Yeah, typically it depends on the sector like auto customers tend to stick on for long some of the customers Want a migration every year? Okay And I think we're running out of time So we'll call it. Thank you very much