 Okay, so my name is Josh Boyer. I am theoretically the team lead of the Fedora kernel team. That really just kind of means I'm the contact point and I shield people from various nasty comments as much as possible. This is the traditional state of the Fedora kernel or Fedora kernel talk. It is not about like super cool technologies in the kernel. I'm not gonna tell you how RCU works. If you would like to know, read LWIN, excellent articles on LWIN. Today we are going to do a little bit of a release overview on how we operate the kernel in Fedora in terms of rolling out new versions and things like that. We're gonna talk a little bit about the team. It's changed over the last year or so. We're gonna have some focus areas that we're currently either working on or looking to get into in terms of what our team is doing in the kernel to make Fedora better. And then we're gonna talk about some future kernel developments. That includes like Fedora 23 and some of the new shiny things that showed up in the kernel over the past year. So release overview. We have one kernel pretty much. That's really two. But we have one kernel for every release, right? So Fedora 22 is on 4.1.5 is in testing. Yes, thank you. But right now, 4.1.4, 4.1.5 is in testing. So 4.1 is the major version. Fedora 21 is also on the same. Fedora 22, I already said that. Fedora 23 is on, we're in that weird branch state in Fedora where we just had alpha. But it's on like the 4.2 RC candidate kernels. So what we do is we kind of rebase as the new upstream release comes out and we roll it back in a graduated fashion, right? So when 4.2 comes out, Fedora 23 will get it. Rawhide is continuously rebased every day. Fedora 22 will get 4.2 probably around the 4.2.2-ish time frame, so a couple of weeks after release. And then Fedora 21 will be a week or two after that. And we do it that way so that the oldest, most stable release kind of isn't, it's not broken right away. And if there are major issues with the rebase, we tend to fix them before we roll out the next one. We do it for a few different reasons. Primarily we do it so that we stay close to upstream because the developers are either still working on it or actually they haven't heard that whole release out of their mind. And we do it to consolidate bugs, basically. Pretty simple. This next slide here is, it's an example of why I'm not a graphic designer. But it's a really terrible, not updated flow chart from the first time I did this presentation about how we do the rebases. And I'm not gonna make you stare at it, I'm not gonna read it to you. I'll post the slides on the schedule description afterwards. And you can marvel at my terrible skills. So right now, like I said, we're on 4.2 for Rawhide and Fedora 23. I didn't add it on this slide. That must have meant I started my slides before Alpha came out. Wow, I'm impressed with myself. Fedora 22 started with 4.0.4. The way we pick the kernel for the release is kind of, it's kind of throwing a dart and sticking it at the wall because it depends on upstream kernel cycle. It depends on the Fedora cycle and depends on how willing QA is to let us squeeze stuff in at the last minute or conversely, how willing we are to subject ourselves to the punishment that comes afterwards. So as you can see, Fedora 20, which just went into life, went from 3.11 all the way through the 4.0 kernel and it worked out pretty well. We had regressions as we go and we get those when we try to fix them up, isn't it? So any questions on how we handle the kernel in Fedora? It's pretty fast, but it's also the same way I've talked about for the past three years. Team stuff. So like I said, our team has changed right now Laura Abbott is our newest hire. Hi, Laura. Justin Forbes has been on the team almost as long as I have, right? Or did you join at the same time? Yeah, you were doing the bird stuff. Yeah, so Justin is four years, I think so. He's been on the team since the first block in a way. So at least three, and then myself, and then we have various architecture contributors. Peter Robinson, who is actually giving a talk on power and arm right now in a different room. So if you're interested in arm, that's probably where you wanna be because I'm not gonna talk about it. Is one of the only other people who actually has a consistent amount of commits in our Fedora kernel. He handles all the cool 32 bit board stuff for us. And we love him because that means we don't have to worry about it. We have Dan Horak, who does, he doesn't really do kernel stuff, but he handles S390 and he's more than happy to point out upstream patches or broken issues on S390. And then we have a very large contingent of Red Hat engineers behind us. And when I say that, I don't mean that it's like RHEL, right? RHEL has a literal arming of kernel engineers for various things. But we do get to leverage them when we ask them nicely to look at certain bugs. And we try not to do it too often because they're pretty much buried all the time in upstream and RHEL work. But when we do approach them, they're very helpful to have. So what do we do all day on the team? In the past, the next slide, the answer to this question has been about four or five slides of bug numbers, charts, graphs, showing all this other stuff. So today, there are no bugs. We don't do bugs. They're all fixed. So I'm not gonna talk about bugs. Sorry to disappoint. If you wanted to see updated graphs, I can point you at bugzilla and the queries I use to generate them and you can enjoy them all you want. There are 702 open. On any given day, there are around 700 bugs open across all the releases. So no more bugs. We're not talking about bugs. Let's talk about retrace, right? Yeah, what's retrace? Everybody know what that is? Good. Anybody not know what retrace is? Okay, retrace is, it's me being clever and saying we're not gonna talk about bugs, but really we're gonna talk about bugs. No, it's this infrastructure. So in Fedora, there is a automated bug reporting tool, ABRT. So if you get a kernel crash and it's not one that locks your machine up, say your wireless driver craps out and you get a backtrace, it'll collect that and send it to the ABRT servers and the retrace servers. So it collects all these reports in an automated fashion and then it points out how many of each instance there are. And if you go look at the retrace server, it's kind of frightening because you'll see like one specific backtrace from a wireless driver will have like 30,000 hits, which means 30,000 machines or instances has happened just for that one. And then if you look at all of them for the kernel combined, it's just as frightening as Bugzilla is, but we switched to it this year or in the process of doing that because Bugzilla is a mess. Bugzilla, there is humans involved, okay? So you get the automated report, you get the backtrace and you start poking at it and you realize there isn't enough information to even start looking at it and then you start asking people for more information. Sometimes they're really great. Justin just had a talk last session about how to get your bugs fixed, but sometimes they're not. Sometimes people are like, you know, my wireless doesn't work, just fix this now. And you ask them for information, they don't want to be participating. So it's really, really hard to get a broad overview of which bugs are hitting the most number of people by looking at Bugzilla because it's specific to an individual report. So retrace offers a high level data where we can look at it and say, okay, this i9-15 bug, you know, there's 30,000 machines hitting it or this wireless driver, there's 40,000 machines hitting it, whatever the case may be. And Laura has done some work since she joined us to help us cut down on bogus data and kind of help prioritize where we're gonna look. So we use that now. Problem areas we see, i9-15 right now. So in the last three kernels, so 4041 and what will be 42, they're doing this work in the i9-15 area and it's really for all kernel mode setting drivers where they're doing atomic mode setting. So, you know, you change panels, you change resolutions, whatever. It's all gonna be atomic now. In the past, it's kind of been this collage between X and the kernel and how it does it. It's complicated, but it hasn't been atomic. So sometimes you get things that are out of sync with each other and then you get bugs and it's nasty. So as kind of a byproduct of that or a side effect, the i9-15 drivers kind of destabilized a little bit as they work this in because it really is a complex problem. So we're seeing a lot of issues there. We're doing our best. Suspend Resume is a traditional one that we always have problems with. Right now they seem to be tied to the i9-15 problems, but they are also, you can have Suspend Resume problems for any number of reasons. Single driver not failing to go into Suspend is basically all it takes. Or you have bogus firmware. We really don't like those because you can't fix the firmware on their machines and people don't understand that sometimes. Wireless used to be like at the top of the list. As long as you're using the open source drivers, it's not that bad. That being said, the Intel wireless driver does tend to freak out a little bit these days, but it ebbs and flows. And then platform drivers is a typical one. So your function keys don't work. Like in fact, this laptop here, I can hit the brightness keys with the function keys and it doesn't work at all. I think Owen actually has the same laptop with the same problem. I have a patch to fix it and I send it and the i9-15 driver, people were like, okay, that's good, but you need to do it this way. And doing it that way requires rewriting like half of the ACPI i9-15 infrastructures so I'm not gonna do that, sorry. Fortunately, the little GNOME slider works just fine. And then Bluetooth. Bluetooth is usually an area where people suspend, they resume, and Bluetooth, that's not this nice back trace. Why? It depends on the chips that depends on the firmware. So these are the problems we see and we're using retrace to kind of help us prioritize. So what do we need to buy priorities? In terms of priorities, this is what we have. We focus on the most severe ones, obviously, first and that kind of goes counter to the number of people being impacted, but we do that to prevent, you know, CVEs from turning into major events. High retrace count reports, like I said, corruption issues, fortunately in the kernel, unless you're using butterFS, we haven't really had any corruption issues that are major as of late. I think there might have been one EXT4 bug in the past year that I could think of. Yeah. And there might have been one XFS issue where you mounted the file system in a certain way that the metadata didn't match and something minor happened. But, you know, corruption does happen. So we focus on those when they come in. And then general kernel crashes. Again, brightness, touchpad, they all kind of fall out after that. Sound is down towards the bottom of the list. It should be higher, but we just don't have the time to get to it, right? It's really frustrating when people buy a machine and they plug in their headphones and they can't hear anything. I understand that. I would be frustrated as well. We just don't have any expertise with the also subsystem. But our FS is on our list contrary to what everybody thinks. It's not that we hate it. It's just that we don't promote it by default. But it is tied to basically reporting to upstream saying, we're seeing this, are you guys seeing this? And they're really responsive when we do that, but they see a lot of issues. So they're also the hope one. And then down at the very, very bottom, after global warming on Mars, we have I686. We don't focus on Intel I686 bugs at all anymore. We sent out an email to the development list in February and said, hey, 32-bit Intel I686, if this is gonna survive, then the community needs to step up and do it. And everybody ignored us, basically. So it's just not a priority for our team. It's not that we don't have nostalgia for it and it's not that we don't even have machines. I have a 32-bit machine at home. But if it had an issue where it wouldn't boot, I would go buy a cheap 64-bit machine instead of fixing it. That's where we are today. If you really enjoy I686, please find me afterwards and I'll be happy to talk to you about it. Yeah, it's fine. There's nothing wrong with people telling us that it's broken. I have no issues with that. I would love to know that information because then I can say, look, this is what needs to get fixed. And it turns out that that issue was in the kernel for two months and nobody noticed. So, right, right. So, it's just, it's not something we focus on. It's not that we hate it. It's not that we want it to die. We just don't have the cycles to spend on it. That's all. Focus areas. So focus areas are, I mean, they're not a new concept, but it's something new that we're trying this year because staring at retraces, staring at bugs, we can do it, but it's enough to drag you crazy. There's always going to be bugs. We're not going to fix them all, unfortunately. So, what are some of the things we're looking at? We're looking at power management. This one came from Christian who conveniently left, but Matias is still here. So this one came up, not actually, it was fairly recently in terms of Fedora timeframes. What can we do to have better power management? Suspend, resume, obviously is one of them. If it doesn't work, then your power management stinks because if you fold your laptop closed and you stick it in your bag and it doesn't suspend, then when you get home, either it's overheated because it stayed in your bag and stayed on, or hibernation failed, it doesn't matter. So that's something we're not charging forward on, but it's something we're looking at right now. It's hard to figure out exactly what we want to do there. More automation. So we have a number of different projects. We already have the kernel test harness and infrastructure so that as a build comes out of Koji, which is our build system if you don't know what that is, Justin wrote this test harness and framework. It picks it up, it boots it, it runs a regression test. All in automated fashion has results on the web and it's actually very, very useful. Because before we had that, it was one person would build a kernel and we booted on our local machine or few machines and if it worked, everybody else gets it, right? So we're improving our test coverage through automation. The exploded Git tree that we had, we started at the end of after last flock, I think. So Fedora's kernel is packaged as a traditional RPM with like a tar ball and patches applied to it and stuff like that. And all the kernel developers we would talk to would say, this is terrible. How do I recreate your kernel? Because you guys are carrying patches. Which is absolutely true, but we don't carry a whole lot of patches I think. We have on any given day in raw hide somewhere between 50 and 80 patches and most of them are very, very minor. So I created an exploded Git tree that developers were used to using and that went fine until about a month and a half ago when we tried to switch to using that instead of like the traditional RPM setup as the canonical source and it worked, but it was really, really cumbersome to take that and go back into the Fedora package that everybody actually consumes. So we're gonna, I'm going to step back and look at automating the creation of the exploded tree instead of vice versa and then testing a cover with Justin. He's actually testing Linux Next. If you don't know what Linux Next is, does anybody not know what Linux Next is? Good, excellent. I see some confused looks, but it's basically what it sounds like. It's what will be the next version of Linux. So he's testing that, which we've never done before. We aren't gonna ship it because that would be a little too crazy. But, you know, testing it now means we find the bugs and we report them upstream while they're still developing them before they get into Linus Torval's tree. Which means you don't see them, which is excellent. And so I have on here upstream and it's kind of a depressing goal for a focus area in a number of ways, but our daily job is not developing kernel code. It's not even necessarily fixing kernel code at this point, but that's something we're trying to get to. Over the past year, I might have three commits in the upstream kernel and for a kernel maintainer, that's sad days. I mean, you know, packaging is fun and I'm glad I'm participating in Fedora and providing value, but it's not what we want to be doing. So we're trying to have like a getting back to roots, going back upstream and making sure we're actually making an impact there as well. So that's one of the focus areas. And then at the bottom, I have our yearly plan. So I'm part of the, our whole team is part of the Fedora engineering team. It's run by Paul Freels. We try to be open and transparent what we're doing. We are some of the few people who are blessed to be working on Fedora, 100% of our time is what we get paid to do. And so kind of as a, you know, a thank you or encouraging participation part of that, what do we do? We put it on the wiki and there's our plan. So if you're interested in what we're looking at beyond what I have on the slides, you can certainly click on there. You can email us on the Fedora kernel list and ask questions and we're more than happy to answer them. Fedora 23 Outlook. All right. So Fedora 23 Alpha just came out. Yay! I think it shipped with 4.2 RC5, right? With like one extra patch to fix the I386 bug. We're probably gonna wind up releasing that with the 4.2 kernel because the development timeframe for 4.3, particularly now that we're getting into conference season for kernel developers is probably gonna lag a little bit and it'll just cut it too close to the end point. So 4.2, it'll be like a 4.2 stable release. So geez, by then it'll probably be like 4.2, 6 maybe if it lasts that long. So, but 4.3 is possible. Dependent. Some other changes that happened. So Fedora 22 came out with, was it Fedora 22 or Fedora 21 where we did kernel core and kernel modules? I think it was 22, 21. All right. So we don't tend to change the packaging of the kernel very much, but in Fedora 21 we split into kernel core which is like, it's not like a pure true core but it's a much smaller package basically usable for most VM machines and then the rest of them are in kernel modules. This time around, I was talking with Harald Horrier and his team and he said, why are we shipping stuff in slash boot? We should only be shipping in slash user slash lib, whatever. And I said, well Harald, you have to have a kernel in slash boot so Grubb can grab it and boot. You can't not ship in slash boot. So after about two weeks of trying to figure out what he was trying to do, really what he wanted was all the packages in Fedora should only install in slash user essentially so that you can go in and do like the equivalent of a factory reset and all the other directories go back to whatever they were before you installed Fedora. Makes sense. So we worked out a way so that we shipped the actual kernel part that is normally in slash boot in the RPM package itself. It's in user lib modules and then the kernel version string. And then when we install the RPM, there's some scriptlets to go and copy it over to slash boot. So it's not that we're not installing a slash boot, it's just that we do it in a weird way so you can do factory resets. And his team finds that useful. Whether that'll boil out into something that you see in Fedora, I don't know. Probably not. Somebody asked me of the day if we'd see it in containers and I'm a kernel guy and I was like, oh yeah, maybe and then I realized you don't need a kernel in containers. So probably not, but I don't know. Harold and his team come up with some crazy stuff so maybe you'll have some use out of that. Best case you just don't even notice. Minor tweaks. So just moving some modules around inside the content of kernel core and kernel modules. Nothing really big. And since the question always comes up, we still do not recommend Plutter FS as the default file system for your machine. Every year we have a conversation in some form. I think we asked on the desktop list again this year somebody did about whether we want it for workstation. So I emailed Joseph. Joseph, how do you say his last name? Do you know? Anyway, Joseph is one of the main Plutter FS developers. He used to work for Red Hat. He moved on to Fusion I.O., which was like this flash-based data center type storage company and then he's now at Facebook. Yeah. So I emailed him and said, hey, this question's gonna come up. You're the primary developer. What do you recommend? And he said, not yet. And so what they're doing at Facebook is they're rolling it out kind of behind the scenes on their storage areas and trying to scale it to 40 terabytes across one file system, which is awesome. And I've seen the commits go in for that. There's fixing a lot of corner cases still, some data corruption bugs at scale, right? So I emailed him and I said, Joseph, that's awesome, but nobody in Fedora is gonna have a 40 terabyte laptop. I'm sorry, but that just doesn't, it's not the same kind of workload that we see. So he said, they are doing other things that will produce similar workloads to what you would normally do in a laptop, but they're not there yet. Once they do it in Facebook at scale, they'll shake out all the bugs and then it'll be maybe time. So another six months, which has typically been the answer ever he released for the past three or four releases. Do you know of the Sousa guys at 8.2 as I played? They would not repeat that on the internet, right? Yeah, Joseph pointed out that Sousa, so in open Sousa, it's just the ButterFS driver just like it is in Fedora. And I think they have defaulted to it now or it was just with their latest release. Well, in SLEZ, what they did is they turned off all the fancy stuff. So there's no like transparent compression, you can't do RAID, it's basically snapshots and that's it. They only ship it on the root partition basically, right? So user data is still on like EAC4 or Exifast or something, I don't know. But we can't do that in Fedora because we've already shipped the driver for however long it's been out there since Fedora 15 or 16. And if we turn off those features then people who are using them will be broke. So we can't kind of cheat and get there before it's ready. And the open Sousa guys have worked on it for a while. They're definitely fixing things. They've, I mean, I see posts from, you know, at Sousa.com quite often for fixing bugs. So I'm sure they're running into issues in finding them, which is excellent because usually it's the way around, right? Fedora tends to be the distro that finds all the bugs and gets them fixed upstream just because they're so bleeding itch. But yeah, he just said it's not ready yet. Whether it will be ready around Fedora 24 is a wait and see. But, you know, unless you're really kind of hankering for snapshots or some of the other high level features that are awesome about it, it's really kind of a, it's like, why do you need it? Right. Yeah, I mean, if you want snapshots and rollbacks, you can do it with device snapper. You can do it with thin provisioning and things like that. It's not as integrated into the OS as we would like or simple from a command line standpoint to do. But I've been told by Steven Gallagher that they are working on it with a project called StorageD or something like that. I mean, the problem with Fedora is not really, might be the integrated by one of the features. Right. It's not really integrated from the desktop either. That's partly because it's not bad. Yes. We are told not to use it, so it would be integrated. Right, yep, absolutely. And I totally see the value of integrating it so that you can do transparent rollbacks and make it really simple. And, you know, if you want to do like time slider stuff, that'd be cool. It's just, every time I ask the person that works on it for his job, he tells me it's not ready. So I'm not gonna be like, yes, let's ship it. It's just, it's not there. But there are some other shiny things that have landed in the kernel. OverlayFS. So for the longest time, what's it called? AUFS was the kind of like the out-of-tree union filing system that everybody carried, Fedora even carried it at one point a long time ago for their, it was either their live image, I think their live images used it, long, long time ago. And then, sorry, what? Docker. Yes, Docker has a back-end layer that uses OverlayFS now. Didn't Alex do that? Alex Larson? Yeah, Alex Larson kind of wrote that. And it works. It was merged in 4.0. There's bugs here and there, but it's kind of the only, you know, it's the only upstream union filing system that we have. So. And I thought I would switch from ButterFS to... Yep, yep, butter switched what it does on the back end from ButterFS snapshots and things like that to OverlayFS to use that instead. It does now. Didn't they fix it in, didn't they fix OverlayFS and XFS? We did. Yeah, I believe so. There were other things. So XFS and Docker itself didn't get along because of namespaces. Username spaces did not work with XFS, but that's also been fixed. As of like 3.19? Yeah. So XFS should work. If it doesn't work, I would humbly be surprised and I'll go look at that because it should. KdBus. Leonard likes to talk about KdBus a lot. So if you've seen any presentation he's given in the past year or so, it's been about KdBus for the most part. Essentially all it is, is an internal IPC mechanism that kind of replaces user space debus. And it's kind of contentious. They posted it for inclusion in 4.1 and it was immediately shot down. Lots of upstream kernel developers were not very happy with how it was posted and how it was kind of positioned for inclusion. So they took 4.2 off. They said, okay, we're gonna go work on some of the feedback we got. We're not gonna post it for inclusion in 4.2. Maybe they'll try again in 4.3. Harold again, asked me, what can we do to get KdBus into Fedora in a way that it's actually usable? We used to have, we still do, have a repository called kernel playground where we kind of pulled in things that weren't quite ready upstream or things that were maybe in staging that we weren't willing to enable Fedora properly. And we pulled KdBus in there. And it worked. You can install it, but it wasn't getting the test coverage that they kind of needed. So what we did is we made this deal where we pulling KdBus into the raw hide kernel and raw hide kernel only. So if you go and you boot Fedora 23 alpha, KdBus is not in there because we said we're not gonna ship something in a release branch before it's merged upstream. So it's only raw hide. Fortunately, you can use raw hide kernels on Fedora 23 just fine. So if you wanna play with it, just install the raw hide. And then boot with KdBus equals one on the command line, kernel command line. Probably wanna set SeLinux to permissive if you're gonna do anything other than just boot. If you actually wanna log in, probably need to do that. SeLinux is being worked on. It needs kernel level support for how KdBus works. And it isn't there yet, but they are working on it. Is it something that absolutely has to happen? No. But when KdBus is merged, it will enable a lot of different things that you can't do today just because of the way if you have your init process start and it's not also your main system bus and you start your system bus later, lots of weird stuff happens when you restart the bus. So by shoving it into the kernel, you can just enable a lot more things. And then live patching. So major and I were just talking about this before the talk started. Live patching is commonly known as case splice, which is an Oracle product at the moment. It's basically exactly what it sounds like. You compile this little tiny piece of code and you can fix a bug without having to reboot. Fedora does not enable it. And it's simply because the amount of time and energy to produce those tiny little patches when we rebase the kernel as frequently as we do is not worth it. It is enabled in that kernel playground repository. So if you wanna play with it, you can enable it or you can go use it and then build your own K-patch patches. But we don't, I mean we just don't ship it. So for things like RHEL, which has a really long lifecycle, short, long time between updates, it makes a lot more sense. Whether or not RHEL is gonna actually use K-patch and live patching, I don't know. I don't get paid on RHEL stuff. So they might, it makes sense there, but it does it in Fedora. So that's pretty much the highlights list. Trying to think if there was one more that I was just thinking about or not. I can't think of it at the moment. So questions? Anybody? This talk goes much better when people ask me hard questions or any questions. Yeah. So like I said, it was posted for 401 that was shot down immediately. There have been patches that have come in since then to address a lot of the feedback on. There's a guy by the name of Andy Ludomirski, Ludomirski. That does a lot of like low level x86 stuff, but he also was really interested in like security aspects of things. Oh, that's what it was. Okay, I'll answer your question and then go back. So he's been reviewing the code pretty heavily and he keeps coming up with ways to boot a machine with KDBus, run a simple client and bring lots of destruction to the machine. When you take your message buffers and you move them from user space memory to kernel space memory, it's a lot easier to starve things, apparently. So it turns out that there's some bugs that he's still finding. Whether or not it gets merged for 4.3, we'll see. The way that it was kind of positioned was this is kind of like binder. You know what binder is? It's like an Android IPC system. The way Greg Crow Hartman pulled binder in is he basically said, look, there is a massive number of users for this piece of code. It doesn't impact anything outside of itself, so we're gonna pull it in. And it had some objections to it. There was some high profile kernel developers that did not wanna see binder get merged, but Linus can actually be pretty pragmatic at times. So he said, fine, if Greg's gonna maintain it and he's willing to spend the effort to do that, fine. That's kind of how KDBus was positioned. It just, it didn't work when they tried that 4.1. So the 4.3 development window will open probably in about a month and then we'll see what happens then. I'm kind of optimistic that something similar will happen because they are kind of taking the feedback that Andy and others have provided, but I don't know. I mean, if anybody knew, they would be Greg and Linus and I don't, I mean, they're pretty transparent. I don't think they're, you know, privately emailing to get it in. So we'll see what happens. Any other questions before I go back to what I was thinking of? That's an excellent question. Will real-time kernel patches ever be available? In Fedora? No, probably not. There was a discussion a couple months ago on one of the development IRC channels about somebody who wanted to provide a real-time kernel and they asked me what they could do to get it in and I just said, no, sorry. Again, it's a cycles thing, right? We're having enough trouble making sure everybody's machines work without real-time aspects to them. I don't want to provide a real-time kernel and have people expect that this robot arm that's moving is not gonna crush somebody's skull, right? So that being said, after a lengthy discussion, I said, if you want to have a real-time kernel in Fedora and you want to provide it in a Copert, which is, so if you know what, yeah. So I said, you're more than welcome to do that, it's fine. I don't care, just as long as it doesn't conflict with the actual kernel proper. It's not that we're not interested in a real-time necessarily, it's that it's a very hard problem to solve. Red Hat has paid one of our engineers, we have a whole real-time team, we have a real-time product. We've paid the kernel engineers for quite a while now to get those patches worked on and upstream and they still aren't there. So it's not only a hard problem just in terms of cycles of support, but it's a hard problem just getting the patches upstream because some of them, you know, they've made very, very good progress and now that the patches they have left are the really, really kind of needy-gritty hard ones, and so we're just not willing to carry that in Fedora. It's just the number of users we would get is very small versus the amount of effort we would have to spend rebasing patches, right? Yeah, yeah, I mean it's not like you can add the patches and have one kernel provide real-time and not real-time, it doesn't work that way. So it's a good question. I hope somebody does eventually build that copper. I don't think it's been done yet. Yeah, they've always had a real-time kernel that's kind of been based on Fedora's, I think. And I don't, maybe it's successful, maybe it's not, I don't know, but I know it's big in the audio projects and stuff like that. Any other questions? Anybody want to rant at me for not fixing a bug that they buy out? That's it. All right, well, thank you for coming.