 All right, I guess we'll get started. I apologize for not having slides up because apparently it doesn't want to allow me to share slides here. It only wants to let me share a Chrome tab and have them in Chrome. I'm Justin Forbes for people who don't know who I am, Fedora Kernel Maintainer. I am pretty much the gatekeeper of the Fedora Kernel but I am not the only one responsible for the Fedora Kernel. We've got great contributions from people like Peter Robinson, Hans Geod, and others in the community which help keep the Fedora Kernel ticking as it should be. So today we're talking about what's new in the Fedora Kernel and some things have changed in process and how things are done. So I want to cover both the current affairs, what is Kernel Arc and why that matters, what's next for the Fedora Kernel and then how you can help. So current affairs, a big and important one and I'll type this in chat just so people can see it. It's Kernel Test Week or the 5.8 Kernel is essentially a week from Monday. It's August 25th or sorry, August 17th through 21st. 5.8 is important for a couple of reasons. We always have a test week before we rebase existing distributions to it. So we don't want to rebase Fedora 32 or 31 till after we've had a test week gotten some general feedback. And it's also important in this case because 5.8 is going to be the release kernel for Fedora 33. Fedora 33 because it hasn't branched yet, it kind of puts us in a weird state which means raw hide right now is actually not moving forward. It's sitting at 5.8 until Tuesday once the branch has happened then raw hide goes back to Linus's tree and 5.8 will be maintained inside of the Fedora 33 branch. Right now, obviously Fedora 31 and 32 are on 5.7 base kernels. They're 5.7.14 I believe is an updates testing now. That could always use some testing and feedback. And I wanted to put a brief note on kernel headers and I'm going to put this little URL in the chat. I'll try to get these slides online somewhere after. But for those of you who aren't aware, used to be that the Fedora kernel package would build the Fedora kernel package, the kernel headers package and the kernel tools package. It hasn't been that way for quite a while. The default for a long time though was to continue to build kernel headers and kernel tools with pretty much every release. And that's completely unnecessary. The reason it's unnecessary is because the only reason to build kernel headers is if the UAPI headers change. Any other headers can change, but the UAPI headers are the ones that matter because kernel headers is what's used to build kernel space packages. It's not what's used to build modules. People get confused frequently and complain that the kernel headers package is not in line with what the current kernel package is and they need to build their driver headers. So for those of you who want a more detailed explanation of that, let me see if I can grab this link better. I'll add a link in a little bit for the blog post that Laura ever did a while ago. Another thing that I wanted to discuss was the ELN kernel and what is the ELN kernel and why it matters. ELN is, there's actually a talk going on about it right now. Wait, can you hear me okay? Yeah, sorry, the no slides thing I mentioned at the beginning. So I have slides that I'm impressed but for some reason when I try to share apps it only lets me share Chrome tabs. So unfortunately I don't have them in a way to show the slides. The slides are not that important anyway. The only, they're really just bullet points, the URLs for them. There's a blog post from Laura that I wanted to redirect people to with the kernel headers information and otherwise it was just kind of bullet points. So the ELN kernel is essentially what's going to be the rel next kernel, right? It's the development for the next version of rel. One of the reasons it matters is because most packages when they build for ELN versus building for RAHI they build pretty much the exact same way. With the RAHI kernel though with the kernel when you build an ELN it's not Fedora and you're actually building with a rel kernel config, all of the extra stuff that we keep on them Fedora turned off, things of that nature. So why are we doing that? I'll go into a little bit more detail on that when I discussed kernel arc but the long and short of it is we get more eyes. I have one person dedicated to the Fedora kernel and 100% dedicated to the Fedora kernel. We have a very large number of people that are dedicated to the rel kernel and by building the ELN kernel off of RAHI as well, upstream, suddenly have a lot more people who are interested in RAHI. And trying to make sure that config options which maybe have a huge performance impact or something, we get to find that out because they're doing all of that testing right now against RAHI or against ELN. So we also have, if you look on the kernel mailing list you'll see there are patches that are going through that are config changes and those config changes are actually set to default values but the reason is the rel maintainers on that side will review them and say, yes, we want this or no, we don't and frequently here's why they have to be act before they're changed. So we get the data from those acts and acts as well and can help make more informed decisions in Fedora. Just another little blip before we go through the current affairs for people who are curious why you frequently see kernels that have a security update listed with them. There have been 124 CVEs filed against the kernel just this calendar year. Now, those CVEs are frequently absolutely not important for most users. They're things in small drivers. A lot of times they require root to crash a system which I don't know that should get a CVE to begin with. If root can crash a system purposefully they don't really need a kernel bug to do it but they are getting CVEs and they do get addressed. We try to address every single CVE that comes in. So that's why so many kernels will have a security notification with them as to whether or not it's really the ones that are really important will get listed as really important. Otherwise, look at the bugs that are associated with that kernel and see if they perhaps apply to your situation. So kernel arc is something I wanted to discuss a little bit more. Kernel arc is what we have already moved raw hide to and it is in GitLab and it is a source Git tree and not a disk Git tree. And the reason this is really important is because everything is set up for automated building of the disk it now. So here's the URL for kernel arc but all development both on the kernel patches themselves as well as the spec file configs and those things happen in GitLab in this exploded source Git tree and then scripts are run to automatically generate disk it. One of the reasons that's super important is because those scripts will actually overwrite anything that's in disk it at this time. So if something is changed in disk it and not changed in source it then it will likely be forgotten. Realistically, I do look at disk it to make sure that there hasn't been a commit between my last commit there to see if somebody added something and if that needs to be changed. But if you do want to change something and you try to do it in source Git it's just not the proper place to do that. Or in disk it is just not the proper place to do that and it really needs to be done in source Git so we don't forget about it. Right now the process and you'll see there's documentation on that side. If you go off of the docks page it takes you to the actual docks that explain kind of the process but it's very messy right now and that is one of the big pain points of this. Right now there are three trees that all interact with each other in different ways to get a release going and you have to do kernel patches against kernel code itself in the part patches tree. That branch is rebased frequently. So if you do a merge request there's a good chance that once it's rebased it messes up your merge request and then has to be fixed. OS build is for kernel config options and for documentation spec file changes all of the basic stuff there. One of the cool things about that tree is you can take that and let's say you wanna do some testing with Linux Next or with any other upstream tree you can actually just merge that tree in and you'll have access to all of the make this, get make source RPMs all of the things there that eventually lead to CI integration as well. And it's all isolated so there are no kernel patches in there other than one make file patch which should just merge in cleanly. That tree is kept up to date with Linux's tree as well but going forward at some point in the not too distant future we hope we will replace, OS build will exist but nothing will actually happen in OS build all future development will happen in a develop tree and that tree will be a merged forward tree. We'll still have a branch off thing to rebase and build this, get off off but there's a couple of interesting consequences for Fedora there. One is we won't be able to right now we have every single patch broken out as individual patches and I really like that. I think it's clean, it's easy to see what we're doing and we don't want that to go away but it's caused some massive problems. One of the problems is we do have to rebase our patches and our patches is a, yeah, as soon as it's rebased it was rebased a few days ago as merge window came in and suddenly all of Peter Robinson's merge request would no longer apply because it lost references there. So to get away from that and stick with a merge only tree what that means is we're gonna have to generate essentially a diff against Linux. So disk it will look like the upstream tar ball and what's there and then a Fedora patch. Or Red Hat patch whatever we decide to call it and just as one big patch ball. Now I still don't wanna lose all of the data that we have there. So we're gonna have a patch list file there that has every commit listed that's included in the patch ball and what it is and therefore you can look it up but they won't be listed as individual patch files. One of the reasons, so there's right now if you tried to make a tree with the stuff that merged but there's actually a make file change in OS build that comes out as a patch. And because it's not a current re-based patch it's a merged patch there was a change to make file and that patch doesn't apply anymore. It's a simple thing but we just rather get away from all of that and make the workflow a little bit easier for everybody going forward there. It is somewhat a shift from separate Fedora and rel-focused trees. It is actually kind of not it's not the existing Fedora or the rel workflow. I think rel is also trying to move to this workflow. And of course it's an advantage of everything is the same for everybody but it's not exactly their workflow either. It's much easier for people generally to work on exploded trees. You know, we've had for a long time there was a tree maintained at colonel.org by a script Josh Boyer wrote and it worked okay. A lot of times and then sometimes it was just broken for a long period of time. And by doing this, we kind of keep everybody on the same page and keep everything moving forward. So the improvements coming soon like I said, those scripts have been worked on. Obviously I don't like to make any huge changes during the merge window because there's just too much other stuff going on during the merge window. We get dozens sometimes that someone has over a hundred config options come in overnight and those things I'll get looked at. From the Fedora side, I look at every one of those every day before I do a colonel build. And just with the number of patches the amount of trend that goes on during the merge window it's just not a great time to make sweeping changes. So once everything kind of settles down a little bit it'll be the time to make those changes. Fedora test week is next week so it'll probably be after that or sorry a week after next it'll probably be after that. And another interesting thing about this workflow though with ARC and all the discussion, all of those things happen they can happen both in GitLab or the Fedora kernel mailing list there's a mail bridge between the two. So if you don't want to deal with mail all the time if you want to review patches and all of those things you can do all of that on GitLab and for the people who want to stick with an email-based workflow then that works for them as well because there's a mail bridge that generally works. It kind of needs to be fixed. So what's next? The next step really outside of Rawhide and getting those things moved is moving stable Fedora into that type of workflow as well. And the way that we do that is a little bit different. So when we instead of having this one merge tree that kind of goes on forever when say 5.9 comes out then we will create a branch for 5.9. There will not be a separate branch for Fedora 30, 31, 32 whatever because that's not really the way the kernel is maintained. So we'll have just a 5.9 and that 5.9 will suddenly be rebased and all of the patches will be clean on top of that and then it will follow the stable trees upstream as we grab things there. What that does mean is the scripts have to be created there that create a source RPM for each relevant version. So they can do the disk commit for each relevant version. But it's, I think Fedora 22 was the last time we had, hey, here's a series of patches that we cannot include in the older version because of upstream. And if we do have those we'll have a way to work around those, work around that so that we can deal with here's the tree, but oh yeah, when you build this source RPM we need to drop this. Another thing that needs to happen is cleaning up the kernel spec a little bit. If anybody's seen it, it's pretty messy. It's much nicer than it was several years ago but it's still could use some improvement. So that's one of the things that's kind of on the roadmap for the next six months probably. Hopefully it'll happen within that timeframe. And then we also need to update the kernel test app. The kernel test app people use to submit results and all of those things for individual kernel test or when then we have test week, test day. That is, it was written quite a while ago. It's using Fed message and not the new Fedora messaging. It's not using the new authentication. All of those things kind of need to be updated. So that needs to happen. So how can you help? And that's the cool thing about Fedora is we've got all this community and people like to pitch in and it's amazing. So we really do need your help. The easiest way, even if you're not a kernel developer, you don't feel like dealing with upstream. You don't feel like trying to fix bugs yourself. Kernel test week is a huge tool for helping determine whether or not a kernel is ready to roll out or what needs to be fixed before kernels ready to roll out. There we've had the last few, we've had 225 plus tests submitted. We would love to see that be even more. We would love to see people testing not just on a VM or but with real hardware because that's a huge factor on whether or not a kernel is really ready to go. There's so much hardware out there and I've got a number of systems in this room but not that many compared to what all is out there. So if you can test, boot up the kernel, run the test suite, there are honestly the results of the test suite aren't that important. If you run the default test suite, the simple easy default test suite that at least tells us that it's been run, even if you don't run it in root, it tells us it's been run that you successfully booted this kernel and you've booted this kernel on this hardware and I kind of assumed that if you've booted the kernel and run the test suite, then if you ran into an issue, you would probably file a bug or mention it on the test week wiki or mailing list or something like that. So it gives us an indication of these are the people, or we don't know who, but these are the number of people who have run tests and like I said, if we get varying hardware there, it kind of puts us in a really good position to know where we are. Another thing is upstream first. I know that people come to us every once in a while and say, hey, we want this feature implemented or that feature turned on and these things are out of tree. It's just not manageable to be pulling in a lot of other tree things. We would love to see everything upstream first. Actually, it's pretty much a requirement. So if there's something you really want to see in the kernel and it's not in upstream, try to get it upstream. Try to just offer some cheerleading testing, whatever it takes to help out the developer of that feature getting it upstream. That's a fairly important thing. We've had, there's the Ath 11K drivers, we've had some users now, I guess they're shipping hardware. And I just can't turn it on yet. 5.9 actually should have the Ath 11K drivers. So I'm hoping that it gets up there. And last, speak up if there's something you need for all of the config options, particularly in Fedora. When we decide to turn on a driver or turn off a driver, it is based on an educated guess, right? There are certain things we don't want, like we don't turn on staging. I'm not gonna turn on a driver that I don't think there's valid hardware out there for. I'm not gonna turn on drivers that are known to be unstable. But for other things, I kind of look at it and I'm like, oh, I'm not really sure if this chip has valid hardware out there or not. And I've tried to do a little bit of research and if I'm not seeing a lot, I might choose not to turn on that module. If you need that module turned on, request. If you send a mail to the email list or send a merge request to GitLab, those things tell us that, oh, hey, somebody needs this. Now the answer may be that, hey, we turned this off for a reason and here's why. If we can maybe find a way to get past this reason, we can turn it on or the answer may be, hey, I'll have it on in next build. But if we don't hear from you, if we don't know that that's something you want, I don't know that that needs, it's not going to happen. So I need to know that, hey, somebody out there needs this and then I can look at it and say, oh, yeah, happy to do that for you. That's pretty much everything. And I said, I'm sorry about this slide set up. So I'll get these slides. I'm sure it will be online and published, but they were pretty simple. The only other thing I will post in the chat real quick is the link for Laura's blog post about Colonel Headers for people who want more information there. That's there. Anybody have any questions or comments? If not, thank you for your time and have a good rest of your Nest experience.