 So, hi and welcome everybody to my talk about how to make the Linux kernel developers fix your kernel bug. It's a pretty broad idea here, which I'm claiming, but it won't work all the time, but in the end it's, the answer is simple, how to do that. It's just a decent report and most of the time that we'll do the trick. And now does it even report actually look like, so I made one up, this one is pretty decent, it's short and has everything in it, we'll take a closer look at it in the during the presentation. And the following made up report on the other hand, that's, that isn't good. And there's multiple issues. And the thing is, in the end it most likely would be ignored. So you might wonder now, why are back reports quick getting ignored here. The thing is, the developers are not obliged to fix each and every reported issue. That's just how it is. And simply because it's impossible, would be impossible to make it mandatory anyway, because the kernel is made by volunteers, volunteers as in hobbyists or people, developers on Windows that help volunteering. So yeah, that's why back reports are ignored and developers themselves, because developers themselves and their employees decide what to spend, what to spend their time on. The good mental picture to have in mind here is think of it like a playground that is built and maintained by the community that ensures everything is and stays safe. Just because you help building a play structure like five days ago or five years ago, doesn't mean you're now obliged to improve it further because somebody says, hey, that could be better or there's something we're working right here. And the same is true if you help building this playground for your employer a while ago. And so because maybe you're working for a different employer now and the employer isn't obliged to fix things either because maybe he lost interest in the playground because he can't make any more money from it, maybe by selling copy for the parents that look over their children while they're on the playground and if they stop selling copy to go, yeah, there's still interest for them anymore to help with the playground. The good thing, the thing you should keep in mind is think of a bug report like asking for a paper, a paper form of volunteer. And there's even more good news that sounds a bit strange, but the good news is those volunteers, the curly developers are really committed to help you with all sorts of issues. And that's why even reports with big flaws sometimes will do the trick. But the problem is they can only do that if their time and motivation commits and as most of you know, that's something that's limited for all of us. And yeah, that's become the problem because unfortunately more curly developers are really buried in work. They have long to do lists and lots of reports they have to work through and even get more bug reports every day. And yeah, and then the simple answer is you have to kick something off with to do list and then the reports with flaws are often the first thing that will be ignored. Yeah, and that's why every flaw in your report and increases the risk that your report will be ignored and why you should avoid flaws and that's why I show you actually how to do it by submitting a decent report. There are five important aspects here that I'll cover in the first act of this talk because those are the things that are most important and what most people get wrong here and this is the reason why many bug reports for the mediums kernel are ignored. One of them is for example, you should ensure that the kernel you're using to report your bug with it's actually vanilla. Vanilla means that it's built directly from the sources as distributed by kernel.org and not modified by anyone. There's a simple explanation for why it should be like that because yeah, that's the core of the kernel developers are working on. Some modifications obviously is not what they can support. So to explain it with the playground example, let me please stretch the analogy a little bit because the playground obviously is material but the kernel is immaterial so think of Linux more like a book to build a playground and with a 3D printer that does everything with a snap of a finger and actually even updates existing parts. And actually a playground is also not really the proper size for such an analogy, it's more like a big amusement park, even bigger than this one. This will, because that's really complex and yeah, that's what Linux is to be based. So now think of you, you have built a park with this book and run into trouble and you couldn't play into a friend who's one of the maintainers of that book and he checks the instructions unsuccessfully to find a guru of the error and after some while agrees to fly over across the continent for hours in the plane and doesn't really have time. But then he comes to your park and he sees that it's not built from the book he works on because he used a book that was published by people who actually modified things and are known to add stuff that don't match the kernel, that don't match the quality standards those people that the kernel developers are working on or those that build this Linus land park. So yeah, and that's something you wouldn't want to do to a plant, wouldn't you? And so yeah, in the Linux, that's also the same in the Linux case and that's actually an aspect that's especially important there as any modifications and enhancements in Linux code base can cause issues in other totally unrelated kernel areas. That's why if you're at a out of tree driver or something like that or modify in one area, suddenly your memory management might not work anymore. That happens actually. And the problem here is that most cons used to the wire that actually heavily modified and enhanced, for example, those in redhead enterprise linux or suzie enterprise linux or you want to. And yeah, that makes them pretty unsuitable for reporting bugs to the Linux kernel developer. You have to report problems of those to your vendors, to your vendor that provided you with the kernel. So actually your Linux is bushier, as those often won't help you with problems that are not that important. There's obviously something else you can do. You can install a vanilla kernel yourself instead, simply by compiling them one on your own or using a pre-built one that's often quite easy on the big distribution because there are a rep of the choice with them. So that's what you actually want to do. If you want to install a vanilla kernel when you report a problem to the kernel developer, to the Linux kernel developer, and when you actually write your report later, there's some important detail here. You really should focus on this vanilla kernel in your report, even if you pound it back with the distribution problem. Don't bring it up. It often, most of the time, just complicates the report really, really a lot. It makes it a lot harder to grasp. But yeah, that's one of the reasons why this report in the end might be ignored. So just ignore the distro kernel. It's not relevant to the apps in Linux kernel developers. And which brings me back to the main up report I showed earlier, like this one, that's that was the one that has some bad aspects. And as you can see, if you look close at the at the ticket, at the Baxilla ticket, you can see that the report is about an Ubuntu kernel. So yeah, that's one of the reasons why this report like it would have been ignored if it was an actual report. The other report, the good one I showed early actually immediately in the second sentence makes clear, hey, I did try this on a vanilla kernel. So I'm using the code you work on and everybody cares about and that avoids any doubt by those that we received the report. That was already the first point when it comes to writing a diesel report, which brings us actually to a next to the second one already, which is actually quite closely related. You should really use a fresh need of started here. Why? Because yeah, that's a code that can develop with care about. They don't care about something that they distributed a year ago or something because since then, many bugs were actually were fixed. And to explain it with the playground or the Linus land example again, say you have problems again and your friend again has to play over. And immediately when joining when coming to your park, see that you build a park from a from a book that is two or four years old. And it was never updated. And yeah, and then he is really annoyed because he flew over and he nevertheless looks close and then even finds out that that can contain the fact that your friend actually fixed himself like two years ago, something that he told me forgot about it because that's happened. The developers are fixing bugs and changing things all the time. That's so easy to even forget what you did four weeks ago. So that's why you really need to check with the latest current as you don't want to do that to your friends at the current level except that you want to help you here. So really test with the latest code base. And so what is it's in your personal interest also because that's the first place where all bugs are fixed. Every bug in the Minus is fixed in the in the latest code base first and then backport it to older and stable versions. And yeah, so how to actually find the latest version you want to use for testing that simple code.com.org and not big yellow box and look up at the table that's below us below that. And just there's the top of the table. There's a mainline line. That's the version you want to want to use at least 98% of the time because sometimes things will look like this like this. Then it's OK to use a stable ground. Then that's a case when a new mainline release was first out and the next one is prepared. Then a stable release of a higher version number is actually what you what you can use. Because the other one is still in the early stages might not be the best test at that point and but other times if it looks like this the next mainline release is already up to an RC seven or one RC one already. You don't want to use stable normally because the thing is stable corners have no bucks that will never be fixed. Because sometimes it's simply too risky to backport changes because they are too complex or it's entangled in some other changes. And that's why stable corners sometimes have bucks that will never be fixed. And for stable corners, that's not that much of a problem. That's why using a stable corner can be OK to report a box is not ideal. But it works. But for long term corner, that's really something something pretty annoying. So that's becomes problematic. So you really don't want to use a long term or LTS corners for reporting back. So you really want to check the code as the issue happens in the latest code base. If you don't believe me, actually, there was a famous there was currently developed by the extra for developer Theodore. So actually recently in a mail actually explained, yeah, it doesn't care about bucks and LTS cards or it doesn't care if you can't even build your own car and those those are the back rewards that will be ignored. So don't use the long term corners. But there's a small exception here. You have a regression within within the long term serious safe when switching from five to 15, 27 to 28, then it's OK. But that's a special case that we'll get to back later. So and just like in the first point I raised, it's the same here. Focus on on the freshest car and you test it in your report because mentioning other versions, the second version, most of the time will complicate the report unnecessarily unless you deal with a regression where something used to work on an earlier version. Then it's obviously finding uses to mention a second second version. But other times just focus on the latest version here, you'll test it. The bad bad bug report example I mentioned I showed earlier, for example, used to buy for 15 LTS car that by then was more than a year about a year old already. And so that's one of the reasons why this bug report actually is would be ignored. And the report actually tried a 5.19 kernel. So that makes the bug report already confusing. But the thing is the latest kernel at that point in time already was a 6.0 kernel. So it didn't even check the latest version. And yeah, that's a pretty bad idea. And one of the reasons why bug reports can easily get ignored. The good bug report I showed earlier on the island immediately makes clear that it's that the reporter used the latest RC release to test and doesn't mention any other kernel. So that's pretty basic, pretty straightforward, easy to understand for everyone that comes across this report. Yeah, that concludes the second important point and brings us to the third. You need to ensure your kernels and system integrity here. Why? Because, yeah, something local might be causing the problem you're facing. So let's get back to the amusement park. Linesland, say the water and rollercoaster stop somewhere along the track often. And you can't have no idea why and your friend either. And he flies again yet over and spots a mobile attraction in the corner of your park, which he allowed to come by each and every day and actually use some of the park's infrastructure. Yeah. And the thing is the friend highly suspects that the workers of the mobile attraction assess the water pipes and the power grid in irregular ways that are causing the the problems you see what the rollercoaster is there, but he's actually not allowed to look closer at the mobile attraction as the owners consider that secrets. And yeah, in the end, there's nothing your friend can do about it because the problem is not in the code or in the instructions they maintain. That's the same with the kernel. And the good thing is actually the kernel can detect a few things that make it that way it into a state where what bad to report bugs that is actually tainting itself. If there's some if something appears, that's really bad and you don't want to see in any bug reports. And if there's fine in the profile system actually contains a zero, then everything is fine. That's actually how it looks. It looks like there's something else in it, for example, the one or even higher number, then something is wrong until you should investigate what's causing what's causing this. The one actually is frequently caused by ambiguous own GPU drivers. But the thing is that it's the same for both GPU drivers. They ship these days. The license doesn't matter. They're open source driver that indeed this is also a problem. The tain number is different, but it's a problem because yeah, kernels without street drivers are not vanilla anymore. And that makes them unsuitable for reporting bugs to the kernel developers. Hence the install such drivers reboot and check into the issue is actually still present and yeah, but the thing is those drivers are not the only reasons why your kernel might be tainted. There are many other incidents that contain your kernel and most of them make it unsuitable for reporting bugs, but not all of them. The most famous one that makes that isn't the problems actually when the first oops, warning or something like that happens. So that does are the screens that look like this. And those cases, when it's when it's actually the first such problem that shows up, then it's something that's OK to report. If it's a second one already, then it's could be a follow up error. And that's already then the problem that isn't worth reporting, at least most of the time. That's actually why in such a stack trace, it's actually a small area where it's actually where the kernel actually is indicating if it's tainted. But the thing is what those letters mean here and what those numbers mean. I can't explain that all here because they will take too long. If the kernel is tainted, check the docs. There's actually a report and issues documented in Linux kernel. It has a section that's dedicated to this time flag and checking it and which actually explains what you need to do and what you can make can do to fix this and actually points you to a page, a different page for a script to decode the numbers of the letters. And then you can see what actually is the reason why your kernel is tainted. Yeah, so your bug reports, you also should make it clear that. Yeah, the kind of wasn't tainted, for example. And the bad bug report I showed earlier, the user is actually mentioning that he's using a g-pod card that actually, at that point of time, was only really usable by using and biggest proprietary drivers. So that's a pretty strong indicator that user actually uses a tainted kernel might be a reason enough to get a bug report ignored already. The good reports on the other hand actually make it clear also just by pretty mentioning it at one point that the kernel was untainted. And yeah, that makes the kernel developers already quite happy. There are a few other things that the kernel can't detect, but that could could could lead to trouble. And that's you should detect those in your own interest, because maybe the bug is caught by them. For example, is the hardware still working reliably and specified? And is your current environment well? You could, for example, test the memory. You should stop overclocking if you have something a stack trace that indicates something with a file system is wrong. You want to ever check the volumes and things like that. The good indicator here, what you also should check is the output from t-message means age and look out for everything red or bold. As you can see in this picture, there are often indicators for what might be wrong. And, for example, a firmware file or something is missing and also gives you will give you some error message that is helpful to search on the Internet with to check if the issue was already reported of this work around so that can save everybody a lot of time. So that was the third point and which brings us to the next one, which actually is a bit quite a sad state because there are things are really unnecessarily complicated. You need to check where you need to submit your report to because there's no single place you have to check what's the right place because developers can't have their eyes everywhere. So I'm overly stretching the analogy here. Just assume you're reporting a problem to some web forum of your friend. You and your friend used to do a use to use five years ago. And when you report a bug where he doesn't react, yeah, maybe he doesn't use that forum anymore. And yeah, that's your fault and not his fault if he then doesn't help you. Yeah, but as I said, for Linux finding the right place, sadly, it's a bit hard because it's on depends on the subsystem. We actually suspect that the issue originates. For example, if it's a graphics driver, it's that's that's a subsystem for the graphics driver. The thing is, there's this buxilla.con.org, which might look like the central bug tracker, but it's actually not. That's something you really should keep in mind, which you actually also notice when you follow the link on the front page because it leads you to the reporting YouTube document I already mentioned and which actually tells you that using buxilla is most of the time a bad idea because many of the bugs file they ignore because they are never reaching the developers that should handle them. Explaining that is a bit complicated. Just believe me how it is. There's an LWM that article about it from one month ago that explains some of it. Yeah, but in the end, the short story is most of the time is it's the wrong place. And finding the right place is actually explained there. The right place is actually explained in the maintenance file of the kernel. This is actually linked there. And it has a lot of entries, more than 2000 these days that look like that. And yeah, you can also indicate by looking at the screenshot, most of the maintainers actually prefer to to get bug reports by email. Most of the maintenance and substance and do that with a mailing list in CC. And as you can see, in the case of the butter files, ButterFest file system, these three people, you should mail these three people and CC the ButterFest mailing list if you have a problem and they should handle handling a problem or delegate to other people. Occasionally, the maintenance file will point you to some some bug tracker actually because some subsystems actually use them. One of them, for example, is the maintenance for the graphics drivers for Intel chips or the new driver. They actually have a github instance and they those developers wanted to file bugs there, but that's only a very few number of subsystems that do it. And they're actually also quite a few small number of subsystems that want you to file bugs or allow you to file bugs in Baxilla.org. That's mainly ACPI, PCI and power management. And then the maintainers file will point at them like this. But even there, you can email the maintainers. That's totally OK. Yeah. But for other subsystems, you can file bugs in Baxilla, but most of the time those bugs never reach the actual developers. And yeah, that's in the end why they will be ignored. So that you want to avoid that because that's not what you want. That's actually what the make up report I showed earlier. It's obviously from Baxilla.org. And it's actually filed against the network wireless Intel drivers. And yeah, those developers don't want bugs filed in Baxilla.com.org. Sometimes they see bug reports there, but that also doesn't mean that they are reacting there because there's no guarantee. Yeah. And what you should have done is shown in the right example. It's simply mailed to the maintainers of the driver in question. And that's what gets you to the best results. Because development is done by email. And that's the usual interface people have to deal with them. And that's actually what works. That actually brings us to the last point that's important. You should really depict your problem adequately because, yeah, developers like everybody values that time and their attention span is limited also. So assuming a friend shows you a report from a common friend from school days, that's totally confusing, multiple pages long. You read the subject in the paragraph and don't get an idea what it's about. The whole text is confusing, full of unnecessary and distracting details. That's five attachments and 20 links. And it's also written in an unfriendly, demanding or in a bearish way. Yeah. If you then you would also suggest to your friend, OK, if you get such bug reports, I would also ignore them. Because, yeah, there's so much more you can do in a day. If that guy isn't nice and don't help him, you don't have to. The thing is how to write a good or a good report is worth its own quite long talk. And I would cover just a few of the things here. Try the one thing that's really important, try to keep the picture short. Either a short is possible, but with all the relevant details. The bug report I showed here, the main app report actually does it basically in one one and a half paragraph. You could say it contains every important thing. And if the problem is really that complicated, that there was more detailed description, detailed description, write it and put a TLVR on top of it that sums everything up in one paragraph. Because the first paragraph would be the way is the thing that most people read and to decide that the rest of the mail is actually worth looking close at. And so you really want to grab the attention of those readers there. You should also take some time to find a good and descriptive subject, because that's the only thing people will read. This is here, again, the example from the good bug report. And state of things that I mentioned earlier already. So if the colon is vanilla, it's changed out of this version. You also mentioned that this row and one relevant all details about the hardware as well, for example, this made up bug report actually links to the to the report of the router and question that does work on the Wi-Fi that actually will help to be relevant in this case. Yeah. You could also you should also upload a link, all relevant logs and details. You can even also attach them. But sometimes the mail is when it comes to emails, the mail gets long and big. So uploading a link is that what most of the time is the thing you want to use. What's relevant depends on the situation. The outcome of the message, I mean, minus and almost always is relevant. The current conflict is often to what else is important. Depends on the situation. So sometimes Alice USB, Alice SCSI, what we need. But not to overload the report. If something is missing, developers will ask for it if you manage to grab the attention. So how it can flow. Yeah, here I already showed that the message of the current conflict will be uploaded by the user and developers can download there. Once you get to the report, compiled we will do the subject and first paragraph again, because getting those right is really crucial because that's the only thing many people will look at and review the tone too. Because that's that's really important when you get these emails that they will oppose. How is the tone? So think of it like asking a volunteer for a favor. Volunteers that are known to be stressed and short on time, but they nevertheless want to motivate them to spend their time on your house. So choose a tone that makes that possible because together with the information provided by doing your part of the job, that makes attractive and easy for the volunteers to help you. That actually concludes the first act of this talk. And are there any questions already, Sean? There are a couple. I am fielding them. They are general questions. So I have been able to fill them. So there is one question, though. You might be able to answer how to attract attention to the issue in a proper manner. Answers, like always, life depends, certainly. Sorry. Normally, if you what you should try is compile a report and show it to somebody who's not involved, but might understand what it's about and try to check if that person understands what it is about or if it's if the report and the description of the problem is actually right, because if it works for that person, it likely will work for the developers also. And yeah, the important thing here is also really mailing or using the right path to reporters because, yeah, most of the time it will be e-mail and they are getting the first paragraph and the subject right and the recipient. So sending the mayor to the right people, that's really important here. So if I may, I can add a couple of things here. Think of it this way that if you took your car to a mechanic and you're trying to describe a problem that some rattle, so you'll have to describe when you are seeing it, what's happening. You can start the car. So think like a developer that somebody else is reporting a problem to you, what information you would want to be able to diagnose it. So that's the one analogy I can think of a car. The other thing you want to do in a report is gather all the information that they might need. D-message. Tosman already mentioned that. And then configuration, kernel configuration is very important. Figure out what kernel configuration. You can see it in the boot slash boot. Usually you'll see the configurations there. So highlight that which configuration architecture you are running on. Sometimes you could be sending a report that might be a generic, but a generic meaning memory management or file system, but it could be architecture dependent in which case it could be ARM or Intel or AMD. So it could be dependent on the architecture. That's what I would say. D-message, kernel log and all of the information that you would want. And any runtime things you might be doing at that time, what are you running? A specific application. Maybe graphics driver, are you doing it playing a video? Or is that triggering the problems? So all of that or something. Yeah. Yes. So sorry. Yeah, I really like the car example. That's a good good explanation because you have to somebody else that's not sitting next to you need to understand and be able. What is this about and you have to provide some hints how to find it. Actually, yeah. So let me move on to the second act, which is actually a lot shorter, which is actually influenced by a short interlude because yeah, are all issues equal? I'll bring you back to the playground example. For example, assume something starts to rot and becomes dangerous. Is it that's obviously something that needs to be handled. So they are ignoring the back would actually be bad. And what actually happens if somebody multiplies a play structure and the kids hate it afterwards? That's those are things that are happening with the current also. And that's actually what the kind of issue you have also matters somewhat when you're reporting the back. So because there are definitely issues that developers shouldn't ignore because they are obliged to address. And obviously one of them goes to security vulnerabilities. I'm not going down to that because that doesn't happen every day. If you have one of those check the documentation, there's actually a page dedicated with that explains it. There's also devastating bugs that are bad for Linux fame. Like something really bad like data is lost or damaged or even hardware is picked and those are really luckily rare. But yeah, they happen if you face one of those, make it obvious in your report, don't delay your report and maybe even CC Linus to make sure it gets the proper attention because sometimes it's needed. He will know who to send the report to if you didn't pick the right persons. Then also regression regression is actually what something breaks when updating a kernel, say from 5.15 to 5.16. That is not allowed in Linux because the Linux one has a we don't cause regression to rule is actually the first and maybe the only real hard rule of the Linux currently developed development is actually coined and enforced by Linus. He wants to take the peer out of updating. And the thing is, there are some fine print involved here and when it comes to this rule, you have to, for example, use a kernel with similar configuration. When updating those fine print is actually also explained in the dedicated document I added to the Linux kernel documentation a year ago or something actually linked here and that's mentioning everything important. You want to know if you face a regression or what you should additionally do to make sure the regression is handled appropriately and more about this hopefully in another mentorship sessions where I'll explain that in more detail because the regression packing is actually what I'm working on. At that point, you might wonder, yeah, if developers are allowed to know bugs, who handles these bugs? It depends on often it will be the author of the code or, for example, if it's a regression and it's known what change is causing it and the author of that change is actually responsible for fixing it. But there's a fallback, the Linux kernel maintainers that take care of those subsystems they have to step in and if they don't do it, Linux themselves will actually take care of the bug or find somebody that will do us. Occasionally, he looks really, sometimes he has a bug report, he fixes himself because somebody else doesn't do. But yeah, they can't handle everything. And that's why they're also issued that are totally ignored or most of the time ignored, which, for example, can happen due to known deficits because there are many, many incomplete drivers in the kernel because there's no volunteer will be enough time or wanting motivation to improve things or lack of documentation of a real-world issue prevents improvements to the driver. So, yeah, that's why the Linux kernel has incomplete drivers that don't support everything, the hardware offers and that's why they want to check the Internet when writing a bug report for known deficits because of sending a wasting time on a report that won't change anything because it's a known known deficit is not worth it for you. If in a doubt, simply send a quick, this is known inquiry to the developers in question before writing a proper non-linked report, that's not a problem. Besides those known deficits, there's another reason why bugs sometimes get ignored. There is code in the Linux kernel that doesn't have an active maintainer. The code, nevertheless, because remains in the kernel because it's too useful for people and removing it would cause a regression. So it stays unless it becomes a problem for one reason or another. But you can see that in a maintainer's fire, then if there's status is odd fixes, then it's nearly often there often. There's just somebody who keeps an eye on things and sending a report. In that case, definitely is a good idea. Because of a bit of luck, that person will look into it and fix it or at least try to help you to fix your to help yourself or something like that. Yeah. And then there's also code in Linux is totally often. That's also a perspective line of the maintainer's fire where it explains. But even then sending a quick brief report that outlines a problem is often worth it because maybe you find others that are affected and you can help each other to find a solution or maybe one of you was actually a developer and can submit a fix to actually fix the problem. Yeah. Besides these two extremes, they're actually a wobbly, wobbly area, a big wobbly, wobbly area of other issues. And yeah, what matters there is simply quickly explain actually the quality of your of your report and the most important things about that we already covered. But what we didn't actually do there when we covered that is explain how to actually approach the thing and there to explain that a bit, let's do a quick actually explains how to ideally handle the back report from start to finish. That's actually something that's in the current documentation also. There's this reporting issues document I mentioned a few times already is it explains what you need to do step by step and actually try to catch local problems early to make sure you don't waste time on compiling your own Carl and then notice, hey, I'm having a set up that is not worth reporting anyway, because Carl is tainted and I need to be the driver and can't uninstall it or uninstall it or something like that. And the step by step guide actually is compared by a reference section that explains each of the steps in more detail. And we can quickly do this goes through this list of things you do during the step by step guide to give you a brief impression of what you actually need to do and the first 11 steps actually just all preparations. Well, the first one, for example, is just checking if you have a considerable reporting box or willing to install one just to keep those users immediately out that are not willing to install a vanilla car because that's what they need to do in the end anyway. Search by existing bugs to join that's something that you obviously need to do all the time when reporting bugs. That's not different for the common. And that's why the five steps in the beginning I focused on that didn't go into that detail and what you should do there is simply using a paper search engine to check for existing bugs and also check the lot of kernel.org which shows many mailing list archives for Linux kernel mailing list. There are dozens of them and try to check there if some if you find a message where your bug is destroyed and you can obviously also search but similar kernel.org even if many bugs are not reported there. Maybe there's something you find there or even the good web instance in case you have a graphics driver programs. As usual, when searching for existing bugs, remember to bury your search terms a few times. That's really something that helps a lot and something a lot of people neglect. But they make their life unnecessarily hard. The search step is actually classifying the kind of problem. We already covered that because if you have a regression and you can demand that it's getting fixed or if it's something that won't be fixed in the end anyway, then you won't know here at this point, which is in your own interest. You should also check your setup if your setup might be causing the problem. Regarding the tainted stuff, we already covered that and creating a backup before piddly but to sense them obviously is always a good idea. That's also why the step by step mentioned it and that's actually right in front of the next step because you need to ensure you're not using any externally developed modules when we're reporting bugs. So you should remove any drivers that need it. For example, you're a media driver at that point. And then restart and then you should, for example, check again if the kernel is tainted before the issue appears. For reasons we already I already explained in more detail earlier. Now it's got time to write down how to reproduce the issue. And so you know what how to actually check with other calls if the issue is actually happening there. And also a good point to check if it's actually a regression in a stable series, that's why checking earlier if it's a regression is a good idea. Because if it is, you can use a stock card that's actually linked in the reporting issue document because if it's a regression, you can can report it briefly and that's often good enough and that will do the trick there. The details are in the document there. The 10th step is actually locating the driver subsystem that seems to cause the issue using the maintenance file, as explained earlier, because you need that again to check the archives of that place for existing reports and that's actually the last step of the preparation. Now you're getting to the to the part where you actually do the real real work is like testing and reporting. And that starts with making sure you use or actually install the latest vanilla currently mainly as explained earlier. Then you should check again with this call once you put it is it does it take itself because maybe that's kind of it's different in that regard and so you really should check that this at that point again because now it really matters and then reproduce the issue with this kind of check if it was fixed in between there. And if it wasn't fixed in between, you should now optimize the depiction of the issue of how to work with producers because that serves as a basis for your report later. And if you deal with a stack trace, you should really consider decoding it. That's not that important. The spectrum is about things and in an oops or warning or a chronic panic. It actually shows how in which line of code the error actually happened and how it was called. But if you don't decode it, things look like this with numbers, with hex numbers, with addresses there that don't help that much. And by decoding them, you get more information about that's nice to have, really would be good if if you're regularly reporting bugs. If you only occasionally report bugs, it's OK to say, hey, I can't decode it, but I didn't do it and just offer to do it later if it's needed. Because if you're not used to these things, it's something that takes a little bit of time to figure out. It's the same with the regressions. For regressions, you should actually specify when the regressions started, so to specify which version was not affected and which version was the first one where the problem actually occurred. You should narrow down this range as much as possible, which actually means you should use the git bisect to find the change that is called in the problem. But that's also some work. So that's also something you can say, hey, I have this this problem. Is this known? And if not, I can do a bisection simply reporting that your issue like that is totally fine. But if the problem is not known, you need to do the bisection because that's often the only way to find out more about this. Hopefully in another mentorship session in a few months, maybe. And now you're actually at the point where you can compile and submit to the report as outlined earlier. But the thing is that is not the end of the process because now you need to wait for reactions and keep the ball rolling because that's when the fun starts. You maybe could say because, yeah, things developers might ask questions or want you to test something to narrow it down where the problem actually happens. And you should try to answer those questions publicly and in a timely manner as soon as possible, because that's actually needed to to to track down the problem often because developers can't check each kind of reproduces each and everybody in their labs, they need your help there. And while doing that in that place, you also should keep in mind that the developers are overloaded with work and sometimes go on vacation and to report that didn't get a reply within two or three weeks. It's like the data and for government already. It's certainly a pain to prod it there then in a kind and friendly way. But before doing so, you really should check if there was the report you said was really given because maybe it wasn't or has some flaws. And that was a reason why what they thought. So that's a good point to recheck that if you need to product. Also, if new kind of versions were released in between, you might want to check if the issue was actually fixed because maybe it was fixed and you were not told about the things that happens all the time because there are so many different errors or bugs happen, sometimes somebody fixes something without being aware of any reports about the problem and retesting with new kinds is especially important that an RC one one mainline release actually came out in between because a lot of changes to get in there. And that's every time one such an RC one mainline release gets out. You really want to check if the issue is still present there and set a quick status update to your report and say, hey, in the latest mainline card, the bug is gone or still present. And because, yeah, that would keep keep the bug in and present for the for the developers to make sure they have an honor to do this. And yeah, also, if you don't get any help of unsatisfying this way, try to help yourself, the reporting issues document actually has a few tips there because there are so many people in the Venus community. It's also possible to find help by help helping each other and not relying on the on the on the developers too much there. That's it. If you want to meet more details on any of that, really check out this reporting issues document. It has a lot of details on all the steps if you need them. But that's the same time is hopefully easy to read because a step by step guide is really short and helps you to avoid any problem. So it's really written in your own interest, which brings me to the end of this talk, actually, to the conclusion. So let's sum things up a little bit. The most important thing is really the mindset. Think of the bug report like asking for a paper, a paper form of volunteer that you want to help you. That's really if you have that in mind, you're not dealing with you didn't buy anything and now can can demand that it's getting fixed because you didn't get what you promised for your money. Because the Linux is done by volunteers. So you really have to do your share of the job and make it easy for others to help you. And then most of the time, if you're kind, it will be helped because the developers have really a big strong interest in making everything work. Yeah. And that's why a decent report is so important. And also in your own interest to get it fixed. Yeah. Reporting issues document in the car that explains how to do that. Among others, yeah, tells you how to submit the report to the appropriate place, which as I said, that's something a lot of people get wrong. When they fire the box in Baxila, the front org, and they get ignored there. That's really sad to see. Hopefully the situation there will improve soon. There are some ideas to to to make it work better. Another thing you also should it's also important when you're when you're compiling your report is that it should cover all aspects that are important for developers, but are still easy and quickly to grasp for everyone because, yeah, everything else just makes it hard for them and that's the risk that they are getting ignored. And yeah, if you have and if you deal with a severe issue or a regression, you really should make it obvious in a report, for example, but mentioning it in the subject or with tech or something. But the most important things are really you should avoid any red flags by doing the following. You should really test and report of a car that's really fresh. So mainline, untainted and vanilla. And mention in your reports that you did so because that's what most people do wrong and that's that those are the more the reasons why many public parts that we are not, which actually brings me to the end and questions. I hope you have questions. I hope I didn't talk you into the ground and I hope my voice still works. Yeah. Thank you. We have one question in the question and answer. When it when it can be reasonable to close existing report and open new one instead? So is it a report you submitted yourself or somebody else? So that's related to some other thing. So assume there's a report by somebody else that was ignored and it looks to be about the same problem. Then it might be a good idea to actually submit a totally new report and just pointing to the old report mentioning that it looks related, but might or might not be. Because that gives you a fresh start and that often helps everybody to avoid confusion and sometimes bugs that look the same actually different bugs. So that's the case here. You also don't get sidetracked or and two different bugs discussed in one third or one bug sale ticket. That's totally confusing for everybody and that's really hard. If it and my risk that the bug is not getting fixed in the end, if it's a report you sent yourself, it's actually and it was ignored for like one or two months. It's actually or if it became complicated because others joined in and through other information in there and nobody can understand anything anymore, then it might also be a good idea to give it a fresh start, but what you really should, what's really important is linking to the old bug and mentioning, hey, we discussed this there already. Don't have any more questions in there. Could have talked for longer. Try to stay within the 45 minutes. But to talk about regressions of a similar was anyone. Back story of why but similar tickets I know of anybody who wonders. Yeah, it's simply happened there were a few people that thought having a bug tracker was was a good idea like maybe 20 years ago. But some of the current developers actually didn't want to participate, and that's why some some middlemen were actually meant to help with looking at the box and forwarding it to the real maintainers that didn't want to bother with with a web-based bug tracker. And yeah, the thing is that those volunteers, those middle in between people that never really work on the scale and the people that set up Godzilla also moved on. And many of the current developers stayed away and didn't want to deal with it. And but at the same time, some some such system actually started using Godzilla, and that's why it couldn't be shut down at that point. And yeah, that's actually why why Godzilla is still existing and still maintained to make sure they can use it. And as I said, there's a plan to hopefully improve the situation. There's some changes to Godzilla and hopefully get more traction into the idea. And then hopefully bugs will be forwarded to the maintenance again in the near future. I have been following when you were as appropriate, I have been putting links in Boston, LWN article. I also put the link in the chat. One more thing I can I thought about is that you you you covered this, I think, as a maintainer from my perspective, if somebody reports a problem and then offers to stay engaged during the fixing process and offers to test, especially if it is if it is something that I do not have hardware for that motivates maintainers, for sure, and maintainers and authors, developers, for sure. Because then they go, OK, whatever we are fixing, there is somebody that's offering to test it in their environment and telling us that it works. So stay staying engaged to the process and making efforts to help with the bug fixing process is helpful. Yeah, that's what I meant. And five point nineteen, really, submitting the report is not when it's finished. That's actually when the fund starts you doing these things and helping with the fixing process was by showing that you're engaged and everything. That's really important. That's what most people forget there. They simply think submitting the bug report is enough and then the developers will handle it and then they don't show up anymore. There's also a recipe to get your bug report. Not really not, but if you don't do anything when people are when developers ask questions, yeah, then the bug will be and won't likely be fixed because somebody has to test the fixes, as you say, yeah. And then what's in it for you, really, you will get reported by and reported and tested by tags in the kernel. So there is something for reporting bugs that you are getting in back. I don't know if that you mentioned that, but it's an incentive for a lot of people to get your name into the product and to get the bug fixed. That's also something you want. Yeah. So any other questions from anybody about any of the any of the information presented here or anything else that you want to know about the kernel development process in general, and then also bug fixing process and stable kernels. Maybe anything else that you want to know. Thaustin, you are coming back to talk about regression specific in a couple of months. So yeah. And and your work is also volunteer work, if I understand correctly. So we're all a lot of us are volunteers here doing the work. Yeah, it's a mix. For example, this reporting issues document in the kernel, I actually wrote that one two years ago during a sabbatical. So I did it all in my free time. Didn't get a dollar for it. These days, regression checking, I get some support there and that also helps a lot. But things like these are related to that. But yeah, but in the end, I'm here on my own time. And because I think teaching people how to to to proper to how to report bugs, probably is something that's needed because I see bad bug reports all the time that get ignored. And if we really want to to get rid of regressions also and all the bugs that show up and then we need to enable more people to to report it, it will will never be perfect. And there will always be flaws in reports, but that's not another problem. So the developers are humans on the other side. They can live with a floor or two, at least if it's not something really, really bad, like using a long term kernel or something or a tainted kernel of the NVIDIA drivers, then your bug reports are likely ignored. But if you forget something in the description, that's that's not a problem that developer will simply ask, can you send me this and that? And then things will work out in the end if you stay engaged as as reporter. There is one question in the chat. Somebody, Anup Parik, is wanting to know how to get involved with the bugzilla and the process that was just described. How do I get started? Anup, I think what would help us would be in what capacity are you looking to get engaged in the bugzilla process? Person, you can go ahead and answer your question. If you need more information, Anup can provide. Yeah, the bugzilla situation. In the end, what bugzilla needs is really people that know a lot about the kernel. They're also willing to look at bug reports and check check if they have any of the flaws I mentioned and actually tell users to improve their reports and when they are really good reports and forwarding them to the right people, because that needs knowledge about the kernel. So you and those people are likely hard to find. But maybe we need to teach them. But the thing is, there's some infrastructure changes planned by Konstantin, who's the admin for kernel.org and bugzilla and his team. He tries to do some changes to keep the subsystem list and sync and these things. I don't know how far he got with that, something that will likely happen over the next few months. And I guess at that point he or somebody else will send out a request for help by people that want to help help others here with writing good bug reports and making the kernel better. So in other words, I think support, looking at the playing the role of a support engineer in some ways, right? Looking at the bug reports and say if they are filed correctly and try to get the information from the reporter and then making the right connections. Yeah, exactly. Also checking if everything is there, something is missing. That's really important. Kind of first level support that checks for obvious problems before handing the problem over to the second level support of two people of more knowledge. Don't be shy. Ask questions. Yes. My voice still is working. Yes. Thank you. Thank you, Thasur. This effort that you started is this has been immensely helpful for the kernel community. That's one of the reasons I have I wanted you to come in and talk about your this particular topic so that we can make it widely available even to new developers as they are aspiring developers and then as they can. Yeah, as they are getting involved. It's yeah. It's something even experienced current developers sometimes get things wrong, wrong. Even they sometimes report back in back to that kind of that organ. They get ignored there or and because even some kind of developers are not not aware that some parts of report are not there. The thing is, if somebody is interested in this topic and wants to help maybe looking at the reporting issues documented in the kernel and looking at it with fresh eyes and and giving me feedback how to improve it and make things even easier for people to to to understand that would be something that is also something somebody could work on to to make this even even better. Somebody who is interested in writing a script to actually submit back reports that actually codifies some of the things I outlined in this talk. So if you could, for example, what whatever in mind is some some some small web app or Python application that helps you compile a report that actually checks for things I mentioned like is the content it is the latest version and that actually guides you through the through creating a good report. That's also something that that would be great to have if somebody looks for for a fun project to write. That is a great idea. Gathering up, it could even be a shell script gathering up all the information you need from a shell script. But I think it's it's a little bit more complicated. Some some more guidance might be good, but it could be a shell script. I think of something that works locally and on the web easily because otherwise you have that there will always be people that want to do it on a common line or with a query and there are always people that want to do it on the web. And so you have only one code base to for these two things are that's at least what I think would be helpful here. Right, that's a good idea. So if anybody is looking for a project to do a fun project to do in the audience, hey, you have an idea there. Any other questions? Everybody is reporting is reading reported issues in the current sources because it could answer nearly everything. That's also why it's quite long. Looks a bit scary, but I really try to write it to make it not scary. I put all those links in the chat. So yeah, you can review that and admin guide under documentation. Admit guide guide is where you will see a bunch of that information. And there are scripts. I think there is a kernel trained script under tools, debugging, I think that that can be done. That will decode the field to explain you explain why the content is needed, because it actually can be multiple issues. The script helps there. Pretty scary, scary. Best code, actually. And then there are the decode stack trace is under main scripts under the main directory. Yeah, I always wondered why kernel check check taint is under debugging, because probably belonged more under the main scripts, but looks like that's where it is right now. Yeah, scripts is a directory where scripts for developers and scripts for users are mixed up. Maybe that script should actually be moved to the tools directory. So the distributions actually install that. Oh, yes. OK. So you mean that even the decoding stack trace could also move under tools? Maybe. I mean, for the decoding, you need to debug in for symbols but you don't need the sources anymore. So so the script should be available there for regular users as well. And maybe having it in the scripts is not the best idea. But in the end, it doesn't matter that people that get that far are likely able to handle that. And somebody was asking earlier, how can I tell if I have a kernel that has gone through modifications? I think the taint does help. Your thoughts on that, Tostan, I answered it, saying that you could run your name to get the build date and sources and so on. But maybe you can add your thoughts. Basically, if the documentation for the kernel doesn't say that it's vanilla, it most likely was modified and enhanced. There are a few kernels, especially those kinds of distributions. And most of the time are modified, but there are few, few gray areas there. For example, there are Arch Linux Fedora and open source and Tumble with kernels. They follow the latest stable series quite closely and they don't modify their kernels much. So with those, you actually you can't see that they are vanilla and they are strictly speaking also are not vanilla, but they might nevertheless, for many bugs, are good enough. The problem of those kernels is actually that those are the stable kernels and you normally want to install vanilla kernel, the latest mainline kernel, at least, ideally. And so maybe you should look out for a representative that actually ships the latest vanilla mainline kernel. And I'm pretty sure there is one for Arch Linux, for Fedora. I'm maintaining such a representative myself. And for open source, there's also something in the open build service where you always can easily install RPMs or packages with the latest mainline kernel in a vanilla form, which actually most of the time they have actually vanilla somewhere that name to indicate that they are vanilla. And those are good enough to report a bug. That's totally fine. Just be aware that sometimes later, a fantastic might need to compile a kernel anyway. To do that, I actually have documentation written up and not yet submitted that hopefully will help users in that regard as well. To handle that task better in the future. Any other question? It's not that hard. It sounds hard, but it's if you don't go the complicated way, it's relatively easy for people that know the way around Linux and the common line. Yeah, it is rather easy to build the kernel and then running your latest kernel. I always run, I'm running RCA right now on this laptop that I'm running on Zoom, so. And they are not dangerous. I'll ask you sometimes, sometimes every two or three years, sometimes as a not that nice bug, but most of the time they are really as stable as stable. So sometimes I think they are more stable because that's what all the kernel developers use and that's where all the testing goes. So sometimes I have the impression that those mainline RC kernels are more stable than the latest stable kernels. That is correct. Yeah, I that's that's why I don't run any distro kernels. Usually I am always running on my primary laptop. I don't know I'm running RCA 10. Yeah, I did the same. And then also you kind of help out you sometimes what happens is something that might show up as a problem for you. You are on top of finding it quickly and getting that fixed in the release that is currently being integrated and paid attention to. So there are benefits to the users as well by staying with the mainline. And you are participating in the process as a result. And I mean, faster and I am I are vested because we have part of the community, but even for others. So that would be helpful. Yeah, it's a good way to give something back to you. I mean, you get a really complex piece of software for free. And by helping testing it by using pre-compiled kernels from a package or something or compiling a regular compiling mainline currency yourself, you really have the opportunity to give something back to the developers and help the world getting a bit better. And prevent others from running into into problems. So it's a good way to give something back. OK, so I don't have the impression that the latest kernel RCS are also that stable for embedded devices. Yes, embedded space is a little bit different. Do you have any comments on that, Dustin? Yeah, to be honest, I'm not that I'm more from the perceived space and not from the embedded space. But from the reports, I see it's true. But that's also the reason for that is that most of the embedded people don't run our sitcoms. So and a lot of behind. So ignore the packages from the base, from the sub providers that are years old. Even if you run mainline on it, latest stable, that's a bit of problematic because so few people do that because there are so many embedded boards out there. That would really be helpful if each embedded board had like two or three people that regularly run mainline on it. And if they report problems in time, yeah, then the problems will vanish. But if they pile up over two or three releases and then you try to go to the next mainline release, then it gets complicated because problems are mixed in and finding the root cause is hard. So what the embedded people really need to do is increase their testing things there. Yes, absolutely. So when I worked for companies that relied on embedded as their development platform and a deployment platform, there was an effort to I used to run latest offices on those to find the problems and then also test strings. A lot of our test strings do have a lot of embedded boards in them, so we do keep finding problems. Like, for example, the narrow has a testing that has several embedded things and Cardinal CI has them. So we do keep up on those testing and finding problems. It is not as unstable as people think, because if you look, watch the stable Cardinal mailing, stable release mailing list, all of those are tested on a ring so that are have several pieces of embedded hardware in them. So it's just a matter of checking and looking at it and then also play with it yourself and see how they fare and you can you can report problems. The important thing is also if you help testing mainlines, then the errors, the problems that were introduced, that those were introduced recently, that makes it a lot easier to fix them. And especially when it comes to regressions, because if they only pound after one year or something, then something else, then you can't remove the culprit that is causing the regression to change, that's causing it so easily anymore because it might cause a regression problem for people. That's also why testing the latest mainline is helpful for everybody if you are on a on a on a rare platform. For example, one that isn't used in your in those test labs, you mentioned, because there are so many embedded boards out there. Bicycle becomes easier if you're kind of, if it's closer to the introduction point. There is one more. I think it's not so easy to produce a vanilla kernel for embedded hardware. Typically you use some build root environment around it and then that is not vanilla anymore. You are right that I play with. I use Raspberry Pi and other embedded devices for testing. And for some projects, I have used it in the past and I continue to use. It is difficult in some ways to deploy a mainline kernel on on on embedded devices. However, many mainline embedded device maintainers, what they do is they keep integrating mainline changes into into their get. What I would recommend is check their get Raspberry Pi maintains their get, for example, or other similarly other vendors. They will have a get that they keep on updating it and keep it very current. So you can download that get and build that one and it will be close to mainline because they will add they might not have mainline to some of their specific changes, hardware, driver changes, maybe perhaps. So what they do is they'll maintain a separate get for things that they haven't. So that's the closest you can get to mainline. Yeah, that's that's a good advice for those all those embedded ones. Yeah. And then I also keep multiple micro SD cards because I keep swapping them out when I do testing on Raspberry Pi because I'll keep a distro one so I can just recover from it. And then I also keep the separate one for mainline upstream testing. To be as I said, on my regular laptop, I use our sequence all the time. And I think I had now five or six mainline cycles where I didn't have any problem with the mainline just this cycle that was one where I had actually two bad problems that affected my machine. But often using mainline on PCs just works totally fine. Yes, laptop. Yeah, I don't usually not a problem. Desktop and not a problem at all. That made for a good discussion. Any more questions? We talked everybody to the ground. It's too early in the US and in Europe. Everybody has a long day already. There are no questions. Yeah. Oh, OK. Oh, yes, looks like thank you for coming in. So I guess I hope you enjoyed everything. And we can hand it back to Pandas for the closing. Thank you both so much for your time today and thank you everyone for joining us. As a reminder, this recording will be on the Linux Foundation's YouTube page later today and a copy of the presentation slides will be added to the Linux Foundation website. We hope you are able to join us for future mentorship sessions. Have a wonderful day. Have a wonderful day. Bye bye. Thank you. Bye.