 Okay, we're all set. We can get started now. Welcome to the How to Get Your Colonel Bugs fix talk. One of the reasons we decided to have this talk is that we need to have this talk is we've got a lot of Colonel Bugs. We have a lot of people upset that their Colonel Bug is not getting top priority, is not getting fixed. And there are things that you can do to help us fix your Colonel Bugs. So, the state of our current Colonel Bugzilla, as of August 1st, there are 702 total open Colonel Bugs. There are three people on the permanent team. That makes it fairly difficult to address every single bug. Obviously, we look at everything as it comes in, but it becomes a question of, well, what is the priority on this? What kind of information is there? How much work am I going to have to do to see if this is even something that's reproducible that can be addressed in a timely manner, things like that? Bugs that come in with no information, things that are just straight up AVRT bugs with no context around them, those types of things. Honestly, if we see them, they go in and they're kind of at the bottom of the queue. We don't pay a lot of attention to them until we have cycles to go back and say, hey, let's get some more information here, things like that. So we would like to get bugs that are easier to fix. And so how can you help us do that? How can you make sure that your bug, your little bug in this massive haystack of bugs, gets fixed? The most important thing is file better bugs. That means you might have to do some work. Yes, AVRT will pop up and say, hey, you've got a Colonel Loops and we'd like to file a bug for you. And it will file a bug for you. There are fields there where you can add more information. Please do. Importantly, see if you can reliably reproduce the problem. If you can reproduce the problem and come down to a small set of steps, that this is what I have to reproduce the problem every time, it makes it much easier for us to fix. Now some problems are not that easy to reproduce. I've seen problems that take two days to reproduce, and even then it wasn't quite reliable. So it's not something you can do every time, but if you can do it and you can include the information on this as how we reproduce this problem, it's going to make it easier for us to find the issue, fix the issue. Another thing to do is see if people are having a similar issue. Google is your friend. There are tons of other distributions that have bug tracking systems of different types. There are several different kind of mailing lists. Chances are if you've hit a bug, unless you have a problem with your hardware, you are probably not the only one who has ever hit that bug. So, a quick Google search, you might find a link to a launchpad entry or a criminal discussion. Okay, I'll discuss things like that. If you can include that information in the bug entry, that gives us a great place to start working. You don't have to be a criminal developer to understand the, there's a discussion happening around that similar bug, and then we can go in and figure out the problem. Or perhaps even better, someone else has already figured out the problem for us and that will be a patch there. The other thing that's really important, because the Fedora kernel moves quickly, is make sure that it's an issue with the most recent kernel. We get a lot of bugs filed on, we're still getting bugs filed on 3.14 for Fedora 21, 4.1.5. That's an even better solution, but sometimes that requires some extra steps, and I actually do mention it in here. We would love for you to try the raw item, and we'll see if it works. That might require configuring other repository things like that, but everybody should at least try the most recent kernel. A lot of code churn happens in the kernel, and the difference between 3.14 and 4.1.5 is massive, so a lot of things get fixed. Unfortunately, there might be some new bugs introduced in there as well, it happens, but if you've not tried the most recent kernel, it's kind of a waste of time to go through and say, is this still a problem? What was the problem in 3.14? Upstream, people don't really care about 3.14 anymore. In fact, 4.0, I think, just had its last stable release, so it's done. Nobody cares about 4.0 anymore. 4.1 is kind of it, and then everybody's working on 4.2. If we are still supporting that, let's say 4.20 before it was in black, we did not push out the rebase there. If there's an issue with 4.0 on 4.20, then yes, we still care about it, whether or not upstream cares. We still care because we're still supporting that kernel, but those are fairly rare instances. Usually, we move along with upstream, again, because there's three of us and lots of bugs. Anything upstream can help us with, we're happy to have. The other thing is include relevant information. AVRT, and a lot of times people filing their own bugs will just, here's the UPS that I got. That's the only information they give. What we would like to know is not only what is the UPS, but what is your D-message output. If this started happening with this particular kernel version, we need to know that. I just ran 4.1.4 and this was not happening, and it started happening with 4.1.5. That narrows it down a whole lot. If it's a device of some sort, PCI device, USB device, we might need the LS-USB output, LS-PCI output. Those things are great to include proactively because when that bug first gets filed, we can look through and say, well, here's a lot of information. We have the UPS, we have a lot of information surrounding it, and we might be able to see the problem right away. If not, at least we know that you are providing information and going to be fairly responsive as far as taking care of that bug. We might ask for more information. Not a non-technical. You guys have, like, you know, run SOS and collect all the stuff for us and just attach it to your board. We don't. There used to be. Right. And why doesn't it work for that? Right. Well, it does, or it used to, right? It doesn't collect. But they kind of turned it back. Well, because it's always a privacy issue first, if you include too much that it might have something. No, I think that's... Yeah, that's true. Yeah. There used to be a great tool. No, but the ones that it's not true for, we need to know it and turn it back. I mean, there's always an argument. Many people are considering including the hardware and the books and maybe actually all of it. Well, it does an auto file. In Fedora, in Ubuntu, there was one simple tool. It was app-collect or something like this, which you had to run shell, run this tool, give it my number and press enter. It authenticated to a launch bar and added LSTPCI, USB, DMS, every, the default set of stuff. Right. It was even probably sending ACP items. Right. Well, it was just one standard set of old data without giving, without asking user what you have to provide. Well, what is the problem? It's not the problem that way. Yeah. It's just like the data and maybe it's there. Yeah, it's better than that. But isn't there just the assistant phone? Because, anyway, there's often an FKFF assistant phone, which doesn't watch stuff automatically. I'm not sure exactly what it is. But that's part of that face. I thought that was mentioned. Yeah. It will dump some info. I was not aware of that. It would be good to see what it collects. But the other thing that I'd also notice is that it's true. The average user may not know how to do that. But if they come back to us and say, we don't know how to do that, it's pretty easy to be able to tell the assistant director, say, open a terminal and type this command. Right. And I guess the goal is to save up over a terminal. Well, they don't necessarily have to. It depends on who's triaging the bugs. Well, that's where we are triaging the bugs. Nobody else does. We used to have the bug triage page, right? Yeah. So maybe there's a kernel equivalent. We do have a kernel bug triage page that walks you through. It has links to all of the bugzilla queries for untriage bugs. Nobody used it. Every once in a while, we'll get somebody who comes along and does a great job for a week or two, and then they disappear. So, you know, I understand. It's overwhelming. It's not easy. It's a thankless job, except we really, really appreciate it. So thank you when you do that. But we haven't had that in a while. I think last year, somebody did it for a month or so. That was the last result. Is there like a bug bug for the kernel bugs always that you go in and like automatically throw in a comment that says, you entered a kernel bug. Thank you so much. Now go read the triage page. And here's some of the other stuff we need to really look at the bug. We'll input that. I don't know. I wonder again about if there's another opening on the front side. We can change it to just say, by the way, here's where it's better. Right. And that means interaction with that team. I mean, I'm sure they'll listen. They're actually very responsive. They've done a great job in filtering out a lot of the craft and making sure we're not getting bugs filed with, tainted and things like that that are important. That actually is another very important point is if your problem happens and you've got a tainted kernel and a video driver or anything like that, you're going to have to reproduce it without that. Or honestly, we're just not going to look at it. I will say that tainted doesn't mean necessarily what it used to. I mean, there's a tainted flag that says you oops before. Right. Well, so there are multiple taint flags and that's another important one though. The one that says you oops before means your system is in an unknown state. You don't want to know the original loops. You don't grab that last one that has the tainted flag. Go to the original one that's not tainted and that tells you what started the problem. Right. Except a lot of us have machines that just throw an oops and boot just because there's some wrong with the ACPI table and that doesn't necessarily render any further or more useful. So it's a tough thing. That's the engineering problem we can solve. Yes. I mean, that's... See if you find enough copies of this that they grab. There's some contrast there and somebody doesn't split. Yeah, right. I mean, there are plenty of examples of that on the board. I think there's, for example, there's the IONU one that always sends a worm up at, you know, local DMRs. The deck that says your kernel is fine, but it's still worn, which they're bringing to pink lines. Right. Right. I mean, we can link those together by default in Mozilla. I mean, say, this first you file both of these and these are probably causally causal to candidates. Right. And then separate them out based on which parts repeat on other systems. We used to have two. And Dave was really good at it just by looking. And we haven't looked. There are people at Back Parkway that file a ton of bugs. And it was weird. I guess Dave remembered every strange bug and identified this person has filed 18 bugs at the last. And they won't check their hardware like we've asked. So we just kind of need to ignore them because it's bad hardware. And he was really good at that. Several times he would comment that. And people were like, oh, yeah, I replaced my memory and it's all gone away. Some extra helpful items, the things that are going above and beyond just doing a quick Google searching and seeing if other people are having a problem or making sure that we get the relevant information around. If there's discussion upstream about the issue, the Google search will probably help you find it. You'll link it to the bottom. Then that helps us a lot. If it doesn't exist, you can start one. There is a maintainers file in the kernel and we don't actually ship it by default on the system. But I think I should probably just go ahead and add it on the kernel Wiki. This is a link to the maintainers file. So you can go and see who's the maintainer for this subsystem. I know that it's kind of daunting if you are not a kernel developer, not a member of the kernel community to send an email with a bug report. But really, it can be a very simple email to the maintainers list to LKML but not as useful because it's very noisy. Saying, hey, I ran into this problem with this kernel version and as long as it is the most recent stable kernel version of what we're shipping, we'll probably get some good response on it. Yeah, it should at least be interesting. If you send that first email, even if you don't want to continue in the discussion or the discussion goes above your level, you have a link to... This is the email essential list. We can follow in on that discussion as well and hopefully get things happening there. A lot of times, that's what we're going to have to do anyway. We don't have the cycles to go through everything. We don't have the hardware that everybody has. So we're going to have to start the discussion on the stream anyway. And if you've already started it for us, that's a great head start. Yeah, but did you guys not read that? That's true. That's true. I mean, the flame part of it, we can do it pretty well. Yeah, I'm sure we have a user that's probably... Yeah. It's actually the opposite I've found that for the most part, if you're a random person and you're attempting to make a reasonable bug report, you'll generally be accepted fairly well It's sort of the theory that if you're an experienced person who keeps growing up or you're a new person who can communicate with you, then you'll get yours up. Right. So I will say I have not had all the luck. Well, you may or may not have luck, but if the conversation has started, then if somebody replies, then maybe we can jump in as well. And also the upstream mail was tapped pretty quickly. So later when somebody hits that problem, they're going to find that post and say, hey, I must have seen this. Right. It helps a lot. If you do find a patch somewhere, even if you're not sure that it fixes the issue, you think, hey, this patch, you can point that out as well. It's much easier for us to look at it and say, oh, well, no, that patch doesn't fix the issue or yes, that patch looks like it might fix the issue, but it's been superseded by this patch, things like that. We can address that. It's still a huge help if you include them. And if you include a link to a valid patch that definitely fixes the issue or probably looks like it fixes the issue, we're probably going to build you a kernel very quickly, a scratch kernel, and say, test this. And if it works, then it gets into the next release. We're releasing kernels pretty much weekly. So that gets an issue fixed very quickly. If you can tell us the most recent kernel that did not have the issue, that's great. If you installed on 3.14, and for 3.21 you just updated from a fresh install and it jumps to 4.15, there's an issue there. Well, maybe you should go back and check the first 4.1 release we did for it, check the last 4.0 release we did for it. If you can just check those things very quickly and then say, hey, this kernel had it, this one did not, that really helps us narrow it down. And then, as someone else mentioned earlier, if you can test with the raw hide kernel, that is a snapshot of Linux's street daily. Well, almost daily. And so if you can test with that and the issue has gone away, that tells us that it's fixed upstream and makes it a little bit easier for us to first track down where it was fixed and try to get that patch back for it in. And for an issue that's causing a problem for users, we probably also want to get that into the stable tree so that not only do our users receive the fix, but everybody running stable kernel gets the fix. General tips. A little bit of kindness goes a long way. We don't like to see kernel issues any more than you guys like to see kernel issues. I know they're affecting your systems, but we're not sitting there laughing at kernel bugs. We would rather there not be any, and that doesn't mean we would rather you not file any. It means we would rather there actually not be any bugs impacting our users. It's an impossible goal, but we do want to help. Due to volume, a lot of issues do take some time to get to, but if you're nice about it, we are certainly much more likely to get to your issue more quickly than people who are throwing a fit. Why is this not being taken care of? Especially when it's not something that's affecting many users. The reality is if you are one user who has filed a bug, and you get one problem, your sound device or something like that, it's not going to take priority to a bug that 15 people have had to affect including or affecting, especially things like data or things of that nature. And be responsive. When we ask for information, it means we're actually looking at your bug. Right now, I understand email takes a little bit of time. If we ask for more information and if you can respond within a few days, that's great. If you're taking two to three weeks to respond, a lot of times it's going off of our radar again. Plus it's out of date now. Well, sure. That can be the case as well. In two to three weeks, we've probably done two more kernel bumps. Hit and run bug filings are bad for everyone. And there are a lot of bugs that get filed as a, hey, here's this. I ran into this issue and we never hear from the user again. We can ask for it. We can ask it. The only time they ever get closed is when we do the auto, we've done a full rebase. We would like you to test this. Nobody responds within three weeks. They've been sending you need info and then they get closed. And we don't know if it was actually fixed or not. Those types of bugs add to the volume significantly. They add to the noise. And it makes it harder for us to get to other bugs. The time that we spend looking at that bug ends up being wasted time because you're not going to help us fix the bug. You're not going to help us help you. And that doesn't mean that we need you to go and do the code. We just need some information. Please be responsive. Bonus points. If you can find a patch, that is the quickest way to get a bug fixed. I'm not saying you will all the time. In fact, you probably won't a lot of the time. But if you can include one, you get big bonus points for that. It's going to be fixed very quickly. The second is write a test. We've been talking for a couple years now about the test suite that we run. Every kernel gets tested as soon as it's built. For issues that are quick to reproduce, easy to reproduce, it might be worth writing a test. It doesn't require kernel knowledge. There's a link right here on the testing guidelines, but really the testing harness is all in batch. So if you have these commands that can reproduce that problem quickly, then write the test and submit it to the Fedor kernel mailing list. We'll get it included in our test suite, and hopefully make sure that the issue doesn't come back up. So it's not a requirement, and a lot of times we'll look at bugs and say, oh yes, that was quickly reproducible. I can run a test for that really quick. If you do one, that's even better enough. There's a badge for people writing new tests. If not, I've put in a proposal for one. I haven't seen how that fell out. Do you work for new regression suite coaches or are you interested in X86? We are only primary arches right now, except for ARM, and that's because of a lot of hardware, so it's X86. So probably never going to get us any more. No. They could actually... We could find a tiny-ass needle or something like that. Well, actually though, they could run them up. They're pretty well tapped. They could run them up. Another part of the issue with that is, because CoG's not doing those builds directly. Oh, that's right. They would have to be a second monitor that says we need to kick off these jobs because this build's finished. I know with FedMessage, we see them now. So I see every time a 390-year power build is done. But it doesn't... Dude, we don't do anything like that. They could easily run the test suite as well. But what we've done on all of those systems, actually all the VMs that are running the test, we just set it as startup. It grabs the latest kernel, runs the test, and then shuts itself back down. So, I mean, it's pretty easy to add to startup an existing system and say, hey, just test every time you do. And those results, even if community runs them, those results still get submitted to our server. We'd see all of it there. Arches. Is it probably data rack in this case? Yeah. But right now, arches don't... I don't think they show up unless they run from our partners. Because we didn't want a bunch of... If people are testing kernels that they're building in those things, I don't think they'll show up. Yeah. Yeah. So, but we can make those visible. I mean, there's really that much that you can actually do an automated test for given that a significant portion of this is going to be completely hardware related. So... There's not a ton. There actually is a mechanism in the harness where you can say, this is a hardware specific test and it checks to see. So, there's three things you can do with the test. You can either execute it successfully, you can skip it or you can fail it. Obviously, you don't want to fail it. But you can say, hey, is this module installed? Do it. Am I running this hardware? Okay. And if I'm running it, then I run this test. And if I'm not, then I skip the test. You can just run the thing on my hardware to... Right. And if it's in the suite then anybody else who has that hardware can also run it. So, there are badges for running the test suite. Which means we get a lot of users actually running the test suite and submitting results. It's great. So, even if it's something that... If it's hardware that we're not going to get an RBM, somebody out there might have it. And it's worth testing. Is there a master Wiki page that links to all of these? Well, there's the... Just the kernel Wiki page has all of the things on it. So, it's got the testing information. It's got the triage information. It's got... Some of it does need to be updated. I actually went to update something this morning. You'll notice there are a couple of other places we need to update. But that'll happen hopefully in the next week. I'm not sure. We're all going to Plumbers next week too. Yeah. Yeah. The word's language. Seattle. Seattle. So, I'm home Monday and then I'm back out Tuesday night. So, you have the summary. It's pretty simple. We've got a ton of kernel bugs. We've got three people. If we have your help, we can get to your bug more quickly. The more information you give us, the more likely we're going to find a fix. And a little bit of work that you do on your end is actually greatly appreciated by us. So, when we fix a bug, if it's your bug, it makes Fedora better for you, but it also makes it better for anybody else who's winning across that same bug. I'm not willing to put forth the effort. We want to make Fedora as good as it can be. So, what do you mean? People use it. Most people do. Yeah. Most people do. Most people are not just saying... First and foremost, most of them may be fine. They can do that, too. So, you mentioned that occasionally somebody pops up out of the community and does triage and then leaves. Yes. So, it sounds like what you need is five or ten people from the community doing triage and then leaving. Well, I mean, if... we had five people who came through and maybe did a half hour a week of peace, that would be massive. So, it's okay from your standpoint for somebody to drive by, do some triage according to the documents that you laid out, and then maybe they got to go back to work the next week, but they did something. Okay, sure. So, why don't we start through this differently? Why don't we get a badge that's one of those level badges series that is, I stuck around doing internal triage as many times. Some of us don't care about the badges. But there are a lot of people who do. It's working pretty well. Okay. And say, you know, you get the first one for showing up an activity day and learning the triage and doing some triage that day. And every release, we have one of those one activity day. And you get a badge for showing up consecutively. That would be kind of awesome. That would actually be really awesome. We could do some of that. So, we have test days that are formally organized from a bidorsional point for all sorts of subsystems. It would be a good idea to throw in a triage day. Particularly, probably a late alpha or very early beta. Yeah. Well, I mean, you know, they've got hackfests around here. What do you mean by that? Well, you can do it in the room. Well, you don't have to do it in the room. I mean, you can do it in our studio. Oh, of course, yeah. I'm just saying here, we have a whole conference. You know what I mean? That's true. We do, but there's usually there's a lot more development that happens in the hackfest on Saturday that's really going to require people being in the same room. Yeah, that's true. It's really helped by that. Whereas, I think we would be taking away time from the people who are showing up doing other things. I'm checking the page to see if my questions are answered before I ask. There's a couple issues. I mean, assuming we're of the technical level that we can build our own damn kernels to test things. Right. You run into an issue where the fedora kernel is not vanilla. So you can't really get bisected for one thing. I've actually been working on that. I have a set of scripts that I've sort of been using to be able to bisect for workshop. And I had at least one person do a trial and run and be defined about what he did to work around it and what he did to identify it. Are you the new kernel person that's come on board whose name I forget? Okay. This is Laura. Yes. This is kernel person three. So is kernel person two here as well? Yes, Josh. He's in the... I know it doesn't really matter if I see what people look like. He'll be doing the stick of the kernel this afternoon, too, which is kind of the talk we do every year. There's no one as well. Oh, yes. We'll be a big investor as well. Yeah. So, let's see. So, you're talking about get bisectability. I have this problem where... I know how to build Fedora packages. So, I'll tweak something in the kernel and then I'll kick off a mock build and wait three hours for the whole kernel package to dump out. Right. I know that the kernel... You can just build source in there without having to go through all the pain in the rear of doing that. But I have no idea how you actually get that installed as a regular kernel so that I can boot into it and then boot back out of it. Right. So, there's actually a great beatme doc in the kernel source that kind of goes through... You can also take a shell script in the package directory that runs the rpm-bd-without-without-without. Oh, right, right. And that makes the smallest one to be a little faster. That's true. Yeah. I mean, I don't even need an RPM if I know that it's going to show up in the grug menu after I finish and make it install. Right. But I have no idea if it actually does that. There's actually a script that should work. Okay. So, once you copy the kernel to boot and then you make it install... Yeah. What? Yeah. What I'm wondering is, I don't know if we've patched to make install to work correctly with everything being in user now and I don't know if new kernel script requires it to work. So, it's working for me on f23. I haven't tried f23 or raw. Well, f23 or raw it should work. It's the question, right? No, it's f21 that's the question. Okay. But it's working for you on f23 and raw. Yeah. That's enough, yeah. Right. So, it's probably the last thing that you've done on that. Right. It's also kind of annoying that we have to copy it to do that when you're clearly doing a quick thing there. But... Yeah. I've heard of a team not that good. The worst veterans have questions that capture some of the stuff somewhere so that you're not experiencing this. Yeah. Right. And that's what this page is for. I want to make sure that... Yeah. There is a deal, so... There's a lot we shouldn't do there, probably. Because right now, like, if you've done that, you've got this kernel sitting inside forever and it's in your revenue forever. And we don't really have a way to say, hey, I'm installing test kernel. Now I'm done with that test kernel in that kind of phrase, I think. There is a way to... Which we could write relatively easily. We could. There's also a pretty simple way to do an argument build. It's just a quicker build. Yeah. So if you tell it you don't want the debug, you don't want... I mean, that's a lot of it. Yeah. So if you're making a release kernel, you're also building a debug kernel. Right. Yeah. And that takes a long time. It does. So I'll just... There's something else going on here. This conversation seems to be both on... both are clued with kernel bugs. Right. Yeah. You know, I'm seeing a bug of an X that started with 3.18. Right. And I... Well... I try every kernel and I... oops, it happens again. Right. Like a week, I go back to 3.16. Yeah. And it's stable. Okay, so that's... That's a... Yeah, that's another bug. How do I know who's faulted it? I've added my 2 cents to the bugzilla... Right. ...to the board. I've added my 2 cents to the free desktop one. And I think it's kind of getting... the bug is kind of getting lost because everybody's saying... Right. ...my problem. Well, so the... Most of the X bits are... I mean, a lot of it is kernel now. So it's... It's... Usually the same people were looking at both sides of it. And so it's okay to file a bug as kernel now. Well, for... Actually, from a Trios Jam point, when a bug is filed against kernel and it's in DRM, it actually gets refiled against the X component anyway because that's how our graphics guys want it. Okay. So it would be... It would be refiled against, you know, an M15 or Radeon or whatever, and the appropriate maintenance will take a look at that there. That's the method that they... Okay. They can't really put it in the wrong place. Yeah. The lesson here is do whatever seems right. It'll get right. Okay. Well, the problem is you're talking about, you know, we can't do the Trios, which means we can't because there's not enough cycles. So we can't necessarily make sure the bug gets redirected because nobody has a chance to look at it. Right. And so if I'm doing Trios, how the hell do I know whose bug it is? So there's... The Trios page includes some of that information. No, does it? Yeah. I guess I'll... It walks through a lot of that. So how are you guys finding ABRT, right? Is that helpful? Is it helping you figure out how to prioritize which bugs you're looking for? Well, with the retrace changes that have been made recently, it's gotten a lot better, and Laura's been doing some work as well, too. I did some work on there, but still on the daily basis, I'd probably say in terms of notifications, there's still a lot of noise. I think specifically ABRT gave still this notification for very old kernels. So, for example, like the point about 3.14, we took a notification for that, for at least not a daily basis about, you know, this is 10 and it's 100, which still happens. And then it's still... The other thing I found is that ABRT, it's still... Unfortunately, it only gives a back trace since it's harder than hard to deal with the rest of the kernel DNS unless someone actually files a bug. So if they just have a ABRT report, then they only give a back trace, which there's a few of them where it's just a little bit more information that could actually be given on how much harder to deal with how it is. So it's a step in the right direction, but I still found it to be very hard to sort through to figure out what is actually doing something. I think it gives me that ABRT is still passive. So if someone is actually taking an initiative to file a bug, then it's still more likely that we're going to need to take a look at it. So, yeah. So, can you guys do... Well, just a few months from now, I was working for you and I don't really touch it anymore. I'm not a big fan of you. I'm still very familiar with code and I really can't help you. But I believe the guys... Because we always are longing for a question from you, like from Chrome people because you are the only two group that were able to specify their request rule. And for the first time, that idea that Chrome... because every year, Josh, he was talking about that. He was saying he's not touching none of your requests. We have impression that your request hasn't worked. We are much happier than... It sounds like what you're saying. It's just giving us a little bit more. It's telling the theme of the solution. And that answers what we're doing. It's really good. But I think it's still sort of a... There's definitely more work to be done. I think it's definitely helpful to at least get somebody to focus on it. I mean, it's sort of probably a comfort wise. I'm probably saying that I don't remember a couple of things that I've actually found that have actually been really good in effect to being able to point to God. Well, isn't that accurate? So it's okay. This looks like a pretty obvious thing for... Oh, I saw a picture somewhere else. It's helpful when it's all day, but I probably say it's less a problem with AVRT probably before the audience or the kind of bug in general that this is a problem. Well, I mean, the biggest problem that I think we have with AVRT is not actually an AVRT problem. It's just a... Because of the way that it pops up, people file bugs and they click to say file the bug and have AVRT file the bug and it seems just assumed they're done. So we get a lot of bugs or AVRT file bugs that they file and then walk away and we never hear anything from them again. You ask for more information, you don't get there. Would you classify that as better than nothing or you wish it would be better if people just didn't file them at all? That's actually a really tough call. Sometimes... It depends on the bug. There's some of them that are, in fact, pretty self-contained. You can tell exactly what's going on, but the problem is that it's sort of a... obscure... If it's something like if it's specified warning, then you can do it, Michelle, but it's just like a random general protection call. It's a lot of trouble then. It's also sometimes helpful when you have somebody who does actually file the bug and then you've got all these other AVRT reports that they're hit and runs, but there are a lot of people hitting this problem and I do have somebody who's responsive and going to help me fix it now. So they're useful a lot of the time and then a lot of times you do get the... It was a one-off, it'll never happen again. You've got this bug file, they're not responsive about it, so it's just noise, but I don't know that you can... I don't know a way to effectively get rid of that noise without getting into the file. Well, somebody just has to go and say, look, we asked for info, it's been a month, it's been a month, something like that. Part of that is something that we need to work on a different scale than talking about the kernel bugs. What I hear you saying is we give an user a mechanism wherein they can report that a fault has happened without becoming involved in reporting the product. Well, I mean... And now they're not responding, because they're not involved. But doesn't a lot of that not go to Bugzilla now? Does the BBRT just collect the stuff or isolate it? Yeah, retrace it. And that retraces... getting much better for us. We're just going to masculade you to the point where it's in the way as much. That doesn't increase the incidence of solving the user's problem. That decreases. You think so? Not necessarily the number of things you're addressing is low. Okay. The 3-House page is relatively complete. I was going to add, but Bugzilla links are so... Where is the 22? If you go back and see... It's right on there. It's the very top of the page. Okay. I'm looking at the link. I'm just trying to figure out... Yeah, here, not actually doing the source, but I'm saying go back and actually view it normally without the source. I'm just trying to add the f23 link, because it's not there right now. Right. So that was what I was saying part of what was out of date. The f23 stuff needs to be added. I was just going to edit it in right now if I could figure out... Even better.