 Okay, everyone, welcome to Exploring Our Bugs. I'm Ben Cotton, the Fedora Program Manager. Before we get into the meat of the talk, I do wanna share a little bit of housekeeping notes. I use he, him pronouns. This talk, which is linked in the hop and chat if you're watching live as this is happening. The slides are there. This talk is licensed under the Creative Commons Attribution Share Like 4.0 license. So do with it what you will subject to those license. If you have nice things to say, which include like constructive criticism, I know I'm not perfect, there's my Twitter handle. It's also on every slide. If you have not nice things to say, you are of course welcome to keep those to yourself. I will ask for questions at the end. Hopefully we'll have time. Did cut a lot of slides out of this to make room. So feel free to use the Q&A tab. I'm gonna try and ignore the chat unless it's to say, hey, Ben, your audio has fallen apart, in which case I'll refresh real quick in the middle of the talk. So what is this talk? Well, it's kind of what it says. It's a look at the bugs filed from Fedora Linux 19 through Fedora Linux 34. And you might say, why that range? And the answer is Fedora Linux 19 was the first one where we use the end of life closure on bugs. And so that really felt like a good sort of modern era of starting point. We could certainly go all the way back to Fedora Core one, but just because we've handled things differently over time, it might make comparisons a little hard to really draw conclusions from. And of course, Fedora Linux 34 is the most recently end of life version. I stuck with those because it's a lot easier to get the CSV file when all the bugs are closed and theoretically they will remain that way. And then to try and dynamically redo this every few weeks with our open bugs. If you watched last year's talk, you'll remember that I excluded raw hide. Now some of the graphs have raw hide and I'll talk a bit more at the end about the future incorporating raw hide bugs. But for now, some of them do, I think I've called them all out when raw hide is included. But like last year's version, this is really based on curiosity, not convincing. I wanted to look at like, what do we see from the bugzilla data and what kind of conclusions can we draw? I didn't really come into it with like, I wanna prove to the community that we should, you know, whatever. And I think this is probably gonna ask more questions than it answers, although maybe not explicitly, but it should be at least get us thinking about things as opposed to saying, you know, prescriptively what we should do. So, quick, some quick notes on raw hide. Right now the Jupyter Notebook, which there's a link to in the last slide, raw hide is separate. As I started working on this, I realized there are actually a lot of places where I'm basically making the same thing, just one with raw hide and one without. So in the post nest deep breath moment before the release process really starts ramping up in like four days, I'll probably go through and consolidate that into one notebook so people can share it and do their own analysis or interpretation anyway. But the way I lumped the raw hide bugs in with a release is it's basically based on branch dates. And so like there's a little bit of edge case, like, you know, did the bug get closed before after that particular bug got branched because it takes a time to iterate through all of them. For what we're doing here, we don't need that level of precision. And like I said, not all of the data and the graphs and the tables and stuff will include raw hide. All right, so let's look at stuff. Some basic things are like, how many bug reports do we get for a release? That seems like the most obvious question to ask. And as you can see, we had a lot and then kind of dropped and now it's coming back up again. And you might be like, oh no, we're getting a lot more bug reports, but Carl Fogel in his book, Producing Open Source Software, which is a great book, says an accessible bug database is one of the strongest signs that a project should be taken seriously and the higher the number of bugs in the database, the better the project looks. Basically, the more people who are using Fedora Linux, the more bugs they're gonna find and report. So the fact that we're seeing arise is potentially a good thing, especially because really I feel like starting with Fedora Linux 32 and continuing up through the present, that's really when we started seeing some of that organic positive press from a lot of the Linux enthusiast groups. So this really does sort of intuitively correlate to that. All right, so now let's add in Rawhide. So just kind of looking at it, you see, okay, Rawhide is ingrained in the lower part of the bar. The released bugs are in blue and released is kind of weird because it's actually from the branch date forward. So sometimes bugs get fixed between the branch and the GA. There's a lot of caveats of this talk, sorry. But you can kind of look and see, all right, well, Rawhide is a relatively stable-ish percentage, right? So let's just look at the percentage. And the general trend is for Rawhide to make up a higher percentage of bugs over time, but it looks like we might be starting back on a downward trend again. And if we think about it, there's probably a relatively consistent amount of Rawhide bugs we're gonna find through our own internal testing and stuff. And a lot of Rawhide bugs are more process related to things like build failures and stuff like that. So unless we get a lot of people running Rawhide as their daily driver, we're not gonna see a big increase in Rawhide bugs, but we're seeing a drop in total bugs. So it makes sense that that's gonna lead to a higher percentage in Rawhide bugs now that we're seeing a higher number of bug reports again, the Rawhide percentage goes down. One other thing that I kind of thought was interesting, and I don't know what to make of it, is just the number of components with a reported bug, sometime during the release process. And I did throw Rawhide in here because not every component has a bug filed against it every release. And there's sort of a general upward trend, which is probably an increase in components over time. But there is a lot of fluctuation. So what does that mean? No idea. But which components get the most bugs? Well, as you'd kind of expect, it's the stuff that people interact with the most. If Justin Forbes is in the room, he will immediately be not surprised that Colonel is in the top. And a lot of the stuff is, you know, Michelle, okay, well, that's, you know, on our most used variant. So of course, that makes sense. What's the fewest bugs? Well, almost 99% of the components have fewer than 100 bugs over the lifetime of this dataset. So from Fedora Linux 19 through 34. And if a component never had any bugs, it doesn't even show up because I didn't like produce a list of all components and shove it in. So it's basically, there's a hunt, you know, 99% of the components with at least one bug had fewer than 100 and almost 80 or a little over 80% had less than 10. So the work is not evenly distributed. One thing I added this year is I just kind of wanted to see how many bug reports get reopened. Sometimes they're closed a little aggressively and so somebody will reopen them. This also can maybe give us a little bit of a measure of bugs closed end of life and then reopened. There are reasons it's kind of hard to suss that out that I can talk about if we have time. But again, this is based on when the bug was closed. So something that closed in 31 may have actually been originally opened in 26 and how do we interpret that? But anyway, there's a general downward trend and I can think of two plausible explanations off the top of my head. One is that we're getting better at actually fixing the bugs and not being like, oh, oops, it's partially fixed, try again. Or it means that people have given up when it doesn't work the first time. Either it gets closed end of life or there's a partial fix and they're like, you know what, I'm not even gonna bother anymore. One is a very happy outcome, the other isn't and I'm not sure which it is just based on this. I'd like to think it's the happy outcome. So another thing we can look at is severity. According to the bugzilla documentation, severity is the user impact. We also have a priority field, which is basically the developer's ranking of how quickly they're gonna fix it. I don't know that people always use those fields the same way, but I wanted to look at it. So this is not much different from last year. The urgent is a small slice and most people never specify a severity. When they do, we kind of see the distribution you expect. Low is actually smaller. I think in terms of actual bugs, there's a lot more low priority or low severity bugs, but people maybe just don't bother reporting them. So it's kind of important to keep in mind these are bug reports, not bugs. Even though I will use the word bugs a lot. So for this year, I did add just a look at the severity over time. And the urgent, which is the bottom line and sort of the magenta is very stable. It really doesn't have a lot of fluctuation compared to some of the bigger fluctuations we've had in just the total number of bug reports. And my suspicion here that I'm gonna go out on a limb and say is a fundamental truth is that we actually catch most of the urgent bugs either in our own testing or through user reports. And so doesn't matter how many more users we add, we're not going to get that many more urgent or even high severity bugs because we've already kind of caught those. So any additional bug reports is gonna come in the medium to low severity. I think that's justifiable. So one of the things to look at and it becomes more important as we expand our user base is the number of duplicate bugs. And you can see over time, the number of duplicate bugs remains relatively steady. I have some theories about like the personal, like the dynamics of how people do that. And I think, basically the percentage of duplicate bug reports isn't particularly correlated to the volume of reports in like certain thresholds. This is more of a, I don't know. One thing that did look interesting to me is sort of how we have these big spikes. Why? Couldn't say. Run through a couple of tables real quick. Fewest duplicates per component. I don't know why these get fewer. Maybe they just don't get marked as duplicate because they have so many bugs. So there are actually many duplicates that never get touched. It's kind of hard to go through without spending time and like digging through the reports and actually post factoed making that determination. What I do find interesting is that 66 components which is three more than last time had nothing but duplicate bugs. So these are either somebody mismarked it or somebody, the bug was filed against another component as well and the one in that component was marked duplicate. I don't know what to do with that other than it's like, hey, that's kind of weird. So if we exclude the 100% duplicates, these are the most. Again, there's no real meaning here that I can suss out. But it was interesting to me that unlike the previous table where about half of the components had an increase and half had a decrease versus last year, this case almost all the components saw a drop in their percentage. All right, so let's talk about the more interesting stuff. Bug resolution. Like last year, I kept the same categories. We have happy resolutions where things get fixed essentially we have resolutions that are sad for the user where it's like, no, we're not gonna fix that go away. We have resolutions that are sad for the maintainer which are basically like, hey, you wasted my time with this bug report, what's going on? And I'm excluding duplicate here because I don't think they actually add to it. And again, we can argue about which of these closure categories belong where or if there should be more resolution buckets. But for now, this is what we're going with. And we can see that over time, we're pretty consistent in our happy resolutions. But that means our sad resolutions are going up recently as we get more bug reports. And in fact, if you look at the percentage of bugs closed end of life, that's really bad the last few releases. I don't have a great solution for how we can fix that but it's bad user experience. And I would love to see us fix that in part because I see, I'm the one who sees a lot of the for do or never fixes my bugs. I'm gonna go switch to something else. You hate me, what jerks complaints. Another thing that was really interesting to me is like it seems up until recently very periodic. And I did some digging last year to try and figure out like how does that correlate to like rel release dates or something like, what's the external factor? And I couldn't find one. So who knows. So we looked at just raw hide bugs because that was for released bugs. As you might expect, raw hide bugs are mostly happy because that's stuff that we've sort of filed for ourselves and are fixing. But it's good to see that. So not only is it important how we close the bug but how quickly we close the bug report. So I looked at time to resolution. And last year I had a lot more slides. I had to cut some down. The Jupiter Notebook has a whole bunch of graphs that didn't make the cut this year. But as you can see, there's a very large spike right at the beginning. There are I think about 30 day bins or so. So it's a lot of bugs, vast majority of bug reports get closed very quickly. Those are probably sort of self-process kind of stuff. But I graphed this on a log scale before and it's very linear. So you do see a very quick decrease. I think a more meaningful thing is looking for the center, so how quickly are we closing bugs over time and we are getting better. We did have a little bit of a regression in 33 but the general trend is that we are closing bug reports faster, which is good. If we look at just the happy ones, so basically are we getting better at fixing them? Yes, we definitely are and that's great to see. But are we breaking people's hearts faster? Yes, we're also getting way faster at closing reports basically won't fix or end of life. I'd say that's a good thing because then we're not languishing, like the bug reports aren't languishing and people don't feel like, oh no, maybe it'll get fixed, maybe it won't like, here's the answer, whether you like it or not. And it's interesting to kind of look at like about the 400 day mark, like that's sort of the longest a bug will be open before it reaches, if it was open on release day to end of life. So we're getting down to the less than a year category. I guess that's good. One thing I added also for this year was looking at our security bugs and brace yourselves. This next slide is gonna look really scary. It's actually not as scary as it looks, even though we have a huge jump in security bugs. I was actually at the scale conference last weekend talking with somebody who happens to work in Red Hat's product security team and they were telling me how product security has gotten really a lot more involved in filing bugs for like every CVE in upstream projects that are shipped in Fedora Linux. And in particular, they asked about the Go ecosystem and a big chunk of these reports are from the Go ecosystem. There's probably a lot more digging into this to do, but you know, this isn't like, oh my God, we've just gotten really insecure. It's more that we're more aware of security vulnerabilities. And again, this doesn't break it out by the severity, which would probably be a good addition. So how are we doing it fixing them? Well, like non-security bugs, it's pretty steady, which means we have a whole bunch more unfixed security bugs. And again, it could be that these things were fixed in an upstream release and the maintainer added that new release and either didn't know or just didn't remember to tag that bugzilla in the new release. And so a lot of these may have been fixed up by the work of upstream and just not been updated. There's a whole long conversation we can have about that as well. So if we look at the time to fix, again, we're sort of seeing a general decline. That's good. We're getting faster, we like that. So the last set of bugs I wanted to look at are process bugs that are sort of a part of various development activities we have in the project. And again, this is also new compared to last year. So we have bugs for when a package fails to build from source. And these are often the result of changes to compilers or library packages or things like that. Just basically something that builds no longer builds. You can see it's generally fairly well distributed between raw hide and being closed in raw hide and being closed post branch. It would be nice if more of those were fixed in raw hide, particularly in that window between the mass rebuild and branch day. It doesn't happen right now. More packages are gonna be helpful if we can do that. So how do we resolve these? And again, this is including raw hide. It turns out, it's like we're only fixing about half the fail to build from source. And again, I'm hoping that this is partly an automation error or something where at some point the package does start building and the bug report doesn't get closed. Somebody who knows a little more about that process, Miro, if you're in the room or somebody from RelEng, hopefully this isn't as sad as it looks. Similarly, I wanted to look at the fail to install bugs. So that's basically the package built, but it can't install because of dependency changed and won't resolve for whatever reason. And it turns out we haven't really been tracking this meaningfully until very recently. Fedora Linux 28 is the first time we actually had a fails to install tracker bug, which is kind of interesting, but you can see we really don't have many reports. And this is basically searching for the string FTI or fails to install in the subject of the bug report. So it's an imperfect measure, but really in the last few releases have we apparently started noticing or caring about it or just gotten really bad at it, who knows. And again, we're fixing a lot fewer of these than we want, which basically puts these packages at risk of being dropped from the distribution because if they fail to install, they get retired eventually. So another thing I wanted to look at is blocker bugs. So we have release criteria and we go through before a beta release and the final release and evaluate, does it do these things or not do these things? And we don't ship until all the blockers are resolved. Over time, we are apparently just getting fewer of those. I think it's hopefully because we've had a lot more testing, we're doing a lot more both manual and automated testing earlier in the process. And so we just never get to that point. But we tend to accept a lot more than we reject, which I think is probably a good thing, although I would actually like to see more sort of questionable blockers get proposed because I'd rather decide, no, that's not a blocker than ship with it and be like, oh, we should have blocked the release on that. Similarly, we have freeze exceptions, which these are things where a lot happens to bug that ships on the live media. We don't want to block the release on it, but it'd be really good to fix it if we can. The rejected freeze exceptions, interestingly, are pretty stable. The accepted kind of go up and down. We've seen a lot more recently, which does kind of this correlate to the gut feeling I had about it. I don't know that that's good or bad. I think it does indicate perhaps a little more comfort with our testing so that we're less cautious about what we accept in terms of being a freeze exception because we're less worried that it'll accidentally break something. And then also common bugs or common issues as Matthew has liked to point out. This is basically a sort of known issues list that we generate recently started using Ask Fedora to generate some of these. So it'd be really interesting to see a year from now what this graph looks like. But over time, we are definitely showing fewer common bugs in general. And I don't know if this is that we're getting better at fixing the ones that we would want to highlight people or if we're just worse at highlighting them. I think it's the former, it could be the latter. And then I just wanted to throw in the prioritized bugs graph mostly to remind people that this process exists. There's not much meaningful to be derived from it, but I just wanted to put that in there. Basically the idea is these are for bugs, particularly post release that are just sort of obnoxious, affect a lot of people and just sort of make us look bad that we want to try and have Matthew and I kind of lean on people a little bit more, like, hey, can we move this to the top of the stack? And if the people are, the maintainers happen to be Red Hat employees and we can go to their management and say, hey, please prioritize this for the person or the team. Okay, so what's next? Like last year, I'll do some community blog posts with a summary of some of the points that I wanted to highlight, maybe a few things I left out. I would love in the next few minutes later on in the office hours I do on Wednesdays in the community blog comments, wherever. I'd love to know your theories about why some of these things are. There will be more graphs. I will hopefully do this talk again next year and make it different enough that it's worth attending again. And like I mentioned, I'm gonna unify the Jupyter notebook because I really, it's helpful because then you can download the notebook. You can grab the CSV files out of the repo. You can make the graphs yourself and you can twiddle things if you want to focus on certain areas or just sort of interrogate the data yourself. So with that, I have a couple of minutes left and if there are questions in the Q&A, I will take them now. There are not questions in the Q&A or at least there's no little red dot. There are not. So if you have questions, now's a great time to ask them. And if not, I'll just stand here and act silly and hopefully whoever edits the video will cut this part out. Kevin asks, have you considered also looking at Apple bugs? Honestly, I haven't. I think I talked to maybe Troy or Carl about it at one point. It's certainly something that could do. It'd be easy to add that. I think the longer nature of Apple makes it a little harder to react to things. Matthew asks, what data or metadata that we don't have would be nice to track in a future possible bug tracker? I would love to be able to know how many times the version changed on a bug. Right now, that's something that can be done on an individual bug basis by just querying the bug's history. But that would mean iterating over the 300 something thousand bugs and adding that together. That's very computationally expensive and I feel like the bugzilla maintainers would just disable my IP. So it'd be really nice to be able to just get that as a field on the bug. Jose asks, how could we get better information on some of the important issues you raised? I don't know that we can directly. I think trying to correlate them to user surveys and contributor surveys that we do on an annual basis will help. Starting with Thorolinix 35, I've sent out retrospective surveys and they haven't actually even touched the data for 36 yet. This has been a very hectic summer for me personally. But I think trying to, we pull some, start pulling some conclusions from that and sort of making reasonable inferences based on the developer, the tester experience and then user feedback to sort of see which things we do are affecting the shape of the graphs. On an all, I'm not sure the question is, did you ever consider to check bugs that are asking improvement? I'm not sure what you mean by that. So improving a feature in a certain software package, that's something that we could do. I don't think it was enabled until recently but bugzilla has basically a setting for, is this a bug, a feature request, things like that. But we don't have a lot of good historical data on that. So it'd be really hard to reliably sort out historically the difference between feature requests and bugs versus questions and things like that. But that is an important distinction that we don't really have the ability to make at this point, unfortunately. But it looks like my time is up. So thank you everyone for your attendance and thank you for the questions. I look forward to having more discussions later on. Thank you.