 Well, hello, like was said, I'm going to be presenting on getting started in Fedora QA or just a quick primer for, well, what I like to call tips and tricks. Things that I found useful in my endeavors in testing. So let's get started. The scope of the talk, it's going to be very high level. Like I said, it'll be tips and tricks that I personally found valuable that I think other people could use. There will be a technical side and then a human side and then questions at the end. The way my brain works is very hit or miss. So we might move along quickly, but hopefully I get my point across. So here's the testing. What do you need? I need a Fedora account and a Bugzilla account, really. One thing I did forget to say is this presentation is on the DevConf website. So if you want to download it, all of these blue links are hyperlinks to the associated information. But this is why we do it. These all came from the Fedora Reddit page. This one, this guy had a bug having problems with it and it started working. This guy not only was able to upgrade, but it felt solid too. And this one's my personal favorite. He wants to switch, so must be doing something, right? So the first thing, I want to give a little background information. And so I will start with a question. What does it compose? It composes a compilation of packages that are rolled together into a bootable image. Ideally, it happens every day, but sometimes for various reasons, one or another, that doesn't happen. The composers are called Rahide until a composer's chosen to be branched. And from that branch, you'll get the next numbered Fedora release. So at some point in the line, I think it's at the end of February for Fedora 30, will branch from Rahide and that will become Fedora 30. There's three that I found, three main categories of testing in Fedora. One is validation testing, which is running individual tests against a certain compose. There's bugzilla or playing with the blocker bugs, which we'll get into in a minute, and the freezing exceptions, and the updates testing repository where all the packages that are not yet stable still need what we call karma to go stable, sit. So validation testing, we run different tests to validate a compose, and we use the release criteria as the standards that we work against. If a bug appears to violate one of them, that will become a blocker. In order to test, you follow the test matrices here, let's see, which looks like this. This is for the most recent compose of Rahide, and on this page, you can download any image you want, and then here are the tests, and let's see. Just for example, we'll go with this test. This test is one of the tests that we have that hasn't been running a while because we don't have the proper hardware. Oh, yeah. The test will have a brief description, how you need to set it up, and then how to test, and some expected results. And if you run this, and assuming that you get the expected results, you will, let's see, you can edit the wiki page here to show your input, your pass, your fail, your in progress, and it'll end up looking like this if you pass, or I don't think any of these have failed. So a failure with a link to the bug in bugzilla, or a note, and here's an example of one that's linked to the bugzilla page. So on that topic, that fiber channel over ethernet, if anybody's got that kind of hardware at home and you feel like helping us out, we could sure use the help, because none of us have that. So we track bugs on bugzilla. This is where you can submit a bug if you have a problem. It's also where you can get feedback on bugs, or people can give feedback, or requests for enhancements. After the branch, well, depending, we'll have blocker bugs meeting every Monday at 1600 UTC, where we discuss all the bugs that are currently blocking the release, and we'll vote on them, whether they violate any of the release criteria. It's a good way to contribute if you've never contributed to QA before. And something that I found especially helpful was to act as the secretary during a blocker bugs meeting. The secretary really just monitors the meeting. You can input, but it's your job to monitor. And then once the meeting's over, go back and update each bug with what decision was made during the meeting. It's a good way to teach you how the process works and how to use bugzilla as well. The third way of contributing would be the updates testing repository, which is where packages pending review and ultimately inclusion into Fedoris, the stable repository, wait until they're tested adequately. And the general rule is three positive votes that a package is working and it will be pushed stable, but it's often the case that packages don't get tested and they sit for a long period of time before something happens to them. There's multiple ways to configure to test from the repository. You can permanently enable the repository or just enable it when you want to test updates from the repository and there would be an example how to do that. The easiest way I found to test updates and then submit the information was using Fedora Easy Karma, which is a tool that, it's a command line tool that allows you to quickly comment and add or subtract karma from updates. I found that if I used the karma page, the actual web page, I could do maybe 10 an hour, but using Fedora Easy Karma you can do 10 in five minutes if you're really moving. You have to add your Fedora account ID into this file that doesn't exist until you make it. So now let's go from the technical side more to the personal side. It can seem kind of daunting to jump into a meeting or to post your results, but it's really not. It's not difficult and it's not something that you should be worried about. Fedora totally relies on community input. Red Hat does support the project, but I think there's maybe 10 employees and there's no way that they could, we could get everything that needs to get tested or needs to get done in the QA community without the community. And if somebody's, I've found the best way to gain a foothold is to have a differing opinion and have the ability to defend it that will make people understand that, hey, maybe the way that we've been thinking about things is not the only way. And I say common sense rules apply, meaning the more you participate, the more you'll be known and the more your input will influence the whole process. So how do test results help? Just from this last release, Fedora 29, which branched or was frozen August 28th and it was released October 30th, there were 1,074 bugs reported. There were 671 tests run on the validation test matrix and there were 1,685 individual karma given to updates testing or updates in the updates testing repo, which averages to 17 bugs a day, 11 tests a day or 27 karma a day, which for one person or two people or even a team of 10, that's a lot. So having the community input there is what makes this possible. To wrap up, I have some tips. If you're a remote worker like I am, I've found that finding somebody in town to talk to about this really helps me kind of see the bigger picture. I'm lucky to have a coworker in town and we'll meet up maybe once a month for coffee and just talk about things in Fedora land. And that really seems to help me. Something else I found was to make a list of IRC names and the associated position within Fedora that they hold and their email because trying to keep track of that is really difficult, especially when you're trying to find the guy who works on that one thing that you had something to do with once upon a time. And again, don't be afraid to jump into a meeting. That's the best way that I've found to make yourself known and to gain some traction within the community. And when you're acting as the Blockerbugs secretary, make sure your amendments to the whiteboard are 100% correct. Otherwise, you'll crash the Blockerbugs page. But that's all I've got. Are there any questions? A good way you can help pests is test days, which are unregulated. Usually once we get to about the sort of branch point or beta, we'll be assuming pest days for various things, like you'll have a known pest day, or a networking pest day, or a federal pest day. We need to get an answer on mailing lists. We'll get a list of pests that I'm not sure where else. This is a matter of whether you use the guy you're going to be using. We've got a lot of Twitter and various social communities. And those are generally really good methods to get around that, because they're focused on one thing. We'll be a bunch of other people doing the event at the same time as you, and we try to have at least one relation to develop a presence of the event. So we basically jump on IRC, build the instructions on what we take about how you do the pest, and you just provide feedback there. Adam was talking about test days, in case you didn't hear. After the beta, sure. After about the beta, we'll have test days where a specific package will be targeted to be tested, and for one day that is the goal, just testing this package. And that's a great way to help, especially with new changes coming into Fedora. We'll often put those through a test day to make sure, for example, in 29, there was a DNF3 test day. A lot of bugs were found because of the test day. So that's a great way to add anything else. The question was, if nobody tests the fiber channel over ethernet, why does it remain in the test matrix? To be a reasonable way, which of the stuff is not tested for a while, and maybe proposed that some changes should be made? Yeah, the addition was, is there a way to see what the coverage is on any given test case? And Adam's got a website that documents this pretty well, actually. Okay, I'll pull it up. This is a view over time of each test in each category from the matrix, with coverage pass-fail over here, and whether or not that individual test is, or what release criteria that test falls under. But this is a good way. In fact, this is the way that the QA team uses to prove coverage at a go-no-go meeting towards the very end of the release cycle. Gray dots here meaning no one's run the test. So a good idea if you want to, to run that particular test. Notice too that a lot of these tests with a release criterion level have good coverage, but the ones here that are optional don't because they're optional, so they're not release blocking. Any other questions?