 Welcome to the check your Fedora before going out or how we validate Fedora, it's a talk about validation and release testing so the validation tests are used to check the overall quality of Fedora and They have basically the following features we define them in release validation test plan The link to it is over there if you are interested They have or they are part of Fedora release criteria We automate some of them some of them must be performed manually they run for each build and There are several fields of interest that we do and We put the results into test matrices. I'm going to show you the stuff There are every day there is a night build of Fedora, so The new version gets built and published and tests are performed ideally All tests would be performed on the nightly build but because we cannot do it as there are Not many people doing it Usually only automated tests run every day The tests are in groups Installation base desktop server and cloud How do they vary so one test It is not just one test, but there is one test and it has different varieties So actually one test could be tested on Different installation media either on ISO USB or even CD. We still Sometimes test CDs the architecture is important Whether it's bare metal or in a virtual machine Whether it runs on BIOS or a ufi whether the test is blocking or not blocking and Some more variations We test for different milestones too there is the alpha quality which I Have heard that now we are going to To call in a term always ready Rohit Which means that you can be on Rohit and it will not break at least not terribly break and This is or some part of the goal has been already achieved by automated testing every day then there is the beta milestone and the file mile the final milestone and The the release criteria are different For alpha for beta and final Each of the following includes the previous so if the release criteria For alpha that we call basic is Not met then of course beta is not met and final is not met But when basic is met and beta is met then final might be also met Unless there is some other problem Always ready Rohit has alpha quality, but as I said the alpha quality should be should be usable Mostly new versions of packages are tested In Bodhi for example and users can give them karma I'm not sure whether you know body or not, but You can if there is an if there is a package and you are interested in the package You can open it You can give it karma write some comments if it doesn't work You can give it negative karma if the package receives three positive karmas then it will be pushed to stable if it doesn't receive any karma It will wait for one week. I think and then it's pushed to stable automatically This is Probably what the majority of packages tend to do like they are pushed to stable automatically because Nobody gives karma then of course this Leads sometimes two problems that a package appears in stable with some problems that could have been caught During the testing stage if you are interested in testing it Then you just switch on the updates testing repository and all testing packages are there So you install them through that repository and then you might Send karma one negative karma negates everything So if there is one negative karma the update will not be pushed to stable So it's good if you catch a problem on Rohide or even Fedora 36 Fedora release Updates and you want to prevent the update from coming into stable Just give it a negative karma if you can then Twice a year there is the Fedora release twice a year. So there is a Special time that's that's called branching out of Fedora Fedora gets branched and We prepare the beta release Which basically is the beta freeze? Then in beta freeze no updates are taken into into the release anymore unless They are they have blocker status or they have Freeze exception status Intense testing begins because We try to do as much testing as possible not just automated but also manual and then when we think it's Almost good. We create a release candidate and we vote whether the release candidate is go or no go Community can take part on each of those steps. There are meetings There are talks discussions and anybody is invited to to join the meetings. So if you think that that You need to say something and then that you Have some remarks or requirements You can just appear on those meetings. They usually Happen over over IRC on Libera chat The same holds true for final except that The intense testing is even more intense and Usually this is when we discovered the most terrible bugs and Then Fedora releases get get delayed The release criteria are Basic beta or final. I will just give you Some overview for example Here there is a oh It's not there yet Not there yet So Maybe Fedora 36 Yes, so Here you can see for example the beta release criteria For example What working sound one of the criteria is that The installed system must be able to play back sound with G streamer based applications We usually take that like it must be able to play sound So if your system doesn't play sound on beta It's bad and that is a no go And if it's on beta it means it's on final to so if the release doesn't play sound then You can't release Fedora The tests are defined by those release criteria. So we usually have a test case described how you should Set the environment how you should perform the test and what are the expected results So if you want to perform the test you can you just follow the instructions and At the end you know Audio works or it doesn't so if it works you just say yeah I think the test is passed because audio works if it doesn't you report bug and You tell that or you say that it doesn't work When we for example take a look on the installation test metrics, so it looks like this You can see that There are cells Okay, this might be a little bit funny for you or for people who are used to use some Framework to do testing It's the problem with Fedora is that One of the rules of Fedora is that everything that is used to produce Fedora must be open source so there is not a test framework that would be open source and suitable for what we Have here at the wiki So it's a problematic thing because we somehow think like we need to deploy a test framework and we should modernize and Nothing is available and We don't have the time to to program something by ourselves This has been Working for years and it's it's good It does the job But it's you know somehow Somehow very you know Beginners like or something but it does its job. So you can see that some of the tests are already filled in These are from the 27th May so that this was the 27th main night built and you see that most of the tests are performed by the coconut user that the coconut user is the automation tool that does the automated testing and There is a there is a joke about its name When it was deployed for the first time the idea was that we will go somewhere sit on the beach and drink Coconut liquors That doesn't happen But the bot is named coconut So you see that most of the of the activities Between releases are performed by the automated automated bot The problem if it's in the release criteria So if the release criteria is not met Then we call it its release blocking and that means you can't release Fedora with this Issue in it Currently, this is the list of the blocking stuff Workstation is blocking server is blocking cloud KDE For arm, it's the minimal Fedora or the X FCE And the blocker bug process of course is the process when we discuss the bugs if they break criteria or not and whether they are release blocking or not So sometimes it's like somebody reports. I have this fancy laptop With a fancy sound card and there is no sound and we say okay What about Lenovo laptops sound okay Dell sound okay the majority of the of the laptops are fine Most of the users report. It's fine. So we think this is not as severe as to block on this and It's not really is blocking all though for that user. It's a problem the results are written in that mattress We talked about this Oh The release is ready when all blocking tests have passed and no other serious issues have been found Anything else does not matter and this is important When the following does not work. We still do the release for example unsupported hardware Third-party applications or non-blocking desktop environments or non-blocking Fedora flavors If there is a bug in it, we still do the release It sometimes bites us terribly it bites the other team also because They say like oh, we don't feel like we should be blocking. We don't want to be blocking So we don't care We don't have time to do non-blocking stuff usually and then they say oh wait Silver blue doesn't work. It's terrible. There is such a huge bug. How come you didn't notice? Well, we didn't because we were doing Fedora workstation, which was blocking and not silver blue that was not blocking but there are of course risks that That We need to access somehow There is one fixed workflow because the automation only performs the test the only single way all the time Limited hardware redhead employees usually receive Lenovo laptops no problems on that if there are problems We find it but not on other laptops because I don't have 10 laptops at home for example Much more virtualized tests, of course, nobody wants to reinstall the system all the time or lack of resources and little coverage for non-blocking parts of Fedora, which is what we discussed a little bit earlier This is the f35 statistics taken from the heroes of Fedora heroes of Fedora is sort of Who did what and how many people did what and there is a You know like a The first place the second place the third place and so so the for f35 better There were there were only 12 testers doing the stuff this release validation stuff. They did almost 500 tests The most active tester one of them Did 164 so in average it was 40 and for final it was 21 Testers so which is a little better, but then they did only 35 tests in average the most active tester by the way, which was the same one did 216 tests and it wasn't me of course So we need you Fedora needs you and you need to test with us or Go against us or you know Please the only thing you need to do is use Fedora and use it early use it since better and Just use your favorite applications and whenever you spot an issue Think about those release criteria's if it's blocking let us know if you think it's fine If there is the test case if we have it in the matrices please fill it in and report bugs and If you do this then we will know about the bugs and we will have enough time to fix it If we don't know about this, maybe it will not be found Maybe it will too late. Maybe it won't you know, so if you want Please join You can contact anyone from the Fedora QA team For example come to Fedora slash QA at Libera chat You could block a report blocker bugs and vote for them if you want it and remember that the coconut Does a lot, but it can't do anything If you want to start somewhere, it's the release validation test plan You can read about how we do the release validation and where to join and the last thing is Especially for people who Like what we hear is you know, I am not that experienced in linux. Of course you all are but maybe your friend aren't If somebody is not that experienced in linux, it's a perfect stuff to get more expertise because we have those cases and they are described step by step What to do and it's commented so If those people test like let's test lvm how we do lvm They might learn how to do lvm using our tests And they can learn about linux. Okay. Thank you very much questions Do you have uh, do the test? Upgrade procedure because I know guys sometimes have problems with drivers on Lenovo with When upgrade from old version to new version mm-hmm Okay, we do The problem is It's here It's for the upgrade milestone beta Test case upgrade gnome software current workstation And previous workstation. What we do is we test upgrades the problem is And we test the current workstation, which is like f 36 So we will test updates from f 36 to f 37 and we will test upgrades From f 35 to f 37 The problem is I have two laptops And I use a limited set of packages on them because I am Keen in some applications and I don't care for the rest of them and I don't have them installed So I only see One and two real cases of upgrade That's the same for my colleague. That's the same for another one. So we might see like 20 or 25 different upgrades And they all pass Sometimes they don't coconut uses a fresh installation of f 35 for example So it installs the f 35 It updates it to the latest state of packages Using dnf update or something And then it either uses the gnome software or dnf to upgrade to 36 or the 37 the new release And that's mostly fine because there are no problematic packages on a fresh installation So this is exactly the case where we'd need people from the community to say I Have this it doesn't work Maybe it's this package or I get this package is Doing problems dnf by the way reports nicely which package is the culprit why it can't upgrade Yeah, nvidia drivers are a problem for everybody because nvidia doesn't want to give Their drivers to the open source community So And it's Therefore it's not blocking so we can't do anything about it What I recommend is If the nouveau drivers work for you use them Instead if you don't need the the nvidia like the full scale of nvidia And then it's mostly fine Yeah, but now maybe Somebody I read an article about nvidia Giving something Like a core module being open source Maybe it will help the situation but yeah nvidia Drivers might be a problem and this is what we as fedora cannot fix So, okay, I think That's it. Thank you very much for your attention