 Hey, so I'm Shumanthro. This is a do session mostly. So we are going to make sure that you run some of the tests. That's what the whole intention of the session is. But to start with, let's start off with the very idea of why this session is and why you guys are here. So the whole idea that started was we kind of test Fedora on different architectures and different hardware. And now we have generally me as a Red Hatter, I test it on a T460S and mostly most of the Red Hatter do that. So we hardly face issues, but when it comes to off the shelf community hardware, there are tons and tons of hardware where we keep getting issues which are not that great to hear about. But then they complain a lot about Wi-Fi drivers not working, some chips are not working. And that kind of tells us that, hey, what we probably need to fix some of those stuffs. So the whole idea of the session would be kind of to make sure that anyone being a Red Hatter, non-Red Hatter can start learning how to test basic stuffs and then report them back to the kernel team, Fedora kernel team. So essentially people start understanding, okay, these are the things which failed, these are the things which passed, things like that. So to start off with most of my whatever that I'm talking about was covered in the last session by Lora, half of it, precisely. Start with, let's go to this page. So there's this thing called kernel testing initiative, which people, so there's this initiative called kernel testing initiative which was started by Fedora kernel team. And this wiki page resides where mostly everything about how to test and what to test and what to look for kind of resides. This is the whole initiative programs which is there. Now this basically is one thing that I suggest most of the community people to get started off with when they are trying to kind of getting started with testing kernels. Now in this link there's one specific pointer which would give you a guideline sort of for regression testing and then it would point you to a bunch of test suits, so which is this kernel regression test suit. So it's a fairly simple process. There's a git repo which you clone and then you install three specific packages. So Fedora, Python Fedora for reporting, GCC and git for basically git for cloning and GCC for like whatever the process is when you start running this run test.sh. So that's what is the very bare minimum prerequisite that you have to start off with. So once you have done cloning this part, if you once you have cloned this you would probably end up having this kind of something like that. So this is going to have a performance where you're going to see a bunch of tests there and then there is going to be this script called run test.sh which you're going to run as a person like as on a VM or like on a bare middle wherever you want to test. So I ran the test and let me run it again, just for this. So it's going to take some time. So I did not configure the config file. You can actually put up your fast details over there which would basically directly upload it to the web app. I did not put it there but if you want you can just edit the configuration. The steps are pretty much in this document itself. Steps are pretty much this. And the document actually does need to be updated. There are a couple new things we've added to config now like the option to test the that the nvidia module works with new kernels or that it compiles against them anyway. And then if you could send me a list of changes I would probably do. Yeah, I mean they're actually in the config. It's just they're documented in the config. I just haven't put them on the wiki so. Okay, sure. So that's going to run and it's basically going to end up giving you a link to a text file which you could then upload. That's basically what the whole process goes like. Moving forward, say it has passed most of it. Yeah. So what I wanted to do with this is keep on having Fedora QA onboarding calls where new contributors come in and they kind of learn how to test multiple things in Fedora. One of the things that I focus on in those kind of places is I kind of focus on getting the kernel tested. So I have a bunch of community people in like in the QA team who are involved and we get the kernel tested and they upload their logs and they do it on a regular basis. So yeah, that's kind of great for us. But what we wanted out of it was if we could have that's what I proposed in the last session we're going to have a test day, a specific day where people will probably be testing things and filing bugs and later on probably we could mature that model to a bit where we can have this as a very specific calendar event for every release, every cycle, some specific time we have this test day running. So mostly like we have regular test days on i18n and cloud that's basically those are two test days we have for every cycle no matter what. So I wanted kernel to be a part of one of those test days which we run for every cycle for every release. So that's what mostly the session was about. The whole idea was to do a do session which is basically running it on the hardware. So that's what it is and I would like to know the feedbacks and how you think that we could help testing more rigorously and how do you think it would be it would be a more connect between we could have the non red headers and the people who use off the shelf hardware to kind of have more like idea about pricing like you guys are talking about if you can give me a kind of a rough sketch of how you guys do it we'd be more than happy to put it down to the family members in here or there's a specific thing that you can try it and yeah that that probably will help a lot going forward. That would be fantastic so there's actually a so there's like a kernel of bug triage page actually we've got the kernel page okay which walks through the basics of how to do bug triage there. For testing more things you know we've we've asked and asked our president at block what two or three times asking people to write and submit tests to this particular test suite. Now if you were in the last session you heard that a lot of red hats internal test are about to become external tests and there's there's an initiative there and those are going to be brought in and and some quite a few resources but just because we are throwing red hats throwing resources at those at this problem from a here's some hardware to run a bunch of tests on doesn't make test data from end users less valuable because they're not going to be running what the average it's going to be enterprise hardware it's not going to be you what the average users so we do need users running these tests and my my goal is to integrate everything that they're doing into this test suite as well okay but there's a lot of tests that will not exist there tests that you know if you've got a specific piece of hardware yeah you might want to run a test on it and so there's there's a page in the how to write a test uh wiki yes that kind of talks about you know you can search for a module is this module inserted and if the module is loaded you know that if the driver for this device is loaded then I want to go ahead and run those tests and if not you skip it you don't want to fail a test because it's not so um you know those types of things if the community gets involved they say well hey I have this piece of hardware I would like to make sure that it continues to work in this way great ready test uh and submit that okay for this test day stuff I know that Fedora QA is is much more concerned with release cycles the kernel though works differently than most things on release cycles we rebase frequently and so it would I think be even more useful and I'm wondering if it's too much of a workload to have those test days be every time there is a upstream linus release which is about once every three months I mean the reason I feel like 4.13 should be coming out soon right and it's probably too quick to organize a test day on that but possibly sunday possibly the week after so and and I know Laura and I are going to be in and Palmer is in LA the week after uh okay but yeah but the next one will be 4.14 and and roughly three months okay or 10 weeks yeah 4.13 by default and to organize them based on their release cycle but there's very little yeah I mean I was doing the work stuff I ran test days for bird they were based on release cycles because that was how you did it this is the new package that's going in and or the new set of packages is going in and new features for features going in that need to be tested and they weren't changing during the release yeah the next the next version will get it so when do you propose the this one to me so there's a website called phpcrystalball.org which is the idea of being able to forecast when kernels are going to release oh I thought you were joking no so the thing is an average kernel cycle goes with a two week merge window and then rc1 is at the end of that two weeks and every week after that you get another rc typically I would say they average 7.5 you get a lot of times you get rc7 and the next week will be final although it's a lot of times you get an rc8 and the next week will be final but like is the not or the question that I think the question is like should we do this between the final rc and when updates are able to release the kernel and don't so because of the way that the things work up from there I don't think that that's going to do is as much good I think it's the I think the optimal time is between final release dot two dot three okay so the rc is the best time no in our league no no release because there's there's the whole stabilization process it's a separate tree after that yeah well I was thinking from a timing thing since there's basically a week of variable there if we schedule them for two weeks after over php says that the release is going to be that's going to be somewhere in between actual release and and when we would consider rebasing right so we want to check her so we want to test it thing is also because this would be basically the same event every time we don't need an awful lot of preparation it's not like a brand new complicated test day that we've never run before where we do spend three months discussing how to do it right no if we do this every three months we can turn it into a pretty well organized thing where we can decide the day you know I think the the most work would be we would need to because you want to run on a stable release and not with raw hide and things like that and we do want people to be able to build external models or things like that if we need to so we need to build a version of the kernel for test day against that release and then we need to spin a quick image that's got a few minutes to do it and the next one I mean the coming one well the the coming one the problem there's there's a couple of problems here one is we're back for a week and then well you know what or maybe we could do one later in the 13th cycle yeah which would be abnormal except for the fact that 13 is going to be the release for 27 how you guys pretty much on release day have the kernel ready to go you have to rebase some patches but it's really not on limited release day yeah chances are we just did a snapshot on Friday and then you released on Sunday you might have put one or two patches and usually he doesn't usually the only that's true puts in this aversion change between Friday and Sunday and when he doesn't he apologizes for it yes so yeah it's ready to go to that by that point because we just built it on Friday okay it's not like there's a lot of work you have to do before four not 14.0 is package well there's so this this has to be a little bit of different because because of the way rebases work so it's not just quite a direct copy over there are certain things that have changed in raw hide that may not be enough point six or stuff like that so and especially if we keep going through and doing some config changes those config changes like turning off things we wouldn't turn off things and at 26 that's true even though we turn them off for at 27 because you know someone might actually be using me expecting the work but we're turning off a bunch of stuff and it goes through the whole raw hide cycle and then the whole release cycle and nobody's complaining that we know it's safe to keep that off but we don't want to do it make kernel updates so there's a rebase process there but it's it's not that is it a possibility that you would want to test more than one kernel as part of a test day more than one arch would be nice well arch is tough yeah it's not everybody has i don't know what you're sitting around but i don't know about one of the kernel with the stuff and then the kernel with the things turned off that you expect to turn off for the next release i mean i don't know i mean is that feedback useful to you well you know what it would be awesome if we have people who are everybody's coming to test a kernel yeah and we're trying to test release kernel it's going to be the rebase right right but hey on that same day which is probably right into the merge window could you also since you're taking some time to test that's the raw hide kernel yeah with the raw hide image and that way you get feedback not only on the kernel we're thinking about rebasing you get feedback as soon as the merge window is closed which is going to find i mean there's a there's a lot of yeah yeah there's a lot of cruft in there but it's useful information and it's a time when you can actually well i mean well it's a lot better than a bisect well it can be a you can be a raw hide kernel on our test server has never i just copy that that image to be the new fewer takers but you know a lot of those 412 bugs that those should have been caught if we had people well the policy wasn't the kind of thing you could argue that the you know yet we switch how to maintain the policies maintained it's reactionary you know maybe we should add a test if you're testing new yeah usually we there is in fact i don't know that i would add it as a pass fail test just add the report at the end of the model so that there is but that must be newer than the last time i checked that that might be very worth adding just so that we've got it if they're not yeah what can it hurt i mean well because policy used to be maintained by the same person who maintained the kernel something so those failures on there um are very interesting at 27 and raw hide fail immediately if you look at the failures the security signature failure something happened with the p so i don't know if you remember the debacle where we couldn't actually build kernels for a couple days because we couldn't find them so they're being signed and when i boot up the test system and run the test again they pass and all this is doing is grepping from a spring in the kernel so it's not i don't know why they always fail on this initial boot but it's only just it's only just two every time now go on too many machines now i'm happy to manually go and check it it's show the show the test that it's under a secure boot sb test it's very simple no it's nothing to do it's uh it's signed by pe sign using a process with hardware keys inside those two machines that are only a little bit but it is signing them correctly that the point is i can log into that exact same dm that's at the exact same kernel installation on and run this test again and it passes and as you can see what it's looking for there's no reason that it should give that better permission wouldn't ever change this exact sprint no i just actually will all can't run test again and it passes but it's been every time since they did that update every kind of window since that upgrade only on 27 or 28 which upgrade the p sign update oh pe sign okay i'm rooming with peter i might have to remember we do echo signer echoes the signer it's what we're looking for we don't know i mean show me a log of the test baby what that was a log of the test field what i've seen that showed that the fedora signer is fedora secure boot signer that uh what they might not remember what it says the right thing it doesn't fail it says the right thing yet it's still the only thing that i'm guessing is maybe something with pe sign is returning the right information but it's not exiting the right return code that's right top looking for signature fedora package for signature fedora secure boots on a then blank line and then we'll just look at this same url and i know you can know the control systems move the mast when everyone maybe there's a problem yeah no that's that's not right at all pees pees sign is failing no that is the output it's looking for which is it said in the doctrine big file so that this might be useful for other people who are doing their own tests uh it said in the config file and that is what it's looking for but if you see the echo signer is what we're getting the blank line yeah so yes it's not getting the output it's not getting the sign about it it's working for it all right but if i run the test again it and if you were to run it on your system it would be right about it at 686 64 i mean the debug in that though is you know just dump some more info what is pees sign yes that's right returning it doesn't get captured so it does get captured yeah that's no the um the return code doesn't though that's the thing you see right after and the rest of its output doesn't get captured either it's getting grabbed out so the empty line that yeah it's the echo sign which is the output that's where we should see the text that it can do that the problem is they don't like being too large uploaded to this off-size limitations on here and had something to do with people being able to upload like split some sort of a pirate binary or something and do tiny chunks or or yeah yeah in 1994 but that was what they actually was either 3b or or yeah but that the size of it's like tiny but std is not like yeah we could we could do it on that test but it only fails on the first we would be on those things maybe you can kind of clue as to why it's made well actually it's not even a reboot um we're all high so at 27 shuts down after the test for run those bms shut down but the rollhead beams are on 24 7 they never shut down they just your issues so i mentioned that i would be in this session on the planet and told people if they had like weird laptops and things that don't work sometimes those are usually bloody personal if anybody brought anything i know most specifically so that i wouldn't have that problem i've had a couple people come up to me and you know we we found one yesterday which i forgot