 We got a couple more people coming in, all right. My name is Justin Forbes, I'm the Fedora Kernel maintainer and have been for a disturbingly long time. What I'm going to go through today is, this is the first time we've done the State of the Fedora Kernel talk since NEST 2020, I believe. So there's been a whole lot of change and anybody who's working with the Kernel package would, well, it's ugly. It's ugly for a reason though. So what I'm going to go into in this talk is a little bit on the end user experience, stable kernels versus raw hide kernels, a little bit there. I'll go into a little bit of the Kernel contributor developer experience. And then I'm going to go into people who are debugging testing patches, things like that upstream and how you deal with that. From a Kernel perspective now, if you look at the spec, it's messy. It's not as clear how you do some things. So I'll go through that. I'm going to try this all in 25 minutes. So instead of having a nice flow, it's going to be a little bit disjointed, section to section. So the first thing is stable Fedora kernels. And if you're a kernel user and you're not a developer, you probably notice there are a whole lot of kernel updates. And why are we getting so many kernel updates? One of them is Fedora does try to stay very close to upstream. We push almost every single stable release as an update. If you want to know why we do that, there have been 117 kernel CVEs so far this year. Now, a lot of those CVEs don't affect everybody. Some of them only affect very specific hardware. Some of them affect older kernels. We've already fixed them, but just in 2023 alone, there have been 117 CVEs so far. One of the other things that happens is we get regressions as these updates go through and it's unfortunate. So we got to get bug fixes and try to get those regressions through. One thing I did want to stress here with karma is you are going to see sometimes, oftentimes we will push a kernel that I know breaks users, breaks specific users, right? If somebody says I have problems with my AMD graphics card and it's a specific card or something, I'm not going to hold back fixes for a bunch of other users just because of this one specific problem, this new regression that's come in. The idea is because we released so many kernels so quickly, hopefully that regression will be fixed soon as well. Now, there are exceptions. There was an XFS issue that could corrupt data. We did not push that kernel. We held on to it until we could get that issue fixed and get it pushed. Data corruption, things like that are not things that I'm willing to let go as a regression. But if a specific piece of hardware doesn't work or sounds not working right or something like that, a lot of times other users still need the fixes in those kernels. Another thing about the stable Fedora kernels is if you look at the rail kernel or the rawhide kernel, you'll see there are actually a few extra patches in that. There are some rail specific things and the reason those exist is because of ELN. We build kernels with most things in rawhide. It gets rebuilt for ELN and it's the same package. With the kernel, it's the same disk it, but the ELN kernel build is essentially a rail. It is exactly a rail kernel. It's got a rail config and it does have a couple of patches that we don't have. One of the things that I do as I make sure that when these patches go into rawhide, is anything that's rail specific, anything that is not wanted or needed for a Fedora, Fedora has to have if-defs so that that code path is not executed by Fedora users. There is a config rail differences config item that basically says we're just going to bypass all of these chunks of code that rail is added. Rawhide that's there. Running the kernel should be no different than it would be if those patches weren't there, but it does make it a little bit more difficult if you're dealing with upstream, say I've got a bug and I'm talking to upstream and they want you to apply a patch and the patch doesn't apply cleanly because there's this weird rail code there. So stable Fedora releases, those patches are removed. And part of the reason why is to make sure we stay close to upstream. So the other thing about karma is if you get a, whoops, if you've reported a bug already, if a bug say happened in 6.4.3 and that bug is not fixed in 6.4.4, negative karma is really not helpful there because what that's saying is my bug is more important than any other bugs that might be fixed by this. I'd need the negative karma if it's a new regression because I read all of those. It's good to know we've got a new regression, what that regression is, but for kernels in a row that you're applying negative karma to for a regression that happened five kernels ago isn't necessarily helpful in that regard. So that's kind of the end user experience part. And now we're going to talk a little bit about kernel maintenance. And the reason that kernel maintenance is important is if you want to do your own kernels, if you want to look at our code, if you want to do any of those things, it's much easier. What we're doing now is everything is maintained in a source git repository on GitLab. And that's the kernel arc repository. In that repository we have branches, there's an OS build branch which is the raw hide branch. And then we have a branch for each stable Fedora release. And when I say stable Fedora release, I don't mean Fedora 37, 38. I mean Fedora-6.4 is the 6.4 kernel series. And the kernel is the same for all versions of Fedora that support it. So 37 and 38 are both on the same 6.4 series right now. And it's actually one script that just updates the disk git for both of them when we do a stable build. That sometimes changes when you have a release going into life, like I didn't rebase Fedora 36 when we had less than a month left. So it stayed on the previous version. In that case, that doesn't mean we won't do updates. It just means I'm not going to do an update unless it's a security fix, something like that of some importance. It is end of life from a new code standpoint at that point. Another thing that's interesting about this, though, maintaining everything in source git, it's an exploded kernel tree. And we have scripts that actually generate the disk git, and they do it by creating a source RPM and then just exploding it over the disk git and adding everything again. That means anything that's done in disk git would be overwritten. Now, that's a weird workflow and I know a lot of people aren't used to it. So, you know, luckily we don't have people frequently committing to disk git in kernel. But we do, I do look at the diff every time. So when I commit, I actually have a script that does a commit and does a diff. So I'm looking at the diff before I push, and hopefully if anybody did change anything in disk git, it would be noticed and it could be picked up and brought into source git correctly. It also means, though, kernel doesn't really do pull requests from disk git on pager. We do have on peer. We do have merge request on git lab. We highly encourage them. Several people in the community have contributed and it's been a great experience, at least from my end, and hopefully once people have figured out the process, it's good on their end, too. One of the downsides to doing this this way, well, when you're looking at the kernel spec in disk git now, it's gotten pretty nasty. And I hate to say it, but it's going to get a lot nastier. And the reason that is is because we've introduced variations. There are variations that are there for RHEL now. There are variations that are coming for Fedora. So the real-time kernel is supposed to at some point be merged upstream. I'll say any day now. Clark would really appreciate that, but we don't know when it's going to be. It's been any day now for a year or two. When that happens, there's a good chance we're going to do a Fedora real-time kernel. There's a good chance that we will do a 16k kernel for ARM devices. Those things are all variations. And while it's very easy to manage them from the build side, it gets pretty ugly in the spec. Eventually, I kind of hope to break those things up and make it to where you don't edit the spec or the spec template directly at all. You're just editing chunks which relate to what you need to do to make it all a little bit more manageable. But we're not there yet. Now, the configs portion of all this. If you need to change a configuration item in your kernel, say you want to turn on a driver that we haven't turned on or you need to turn off something, any time you need to change the configs, they're terrifying. There's thousands and thousands of files in the Red Hat directory. There's a Red Hat configs directory. And inside that, you'll see Fedora directory, a common directory, and a RHEL directory. The idea is anything that is unique to Fedora is set in the Fedora directory, and RHEL might be set differently. So anything that would be unique to RHEL is set in the RHEL directory, but anything where they're set the same on both would end up in the common directory. Tries to make it a little bit more manageable, but it's still pretty scary. And it gets you a little bit more interesting when you say, I want to turn on the config option and you find no file there. If there's no file there, it means that adding one and turning it on will do nothing. The reason that is is because that when you build the kernel or when you actually, we have a script called make disconfig check even as well, that you don't have to build the kernel to even get to, it goes through and builds your configs and checks them and make sure that they're valid. If something is unset, it's going to say, hey, there's nothing set here and you have to set it in one way. Now, the reason that you don't have a file there, if something is unset and you're trying to turn something on, you know that's a valid config item, is because there's a dependency that it's missing somewhere. So if I don't turn on Netbender Qualcomm, for instance, there's four drivers underneath there that it doesn't even ask about, it's not going to check because I haven't turned on that dependency. So if you are changing things, and this is important because we even have internal kernel developers that stumble on this, it's just, unless you're dealing with Kconfig frequently, it gets somewhat confusing. If there's not a file there, it means you need to find out what that Kconfig option depends on and perhaps go change that as well. Once you do set your configs to what you want to do, there is a command make disk configs check which builds all configs for all arches and checks them to make sure they're valid. It will tell you if something is missing, sorry, it'll tell you if something is missing, it hasn't been turned on or hasn't been set in either way. It will also tell you if there's a mismatch. So let's say I set a config item as a module and for some reason, due to dependencies, it's forcing it to be built in and can't be a module. It'll say, hey, you set this as module, the generated configs are showing it as built in. You need to go fix that and it won't let you continue without that. Please, please, if you're doing a merge request for the kernel changing any config items, run this command first because if you don't see I will fail, it's going to have to change anyway. The other thing I'm going to say is help is on the way. I know how messy this is, I know how bad this is. Others inside Red Hat know how bad this is and there have been several discussions and there's even been, there's code being generated now by someone within Red Hat that will be made, that will be made public as part of this kernel repository where you will be able to manage configs in a much easier way. You'll have a way to say, I want to turn on this driver and it's going to come back and say, okay, as a result we also have to turn on these things. Is this okay? And that's it. It'll be a lot easier to manage but it's going to take a little bit of time. So I'm hoping by the end of the year, but we'll see. In the last bit, historically people dealing with Fedora kernels, if they get a patch from upstream, they want to test that patch. It used to be a lot clearer how you add patches within the spec file. It's still not horribly difficult. There are two ways that you can really do things. If you're working in disk it, you can actually just add that patch to a branch and make dist srpm and it's going to generate a source rpm for you to test. But if you really want to work within disk it or you're working from a source rpm itself, we have a blank patch called Linux kernel test patch. And you can take a patch or a series of patches and cat that to that file and it's already there. It's already applied so that you don't have to go through and mess with things that way. We also have an option in disk it for config overrides. So if you want to change something in your branch, if you're running a branch somewhere, you want to change a config. You don't want to mess with our configs about conflicts as you update over time. There's a directory called custom overrides. Red Hat configs custom overrides. Follows the same arch layout. When all configs are generated, the last thing that happens is we go through and say anything that's got an override, we apply that last. So the order of operations is common, fedora, and then custom overrides. And so whatever you set there will be the final thing set. Again, you still have the problem of worrying about dependencies, worrying about mismatches, those things. But at least you can maintain your branch with custom overrides and not have to worry about updating and us changing something. The merge is not working correctly. That one quicker than I thought. Sorry, I tried to rush through some of that and I'm happy to discuss anything you would like to discuss as far as questions on any of the topics or anyone? Yes. You said that in raw hide, the rel patches are applied. So if we're using raw hide, we're kind of using the rel kernel? No. Raw hide disk git has rel patches applied. But like I said, anything specific there, so like there's some code to support a different type of multi-pathing that rel used to support. Anyway, upstream has rejected that, so rel patch, they are caring for compatibility reasons or whichever, but that code that enables all of that is if deft in the kernel itself and it's under config rel differences. So rel and thus ELN has config rel differences turned on, meaning that code path exists. But raw hide does not. So functionally, the kernel is the same as it would have been on a stable Fedora branch, but the code is there. Anyone else? I looked at this whole workflow a few times in the past and it always struck me as very, very complex. I mean, I know that there are reasons for this, but like long-term, do you think that there's chances that it will become simpler or do we really need this complexity? Some of the complexity I think is required and some of it we can simplify. Like I said, the config stuff I know is a nightmare. We're trying to simplify that. We're trying to break out the spec in ways that it's easier for you to do a custom variation, do custom kernels, things of that nature. But as far as the doing a merge request through GitLab, that whole workflow, it is kind of I think what it has to be right now. Now part of that is rel requires controls on everything. You know, they have, there's a three act system, all of these things that have to happen for rel patches. And what you'll find out is like if you submit a merge request to Fedora 64 or like any of the stable Fedora branches, all that it doesn't go through any of that process. If you do a code change in raw hide, it does have to go through those processes because it requires acts. And then if you do config changes, as long as it only touches Fedora, again, it bypasses all that process. Now, I also have a bypass lever because I don't actually build kernels in raw hide out of OS build. I build it out of a directory called arc latest. Excuse me. Arc latest is regenerated every night. And what it is is the contents of OS build plus any patches that are not or any merge requests that are not merged, but they're listed as include in release. And the reason that's there is so that I can make sure that Fedora is doing what's best for Fedora without having to wait for rel acts or rel process for those bits. I still have the ability to react and do everything that I need to do and what's best for the Fedora community. Now, everybody on the teams that I work with internally are certainly most of them are actually they're all running Fedora. So they're happy to they think about Fedora, they're happy to have things going for Fedora. But at the end, they're worried about getting code in and doing things for rel in that rel process. My only job is the Fedora kernel. And so that's, you know, I'm the the lookout to make sure, hey, we can't overstep bouncer, we have to make sure that we have ways around all of this to do what's best for Fedora at all times. Is it on? Yes. When can we expect your new book then in the art of kernel maintenance? I don't know that that would be quite the book, but let's say when I started kernel maintenance I had a lot of hair. About upstream and not upstream bug reports. So what is useful to have in Baxilla report and what is not useful assume you have a wireless car that not quite works as an access point. Right. Is it useful to have the Baxilla report there about it? I assume it would be in upstream but I don't really know. So we try to stick as close as possible to upstream. And yes, most of the time it is going to be an upstream issue. Now it's good for us to know that there's several people with these types of issues because one of the things I have to do is watch for patches upstream. So I don't want to bring in patches to the Fedora kernel that have been rejected upstream or have not gotten any comments or anything else. But I am happy to apply patches that fix bugs before they've made it to a stable release upstream. So I'll backport patches from Linux's tree to a stable release or even from Linux next to a stable release to make sure there's a lot of users. Yes, ideally if you can deal with upstream directly it's going to make things better but I still need to know that it's happening. Okay. Anyone else? I've recently started using Rawhide more and I've noticed when I do a system upgrade there will be what looks like two different versions of a kernel or multiple in a day. Just for an example it was 5.0 RC3 and then a long number afterward and then later on in the day I was just checking for other updates and it was the same 6.5 0 RC3 but another long number could you explain that? So the long numbers are the get, it's the get commit ID that that kernel is based on from Linux's tree. Doesn't matter what we've changed on our side this is Linux's tree merged in. The reason you might see two Rawhide kernel builds in a day is it just depends on what time of day the kernel was built. I try to do one every single day but it might be the first thing I do in the morning or it might be something I do later in the afternoon so if you update in the morning you might get yesterday's kernel and then you update in the afternoon you might get another kernel that way but usually it's just one a day. The only time that I would ever do a second is if there's something that's a major major security update or like I said if there's reports of data corruption or something like that that are serious then I might if we've got a fix for that we might push that quickly but usually it's just one a day. It's usually also built after the build usually it'll land after about 9 a.m. central time in the U.S. so well the right right the the other thing we have a script that runs every day that merges in Linux's tree to OS build and that script runs at 4.30 a.m. is my time and then then script an hour after that that builds the ArcLatest directory so even if I'm traveling the scripts run when they do I can manually trigger them if I need to but I usually don't I just wait till they run. Hello I work for the QA team and when I look at kernels at Bowdy they usually updates the stink usually they get a lot of karma quickly a lot of feedback do you have any like approach that you take how long you keep the new kernel in updates testing before you push it? Yes and no so one of the things we do we turn off auto karma we don't want the auto push we realize years and years ago that we would you just skip updates testing we usually get enough karma to push immediately so the typical wait now because you're doing one to two kernels a week is it sits in updates testing no less than one full day and we'll push after that but a lot of times we just kind of wait until the next kernel comes out and we push to stable and then that one will go to updates testing obviously if there's a critical security bug or anything like that then we push it through as quickly as we can sometimes even with Kevin and QA helping out we end up with heroics and people going above and beyond to make sure that we can get a serious security bug tested push to testing given karma and push to stable within number of hours. I'm asking because sometimes it happens and I think it happened recently that some part of our user base is affected for example certain AMD cards no longer boot and usually you can see the negative feedback only after for example two days three days because it doesn't affect everyone so it will take time until the affected people update realize report the feedback so if for example new releases you wait for at least three days or like what's your approach to minimize the impact that so that was an interesting one I didn't want to push that kernel when I did the idea is yes you give it at least a day in updates testing and usually you'll get feedback by then AMD issues originally I thought it affected fewer cards than it did it turned out it impacted more cards and so there was a second kernel that was never pushed and then we had another update that came in that fixed some of the cards but left some of them still broken unfortunately there was also a critical security update there and I mentioned that in the body when I pushed it we had to push it because of this I'm sorry we'll try to get this regression fix as quickly as we can okay and I do have one related question extra so if I see some people complaining on discussion or somewhere that they have a hardware related problem and it seems to be like really specific it's not like tens of people complaining about the same thing what's the best advice that I can give them like report back in bugzilla or reach the upstream if possible but it's much harder to talk to the upstream especially kernel it's somewhat easier with Mesa and stuff because they have GitLab and everything but for kernel and mailing lists and everything it's very difficult for regular contributors so what's the best approach to do Red Hat Bugzilla is probably the best approach there so we wish you could work with upstream but I understand it's difficult for me even to remember this upstream used like for great you want to file the upstream bugzilla he'll use it but you go to another sub system there's a kernel bugzilla yeah you can enter in a bug for it but they'll never look at it but every sub system has a different way of working so that's kind of one of the ways we have to help I'm out of time out of time you can ask the question I'll be around all week and happy to answer any questions or take any contributions do we have Miro Slav here already Miro Slav can you please all rise thank you hello thank you everybody so he's going to speak about the case of SBTX versus Fedora thank you it's usually habit that you sit down when the jury say you may be seated but you are forgiven so members of the jury your duty today will be to determine whether the defendant is guilty or not guilty based only on the facts and the evidence provided in the case and we will be talking about the case of SBTX versus Fedora so thank you for the coming and thank you even the audience will be able to see what's going on in the case of SBTX or SBTX or SBTX or SBTX or SBTX or SBTX or SBTX or SBTX or SBTX or SBTX or SBTX or SBTX yeah he made the system out of the blue because in that time what was the time Tom? 2003 yeah and there was no standard and probably should speak on the mic yeah and in that time there was no standard about licensing at all so when the license was GPL version 2 yeah Tom said GPL v2 string for that, and we live with that for a long, long time. Much time later, we had some problems with the software. That's when the stuff and Sbom started. To standard right now, we have SPDX and Cyclone DX, which is used for the Sbom, and one of the things for that is the licensing. So licensing starts to be standardized, and SPDX used the identifier for separate licenses. So when you use the SPDX identifier, you know exactly what license it is, and it's never a whole family of the licenses like we had in the past in the Fedora. By the way, who knows what Sbom is, and who doesn't know what Sbom is? Okay, so we have some people in the audience. I may refer you to my presentation on devconf.cz this summer where I spoke about what the hell is Sbom, but it basically is something like that, this picture, and it describes your system. Because usually the requirements for the system is just the top level, and you never know what's below. So it's the map. Some people like the metamorphosis list of ingredients, and you know where on the end of the list is the chili. It can spoil the things. Yeah, there's a question. I prefer saying your honor. So in this picture, the fun is of this teeny place, but from the Sbom things, we actually don't thinkers. Sbom doesn't care about sustainability of the things, whether it's sustainable in the future or not. You just describe the system and to note that, for example, this block has some security vulnerability. It just provides you map of the system and know what exactly there is. A lot of things for the Sbom, you can fetch using rpm-query all, and you get all this information. That's like name of the package, upstream website, what files it includes, et cetera. But one thing I think is the licensing, so that's part of the Sbom. And that's why we are doing the change in the Fedora itself, to move to something which is standardized, because every software right now in the world, which do something with the licensing, use some standard, and for what I most see is the SPDX. So if we want to continue further and be first, we need to use the SPDX. So last year, we started Initiative and a Change Proposal to Move to SPDX Identifiers. This is where we are right now, and the blue part is which text we already converted, which is pretty awesome. We are at 40% right now, which is a good number. It's starting slowing down a little bit, but we are doing a lot of stuff. We are adding like five new licenses to SPDX every week or something like that. So it's pretty fast for a human, slow from the whole distribution point of view. And the yellowish and reddish part is the part which we have to finish yet. And the yellow part, I mark it as trivial conversion, which a lot of people hate me for that, because that's trivial from technical point of view. But when you start speaking to lawyers, it doesn't seem to be as such trivial as it seems to be. One of the reasons is that we are changing how we evaluate license. Previously, you can evaluate the license, and now we don't evaluate. So some systems say, projects say, this is GPL version 2, or if you are shipping with Fedora, it's MIT. And previously, you may choose MIT because that was for Fedora. Now you should say GPL or MIT, and it depends on the context. So right now we have to evaluate all the licenses as well. So the trivial part is not so trivial, and the red part is not trivial at all, because we usually don't know to which part it should convert. And the non-trivial part looks like something like this. So if you try to convert MIT and BSD license string, we give you a hint that it's one of the MIT choice and BSD choice. And you actually evaluate the license because previously in the Callaway system, BSD, MIT and other licenses referred to whole family of licenses, which is right now, not possible. And now I want to meet your defendant, and it's a form of quiz, and you will be the jury and decide if it was the right answer. And I want to ask you to all stand up, please. And here we have the question, and this is one of the change-owners. And who is this guy? And if it is David Kanshark, raise your right hand, and if it is David Kanshark, raise your left hand. And I will shortly reveal the correct answer, and if it is incorrect, please sit down. If you guess it correctly, you can stand down. Credit for this quiz go to Radek Vokal, who introduced it in DEF CON. So guess who is it? David Kanshark or David Cantrell? Raise your hand. The change-owners. This is this guy, this guy on the board. It's not so visible. And it's David Cantrell, and he's here in the... Not on the board. And second change-owner, is it Emily Lovejoy or Gillay Lovejoy? This one will be tricky. One, two, three. It's Gillay Lovejoy. And who is this? Is it Miroslav Sukhi or Marek Sukhi? It's Miroslav Sukhi and it's me. And now slightly different. Who is Richard Fontana? Is it the guy on the left side or the guy on the right side? Both are Richard Fontana. This is Richard Fontana, the lawyer, and this is Richard Fontana, the actor. It's easily like Richard Fontana, the lawyer, is working on the change proposal. But this change was tricky and both questions are correct. And when the migration to SPDX in the whole Fedora will be finished based on current approximation and estimation, will it be sooner than October 2024 or later than October next year? And this is hard. And it varies because the estimation started to be summer 2024 and you see the graph was slowing down. So right now the estimation is December 2024. So you can do better if you migrate your packages. And so some people are still standing. So last question. Oh, sorry. Do you like the migration to SPDX identifiers? Yes! The correct answer is yes. And yeah, I was looking actually for standing ovation but you didn't get so many questions, so you are not standing anymore. So my fault picking doing that. So yeah, so that's basically all of I presented because I can speak about it a long, long time and I reserve more time for your questions. So what's your question? And then there we have. It's not so... I will rephrase your question. Okay, thank you very much. So my question is this change only for the licenses or you will publish the SPDX files with or the requirements provided and things they get to the Fedora in the format that will be accessible to the scanners or security scanners and things they get. Uh-huh. Because sorry, tomorrow I have a little bit of a supply chain in Rail and actually Redcat has already started to provide in Beta in one of the portals so I'm curious how it looks in Fedora. So again, the question is like... If Fedora provides the SPDX but not the license thing, the packages but the SPDX has the files for the individual. So as far as I know, Fedora provides SBOMs for the containers made of the Fedora but not for the whole distribution and even the SBOMs for the containers are very easy and not going down into the details and just for the others, like the SBOMs can be really, really huge, it's a huge rabbit hole because you can have SBOMs for the artifacts that's what we are actually producing. Then you have SBOMs which is for building systems which includes not just the map you've seen but other maps which is required for the building, the packages as built requires. You have the SBOM which includes stuff for designing so other stuff which you need to for design it before you start building that. For example, Enscape for drafting the logos and the other way around you can have SBOMs for deployment because your system may need Postgres DB but it can reside on the different system so it doesn't need to be tracked in some scenarios but in some scenarios it needs to be tracked or S3 which is provided by AttaVendor somewhere on the internet so when you say providing the SBOM it's a really wide area but for our change proposal I don't dive into that because that's a really huge topic and right now I'm and we the change owners are focusing just on the changing the license identifiers even that is a huge topic and what we are and even in this area there was a proposal that we do the analysis on every commit to this grid which is a really great idea and one day it will likely happen and there are some tools that actually do that for example, SUSE has the cable system which we probably plan to use or investigate but if we do that right now we would have to work and build the system for both the Callaway system and the SPDX systems and somehow make it live together which will be tricky so I pursued others that we first make great everything to SPDX and then some two years later maybe is my expectation we will do some system make some system which will warn you every time you do change that hey there may be new license or some license disappear so this is the future in the future Federa probably I have a second question but I don't want to I will give chance there and then maybe back to you My question is a bit twofold I was wondering if you have any view on what the most common licenses are in Fedora right now since you are going through all the changes and doing the statistical analysis and I was part of that What is your view on the proposal that has appeared recently to follow suit what Debian does and just sim link to copies of common license tax instead of having multiple copies of the same license So the question is common licenses I don't dare to say each license is common in Fedora We have a lot of MIT, a lot of GPL version 2 or later, a lot of BSD I've seen some pretty long list licenses like 800 characters long license identifier Right now because we are in the middle and we mix the call away system as PDX and the call away system itself it was not defined in any form but one wiki page So later when we finish this work it may be actually easy to determine the whole license for all Fedora distribution because there is a library which evaluates the licenses so we may put all the strings beside and concatenate with all operators and it evaluates the license for all Fedora distribution so that would be awesome and the idea about linking to common licenses actually last week Richard kicked this off on the Fedora legal list So it is proposal it has some problems like what is common license, what is not so for others like GPL version 2 may be common license and you may link to that but there are some other licenses which has the credentials contribution like at the top copyrighted 2024-3 by Miroslav Suki and it's part of the license and license checkers ignores this contribution thing but the license itself says it's free if you include this license in exactly this form in the software so you can't actually link it to some common example where here comes contribution because there has to be that contribution Yeah, yeah, yeah that's the part where things start complicated and whether it's sufficient to link the header or whether you have to include the part it's tricky and right now we have like 20,000 packages to go and we still defer some stuff like copyright only or CC0 already debatable stuff so we just noted and we will still have to do some fixes later in some cases so there is a lot of stuff and not sure whether like saving one kilo of text is right now the biggest priority but yeah, people are trying to address that and it's fine and I welcome such an issue at it Okay, my question is also because there are multiple formats for SBOM and why you choose one over the other for example as you have the second case and why you choose this one, SPDX instead of Yeah, so the SPDX actually was not SBOM's thing at the start it was the license management project so at the start they have huge list of licenses and as far as I know they are still the only one which actually care about the license is too much the Cycon DX is the new one and it's focused on the tracking vulnerabilities so they don't care too much about the licenses and they care about the vulnerabilities in the blocks so I'm not even aware whether they have some list of the licenses they don't defer it so much like the SPDX so my wild guess and I may be incorrect is that in few years even the Cycon DX will use the SPDX list because it's really comprehensive and very well maintained So can I ask Mike? Okay, your honor I was investigating one of my victims and there was a problem in evidence and in layman terms, is there some tooling that can help me with converting my packages to the SPDX valid license? Yes, thank you for the question and it's in the package license validate and if you're on license of Dash Fedora to SPDX with the old system it will tell you to which string you can convert it but there's a problem with license evaluation so you may run the checker and we have license check, scan tools a scalone and every tool has its own problems so there's no thing which I can point this is the best tool you can use and we are still evolving and the ecosystem is evolving Your honor, I just want to add to that one annoying feature of that tool is if you already have an SPDX license it tells you it's not a valid license string I may address it after this presentation because we are already at the end I'm aware of that so we are at the end of the hearing thank you for the coming Thank you very much Thanks a lot, that was fantastic The next we have Tim Flink No Sorry You are the one, right? I'm sorry if I... Don't worry, I was called worse I'm sorry Thank you very much I might just... Fuck it, I'm after putting off the whole thing Sorry Thanks Yeah No Hello everyone We are starting our next talk Please settle down at the back Okay, so please welcome our next speakers Effie Maloney Ifa Sorry I tried Okay, I'm brilliant at that one So the topic is State of Community Applications and Infrastructure Thank you everybody, all the best Clapping before you begin, excellent So I'm not Effie, I'm Ifa But it's close enough, people have called me worse Welcome to the talk on the... Actually Question, because we like quizzes here So can anybody tell me what team takes care of the community apps services and infrastructure for the centres and for our projects and if you're a member of that team you are not allowed to answer I'm not a member of that team Yeah, Nick got in there first Go first You get a prize Yeah Prizes will be thrown Yeah, so Community Platform Engineering Team This is a discussion just around the overview of the team, what we do and Akash is going to go into a bit more detail then on the applications and services that the team actually supports and maintains, etc. So, like I said about us, we are Red Hat's sponsor team We are all paid by Red Hat to take care of the Fedora and Sendos communities and it is a very nice team to be in because everybody who's in that team really does care about those projects and it's nice that we get paid for that We cover a lot of ground on the team We have operations, day to day stuff We do some development into apps and services that may either fall into a little disrepair and we also then maintain lots of apps and services Like I said, we cover a lot of ground We have a couple of little teams in a bigger team that we all make up together We have the Fedora infrastructure team We have a Sentos infrastructure team of one Fabian We have our release engineering team and then we have the initiatives team that spring up if we decide to action any pieces of development work We have some cool stats to share with you as well today and the presentation put together by Akash is all very cool laid out with lots of data and metrics and things So this is the first in-person flock since 2019 It has been a long time since everybody came together for the Fedora project So it's nice to be back, it's my first flock So it's nice to be experiencing it This time you've chosen to come to Ireland So thanks Yeah, three years, 11 months, 25 days Too long to be together, but we are and that's the main thing In the Fedora infrastructure team there's a lot of work that gets done This is just a very quick overview of the amount of work that the team actually powers through So there's been over 3,000 tickets closed from last flock to this, like that's averaging about a thousand dollars a year Crazy numbers, but that just proves how dedicated the team is and how determined they are and helpful with Fedora to support the infrastructure that it's built on Out of that, we do have 16 tickets that came through as requests that actually ended up turning into the initiatives that they were much bigger than actually one, two people for a week, etc So they got passed to me In our Centus Infra So for one man band, he's getting through over a thousand tickets in like a year, which is good going I'm sure there's more, but there's a lot of work that gets through and from there we have four tickets to release initiatives as well Release engineering, again these teams are like we're talking single digits for people There's no double digits, the team itself is made up of double digits, I think we're 20 odd strong, but when we're talking about Fedora and for Centus Infra, release engineering, etc it's a much smaller subset of people So to see them in the thousands and the ticket closures is pretty impressive What does it all good mean? So a bunch of times the things get reported and by the time we look into it, we discard it all, it's recorded by itself, magic sometimes exists in the real world We had 146 magic stuff happening, issues that fixed themselves over the course of the last three years, from the last 10% to this one, that's what it means Thank you Finally then we have the initiatives teams, since I joined CPE about three odd years ago, we've managed to get through 14 initiatives, which is pretty good going but we are always open for people who if they'd like to submit an idea for work that we could potentially do, you can submit it in, there's a repo on Pagier, and we always proof the initiatives before we started and you can find that on the Read the Docs, they're called Fedora Arc Team We won't dwell too long on it We'll see you in a bit Thank you Do you want to talk to the projects part? Yep Thanks, Ifa. These are the projects that the community makes use of, and the community of contributors that we're talking about sure, we do provide our offerings, we are websites, but there's a bunch of things that goes into it, right? Things like building the packages or, well, we were just discussing with Justin outside how we plan on undertaking or we plan on maintaining, let's just say, so we maintain a total number of 74 projects out of which 52 of them are used only by Fedora project, 9 of them by CentOS project only and 13 of them by both the communities We would like to divide these projects into certain categories and that's how we are able to understand how important these projects are to look into say, if they are very important and we might as well lose our sleep fixing it, or if they're not important at all, maybe we'll come back on Monday and fix it, right? Say, four classes, critical one these are the application and service that we developed a code for and we maintain our infrastructure, really, really important for Fedora Linux to be built and distributed like the way it is, right? Anything happens to them for a longer period of time? Oh boy, we don't have Fedora Linux distributed the way it is right now then the critical two these are the ones that we don't write the code for but it's maintained in our infrastructure, we deploy it, we really want to make sure that it has as much uptime as we can make it then the standard ones are the ones that we develop the code for, we maintain our infrastructure, but yeah, maybe it's not as important as the other ones and finally ecosystem ones, these are the ones that well, they enhance the quality of life for the contributors, but they are, well, let's just say not as important. So critical first class, 20 projects, 12 of which by Fedora project only, three of them by CentOS and five of them used by both the communities and well, like I said most important and well, these projects make use of Python, JavaScript and C, so if you want to participate in maintaining these projects, by all means come we can use all the hands we can get and well, if you are aware of these technologies then well, I'll say that we will make a great team. Right, so which projects do you think would fall under critical first class? Like what the hell would be so darn important that people can't live without it? We have one hand raised. Come again? Well, let's find it out. Oh, both he is actually on let's say. Awesome. Yeah, we could throw it but we don't know if it's going to make it to you. Right, so we have these applications that are used by only Fedora project the list that's only used by CentOS and then the ones that are used by both the communities. Go ahead. Come again? Sorry. Oh, okay, right. Cool. Wow. I mean, let's just say that it probably includes both of them. Right. Yeah. Just a listener and not the results so I think it should include the both of them. Yeah, Kojik might come into critical too, I think. Because if we can't yeah, how can we push stuff out there, right? Where does the updates come from? Unless the updates are coming from somewhere else that we don't know of. Right, so these are the statistics for the technologies that we consider for applications. You know, if they have test cases, how many of them have them messaging schemas? How many of these have documentation and as you can see that the numbers are something that could use some assistance and say if you are really looking forward to contribute, I mean not just in the code base but in documentation as well, you're totally welcome to do so. Right, so a question time. What is the purpose of Fedora messaging? Does it help Fedora project contributors to communicate with each other? Or for them to communicate with Linux users? Or for them to keep track of the important contributions that the community members make and finally, to assist each other to send messages when they are really thankful about it. So let's see some hands if anyone knows about it. Let's find it out. It is indeed see. That's one good thing. Right, so it is see moving on to the critical applications too and 21 in total 10 of them used by Fedora project only, 4 of them only used by CentOS project and 10 of them used by both the communities. Right, so is Koji going to make the list? I wonder. Any guesses which project do you think would fall under this critical second class category? Let's find it out. Any more guesses by the way in the meanwhile? Go ahead. Koji is actually used by both the communities. Look at the bottom of the list. So Koji is there and it is my fault. We kind of have to award them. That's fine. I have to. We could actually throw them. There you go. Perfect. So these are the applications that we consider critical to. These are the ones used in Fedora project and CentOS project as well as both the communities. Right, so now that we are speaking too much about Koji, is it used for making Koji and comfortable homes by the KD contributors? Is it what it is used for? Is it used to make fermented rice pastries for the Fedora project contributors? Or is it used to make RPM packages from the source specification that's fed by the packages? So finally is it the official build system of the Fedora project and all the downstream communities? What answer is the most fitting for the purpose of Koji? Anyone? Come again. D. Right. Say, is it D? Yes it is. Right, great. This was a little twist because both C and D are pretty much the neighboring options, but Koji does a lot more than just RPM packages. It can build OSMHs, bunch more stuff. D it is. Right, now we're coming down to something that's a lot less important, the standard applications. These are the ones that the team write the code base for, but let's just say that they're not as important as the critical first class, critical second class are. 17 projects in total, 14 of which are used by Fedora project, two of which are used by CentOS project and one used by both of them. So, just because the importance is lowered. Okay, let's, we'll allow you to make a guess. What's not so important but very important? Efferman? Okay, let's find it out. Hmm, do we see Efferman in the list? Yeah. No, it's not, right? You'll have to give us something. Yeah, we could have waited for some other time, but then again these are the list of applications that are in the standard category and yeah, you see that these are the ones maintained by the Fedora project, used by the Fedora project and the others as well as both the communities. Well, what's the use of Fedora plan and now that we maintained, well, mentioned of it in the standard first class, is it like to, you know, official fork of Pandora, but Pandora is an official keyword so we can't quite use that name no more. Go ahead. Right, this is a truck running people over and then we get teleported to Fedora plan it. Yeah. Fedora plan it. Okay, just to be clear, we don't endorse getting run over by a truck. Yeah. Which one is it? Let's see some hands. We can't just echo in A, right? We don't have that many swags. Oh, you had the opportunity. We won't let you get another one. Yeah. Anyone else? No. That's wrong. Yeah. Well, let's just say that we have one back for you. Oh, yeah, back in the day. Yeah, no, it's definitely not B. It's A. So, yeah, that's the correct answer. Say one for Samantha. There you go. Say the last part is the ecosystem applications then reach the quality of life for the contributors. It seems like CentOS project contributors are so motivated. They're so motivated they don't need those apps no more, right? So, yeah, let's hear it. Go ahead. Let's find it out. Okay, let's have some more answers before we move on to the next slide. Badges. Badges? All apps are important. Right, there you go. Good. Well, let's hope. So, do we see badges here? Oh, probably sure. Do we see kernel test? Oh, we see that as well. Sorry, what did he say? Open QA. Oh, we don't see it here, do we? Let's just take that as an outcome for this talk, right? Yeah, so we get one for that. Oh, Nick got one, yeah, nothing for him. Right, so those are ecosystem application, right? What's the purpose of meatbot logs? Do you get some artificially oak logs cut it into small pieces and print the, I don't know, the artwork of Swordbot? Is that what it's used for? You know. Yeah, I'm going to let you make that pull request. Right. Or is it for people to catch up on their recordings because, well, we like to have meetings and video calls, is that what it's for? Or finally for people to catch up when meetings happen exactly in midnight, not a minute before, not a minute after. Let's see some hands. Good. Is it B? Let's find it out. It is indeed B. Right, cool, awesome. Let's find it. Oh, yeah, I mean, look, well, they are recordings of some kinds, but maybe I had the video recordings in my mind when I wrote that. But yeah, that was all about it from our side. Thank you so much for being here. And do you have any questions? Oh, Sandro, that's one good. Question depends a bit on who made the slides, but I suppose it was a contribution or consideration between the two of you. What the question is for Akash, what happened to the footnotes? You know, the internet connection is slow here, so we're still waiting for the footnotes to load. Trust me, it's here. You just don't see it because it's not loaded yet. Right, any more questions? Go ahead. I just wanted to kind of put in a plug for Kevin's talk or workshop on Friday. Is it? Fedora infrastructure just CPE. There is people from the community can be involved too. I've been involved with infrastructure for a long time since Mike McGrath did it, which what, that's been a while now. So there's going to be a infrastructure onboarding workshop on Friday afternoon, I believe it is. I just wanted to mention that if anybody is interested in getting involved, come join us. Most definitely, yeah. That session is something that you really want to join if you work for Fedora infrastructure or you are looking forward to working with them. Any more questions, comments? Good. Careful. Where are we with getting rid of some old things? We still have the old message burst. What was the other one? We still have PDC. How are we doing with getting rid of things? Careful, Kevin. We are getting rid of things, we are doing it silently so that no one knows who got rid of it. One of those things that we took care of the last month was nuance here. It's the supplemental wall paper voting thing. We had it around for a while but we just figured that no one used it since 32, I guess. Correct me if I'm mistaken. We decided to ask folks who predominantly use it, design team and other folks if they want to use it. They wanted their things back. We are trying to get rid of stuff slowly. We are getting there. I think one of the things that we are really trying to get rid of is Fed message. Certain applications make use of it like badges so we are really trying to build it to not use it. There is Fed or messaging scheme available if you are an application maintainer and you are still dependent on Fed message. I would encourage you to make use of those schemas, import them into your application, hook it up into the new FMN service that's available that the team worked hard on and is now live. That will allow us to close off the older Fed or messaging notification service too. Plug for Fed message or Fed or messaging schemas. If you are using Fed or messaging or if you plan on using Fed or messaging please write schema. I don't think we have any more questions and I think we are at the top of the hour right now so thank you folks for joining again. Hi, everyone. Hope you had good lunch. Now the next session is on using AIML to process automated test result from open QA from Timflink. As you said, I am Tim. I am a big believer in informal presentations. If you have questions, please ask them. I was a little unclear on I am still going to find out whether I put too much information, not enough information. If you have questions, please feel free to raise your hand. If I don't see you, please interrupt me. That's fine. I don't really care. The whole point of this is to communicate and sometimes that's the best way. What I am looking to do is do a bit of introduction, talk a bit about neural networks, go into the experiments or the experiment that we have done, where we are looking to go in the future and leave some time at the end for some questions. Out of curiosity, is anyone coming in here hoping that GPT was going to be making the presentation? Just me. I was kind of hoping, but yeah, it didn't actually work. Just to kind of get started because it's a very popular term right now and in my opinion there's a decent amount of misunderstanding of what it all can cover. What I am looking to talk about is in general an experiment that we did to figure out whether or not it's going to even be possible to be using AML to do triage. And it is in particular using data from tests that are visual. And I just want to make that distinction because a lot of testing is done primarily in the text domain. And the techniques that are done there don't necessarily work with visually oriented test and vice versa. Just to be clear, it's not about, we haven't created this magical tool to automatically triage test failures. Maybe someday, but we aren't actually to that point yet. And nor will, in my opinion, what I'm about to talk about be able to completely replace human triages. This is, in my opinion again, a good way to make people more efficient to help make their jobs easier and make their lives easier. But at the end of the day it's going to always rely on some form of human expertise in order to actually get the job done. So the system that these tests are running in is OpenQA. OpenQA is a system that's primarily based on computer vision. What it's really good at is it takes a screenshot of a machine that's running, and it has a little picture and you basically tell it, look on the screen anywhere for this picture of, for example, a button. If you find this picture or this little snippet somewhere in the screen, move the mouse over there, click on it. And you create your test cases based off of that. And consequently, the information that comes out of it is visual. Yes, there is some text that comes out of it, but it's primarily in the forms of screenshots and it also produces videos of the entire run. The system itself started with SUSE. As far as I know, it's almost the way they do all of their automated testing. But it is Infador now and it is a critical part of our release process. Every update in Bode goes through OpenQA and at this point, every brahide update goes through OpenQA as well. It runs a lot of tests. This particular idea, this started sometime last year, there was a chronic issue within OpenQA where tests would fail unexpectedly when Firefox, which was being used for part of the test, would just quit. And the test would fail because Firefox quit. And we tried looking into it. It took a while. When this started, we hadn't found the root cause. So we had a whole bunch of tests that were failing for reasons that we weren't testing for, which is not terribly useful. So the people who were doing the triage would go through. They'd recognize that this test had failed for a relatively common reason and would mark it as for rescheduling. And the test would be rerun and tell it either passed or failed from an issue that wasn't this Firefox just decided to stop working. And it wasn't something that was causing the system to stop or anything like that, but it was overhead on the people doing the triage and they usually have better things to do with their time than deal with a common issue that hopefully we can find a tool to work around. So we came up to this question of can we create, can we train a machine learning system to detect that particular crash and then automatically do the rescheduling. But this kind of comes to a question because I was talking about this issue last year. As it turned out as I was starting in this, the bug was magically fixed. I'm not sure anyone particularly found this bug, you know, what the root cause was, but there was an update and they stopped working or, I mean, they stopped crashing. So, you know, there's a very valid question of, well, why do we care? You know, this is no longer a valid issue. Why is there a point in continuing with this? Because, you know, the issue's been fixed. It's not going to be of direct use. And it's an easy answer. It created a rare situation where creating the data set is cheap. By and large, when you work with machine learning, the really expensive part is creating your data set. And something that I've learned through doing a whole bunch of machine learning is if you don't have to create your own data set, don't. It's a lot of work and it's not fun. And, you know, at that point there was a real question of can this even work? Can we look at the screenshots coming out of OpenQA and, you know, get enough information to start doing triage work? So, this was a cheap way to create the data set to set up for the experiment that didn't, you know, involve me, you know, harassing the triagers to help me tag, you know, different runs until they stop responding to my emails. So, I mean at this point, does that make sense? You know, it's set up as, you know, we have this recurring issue and it's visual. I guess I haven't gotten much further than that, but any questions at this point? All right, so we did try a couple of things. I'm generally a believer that, you know, more simpler is more better, but it didn't work. You know, tried to do OCR on the pictures and run that through a classifier and that failed spectacularly so I'm not even... I didn't write slides for it. But, you know, it was worth trying because it's the easiest possible thing. But to get into some background on, you know, what I ended up doing is so artificial neural networks the the way they work, it's, you know, it was really inspired by how, you know, the human brain works. You have a bunch of interconnected you can call them neurons that work together to remember and process information. It's generally made up of multiple layers, at least modern ones, but the concept itself is not really that new. It was first proposed in the late 19th century and the first computer-based neural network was done in the 50s. And as time went on, you know, it would be popular in research and then people would realize at least in the 50s that they didn't have the compute power to do it so it would die off for a while. Then, you know, people started talking about it more in, you know, the 70s and then it kind of died off for the same reason. The use of neural networks really didn't start taking off until we had, you know, the growth of in particular GPUs. And the ability to do a whole lot more math a lot more quickly that made this theoretical, oh, we can make a computer thing, you know, work like a brain into something that is a lot more practical and, you know, doable. Roughly I'm trying to, sorry, I'm trying to remember that this is being recorded. So that shows up. And, you know, like I said, this is a very simple diagram. I don't have the time to go too deeply into this, but the basic idea is you have, you know, this in particular is one, but you have your input layer and your output layer. Each of these would generally represent one bit. And you present your input. It is connected to all of the different layers in between through various techniques, and then you get an output. Is the very, very high level of how a neural network would work, or does work. In terms of the types, the, what's called the multi-layer perceptron is very much like the diagram I just showed. It is, it was meant to model, you know, how the brain works and how the neurons in the brain work. It is still very heavily used, but it is tends not to be the primary thing. It tends to be a final layer trying to gather stuff together to produce a, at least in the term, in a classifier sense, a single output. Another is recurrent neural networks. It's usually used for more for natural language processing, and where you have things of indeterminate length. And convolutional neural networks. This is, you know, they really became popular for image classification. Of, you know, give it a set of images and, you know, say, you know, in a set of categories, you know, can the neural network correctly classify this image as a dog or, you know, this image as a bus. You know, that kind of stuff. And that was state of the art in terms of image classification until about 2019. And the stuff passed then, you know, this is a good question of why, and I'm not covering it because I ended up using a CNN. And the reason for it is data. All the stuff that is newer, the stuff that has been state of the art since 2019, requires, like, millions of pieces of data in order to get that performance. And that's you know, getting back to the theme of this was an experiment, and we wanted to do it cheaply. Millions of data points was not an option. So, a classifier. Like I said, it's, you know, given an input, you know, what class does it belong to? I think of it as very similar to a capture. So, you know, you get the, we've all seen these things. It's like, you know, select all the images with crosswalks. So this is, you know, using a human as a classifier. You know, is this, does this image have a sidewalk? You know, yes or no. You know, go on. So each one of these pictures, you know, can be classified as contained sidewalk, or does not contain a sidewalk. And, you know, using neural networks is one potential implementation for an image classifier. So, you know, getting into, you know, like machine learning. You know, from a very, from the high level parts of this is, you know, give it a bunch of examples. You know, it's supposed to learn from that, and then be able to replicate what, basically what you showed it. And that's how a neural network also works, is you give it a bunch of data and the correct answers. You know, it learns from, you know, this was correct, that wasn't correct, and eventually you will, well, you hope you end up with something that can repeat what you've taught it. So, you know, we've got a common term, deep neural networks, it just means more layers. More layers, more computation. And that's really what it comes down to. There's no strict definition from what I've seen. Generally, it's three or more layers to be considered deep. But that's about that. My very quick overview of neural networks makes sense. Does anyone have any questions? Go ahead. I forgot about that. Could you go back a slide? You were saying it's multiple layers. Does that mean multiple passes of the same? Is it multiple passes to determine the image, to classify the image? In retrospect, I probably could have found a better diagram. So in this case, this is a network with three layers. You have your input layer, one hidden layer and an output layer. And that's like more layers would add like more if it was fully connected or something. So it's more process, more steps before it finishes. So it goes from input layer to the next layer to the next layer and eventually you get to your output. So it's not more passes, but it's adding more steps before you get from beginning to end. Does that make sense? Thank you. Regarding this diagram, each participant in the first hidden layer would point to all of the points of the consecutive hidden layers. For a fully connected network, yes. So this is a simplistic diagram. In this case, yes, there are cases where you would use that, but when that stopped being state-of-the-art probably in the 90s. So it's a good way to think of it, but it doesn't always happen in this case. Any other questions? All right, so let's get into the experiment. All right, the data set. Like I said before, one of the big advantages of looking at this particular problem is it was easy and cheap to create the data set. We had an issue with the tests and if the triagers saw that test, they would reschedule it. So making a couple quick assumptions that assuming that all of the jobs that failed this reason were rescheduled and assumed that anything that failed and was rescheduled was due to this issue. Are there, did I probably catch a few that weren't supposed to be in there? Do I miss a few? Yes, but over this time period, that holds true enough to produce a valid data set. From that, it's easy to just point some code at the open QA instance, download all the pictures, download all the videos, and we then have our data set that is categorized into two sets. We have jobs that were supposed to be rescheduled and jobs that weren't. And again, I'm harping on the cost because that's one of the biggest things is I didn't have to go to the people doing the triage and after they described the issue and confirmed that I understood, they didn't have to sit there for hours and hours giving me this job number, this job number, this job number, this job number, those were the ones that you were looking for. So the cost in terms of human time was very much minimized. I gathered all the jobs from August 30 to September 13 of last year, grabbed all the screenshots, any text that was produced in the video. Ended up being a little over 31,000 jobs, the full data set is about 80 gigabytes. Eliminating the data brings it down to about half at 102 gigabytes of mostly pictures. So getting, you know, using a convolutional neural network, it doesn't have really a sense of time. You have to give it all of the data at the same time. So I can't just feed it the first screenshot, the second, third, fourth, so on. And what I ended up doing was creating a composite screenshot just as I'm going to go back and forth, but basically this. So this is, you can kind of see the different screens. This is from a part of the job, these are all the screenshots that produced. Here's the first one, the second one, third one, fourth one, so on. So this ends up being one image that contains all of the screenshots and I can feed that into the network all at once. Just getting into some technicalities. Did some sub-sampling in general with neural networks. If you give it something that's rare, I mean this was I think we were seeing maybe 1% fail. I think it was out of the 31,000 jobs, I think it was 900. So I mean it was 1 30th. If you just feed that ratio into a network while you're training it, it's not going to do very well because it can just always say false and you know 29 30ths of the time it's going to be correct. So I just limited the I took all of the samples where it was supposed to be rescheduled and I think I left it at a ratio of about 5 to 1. So for every job that needed to be rescheduled I took 5 random ones that didn't and created the data set out of that. Left the data set as 80% for training so it took 80% of that to do the training did the testing on the rest of the remaining 20% and used a 3 layer deep CNN to do the classification and again this is the composite screenshot that gets fed in. Hopefully this representation will at least not be too confusing. So you were talking about what was it Jeff you asked about you know adding the layers. So in this case this is one layer, this is a second layer, this is a third layer and this ends up being 3 and 4. We can get into technicalities So basically this last one is dimensionality reduction so this is a vector of length 100 and I needed to get a binary answer so it has to be brought down to 1. So that's what this last one is is dimensionality reduction from 100 to 1. Which is why I was saying it depends on how you look at it I'm not going to spend too much time harping on the specifics of what all of this is doing but it is using convolution to steadily go through and convolve the data that starts in the layer and essentially do feature extraction trying to extract information out of it to the next layer. You can it was left purposely without some specific there's a parameter search space that I left for this. The different sizes of the layers the fully connected that last part where it's kind of pulling the information together and eventually reducing the dimensionality what size of the for convolution max pooling has to do with trying to keep it from having all of the information in one part of where it's looking it will take it would take longer than I have to really go through it it's commonly used part it's part of almost every CNN but I did a great search of all of that which is 432 runs and I just want to make a bit of an aside there's been some conversation on some of the fedora lists about do we really need GPUs and I want to point at this where this is the same machine I ran the same network on the same data and it took 4 minutes to run with a GPU and it took 24 minutes to run on just the CPU so it is dramatic and even just with 4 minutes per run that still took more than a day running constantly and it's not, I mean it's a 13th gen i7 with a 24 gig 3090 it's in terms of what people use for machine learning it's tiny but it's I think that says more about the clusters they use for training stuff like GPT which costs millions of dollars but it's still it's the bringing home why we might really need to have some accelerated option for doing the AI, ML and fedora and more of an illustration of like I said the concept of neural network has been around for a while it didn't really take off until we had machines that could do that much extra math very quickly Leaving aside the last one that took hours do you have an estimate of the power consumption of the 4 minutes versus 24 minutes I imagine I could go look at it but I don't off the top of my head be interested to know because oh yeah my my suspicion is that it takes me guessing this is gut is that it's less energy with the GPU but I can figure it out I don't off the top of my head though so one question two slides before if you were to not take the subset, sub-sample if you were to just take composite images how many composite images would you still require to get 80% efficacy throughout for one single failure I don't understand what you mean by 80% efficacy I don't understand for example if you are training for 80% and you sub-sample the dataset if you were to not sub-sample 80% of what I don't understand you sub-sample the split right to 80% to 20% 80% of training no I sub-sampled I can go check the code I don't remember off the top of my head I believe it's 5 to 1 so there are 5 jobs that did not need to be rescheduled for every one that did need to be rescheduled so that and that's just for training purposes if it's too rare like I said you can just the and I've done this where the neural network will just return 0 for everything and you'll end up with like 90 something 80% accuracy because 90% of the jobs were 0 so that's where the sub-sampling comes from in terms of the split it's just 80% so then from took the original sub-sampled it so there was a 5 to 1 ratio of not rescheduled to rescheduled and then from that dataset took 80% of those and that was used for training and the remaining 20% were used for testing and that's it okay and go forward please and so if I understand correctly it took 28 hours to completely train the network? No it took 4 minutes with the GPU it took 4 minutes but I can run it with different parameters so this is just all like to do an exhaustive search of this entire list with 2 tries each was 432 iterations. OK. And like the 28 hours is the number is the total time to exhaustively search through that parameter list to see which one performed the best. Which combination of parameters performed the best. I see. OK. OK. So if you got through this whole experiment, the idea after the 28 hours, you know which neural network configuration is the most efficient for the task. So if you want to train it on a different failure, you wouldn't have to go through the 28 hours again. You could just. Wouldn't even have to go through the four minutes. OK. I mean, once you have the network trained, you can just run it on more. So that four minutes is to train on the 80% and then test on the 20%. So I don't know off the top of my head. I think it would be, I'm not even sure. I don't want to try to do the math in my head. It's much quicker once it's actually trained. Any other questions? So OKQA has configured such that the screenshots are all 1024 by 768. When I did the first try, I took all, you know, did a composite screenshot and then shrank it down to the same size of 1024 by 768. The results were so bad that I did not write them down. And actually, at that point, I was convinced this was not going to work. And I'm like, all right, let's just try this one more time. Let's double the resolution and shrank it down. The composite screenshot, which could be eight times the size of the original, but shrink it down to 2048 by 1536. When I did that, and then did search for the parameter space, it was getting really good numbers. I was going to put a slide in here on precision and recall. So the first thing about accuracy is basically how many, so it got almost 98% of them correct. But the more important things is going through the precision, you know, did it find, I'm trying to think of the, how many of the true positives did it find? And then recall starts looking into how many false positives were there. This is much easier to explain if you include the diagram that I forgot. But suffice it to say, it did well. It is not showing too many false positives, nor is it showing missing things, like things that should have been a positive that should have been a negative. Which in my mind is important for this use case, because if we are trying to rely on something to say, yes, this is triaged in a certain way. This is not triaged in a certain way. If you get a whole bunch of false negatives or false positives, no one's going to trust it. And it's going to cost them more mental energy than if we hadn't done it in the first place. These are that. OK, so there's a typo in my slide. These are the configuration values that yielded the best results. The kernel size is supposed to be 2. I'm not sure why it says 62,000. But that value is supposed to be 2. And getting through it's like, that's great. We have something that's 98% accurate. But is this useful? Is this something that can be applied elsewhere? And this is a way of saying, not really. Like I said, this was an experiment to see if any of this is feasible. But as an analogy, it's like if I have this network and I spent all this time to train it on pictures and I want it to identify cross-box, that's wonderful. But the next capture I get, I need to identify buses. It's not going to find anything. So that's kind of where we're at. This experiment was an analogy to pointing it at open QA and say, find all the sidewalks. Which is great if you're looking for sidewalks, but that's not all the time. And one of the potential issues is that because I had to use the large screenshots, it takes a lot of memory on the GPU. With the 24 gigs of memory I had in that GPU, I could only do the images. I think it was four or five in a batch. And batch sizes are usually larger than that. But there's only so much memory. Do you have a question? And offloading the images from the system is too slow for this application? Folding? Offloading the images from system memory is like if you don't have enough. It has to be on the GPU in order to run. In order to get that time increase from 24 minutes to four, it has to fit within the graphics card's memory. Okay, thank you. Just curious, on the composite image things, as the human behind the curtain here, I happen to know that in classifying these failures, all you actually needed was the failure image. Did you try running the experiment just using, because I believe from the API you can get only the image that the test failed on. I don't know if you looked into that. I did not know that was a thing. Yeah, it knows. It circles it in red in the web UI. And I believe from the API data you can figure out, this is the one the test failed on, and then run it on that. So, I mean, we could, no, I did not know that was possible. So no, I did not run that. I did not know that was a thing you could do with OpenQA. Next time we do it, we can do it that way. It'd be interesting to try my suspicions that would not work. Because, I mean, you're, I've talked about all this OpenQA trioshes. It's mostly you. So, my... You've just been using they, them pronouns for me so far. Well, I... The, my understanding of the time period that this was happening is it's not, you needed more information than just the one that it failed on? No, for this one, Jen, for this one failure, you would know from the screenshot. From the screenshot, you don't, you didn't need any more context of what job it was, what it was doing. The failed screenshot would always be, you know, oh, it just dropped back to a command prompt and there's a bunch of Xorg server blah there. That's the thing the network would have wanted to figure out. But there are other cases where that wouldn't work, though. Just for this specific failure, it probably would have done, I think. We can, it'd be interesting to try. Like I said, my, my incident, I don't want to, I mean, we can get into the details of it. I don't think that would work. Is my instinct, because I don't think there's enough context in that one image to be able to tell, you know, reschedule don't, but it'd be worth trying because it'd be a lot less computationally expensive. Is that another question? This is kind of a question for Adam. Are there things that are successes that look like that failure screen for other kinds of tests? That's a question for Adam, I don't. It's an, he asked, are there things that look like, are there things that would look like the failure screen but would be a success in another test? Again, I think in this specific case, probably no. So, yeah, as I said in the previous question, but just in case anyone didn't catch it, in this specific case, when the test failed, what we would normally see is, instead of something in a web browser, which is what the test was expecting to see, we would suddenly drop to a console and there would be, you know, when this was actually running X for stupid reasons. So you would have the X server output on the console and probably like a cutoff stream of error messages from X usually. So that's not something that's a success in any other case. Yeah, there's no point at which we want that. So I think it would probably have been fairly clear cut, but for instance, there's another problem I have right now where, if we were only looking at the failure screenshot, that wouldn't work and you probably would need this composite screenshot thing, which is an ingenious approach that I like. Yeah, I just, I'm going back to, like I experimented with OCR and the OCR had so many problems with identifying the text correctly, that was part of my suspicion, but it'd be worth trying. Like I said, more simpler and more better. Hey, so can you just go back to the experiment slide? Like the experiment method, where you did like the... This, the experiment design? Well, the RIT search one, where you mentioned that you performed the RIT, yeah, so in the RIT search, so the way you did what I understand, like for every hyperparameter combination, you trained the network using the training set, and then got the accuracy of whatever you were using the test set and continued that for all your combinations and selected the best one based on the test set performance, is it? Yes, that is correct. But do you think that would kind of like overfit the like or influence your network or the results? That's because test set in general is supposed to be kept separate and at the very end, you need to do that. So maybe a split of, let's say, training, validation and test, so that at the very end, when you select the best hyperparameter, then you apply test at the very final. No, you're correct. And to be honest, I didn't do this as formally as I could have. If I was, you know, especially if I was trying to get this published, I would have to do that because you're right. I'm kind of cheating is a way to put it. I think for this particular context for the fact that we're not trying to produce something that's going to go into production, it's just a question of, you know, is there enough information in these screenshots to try and triage? For specifically that purpose, I think it doesn't matter, but you have a, you're correct, I should have done that. But that answer your question? I'm not trying to say, I mean, I think these results are valid, No, I think that's valid. That's because you have quite a lot of data points. So essentially, it's unlikely to overfit, I guess. But, I mean, and now that you mentioned it, I'm curious because I did record all of the, I mean, getting into more specific things, you know, within that 80%, I think I did another 80, 20 split. So it's only using, so I used that 20 for validation. I did record that, those, the results from the validation parts of it. And I, now that you've mentioned it, I'm curious to see if it would have picked the same hyperparameter combination from the validation set versus the test set. Yeah, thanks. Does that make sense to everyone else? What I just said? And what he was asking? Enough anyways. All right. He basically pointed out that there were some flaws. So there were some flaws in my approach that certainly would not pass muster. If I was trying to get it published or something like that. All right, getting back to, and I keep harping on this because I got some questions about, you know, when is this going to go into production? And it, you know, again, getting back to, it's not going to go into production. It wasn't ever meant to go into production. This was an initial experiment. And the idea was to, is there enough information in these screenshots to do triage just based on throwing them at a neural network? And I think that the answer is yes, it's certainly possible and certainly worth looking into. So data and code. I forgot to push my code public because I found a bug last night and something that doesn't affect stuff but I need to push that public. The data is available. It is a 50 gigabyte tar ball that took four hours for my computer to compress down from a hundred and, yeah, 108 to 50. So you're welcome to grab it. It is a lot of data. And I can publish these slides afterwards. If you'd want to see the code sooner then I get around to pushing it. Just come find me. I'm not trying to, you know, hide it. It's just a, trying to make sure it's correct before it gets pushed. Yeah. Hold on a second. Can you wait up for the? So user conclusion is, yeah, it's possible. So if I would love to do a new classifier to find the buses. So, like, yeah, initial around a lot of investigation but the second time when you want to do a classifier for the buses, how long it would take you and how much time it save you during regular QE. The primary flaw in the approach that I've talked about is that it cannot scale. In order to say, you know, say that, you know, Adam said there's a new issue. It'd be nice to be able to detect. In order, the problem is creating the data set because in order for, you know, this kind of an approach to work, you have to have, you know, at least a thousand labeled instances of it happening and then, you know, do all that kind of stuff. And if we're trying for every single issue to find a thousand examples of it before we start recognizing it. So how long it would take you to create new data set for the buses? That's a, it's a hard question to answer. It depends on how quickly, you know, how frequent the error is. The long-haul intent is the person who is going through marking things as, yes, this is related, no, this isn't. So I don't know if I am misunderstanding your question. I have no idea. Like even the scale, like is it minutes, days, weeks, years? Again, it comes down to the issue. In terms of once the data set is created, it's gonna be the same kind of process of, you know, say a day or two of exploring, you know, of computer time to explore the parameter space and then four minutes to train each would be my suspicion. Assuming everything's about the same size, but the most expensive part of this has nothing, it is not the neural network. It's not the time coding it. It is creating that data set. And so in order to get, you know, 1,000 examples of that particular issue, however often it's happening. So say, you know, it happens 10 times a day, you know, it's gonna be 100 days before you get 1,000 examples. And, you know, is, do you need 1,000 examples? It, that's a hard question to answer. It depends on the issue, but what I'm trying to get at is the expensive part is the data set creation and collecting all of that. So let's look at this from the perspective of recognizing failures that, let's say we don't get them in OpenQA. We can look at the Fedora, a more general view on Fedora. So for example, we have stack traces. And we have a system that from the user's systems uploads these stack traces and we have collection of the stack traces. So for example, this can be used if we marked a certain stack trace that this is real error. This, and we know a bug that is associated with. We can have the information similar to that stack trace identified by similar network run to be belonging to the same bug. So kind of how long would it take to achieve this? This is probably something that Miro is asking. Before I answer the question you asked, I just wanna answer a different question that was implied. There is, that's a whole field of research in itself that is not terribly related to exactly what I talked about. But to answer the question you asked, it honestly, it starts getting a little outside of my knowledge. The problem with honestly, every machine learning problem is data. And it's just kind of getting back to the data set. And is it how long? I don't. The reason why I'm taking the stack traces as an example because they are more confined and easier to identify similarity there. Remember when all of these GPT became sort of public available for experiments, but not really run in open source fashion. What people did, they started looking into stable division, diffusion and similar network systems and started to convert other types of information into the visual. So effectively the beginning of that story was that they converted audio into the video, into the pictures and ran stable diffusion to generate something new with those audio and then converted back to the audio. And this is effectively what I am implying here. Now we can take these stack traces, convert into pictures that you analyze, stack them together and run some sort of analyzer. If we know that this particular stack trace, for example, corresponds to the real problem that we already been getting in past and we have collection of those problems actually classified everywhere. We can take that data that are already associated with the solved bugs and train the system with that to figure out some sort of a similar stack trace happening. It's an applied that to the logs of open QA when the crash actually happens. One of the issues I at least noticed is that with open QA specifically it doesn't always give you that in text form. So like the crashes that it was seeing with X it would show up on the screen and it would not show up in any of the text logs. So yeah, getting back to what you had asked, I don't know, I mean it reminds me of talking to someone who did some interesting research of trying to find malware by converting the bits in a binary to a bitmap and then running visual analysis on that. It's a hammering of it because you're making a much harder, you're trying to solve a much harder problem for something that could probably be solved much easier but it still seems to be where the direction of research is going today. And like I said, this is me, this is outside of my exact realm of knowledge. My instinct is that because, I mean it'd be worth trying but my instinct is that because it's more structured, more information retrieval techniques and less machine learning might be more effective because like I said, the downfall of most machine learning problems is that you need a hundred examples, you need a thousand examples of the same thing before you can train it to find it. And in a lot of cases by the time you have that many, the bug is fixed or it's no longer an issue. And either way, finding someone to identify a hundred duplicates would be difficult. So I don't, am I answering your question? That's an interesting way to do it. I don't know how possible, how likely it is to work. I'm just looking at this from the other perspective. So we have roughly upstreams, then downstreams like Fedora, then downstreams like CentoStream, then downstreams like Rel, which have a span of time before a certain thing comes in. So by the time we missed something in QA in Rel, that problem for sure might have already been seen somewhere upstream, upstream meaning in Fedora and in real upstream of that project. So if we could have had some sort of analysis of these problems we see in upstreams and in Fedora, then at that time to generate these kind of data sets and then reapply them downstreams, we might actually get results that we don't see now in Rel QA or we don't see them in the actual support cases and help there because we see them maybe 12 to 18 months later. No, and that could, yeah, having in a way, I'm paraphrasing what you were getting at but having a way to fingerprint some of those crashes to be able to enter from later that, yeah. Just side note that if you want to look for these traces from, I'm gonna compare it to the different issues already reported, you already have it, it's part of ABRT, FAF, FAF already do that, not based on AI but on the editing distance. I was never too much popular, so if you want to dive into that, you can, it already exists. I'm using the traces example here just because people are aware about ABRT but in reality I would take fragments of the actual execution logs with the variation there like SSSD logs or journal with certain things and focus on those because these are the things that we analyze with support every day and some of them span a lot of time in execution and you do match across multiple logs to find the actual problem, it's not in a single place. So extracting these, correlating them and finding them this way seem to be a bit of promising. I think Steph Walter did this couple years ago with the cockpit and they did find some promising results but again training and then reapplying the results fails because you never find the same problem while in maybe reapplying this across multiple distributions from upstream to downstream might give you actually a repeatability of the problem. That's the reality we see. Customers might see the same issue again and again in downstream 12 to 18 months later after we probably solved this problem in the upstream and this gives you a chance to actually catch something that you forget to back port for example. That's my point, maybe this is a real value here. Okay, and I'm happy to talk about this more but I have five minutes left and we'll see if I get through everything. I should be able to. So getting into, it was an experiment. It was never meant to go into production so what can we do with this? Can we make something out of it? From what I understand and Adam, this is from a conversation we had, something that is possible that I want to take for the next step is can we take those open QA failures and start grouping them by root cause? And so that you don't have to have so much, again trying to revisit this whole thing of making people who are doing the triage work more effective, it takes less of their time because they have other things they need to be doing rather than just triaging the same thing over and over again. I think the value is not so much about taking up people's time. So the Adam W. Restot bot runs maybe every two hours during the day and then there are these eight hour intervals where I have to go sleep and that means that if your update is blocked on one of these failures, you might be waiting eight hours before it gets restarted. If we can do something like this, then your update will get restarted immediately and then we cut out that latency. I'm talking about more general. Not just doesn't need to be restarted as in can we change open QAs so that there's some suggestions up? These five failures look very similar. And trying to group things by root cause and failure not just doesn't need to be reset. Yeah, that could be helpful for sure. So yeah, I agree. I agree it's definitely worth looking down. And I think that the thing that we have to look at is the whole question of how long does it take to get enough data together to classify things versus how long do problems exist? But there are, I mean the X thing lasted for several weeks. This 404 thing has been going on for a couple of weeks now. Yeah, and like I'm talking about something completely different. This is related and not. So the, but the idea is, yeah. It's still the same thing, it's not always the same ones. But one of the things that, I'm sorry to tell you I'm just trying to make sure I finished my slot, finished before time's up. The, there's a system out there called Tango. It's been published in research. Basically it was developed to try and find duplicate results from screen captures of mobile apps. And it used video and text to try and find duplicates within its data set. It's something I do wanna look at. Cause I think it's promising. It's related enough to be worth looking at. But as I'm repeating myself for the 50th time, the problem is data. Modern machine learning stuff needs a lot of data. We have only so many expert hours and only so many emails to set experts before they start ignoring my emails and never talking to me again. So it's a question of this is the real cost. Is this going to look promising enough to spend the time to create the data set to do more research? But one of the things that I also wanna look at is a technique called active learning. Well, I call it technique. An approach called the active learning. And the idea is instead of sending every single sample to an expert to annotate, analyze what your data first and organize things such that you are using them minimally so that you can gain the most amount of information from the least amount of expert time. And that's, so the idea is from here to look at Tango to look at active learning so that we can minimize the amount of time that we need from experts and hopefully find something that can start grouping those failures by root cause with enough confidence that it doesn't end up being noise. So just repeating the conclusion, it's promising but in my opinion but it is only beginning and going forward it's going to start getting expensive. One of the things I do wanna harp on is and I'm leading into this, all of the work that I did was on an Ubuntu machine because this stuff does not run well if at all on Fedora. And we can get into a whole bunch of stuff about why and what can be done. There is kind of a solution coming up that someone's working on, basically the replacement for NVIDIA Docker. You used to be able to pass the GPU through to the container and have all of NVIDIA's proprietary crap in that container and run it on Fedora that way. We stopped shipping Docker. The replacement that can use Podban. I think someone's working on packaging it. But that's a lead up to if you also don't like the fact that this all had to be done on Ubuntu. We're working on stuff to fix that and maybe you should show up for the, there's a, at 4.30 in the room next door, we're gonna do some of a meetup to start talking about some of the stuff and what we're doing to fix it. I think I'm out of time, but questions, comments. I guess if you have, I am out of time. If you have questions or comments, please come find me. Thank you so much, Tim. And thank you for all of you who are here in the Fedora Leads, the Linux distribution development room. We will be picking up again in five more minutes and if you are on your way out for a moment, there is a badge sign on the back desk there. Please stop by and get your flock badge. If you haven't already, you can scan the QR code. So I just wanna remind everybody for that. So we'll be picking up here in about five more minutes. Yep. All right. For those of you who are joining us online and around the world and those of us here at flock today, welcome back. I'm glad that you're here to continue with the Fedora leading Linux distribution development track. It's my pleasure to introduce Michael Cornetchi. Cornetchi. Cornetchi, ah, almost there. And I will hand it over to you for a talk today about what's new in the land of releasemonitoring.org, 2023 edition. Thank you. I was already introduced. So this talk is about what's new in the land of releasemonitoring.org. This is the 2023 edition. As you can see, I adjust the fabulous slides. My name is Michael Cornetchi and I'm mage from releasemonitoring.org. That is not my official role, don't think about it. So first thing first, where is my magic hat? Oh, here it is. Okay. But how we can get it from the slides? Let's see. Oh, it's gone. Where it is? Oh, it's here. Okay, found it. So now I'm the mage. And let's start with a story. So once day I was going to my magic tower. As you can see, it was nice day. Everything sunny. It was like in some kind of paradise. Plenty of flowers. And then I got inside. And I found stairs. So I get up and then going up, I was thinking if what I can see. As you can see, there is a balcony in there. And what do you think I could see from it? Okay, so if nobody knows, it was actually the role of releasemonitoring.org. It's all real. And what it is actually? So there are two parts of the releasemonitoring.org. It's the Anitya and the new hotness. The Anitya, I will show you first because it's just easy to show. Easier to show. Let me just find it. Oh, let me fight with it for a moment. Oh, let me open it. It will be easier. Okay, so here is how it looks. This is the Anitya itself. The Anitya is a front-end for watching for the releases of your favorite projects as you can already read. And this is how it looks. And let's go back to the slides. And what it allows users to add projects to watch for new releases. This will allow you to actually see what projects, what versions are available, what version is the newest, and it will allow you plenty of customization for the projects. It automatically checks for new releases of the upstream projects. You can see what is happening there. You can see hope when the version actually came out. And it sends federal messages to the message bus when the new version is retrieved. So all of this is actually done in the front-end. This is what you can see mostly in the plant of Riesmointing.org. But there is the other, the new hotness. The new hotness is kind of somewhere inside of the Riesmointing.org, but let's say it's floating island above. And what it does, it's listening to messages emitted by Anitya, creates or updates Baxilla when new Ries is built, font, and it can start scratch builds configured. This is what you will get if you are a package and you get notification on Baxilla. It is the issue created by the new hotness. And now we have some magic numbers, who doesn't like numbers for Anitya first. So it will be in this format. I will show you how it was the number in the previous nest and year later and what this is for between the last nest and the new flock. So we will do it a little more interactive. So we had two releases of Anitya two, one year ago, or to the next, to the previous nest. So how many releases do you think we had? It was more, less, more, okay. Yeah, you will be right. So next, we have 266 commits. So do you think we had less commits or more? Yeah, it's less. And it's less because we actually change how the dependencies are solved. So we don't get that much dependency updates than we get before. So plenty of tools were created by both before. And we had 13 contributors before. How many do you think this year? More, no, it's less. But I would say that this is because in the last statistic I probably included the both accounts and those are removed. So the number would be same. And how many issues? Less, less issues? No, it's more. But this means more people are actually using the release monitoring.org, which is great. Okay, and closed issues. Yeah, it's much more, much more than last year. And almost the same number as the created issues, which is nice. And the last version that I presented here was 1.4.1. So how many versions do you think was before? We know that it was six, but what is the version number? It is definitely bigger. So, to that, oh no, 1.1. We had one hot fix and few mineral releases. There wasn't any major change, at least not in the backend. There was some changes in the front end that were actually really big. And how many projects we watch? We had 216,314 last year. So how many? We have no. More. Yeah, definitely more. So 309,000. And how many of those are Federa packages? If 20,600 last year, it's close. We didn't get that much new. But I'm not sure right now how many Federa packages are actually out there, how many Federa packages are actually packaged and maintained right now. So not sure if the number could go that much up. Okay. And now look at what the goblins delivered, so what new features we have. So you will now see the goblins that are delivering things. Those are actually DICE goblins, but this doesn't matter. Let's say that those are the goblins that are working on the RISC 1.0.org and trying to deliver things. I'm one of them. So we had one new thing is that we have sort of the list of distribution in the project view. I can actually show that when I switch to the correct tab. Oh no, okay, here it is. Okay, so if you saw the project previously and see any mappings, it was just how people created them. So we have no way that you can see them alphabetically. So it's much easier to find what you are looking for. And okay, so let's go to the next one. Let me just switch the slideshow. And the next big thing is migrating to bootstrap five. The previous where previously it was with bootstrap three. And it took a great amount to actually migrate it, but the new front end looks much more clear, much more clean, much more modern. So it took work actually was good and satisfying to see that the outcome is actually something we can look at and we can show the people. The next thing that was related to it and why the migrating actually started was we actually had dependency watching for JavaScript packages. And this was actually one of the things that was found in this, that we have really all bootstrap version. Great, source backend. If you know how the Anintia works, or if you don't, there is plenty of backends you can choose from, I will open the guide. As you can see, there are plenty of them. Those are actually all the Gitforges or sources you can watch for. We added the source hat, that was one of the newest one. And one of them is made custom so you can create your own regex that is actually trying to parse the page and trying to find the version based on regex. So let's go back. We have configuration for this trainings. I will show that because that is really nice feature. If you go to any project, let me just go back. And you can see that there are links in the package names and if you click on them, you will get redirected to the repository in the distribution. This was originally created just for few distributions and those few distribution were just having it hard coded in the web page. So the change is that it is no configurable. There is in Anita configuration you can actually set distribution and the link. And it just adds the name of the package that is specified in the package name. So you can use it for anything. You can see here is a Fedora. So let's just look at the Fedora one. And it will show you where it is, how it looks. And it's working for the FlatHub as well. So yeah, it's usable if you just want to know what distribution are actually using it. You must take in account that all of these are actually edited by somebody. There is no automation that is creating the package mappings. So if anybody from the distribution wants to update, it's on you to actually add the correct project. Okay, so let's continue. And we have another bagend. This is the CGIT bagend, just I think as the last one. And the Googs bagend. And now we will look into the crystal ball and we will see what the future holds. So one of the things that we, that is actually wanted by plenty of people, I closed plenty of duplicates for this one, is to have some kind of replacement rules for versions. If you get a version right now, only thing you can do is to remove prefix from the version. And there are some, let's say projects that actually creating really strange version strings. And this is to help them to actually just find the version in it. So kind of reg X that will just, you will set, here is the major version, we use the minor, we use the what fix version or something like that. And the rest will be just thrown out. So the versions are much clearer. Support for different version streams. This is something that is related to both Anidya and Newhotness. The different version streams means that we will have, we will watch separately the versions for 0, 1.0 release via 2.0 release and so on. There's plenty of people actually wanted to get notifications about the older versions, even if there is a newer. So this will help with that. A new authentication backend. This is ongoing work. This is already being done partially. We will go from the Flask OIDC to Outlib. And we already have some work done for Google and GitHub, but we need to add the Outlib support for Fedora. Because the Flask OIDC is using deprecated authentication number, which is not great. But yeah, this is something I need to look into and didn't get to it. And the next is RSS feeds for project. So if you just want to watch one project, we would like to have option for you just to add it as RSS feed and it will notify you when there is a new version. And automatic filling of project details when creating a new project. This is something, if you are familiar with ProtonDB, for example, if you just put there the ID of the game and it will show you all the things that needs to be filled in. So it will just prefill it. And I would like to have something like this for Anitya. You will just put it the URL of the project and it will prefill most of the things for the project. And now some magic numbers for the new hotness. So make it interactive again. So we had four RSS till last nest for the new hotness. Who many do think there was this year? Less or more? More. More? No, it is actually less. And who many commits? Less, more? More. More? No, it's much less. The reason for that is, like I said, in case of Anitya, we changed how the dependencies are resolved. That is one of the things. And the second thing that is actually bank here is that we actually don't have that much issues on the new hotness. Which is another thing. How many contributors do you think we have for the new hotness? We had five. Less? Yeah, we have three. But as I said, this could be the both accounts. I forgot to remove on the last year. Issues created. We had 62 last year. 24. And how many closed issues? More or less? It was less. But because we didn't have that much open issues. So. And version? Do you think the version is bigger or less? No, it's not. I don't think it's the same. Could you want to actually guess what version do we have right now? It's fine 2.24. 1.2.4. And the fourth one is, this is actually because there was only a few hot fixes that needed to be done. And otherwise I didn't need to touch it. There are a few feature requests, but they are not high priority feature request. And for the issues, it's actually, when the issue is reported, it is closed really soon, because it is much better to maintain it and much more stable, it's easier to find what is causing it and fixing it. There are a few that we didn't actually found out what is causing them, but this is something like a heisenbach, so it's hard to find. And what features? Okay, I will ask. How many features do you think there was delivered in the new hotness? Okay, yeah, it's zero. We didn't have any new features, but as I said, it looks like the new hotness is right now stable. There are a few features I'm still waiting for Pagur update to happen for the diskit, for the diskit to actually be useful, because I already have in new hotness option to watch for all new releases, not looking only at the new best one, and option to watch only for the stable releases. But both of those options are dependent on diskit to have new version of Pagur, which didn't happen yet. And what is the future? So I would like to have support for flood packs, for Fedora flood packs. So we will just create new bugs in the Fedora flood pack namespace. Same for Apple. And that is actually for all. So any questions? Any questions about head? No. No, it's not about head. I'll just give him the microphone, so. I know, I just wanted to say your head is fascinating. I mean, do we have any more pins in a flock? This year? Because I would like some pins on the head. I think that they do have some pins for giveaways later. So if you're in some of the social nights, I think they have some Fedora pins. Okay, that sounds great. Okay, so the question was where did I get it? So I ordered it from some eShop. I don't remember. And I liked that the colors are actually in Fedora, white and blue. So it's nice. And any other question? So forgive me if I'm just oblivious. I don't think there's anything in it for like CVE tracking separate from just actual updates. Have you considered being able to like look for CVE fixes and stuff as well? Oh yeah, that is one of the things the CVE fixes are not part of the Anitya. Anitya is just watching for new releases. It doesn't know what they seem to release. In case of GitHub, you can look at the commit that is related to release. So this could be used. You can look at the upstream, but you need to look at the locks of the project to actually see the patches. Yeah, I was more or less just asking is that a feature that's been considered for the future is just automating some of that? At least flagging saying, hey, there's a CV on this. You might wanna go look for fixes. Yeah. Okay, in case of Anitya, Anitya doesn't look for this. I'm not sure if we plan it as which I don't know if we even have issue for it. Let's just kind of go basically. Yeah. Okay. Just right now it's just looking for new releases and the new hotness is the one that is creating the bugs. So it would need this information from somewhere and right now we don't have it, especially for some of the baggants that don't actually share it. We have custom baggants, it's usually just parsing some sites so we don't know what is on the site. Just looking for the version that is new. Okay, so here are some links. I'm not sure where I can share the slides actually. You can put it on the sketch page. Ah, okay. Okay, so I will put it there. And yeah, you can look at those. There is blog post. Oh, okay. Well, it's really a personal interest question because I put in an issue about supporting certain source force projects that release like different streams of artifacts that are tracked independently, they versioned independently. So they like sub-projects from one project. Because I got bitten by lack of this feature and because the new hotness filed a bug with the incorrect version that was the defaults, let's say sub-projects version and not the one I was actually pointing to in this particular project. Yeah, in this case it depends if the projects actually have the DAX different for the projects that are tracked on the same repo. Because if the DAX are just version, we can actually. There are sub-folders. Sub-folders, yeah, okay. Yeah, I'm not sure if we can, we don't support that and it's not that much project that's actually using this. So it's a low priority feature in this case. But yeah, it's one of the, I know that there is issue for something like this. Okay, thank you. So if there isn't anything, I will share the slides later. The blog post is something I didn't update it in a long time but it's why I actually have the head and why I'm here as a mage and you can actually read why it is like that. And there are some links to the repositories and for the messaging schemas that are actually very emitted by Anitya and the new hotness. And that is all. So thank you. Thank you very much, Michael. And if you all are welcome to head downstairs for coffee and tea, there's a social breakdown stairs and we'll be picking it back up in 30 more minutes. And if you haven't already gotten your flock badge, there is a sign on the back of the table there where you can scan the QR code and get a fedora badge for flock. So we'll see you all back in 30 minutes. Thank you so much. All right. All right, well welcome back everyone to the fedora leads and Linux distribution development track here at flock. It's my pleasure today to introduce Adam Williamson who will be doing a presentation today, Fedora CI and automated testing. I'll hand it over to you. Okay, thanks a lot. So yeah, I'm Adam Williamson. I have the Fedora QA team lead for Red Hat. I've been doing this for quite a long time. I started in 2009. As you can see from the next slide in my deck, which is a picture Mo Duffy took in 2010 in Zurich. So there I am, you know, young and optimistic with Peter Robinson and I think Jesse Keating. So yeah, I've been doing this for a while. Just so you guys know, my policy for talks is to hold Q&A and feedback to the end. Otherwise it's hard to get through on time. So I'll definitely make sure to leave space for questions, but just hold them to the end. So yeah, let's get into it. So next slide, I guess that's the next slide. So how do we test Fedora? This is kind of some background. This is all gonna come up later on. So I just wanna establish it. There's different levels you can test that. There's disk it, which is where we keep the sources for the packages. So you can actually run tests before an official build has ever happened. You can run tests on pull requests. You can run nightly tests on the Git repo if you like. You can test after an official build happens in Koji or an unofficial build, you know, any kind of building Koji. That's another place you can test. You can test updates, which are when we put related builds together into an update and say this is the thing that needs to go into the distribution, you can test at that level. And you can test composes. And composes are when we put all the builds together and we make images out of them, we make repositories out of them, we do that nightly for most streams of Fedora. You can run tests after composes happen. So these are kind of the main levels that you can run testing at. So some background to, I mean, the topic of the talk is where we are with automated testing, but you know, how we got there in the first place is kind of useful for the story. So in the oldie days, this is what we did. We basically, in a QA, consents context, we didn't do any testing of package sources. Like maintainers might do it for themselves, but that was outside of the scope of QA. Build testing at the build level, again, we really only had what maintainers did themselves in the check section of the package where a package build can run test scripts and fail if they don't work. At the update level, we had manual testing. So this is all, you know, circa say 2013 or something. There was manual testing of updates, which was kind of haphazard. We have this little mechanism where if a wiki tests in the wiki, especially tagged, it will show up on the update page. So we can kind of give you some guidance. This is a test it might be useful to run on this update, but that was about it. Other than that, people would test the update and say, yeah, it's good. What did they actually test about it? We don't know. They just said, yeah, it's good. Or they didn't, if they said it was bad, they would normally tell us why, which is useful. But yeah, it's very haphazard. There's no way of knowing that what you're testing from update to update is the same. And sometimes an update just wouldn't get any testing. So do we know if it works? No, we don't. We have no idea. And it's very difficult manually to check if an update breaks initial install of the system or the compose process. And we'll get to that later. This is an advantage of, but that was a thing that was very difficult to do with manual testing. For compose testing, that was probably where we used to focus our manual testing the most. So we would put a lot of work into manually doing what we called release validation testing, which was, it's focused on, okay, we have a release candidate. Is this release candidate ready to go? And we would run over a hundred manual tests to decide if that was the case. And that gave us pretty decent coverage. We didn't ship many completely broken releases, but it's very time intensive. It's very boring because you just sit there spinning up virtual machines all day. Camille remembers these days when we would just spend weeks just spin up a virtual machine, run and install this way. Does it work? Okay, do it again. Do it very slightly differently. Do this until you're tired of life. Yeah, so that was an issue. And even the amount of resources we put into it, we couldn't test every compose. There's a compose every day. We would go insane trying to do that. So that was another limitation. The consequence of only being able to test periodically is that you get problems where there's a bug and then by the time you've noticed the bug, there's another bug. So you have these kinds of stacks of bugs. By the time you get to testing a compose, there's these four or five different bugs in it, all of which you could have caught one at a time if you could test every compose. And it's, if you don't test, one of the nice things about testing very frequently is that it's easier to identify what broke something. So if you don't test very frequently, you're looking, say we ran a test on a compose and then we do another test two weeks later and we see a problem. We know that something that changed in the last two weeks caused the problem, which is better than nothing, but that's a lot of things to check. So there's a lot of issues with only doing manual testing. Just some quick sort of history of how we used to do, how we did validation testing and where we got to before we started automating. On the left here, it's, I have to shrink these down a bit, but this is the earliest organized QA testing of Fedora that I could identify as from Fedora core five in God knows when that was 2001 or something. I don't know. There were 26 tests, which I counted out of this table. So then by the time we got to, oh, 2006 I have the note here. By the time we got to Fedora 21 in 2014, which was the release before we started automating things, basically we had 138 tests. That's not all of them. That's just some of them that had to be done manually for every compose, which was that was a lot of work. So we'd gone up six times and that still wasn't testing everything and it was taking up a lot of our resources. So that was the point of which we were like, okay, we need to change something or we're all gonna go insane. So that's kind of where we got up to. And just as a side note, after we started automating things, we've continued to grow the test set. So as of 2023, the Fedora 39 matrices have 202 tests on them as I counted. This is, I've done this talk three times before for anyone who doesn't know. I have a bit longer today and I'm sorry, I can be a bit goofier. And this morning I found a talk that Will Woods did at FudCon Toronto in 2009. And if I'd had a little more time, I was gonna rewrite this entire talk to do it off Will's slide deck, which would be great, but they didn't have the time. So instead I've kind of dumped a few of his slides into my talk just for fun. I don't know the license on these slides, unfortunately. I asked Will, but he hasn't got back to me yet. I'll update that later. So this is from 2009 from Will's slide deck when he was, this is when they were starting AutoQA, which was the automated testing thing at the time. And the fun part is a lot of this stuff is relevant. This is the stuff that 14 years later we have finally managed to fix. So they were discussing, okay, what makes raw hide broken or good? Can we write code to check that? How can we run that code automatically? All of these are the things we have finally managed to get around to solving an automation recently. So this is all the stuff that we've actually managed to implement. It took a while, but we got there. So what do we have now? Where did we get to? What are the automated test systems in Fedora? What are they testing? That's, this is the main meat of where we're getting to. The two main automated test systems are Fedora CI and OpenQA. The point I wanna get across here is why do we have two? What are they both doing? How do they complement each other? So this has evolved over some time. OpenQA is, was effectively came out of that problem of having to do a lot of manual testing, but still not being anywhere near the amount of testing we wanted to get done. At that point in history, AutoQA had evolved into something called Taskatron, and we were working on that, but it wasn't at the point of being able to automate all of those tests for us, and we wanted to do something. So we picked up this tool called OpenQA that the SUS folks wrote and said, look, this can, we can use this right now to help us, we're just gonna pick this up and use it. That was basically a Skunkworks project which ran on someone's computer under a desk. It ran on SUS at first. So that's where OpenQA came out of, and we've just kind of gradually built it up since there. Fedora CI is a more kind of planned effort but had more resources behind it. It had some involvement from the REL side of things. It was kind of an idea to standardize and share how automated testing could work across Fedora, CentOS, REL, and make it so that RedHack can push some of its testing that it was doing internally upstream. So try and provide a kind of shared environment so we can get some of those tests running outside of the RedHack firewall. And Fedora CI is kind of trying to offer tools and workflows to packages in a form that will be kind of familiar to people who are used to working with CI systems upstream, whereas OpenQA is very different from that. So that's kind of the background where we have here. So the way it's worked out is that CI is kind of intended to be self-service. If you're a Fedora packageer, it's like it's a way you can write tests for your thing and get them run at a time in a way that makes sense to you. You can have the tests run on pull requests. You can have the tests run on package builds. The results can come back to you in Diskit or in Bodi. It's not, there's nobody centrally writing the Fedora CI tests in general. That's not how Fedora CI works. So it's kind of, it's a developer workflow is the idea. As I say there, think about providing CI services to people working in Fedora. OpenQA is more, it's testing of Fedora itself. It's not something for Fedora people to use to test their little bit of Fedora that they're working on. It's about making sure Fedora, the thing we define as Fedora is working well. With Fedora CI, the system is the thing. The people who work on CI wake up every day and think about how they can make the system better. With OpenQA, the system isn't the interesting thing. I'm the, me and Lucas are the main people working on OpenQA. We kind of wake up and say are any of the tests failing? How can we fix that? Can we write some interesting new tests? We don't think about improving the test system. So that's kind of the difference there. So get into some more detail on both the test systems. What is OpenQA? OpenQA is a system for testing which it's kind of designed to test the way a human tests. So OpenQA spins up a virtual machine and then it knows how to type. It knows how to see things on the screen. It knows how to click on them. And all of this is happening basically at the level of the virtual machine. It has no idea what operating system it's testing. You can use it to test a firmware interface. You can test Windows. It doesn't hook into any kind of toolkits or anything like that. It's just looking at the screen clicking on things and typing things. And then, yeah, the way it originally works it's all about looking at the screen, matching screenshots. It's been developed these days so that it does have some integration for Linux consoles specifically. So you can run commands at a console. You can get the output from them. You can get the exit code from them in a kind of programmatic way. So you can write tests that are like, okay, run this command, did it pass or fail? That can be a requirement. So it has that capability. What we use it for, as I said on the previous slide, we use it for high level functional testing. It's like we define, we kind of build off the release criteria and things like that. And we say, okay, what are the core requirements for Fedora? What are the things we want to know at all times are these still working? That's what OpenQA focuses on testing. So we run a set of tests on all Fedora composes, also Fedora CoreOS composes, IoT composes, cloud composes, there's probably some other kind of compose, I'm forgetting. Any kind of Fedora compose, we run some tests on it. ELN we're testing now, which is a fairly new thing. It runs a slightly smaller subset of tests on every critical path update for Fedora. Every single critical path update, nearly, will have at least some of the OpenQA tests to run on it, which is pretty big, across all releases, not just stable, but also branched and raw height. So it's running a lot of tests. I'll get to some numbers in a bit, but that's a lot of testing. So OpenQA is great. The screenshot-driven testing is really useful. As I said, what we've started out wanting to do with OpenQA is test the installer, like automate our installation testing. And it's fantastic for that, because almost nothing else can do that, but OpenQA is really good at it. So that was kind of the entry point. But it is also useful. We use it for testing things like desktop applications. Lucas has been working on a lot lately. And it also, we managed to implement like client server testing with it, which is really useful. So we're using it to test things like free IPA, database servers, just because we have a good architecture for having multiple test machines sort of interact with each other. So it's also good for that. Yeah. Do, do, do, do. Just to go off briefly how OpenQA looks, because I wanted to have a sort of more practical element to this. You're not necessarily expected to look at this. It's more kind of something for me to be looking at most of the time or Lucas to be looking at. We see OpenQA is kind of a service that QA provides, but if you are interested, this is kind of how it works. This is what it looks like. On the left is sort of the overview for a given thing we're testing, in this case, a compose. So if you clicked on one of those little dots, the green or orange or red dots, you will get to the interface on the right, which is the details about that specific test. And you can see kind of it's got thumbnails all the time it's running and green ones mean the thing that was meant to happen there happened, either a screenshot match or a command succeeded. Gray ones are kind of informational or it was waiting for something and red one means something failed. So here we've got a real fail test. We click on the red thumbnail and we can see, you know, it's booted, but it's at the emergency mode, which is not where it should be. So we can see what went wrong and then you can get logs out and start investigating failure. Okay, yeah, otherwise I'm gonna mess up my timings. So yeah, that's kind of how OpenQA looks from my end of it. So here's some details on what we're actually testing in OpenQA today because I wanted to give this talk because we've been developing this stuff for a long time now and I'm not sure people kind of understand the extent to which it's grown and what it's covering because we really are testing a lot of stuff now. So for composes, we're covering 75% of the validation test suite, which are those tables I showed earlier, the 130 tests as of, you know, Fedora 21 or 200 something tests now, 75% of those are automated. We do not need to run them manually, which is fantastic, makes life so much easier. We also do run some additional tests that aren't technically release blocking but are useful. So for instance, we test silver blue because it seems important. It's good to know whether it's working. In more detail, you know, what it's actually testing, it runs a lot of install tests in different configurations. It tests the different images. It tests installing different package sets. Tests a lot of partitioning stuff, you know. Can you install the XT4 or can you install XFS? Can you install ButterFS? Can you do thin partitioning? Can you resize existing partitions? All of this stuff that was incredibly annoying to do manually, it does all of that. It installs in different languages. Since the last time I gave this talk, Lucas has implemented a Turkish install test, which is great. French, Turkish, Japanese, Russian, I think one other. And there are all kind of languages that are interesting for various reasons to do with character set rendering right to left. And Turkish is interesting because it has weird K schools. So this lets us cover, you know, oddities that tend to pop up with different languages. It tests UEFI and BIOS, so it just covers a whole combinatorial mess that we don't have to test manually anymore. But we go a long way beyond install testing these days. So what we call the base tests test, so the really core functionality, like can you install a package? Can you remove the package? Can you update the system? Can you log in? Can you log out? Can you reboot? Is SE Linux in enforcing state like it should be? Can you manipulate services? Which again is incredibly boring to test manually, but it enables the service, disables the service, starts the service, stops the service, reboots about five times, and yeah, you don't have to do that by hand anymore, which is great. Does logging work? We also test upgrades, which again is huge because that was a massive time sink to do manually. So we test whether, starting with a clean install of each supported package set for the previous release and the release before that, can you upgrade to the current release? And then we also test, does stuff work after you do the upgrade? So we know for instance that you can upgrade a free IPA server and it will keep working as a free IPA server, which is super valuable. We do quite a lot of graphical desktop testing. We test the desktop itself. We test things like our notifications working. Can you log in, log out? We test from within the desktop. We test whether every single pre-installed application starts up and quits successfully at least. And we do some detailed testing of quite a lot of apps now, which Lucas has been working on. So there's really quite a lot of testing whether the desktop works printing we test, like updating from within the desktop. And we test all of these, as I said. We test them not just from a fresh install, but we do an upgrade and then we test them again. So that's pretty useful coverage. And we test on Silver Blue as well. We have some pretty advanced tests for server functionality. So as I said, we test free IPA very recently. This is a new one as of last week. We test active directory using a SAMBRA active directory server. We test a database server client test. So make sure a postgres is working. We check cockpit pretty extensively. Not on here, but we also have some tests for Podman. So we're testing a lot of stuff. And all of this is being tested on every single nightly compose of Rohide branched. So this is partly just for validation. So we don't have to do so much work for validation. But also it means we know whether this stuff is broken way earlier in the cycle. Like all the time now we know what's broken. Rather than in the past, we would get to two months before release and we'd start testing and we'd find everything that was broken. Some of these tests, like adding the Silver Blue tests, CoroS victory test, and a lot of the server tests were kind of driven by the SIGs. So the server SIG has been very good at working with us and saying, hey, this is what we want to have tested. So we're like, okay, great, we can, that's valuable to test. So we'll add tests for that. So there's that kind of process going on. So that was composes. What do we test for updates? For updates we don't test all of that stuff, frankly, just because we don't have the resources. So we do a subset, but it's a fairly big subset for, so recently it got more complicated because I kind of enhanced things so that it now tries to run only relevant tests for each update. So before we had this problem where there'd be a KDE update and it would run all the known tests, which was kind of dumb because a KDE update isn't gonna break no. So I've kind of tweaked things around and now critical path is defined in groups and everything along the path knows which groups an update is critical path for. So OpenQA can say, hey, this is only in the critical path known group, I'll only schedule the known tests. But assuming an update was in all of the critical path groups, you would get about 60 something tests. And it tests KDE, it tests workstation, test server, not running every single test we wanna compose but running a pretty big subset of them. It does the most important base tests. It tests the free IPA stuff, the Samba 80 stuff, cockpit database tests, it does do an upgrade test. So it's pretty wide coverage. And something that we only do in the update tests is we try and test whether the update will break the compose. Like I said on an earlier slide, this is very difficult to do manually. In automation it's quite easy. So for every single relevant critical path update we build a network installer image. We build a known live image, a KDE live image and we build a silver blue installer image. And then we run an install from them and then we test that it boots properly. So we know for every update, does this break the compose process basically, which is very valuable. That's one of the most important things we get out of the update tests. And yeah, just as a note, as I keep focusing on for OpenQA, the idea is we wanna know, does this update break Fedora? Does it break something important? So it's not so much that we're testing Postgres because we're really interested in databases. It's because database functionality is part of the server release criteria. So it's something that is meant to always work in Fedora server. Therefore we test that it always works. And the test aren't meant to tell you is this update to package X a good or bad version of X? It's more, does this version of X break the things that are important to us in Fedora? That's the idea. The main limitation on how many tests we run is just capacity. If I had more OpenQA machines we could run these tests on every update but I don't, so we can't. And yeah, just to give some of those numbers as I mentioned earlier, like it's getting pretty big. This thing that started out as a tiny little Skunkworks project under a desk. So we have two instances now. We have a prod and a staging instance. Across the two, we have over a hundred simultaneous test executions. We've run over three million tests since 2015. So that's kind of a lot. Discovered hundreds of bugs. I tried to get a precise number but it's hard because often we report the bug upstream. Sometimes I just fix the bug and never file an issue. But there's at least 358 bugs tagged in bugzilla as having been found by OpenQA. And there's a lot more in upstream trackers and things like that. So that's a lot of bugs. On a typical day we'll test one or two composes depending on whether Branched exists. We'll test three CoreOS and Cloud Composes and we'll test about 20 updates. So that's thousands of tests a day for sure. I should have updated the third bullet point. This is recent as of dev comp for a couple of months ago but just some recent things that it found at that time. We had a change go into Fedora 39 to make the EFI system partition bigger which broke a surprising amount of things and OpenQA found most of them and we were able to fix those. There was an update that made Firefox crash on startup so we caught that and that didn't ship to users and give them a broken Firefox. The Arabic translation just disappeared from the installer one day so we caught that. Yeah, that was a fun one. Nome was notifying about dates when running live which is something it's not supposed to do when you're running live. We don't want you to try and install updates because it'll try and install them to memory and the whole thing will hang so it's specifically not supposed to do that and we caught that and fixed it. So it's catching real issues all the time that are getting fixed. And an example of something that we caught with update testing was that a new Utah Linux update mounted the group partition read only which not on silver blue on RPM installs which obviously breaks everything and because of the update testing we were able to make sure that never actually landed in Rohide and nobody got that update and got a broken package system so that was pretty cool. OpenQA resources, yeah. So I've mentioned this kind of in passing but the idea with OpenQA is it's a full service system like as I said you don't really need to go and look at the OpenQA web viewer. I when something fails in OpenQA one of us the QA team will investigate it, try and figure out why it failed. If it was just a blip we restart the test. If it's a real bug we'll kind of investigate it and file a report in a format that's useful for you as a package maintainer whether that's to bugzilla or upstream and we'll try and provide the useful information on why it's failing. So you don't have to go and do that debugging work yourself but that's kind of the idea of how it's meant to work and we develop the tests, we run the system. So we're kind of trying to provide an end-to-end service there. If you need to contact QA about this we have a mailing list, we are on Fedora discussion as well you know the new forum thing and on Fedora chat there is a Fedora QA room where you can find us. The other things are just kind of references if you download the slide deck but the upstream site for OpenQA and there's a downstream, there's a wiki page where I kind of explain the whole setup and if you want to get involved with it you can. I added the last bullet point after DevConf because Steph Walter had a really good talk there about open source services, the idea that you should be able to inspect and change services as much as you can open source code and OpenQA does try to be like that. Everything involved in the deployment of it is open. So if you are interested you can kind of dive in from the wiki page and you can look at and contribute to any part of this system. Moving on to Fedora CI. This slide, all the CI slides were contributed by Miroslav Vagkoti who is one of the sort of leads on Fedora CI and testing forum so thanks a lot to him. At DevConf he co-presented and did these slides way better than I can so you can watch the recording of that if you want to see what he has to say but. For Fedora CI as I mentioned it's not an end-to-end service which is testing whether Fedora is broken. It's a system you can use to kind of improve your testing of your package. So there's a doc site which kind of gives you a quick start in how to onboard yourself to this system. The test it runs are defined in package repositories so you keep the test in disk it alongside your package with a little bit of configuration that tells the system how and when to run them and they're expected to be maintained by the packages. They're not maintained by the team behind Fedora CI. That's not how that works. And it's meant for component level testing. Fedora CI isn't really targeted at the kind of high level integration testing that we're doing with OpenQA. It's more at the lower component level of testing. There's two formats you can write tests in but you should use TMT, STI is going away and yeah you can link to the quick start guide there. There isn't a workshop tomorrow. That was also a dev comp. I should have fixed that. Sorry about that. There are few generic tests that Fedora CI runs which are kind of a hangover from AutoQA slash Taskatron which are tests that get run on every single package. And that's RPM inspect which David Cantrell who maintains that is around. RPM Deplint which tries to check for broken dependencies and installability which just tests whether the package can be installed. And they run in CI just because it's a sensible place to run them. So this is kind of when you can run tests and where you can get results. So Fedora CI can run on pull requests. So if you actually do pull requests for your package which some packages do some don't but if you use pull requests you can have test run. So when a test run or pull request is created there will be a scratch build run. I think it's running coper, I'm not sure. And CI will test that. So it doesn't ever have to touch Koji and you can get the results back on the pull request page which is kind of a good workflow if you're committed to using pull requests for your package. But it also, it tests whenever a package is built in Koji. If there are tests defined for that package it'll run them and then the results will be shown if you create an update from that package build you'll see the results on the Bode page. So that's another place where they get hooked in. And yeah, under the hood stuff isn't so important but testing farm is kind of the back end. It's the kind of combined back end for all of the Fedora CI but also CentOS Stream CI uses testing farm. Some of RAL's testing runs in a testing farm instance so that's part of the project of trying to unify things. Same experience with RAL CI, CentOS Stream CI and packet which is a cool thing if you haven't heard of it. If you buy into packet, if you're a maintainer upstream of the thing you're packaging in Fedora you can kind of do everything in the upstream upstream repo, you can have your spec file defined there and then all of the downstream stuff is kind of done for you by the system. You don't have to do all of that stuff. You will get pull requests for the package when you tag a new version upstream and tests will run on that and you can just say, yeah, this looks good and then the build will happen and the update will happen. It's a really cool system if you control the full stack as it were. Oh yeah, hardware crimes in TMT, that's just if your test needs some kind of specific hardware you can now define that. There are a few things that I kind of see as testing systems, like Fedora CI and OpenQA are the main ones but some of these things kind of feed in as well. So Cache is, it's a system that tests any time a package's dependencies change it will kind of do a test build of it and see if it works, which is a tool for developers but effectively it's quality testing as well so I see that as kind of a quality system. A Fedora release auto test, Lily is not here but she's around, she's a member of our team who we're meeting for the first time here and she wrote this thing which is really cool. We have this problem with some of the requirements, the tests that are required are used very weird hardware like high end enterprise storage stuff that we just don't have lying around in our houses or in the Fedora test system. So Fedora release auto test runs those tests in Red Hat's test farm called Beco which has access to, oh Lily is that, sorry I didn't see you at the back there, good job Lily, which has access to a lot of really exotic hardware that we don't have anywhere else so if she, it used to be a nightmare we would have to find someone to run those tests manually so now those all get run automatically by that system which is great, it saves a huge amount of pain every cycle. This stupid tool I have called RailVal which is related to creating the validation events as just because it was a sensible place to put it actually runs the size checks so if an image is oversized, RailVal is a thing that finds out it's oversized and files a bug on it. Fedora Core OS, because of the history of where Core OS came from, has its own CI system which is kind of cool and does a lot of testing on Fedora Core OS and they have their whole separate release workflow so that's another automated testing system that's out there. I mentioned Packet and Zulu Base CI is kind of part of Fedora CI and the testing farm back end but that is if you have a project hosted on Pagger.io you can get testing for that run via testing farms and stuff, not very important. More guest slides, so yeah, wait, why more guest slides? I must, is this a messed up version of this talk? Dang it, I'm completely ruining my joke here, nevermind, oh, how did I get past this? Skipped over two slides, oh well, no I did, I talked about that, okay, I did, I'm sorry. I've had three hours of sleep, do I? Do you excuse me? So yeah, we have a couple more slides from Will's talk here which I just thought would fit in really well. So this kind of ties into the bits I just talked about. So this was in 2009 remember, so coming soon-ish, this is the stuff they were trying to figure out back then, so easy auto-QA for packages, this was the idea of making it so you can write tests for your package, so that's what Fedora CI does and it works now. 14 years later we got there, it's pretty good. Post ISO build hook, so this is where Will was saying, yeah, we have this huge install test plan which he's talking about all those Wiki test pages. It would be great to be able to automate those, right? We got there, open-QA does that. And use Fedora Message Bus, so at the time, auto-QA was doing this crazy thing called watches, so it would just wake up a script every hour and see if something had changed. So obviously now we have a proper Message Bus and that's how all of these systems work. So yeah, we got all those and they're coming eventually, someday probably maybe Sly, they were like, hmm, how can we do multi-host testing? That seems hard, open-QA does that. We do test HTTBD, we do test NFS installs. Functional testing for GUIs, open-QA is great at that, that's how you script that. So we got there. Well, Domination is still a work in progress, but we're getting there, we'll be there soon, don't worry. Tim, okay, quickly. What about the few collages? The Pina collages, I've drunk a few Pina collages. The results, if you've ever noticed on the Wiki, the results from open-QA are filed under the name Coconut, because one name for this whole thing was Project Coconut, the idea being that we wouldn't have to do any testing anymore, we would just let the robots do it and we'd just sit on the beach and drink Pina collages, so yeah, doesn't quite work that way, but you know, it's better than it used to be. Anyway, getting back to the serious topics. So testing is only, this is something I'm kind of big on, testing is only one part of this whole process. It is useless to run a bunch of tests if nobody ever looks at the results and does anything with them. Like, one of my saddest things is when you go to an upstream project and their little thing says 60% of the tests are failing and have been for the last three years, you know? So it's crucial to make sure the results are going somewhere where somebody is doing something with them. So, and different ways of getting results make sense to different people. As I said, the open-QA web UI is kind of for me and Lucas to go and look at and figure out what's going wrong. We don't necessarily want packages to look at that. What makes sense for packages is to use the interfaces that they're working with, you know, Diskit, Bodi, that's where you want to go and look and get your results, right? So different routes make sense to different people. One really key thing I'm going to talk about, I'm going to start saying the word gating a lot and gating just means, instead of just letting this thing through and testing it and finding out if it's broken, we don't let it through unless the tests pass, which is a really key thing we've been discovering over time. And the point, the thing we're trying to achieve with gating is to catch problems before they infect other processes and to catch them before people see them on their systems. So we don't want problems getting to where composes are broken, you can't run package builds because another broken package got in and broke the build route and we don't want people updating their systems and seeing bugs, so that's why gating is important. So in practice, where do you get results? So as I mentioned, for Fedora CI, the earliest point you're really going to get your results in a way that's useful to packages is in diskit. So if you're using pull requests, you can get the results of tests in there. And this is a fairly old screenshot and it's a little zoomed in, a little squished down, but you can kind of see it. So this is a pull request for a package and then down at the bottom, you've got like three little green blobs and those are results from Fedora CI saying, okay, this is, this one passed its test, this one's fine, so that's one level. The benefit of this level is it's the earliest possible point. So if you're testing at this level, this is before anything gets even into Koji. And if you're using pull requests for everything, this is great because you know exactly this pull request broke it. You can, you have a very tight development loop if you're testing at this point, right? The drawback is that you do have to use pull requests, which a lot of packages are not used to doing. If you're the only person working on a package, it's kind of weird to go and create pull requests and look at the web interface. You just kind of want to Fed package commit, Fed package push, Fed package build. And if you have cases where packages depend upon each other, so you need to update two packages at once and if either of them won't work without the other, then it's kind of difficult at this level to do that. So the other major integration point we have is Bodhi, which is the update system. So this is integration at the update level. So the key advantage is just the thing I was talking about is if you have packages that are interdependent, Bodhi is where you group them together and say these packages go together and if we test at that level, we can test the group of packages that should work together and see if they do work together. And it's also just by default for packages who don't use pull requests, this is where you're probably gonna see your test results. So yeah, it displays the results from OpenQA and Fedora CI. Fairly recently, it's been updated to also show when tests are queued or running. Like before a test actually completed, Bodhi would just tell you the result was missing, which was a bit confusing and scary and that was just because the system hadn't got to it yet but now it does actually tell you it's in the queue or it's currently running. Yeah, we had some issues with this in the past and I've kind of been working on it to try and make it as consistent, as accurate as possible. There are still a few things that we know are problems like for instance, it will let you try and wave the result if a test is running, but that doesn't work. So you can click the wave button all day long and it'll file a waiver but the actual system that decides whether the update can go or not doesn't account for waivers on a running test. So that's something we want to improve but we're basically trying to make this as accurate and consistent as we possibly can. The in-bolt is there because it's kind of a big deal. Since the last time I gave this talk, all updates for all releases are now gated, including Rohit. So for the first time in basically Fedora history, when you do a build for Rohit, it does not go straight into Rohit. It now waits for the OpenQA test to complete and if any of the critical ones fail, your update does not go into Rohit. So this is one of the biggest things we've done for a while so I'm kind of proud of it and also scared of it but that's happening and nobody's killed me yet so I guess it's going all right. So yeah, that just means nothing gets in unless it's passing the test basically which is pretty huge. On a package by package level, you can actually configure the gating requirements. So for the OpenQA tests, they're all kind of the same but if you put tests for your package in CI, you can also add a file in the repository which says gate on those tests and then your update won't go through unless those tests pass as well. As I said, there's a button for waving a failure so if you're sure, if you're really, really sure that a failure doesn't actually mean things are broken, you can wave it. I would like for people to be very sparing with the use of that button because the problem if an update that has failures goes through is that then possibly all subsequent updates may have the same failure which is gonna be an issue so but it is there if we need to use it and yeah, the way Bodhi works is it listens out from messages, Fedora messages from results DB and sort of updates its situation. A couple of things that we know about this at the moment, if you've noticed problems with Fedora CI test results like being wrong or just not showing up, Fedora CI is having issues with DNF 5 and Python 3.12 which are kind of affecting the test running so that's why the CI folks are working on that really hard but just because of the details of that system, it's quite difficult to fix all of that and some OpenQA tests recently have been affected by this annoying problem that Kevin knows all about where we're getting a 404 from a repository which is making the test fail. I'm hoping something I did this morning will kind of make that less of a problem but if you've had an update held up for a few hours that was probably why and we're sorry about that. As a package if you see failures in Bodhi what should you do? This is a question people ask me so I wanted to explain it to you. If the test failure is for a Fedora CI test that's not one of those generic CI tests then that's kind of on you because as we said Fedora CI is something for you to use so you should kind of debug your own failures for Fedora CI fails. When you click on a failure it should take you to the logs. It takes you to the Jenkins page I think and from there you can get to the testing farm page with more details. If it's one of those Fedora CI generic failures check the logs from the failure and see if it actually seems to be caused by your update in which case you should fix it obviously. If not talk to the CI team and see if there's a problem with the system and that's got some links to where you can do that. For OpenQA in general you don't have to do anything. If you're impatient you can go and look at OpenQA and try and debug it yourself but what we aim for is that within 24 hours one of us will investigate it and either fix it or tell you what the problem is in a Bodhi comment, a bug report, an issue report something like that. If that doesn't happen or if you really need to figure it out right now you can contact us via Fedora Chat or via our mailing list or any of the other ways you can get in touch with us and we will try and help. There is also a button in Bodhi to rerun tests. When you click that it sends out a message that both CI and OpenQA listen to and just retrigger all tests for the update. If there's a failure that you're not sure whether it's genuine you can click that button and have all the tests rerun which is much safer than waiving the result and only waive the result if you're really sure it's okay to do so. And for problems with Bodhi itself you can contact the CPE team in the Fedora Applications channel or you can contact QA but we'll probably just go and ask Kevin to fix it. And one more integration point, the Fedora Wiki. We still have those giant Wiki pages full of results and the reason we use it is simply because we need, we still have some of the tests are still manual so we need a place where all the manual and automated results kind of go together and when we're doing the meeting that decides whether we release Fedora we just wanna have something we can look at and see all of the results. So sadly the Wiki is still apparently the best way to do that. So there is a dumb system I wrote which actually reports the results from OpenQA into the Wiki automatically and then humans file their own results into the Wiki and then we look at it and we can see all the results together. RelVal's size check results also go in that way. So behind the scenes this stuff, so what do we got, 15 minutes left so we can go into this a bit. This bit of the talk is a little bit optional but here it is. Behind the scenes there's a lot going on to make all of this work together. It's quite complicated I think because a lot of this stuff came out of an idea we had 10 years ago called Factory 2.0 when at the time the buzz concept was microservices. So I don't know if anyone remembers a few flocks back but we had these giant whiteboards which Ralph Bean would stand in front of with all these little boxes about how Fedora was gonna get built and this bit would talk to this bit and this bit, and this bit. And so about half of that got done and because those bits are still around we still use them. So there's kind of a lot of complication in how all these things talk to each other. The most important thing is Fedora Messaging which is a message bus. So the way CI and OpenQA know how to kick off tests is that they listen out for messages from Koji or from Diskit for pull requests that say hey, a new package build happened or an update is ready for testing so they listen for the message and then they test the thing. And then they also communicate back via messages so both OpenQA and Fedora CI publish messages when they've completed the test. Results DB is where we keep the results so both Fedora CI and OpenQA file their results in Results DB which is kind of part of Tascotron so Tascotron is still with us. Waiver DB is where the waivers live which is just a really simple, it's a database of waivers with a JSON interface. It's very simple. Greenwave is the thing that it decides when we talk about gating the way gating actually works is the Bodie sort of asks Greenwave whether this package is okay basically. And all Greenwave does is look at the results, look at the waivers from Results DB and Waivers DB and say yes or no, it's, you know, if we were writing this from scratch today we'd probably just dump it all in Bodie because the idea was lots of different things would use Greenwave but as it turns out the only thing that uses Greenwave is Bodie but this is how it got written. We have a lot of different pieces. Yeah, so that's kind of how it all, and one, I guess an interesting thing about that is that Fedora CI and OpenQA look like very different systems but there is a lot of kind of integration going on and we have this concept that at the point where you're looking at the results in Bodie it doesn't matter that much which system they ran in because everything is kind of integrated in terms of where the results go and how the gating works so which system the test ran in is kind of just a detail so that's kind of how we looked at integrating those systems. This was the most popular slide that I added in the second version of this talk. If you would like to say you're in this talk but it's been very boring and you've just been looking at cat pictures for the last 40 minutes just look at this slide and you can say you were here. This is the entire talk in one slide. So OpenQA tests updates and composes Fedora CI test package builds, we test composes, we use those manually, updates, get tested through Bodie and we do gate those and yeah, that's the whole thing. So yeah, the slide deck is on shed so you can grab the slide deck if you wanna see all of these and it has my notes on it which have some good stuff on it. Something quick? Over you. Yeah, over there. And just a question that popped up in my mind I received a Buxilla report a few days ago. One of my packages was causing a conflict executable by the same name. Would that be caught by QA gating these days or? No, so that's not something we explicitly test for so it's something that could potentially go into CI I guess is a generic test but we don't have one right now. And I say with OpenQA it's very focused on kind of functional testing, you know, is Fedora broken. So we would catch something like that if it caused something else to break. So for instance, we'll catch dependency problems if they make one of the things we're testing fail but we won't just catch any dependency problem in any package in your update because it might not affect the test. So the same thing there is like if that conflict actually caused, you know, the free IPA test to fail, then we would catch it. But we can't say we would catch all of those issues at this point in time, no. Would it be Koshai eventually catching it? Yeah, possibly Koshai. That's the kind of thing Koshai could be extended to. Sounds like a good test. Yeah, or as I say, it could be a generic test in Fedora CI. It sounds really like a RPM inspect kind of. Yeah, a RPM inspect. I don't think it needs a RPM inspect but like you're saying it's a generic kind of thing. Yeah, the things that we run. It's one of the most defined areas of one. Yeah, yeah. So that would probably be something that would go into the Fedora CI workflow somewhere. I just, that was my question and I have an anecdotal effect. You just mentioned an old FodCon 2009. Yeah. Apparently at two this year. Because if I turn around. I was talking about the Toronto one. Yeah. Yeah, I'm having the other one. Okay. Yeah. All right. Something's with me. So you mentioned the RPM inspect. And I was just looking at the last eight updates that I made and they all fail because RPM inspect is very inflexible. And it lectures me that I should talk to product security at Red Hat to allow a code point in my test file. And the use of a function in my test file I should rethink how I program. And the third message was something else. Yeah. Like, I don't know, I would love to have system D installation tested by RPM inspect, but it doesn't happen because it just refuses to install system D. Yeah. How can we get like manageable opt outs and opt ins of tests? That's a good question. So David Cantrell actually is here and he's the maintainer of RPM inspect. He's here, he did a talk on RPM inspect this morning. So he would be a great person to answer those. I know he's trying to make it as configurable as possible. From the perspective of this talk, RPM inspect is not a gating test unless your package decides to make it one. So nothing will ever be gated on RPM inspect unless you say that in a gating YAML file. What I was looking at the same stuff during David's talk just out of interest. And one thing I noticed is that RPM inspect has like a good inspect or bad. So a lot of the things are inspect which is kind of between a pass and a fail. But anytime we get anything but all goods then the RPM inspect result is reported as a failure. So that's kind of one thing we could maybe improve on. And the other thing, yeah, I know that he will take feature requests for being able to configure to say this, to exclude this warning for your package and say, I think this is fine. So you can get that into RPM inspect at some level but he could give you a much more detailed answer for that part of it. But you can add RPM lint RC to this thing. Yeah, I believe that's kind of a good word. Which say ignore these kind of errors. There's supposed to be some way you can just basically exclude specific failures but I don't know the details of it. If you put foo.spec.rpmlintrc Yeah. In the package. Right there config which error should be ignored, then it will be ignored. But I guess another thing, if RPM inspect doesn't run the installability test, if the static test fails, I guess that could just be changed and it could run both tests always. But I guess David might have a reason not to do that. I'm not sure. In OpenQA we run all the tests, we don't stop if one test fails. So I just want to get through my last slides before we do more questions. The future, so this, it's kind of funny to think about, is this a lot of testing? Because to me it feels like a lot of testing but you can also look at it as not really enough testing because an operating system is huge. There is so much stuff we could test that we're not testing. It's an infinite ocean and we're just trying to make the drop bigger, right? But we can always try and make the drop a bit bigger. We do, yeah, yeah. So our limitations as again as I said is just both the capacity to run tests like OpenQA has a certain amount of hardware and I can't run too many tests or it just gets backed up. But also the capacity to analyze and act on the results. As I said, a test system which is just spewing out failures that no one has time to look at is a useless test system. So we also need people to, I try and keep OpenQA down to the level where we can inspect every important failure and do useful things about it. And I'd say we've come a long way. We're not at the point where we're doing CI for an operating system yet, which is very difficult but we can keep trying to get more. For OpenQA we want to, we keep writing tests, especially Lucas is always writing more tests. So we're going to test more applications. I want to do more podman testing. The podman team is keen on that. We want to test flat packs because there was a case a few weeks ago where a flat pack install just stopped working. We weren't testing it. So we want to test it so we can catch that. We've got validation requirements for toolbox which need to get automated. So we have a whole pipeline of new tests to write. I'm always working on trying to make the grow hydrating thing as smooth as possible. So that's a future plan. More architectures would be nice. We have ARCH64 right now but it's very hardware constrained. So we don't run the update tests on ARCH64 because we don't have enough hardware. It would be great to have S390. We do PPC on staging but not proud again because there isn't enough hardware. We got a cool plan to do bare metal testing in OpenQA using a thing called PyKVM which is a little Raspberry Pi based box which would basically let us treat a real system. It can inject mouse movements and stuff into a real system the way OpenQA does with a virtual machine. So we should be able to kind of run our tests on real hardware using that which would be cool and Lucas has been working on that too. And more tailored update tests is just kind of like since we now have this mechanism for running different tests on different updates we could maybe run the entire install suite for an Anaconda test. An Anaconda update for instance. It would be nice to do that and possibly move it to the cloud which I've been looking at for years but maybe it'll happen. Fedora CI plans. This again is Mirro's slide so I don't know what all of these are but multi-host testing is kind of that thing where you have different tests talking to each other. More kind of back end improvements. Finally get rid of STI and testing for our reservations is like again, reserving specific hardware and just kind of improving their error rate and stuff. So, yep. Oh and another thing I wanna work on is there are things that aren't packages that change and break stuff. So you can make a change to the kick starts and break the compose. And right now we won't catch that because we don't run a test when you change the kick starts. So I would like to write all the integration bits so that when someone has a pull request to the kick starts we will run the open QA tests and find out if it breaks. But it's just, it's a lot of stuff to work. Same thing for comps. Pungy Fedora is the repo where the compose configuration goes. And again, if you change that and get it wrong you break the compose. And Workstation OS Tree config is misleadingly named. It's where all of the OS Tree definitions go. So not just silver blue but also Kinoi, Sericea, all of those. And again, they have sort of configuration which if you change it you break the compose and we don't test any of that. So I would love to be able to test all of those things if we could. And yeah, so thanks to Miroslav again for the CI slides. Thanks to Mo Duffy who made the template I used here. And if anyone has any more questions I think we have like three minutes. Yes. Not a question but a comment. The coming or the goals. TMT now has multi-host testing. Yay. Yes. And there's reservations in testing form. Yay. So yeah, I should have got Mirro to update this slide again. It's kind of fun. It's very recent. That's like in the last month. It's kind of fun because I've done this talk four times and every time like I've had to revise things. It's like, if you go back to the first version of it it's like we weren't doing raw high gating. We were like, CI was totally different. It's kind of fun how fast things are changing. Any more? Oh, any more. I don't know whose hand went up first. Amita. What do you mean by moving to the cloud? Is it the testing moving to the cloud? Yeah. Open QA moving to the cloud or you wanna test the cloud as well? I see what you mean. Yeah, no, running the test system in the cloud, running Open QA itself in cloud. We do do some testing of the cloud in Open QA already. We test cloud images. But yeah, so as I keep saying our constraints, our resources like we have literal hardware systems in the Fedora data center running the Open QA tests and it's quite difficult to get more of them, get them racked, get the networking set up. If we could move it to the cloud, we can scale up and down as much as we have money for as much as Amazon will give to us, right? But so there is actually a project going on with some folks from Amazon and Meta to get an intern, I think a Meta intern who will look at doing this because it's kind of, it's a several week project and I never have several me and Lucas, we never have several weeks to set aside to look at this so it would be really cool if a person can kind of do it and lay the groundwork at least for us to do that because that would let us scale much further. Thank you so much. Just a real quick one that was reminded by the RPM Inspect stuff. Fedora CI, one of the tests that it does is install all the sub packages to make sure they're installable. That fails every time on Fedora Release, the Fedora Release package because it has all the variants and they can flick with each other. So there may be, if there's some way we could fix that, that would actually be great, but I looked at it and I couldn't see how to exclude that, so. Yeah, again, I think it would be really good to talk to David about it because he is very open to like, this is a problem, how can we fix this problem kind of feedback? So I think, and he would definitely be the guy for that. Yeah, do we have time to take the Spigniavs quickly or I don't think we have another session. So I was thinking about it because I feel the same problem in SystemD and basically there should be a way to say, try those groups and open QA, you had this slide where the list of screenshots goes from green to red and back to green, what does it mean that it's green at the end? Let me get back to the screenshot. There's a lot of slides in this talk, my word. I just kept adding those, mothers. No, oh, yeah, okay. So that is a thing in open QA called the post-fail hook. So when a test fails, because we want to get information out to know why it failed. So after the test fails, it runs like a, these things down the left are all the tests, like graphical weight login there or all the kind of test modules that make up a test. When a test fails, it jumps to a special one called the post-fail hook, which just what you use it for is like, grab the journal and upload that, grab a bunch of log files and upload those. We have special ones, like if it goes to the emergency boot prompt, it recognizes that, and instead of trying to get things out the normal way, it gets them out in the serial console, stuff like that. So yeah, that's why. So that green one at the end is actually just the post-fail hook, like succeeding with something. So if you are working from the web UI, when you get a failure, you generally want to look for the last red square, and that's where the test actually failed. And anything that happened after that was just, post-failure informational stuff. Anybody else? All right, thanks a lot for coming out, guys. Thank you so much. I just wanted to thank everyone who came out today. That concludes for this room for this afternoon. So there's other tracks going on and a few more sessions. And I believe some lunch and things like, or dinner. If there's a games night. Games night, that's it. Games night here a little bit. So please feel free to check your schedules online. And again, thank you so much, Adam, for your presentation. Thanks, everybody. Oh, and don't forget to get your badge if you haven't done it already. There's a QR code in the back. I just love reminding everybody to get your badges. Badges are great. If anyone wants to talk to you, there's a Badges talk like now, so. Yeah, actually I'm gonna head down there in a minute. So thank you, everybody.