 So, clarification. It's not my fault. Pingu should be standing here. He has some virus. He didn't want to spread it around. So, I received the whole transcript yesterday. I tried to learn it. If you ask me a question, there is Nils and Michal and I will throw it back at them and they should know what this is actually about. I know some things about the testing site because that's when I got in. But let's get to it. I am talking about gating Roheight and this is the talk where we announced that, yay, we have it running somewhere. So, agenda. First, why we want to gait Roheight and what were the challenges? If anybody saw Pingu's talk on previous flock, I think these are the two things he already talked about. Then I will talk about what actually is, let's say, live, what we've done. Maybe some hints about how to interact with it when we need to suddenly debug because it's stuck somewhere and you actually want to use it. And some roadmap because, of course, it's software so it's never done. Last but not least, Pingu has a surprise for you. So, why gating Roheight? The primary reason we want to have more stable Roheight. So, we want to have something automated when you want to push a new package, new update that checks it didn't really break anything. So, it should have many advantages. It should give us working composites and composites not being broken that often. And hopefully, faster updates when we are actually doing release. And one thing we would actually want to achieve with it is if we manage to implement it correctly, it shouldn't happen to you anymore that you are maintaining your package and somebody broke one of your dependencies and it's your problem. So, we would move to, you broke it, you fixed it, you coordinate, you tell the people downstream that, hey, I'm doing this update. I don't know. I'm doing a surname bump. We need to figure this together instead of just pushing an update and let all of the others deal with it. Like, that's not something that we really have yet but we are moving to that. So, where are the challenges? First, biggest challenge, way Pingu told me is we actually attempted to do something like that a few years ago and we didn't manage. He didn't tell more of the story. So, I don't know what were the problems in the previous iteration. Second, no big sweeping changes to the update system. We definitely don't want to maintain like, I don't know, raw height specific body or something just to have this gating feature. Third, interrupt as little as possible our packages. Like, there would be some changes to package workflow because we are adding this new feature but have it fit into the existing tooling and limit what they need to learn, et cetera. And last but not least, of course, we built this new testing tool for the packages and then our testing tool fails and the packages suffer because they can't get the update in. So, there needs to be some sort of escape hatch. If that happens or if you just know you need to push this, there is a way how to wave the test results to ignore them. So, this was the plan. We need both single build update testing and multi build update testing and we would face it in a face draw out and then we would be able to gather feedback instead of doing like a big bang release. And we really wanted to have a good user experience once we go forward with this. That's something I shouldn't ask in the questions to you guys if those of you who actually had to interact with it like what was your experience. So, we actually had it running now. It has been deployed on body and staging and production for a few weeks and you now can see that builds are gated in Rohide both for single and multi builds. So, how does this work? Let's first look at the single build and this thing is just to scare you off. You don't need to really pay attention. It mostly is to signify that, oh, look how many systems we need to touch to do the updates. Like, all of this part is the testing part and tomorrow morning there is actually a talk just about the result db, greenway, abate and waiver db, the parts that actually do the testing. So, let's look at a simplified flow where you see just the thing that actually happens. This is a simplified implementation. We actually in the end went with something a bit different once we had multi built gating because we wanted to have them fairly similar. So, once you kick off your build, the build is tagged with updates candidate, then it waits to be signed, then it gets re-tagged, updates pending and that's kind of the sign that, all right, now we are waiting for the tests, then the CI system notices to make it clear. If you look at this column that is like populated with all of the stuff, this is actually federal messaging and all of the messages that are sent back and forth to actually facilitate. I think Pingu told me he will share the full resolution. So, I think once this gets published, you should be able to study it in detail and tell us when we have like weird stuff in it. So, Bodhi then queries green wave to see how the testing went and if it passed, it can let package to stable row height or it will gate it. So, does that make sense? Okay, it looks at least two nails. It was one of the people who built it. It makes sense. That's good. So, multi-build. Let's start with the same horrible image. There are kind of two things. The way we try to solve multi-build is we are using site tags and coaching. So, in the beginning, there's like part that deals with site tags and in the end as well. So, this is the simplified thing. So, first you would have to create a site tag and then you build all of the packages that have to go together in one update in the site tag. And once you are happy with everything being built, you can tell Bodhi, all right, this thing, this site tag, that's my update. Run with it. And then CodeG will tag the site tag for RoboSignatory to know that it should sign the packages. And after that, it gets re-tagged with testing pending. And that's how Bodhi knows, all right, now we are handing it off to the CI system and Bodhi queries again the CI to know how well it went. So, this is probably the most significant thing because you just create a site tag. And from what I understand, this haven't really been the case. If you wanted to have a large update, there was a fairly involved process to have site tags or in the largest of projects use them. But we worked with upstream CodeG to simplify the site tag creation so that we can actually use them for this process which we want to be running quite often. One of the things you might have noticed in the previous slide, we are then creating an update in Bodhi. That's something we didn't need to do for the single package because it's just a single package. So, Bodhi can just know that it should create, like continue with the build. But that's something we needed to change. So, now the process for the single build and multi build is more similar. Okay. Like, if you don't want to go to the tomorrow talk about green wave, etc., this is kind of a high-level overview of what actually happens when the tests are running. If I remember correctly, currently, the CI systems is mostly the CentOS CI. We update the above the results to the results DB. And then green wave is the service that can check what was the result of the actual test for the update or if we don't care and if we just want to wave it because, I don't know, some part of the system might be broken or we really need to push the update or something. Like, you need the specific permission to be able to just wave the test results but it's there and that's what the green wave is for. And there is even like CLI command for that. You can do both the updates wave and both the CLI. So, now, if something gets stuck, what can you do? Like, you created your update. How do you know what is actually happening as you saw in the... Let me go back here. As you can see, like, the biggest state change that happens that is visible are the tag changes. So, if you list the tags on your build, then you can see if it is waiting on the body or if it is waiting to be signed or if it is waiting to be tested. There is one more tag. It's just updates test testing, which means that tests are in progress. So, once you list all the tags, you can at least see where it is stuck currently or it might not even be stuck. The test just might be taking too long but it is at least some sort of signifier like, all right, where are we at if you start writing somebody on IRSE, hey, my build is stuck or something like that. Another interesting thing is that because we are relying so heavily on the Fedora messaging system, you could try to massage the thing forward. So, if you created your update and nothing happens and you list your tags and there are none, you could try to tag it manually and once you create for the single package updates candidate tag, the message should be sent and somebody should pick it up and that should kick off the whole process, even if in the beginning something failed and wouldn't work. No, just, yeah, this would work only if you see there are no tags and maybe I will try to kick it off manually. So, if you would just tag it randomly with some of the tags further, it wouldn't work because it, for example, for updates testing, it expects there are signed packages. So, if there are no signed packages, giving it the tag wouldn't help. But if something happened and the first tag is not even created, then doing that manually might help. So, you can basically, yeah, this is what, yeah, the second part is if you have multi-bit build, you are looking at the site tag because with multi-builds you're not just tagging packages. You are, for each step, you are creating entire new site tags to either to wait for to be signed or to wait for the testing to finish. So, this is actually happening in the site tags. In theory, you could actually check this in Bode as well. There is a little spinner in your update UI and if everything works correctly, it connects to the green wave to see what is the status. So, there are basically three states currently we displayed. Either it's pending, which means it didn't go through the RoboSignatory yet or testing, which means it's in the testing part of the whole process and tests are being running or the test finished and it's being gated. And there is a little results tab where you can see how the test looks. And finally, if everything went well, you can see that the update is stable. So, we had to change something based on the user feedback. Once we went with it, some of them were with the release of the new UI in Bode 5 because it seems we inadvertently broke some people's workflows. For example, we added querying to coaching so that you don't need to copy paste all of the builds you want in your update. But it turns out there are many people that have good reasons why they want to copy paste the list of all of their updates. So, we added that back in. You need to have that list like space-separated, but it should work more or less as previously. There are still some performance issues with Koji, especially when we do the querying, because we always ask what is the state. So, if Koji is busy, then it doesn't really show up in Bode because Bode is doing the querying. Another thing was for every single build in raw height update, if you are a maintainer, you would receive like three to five emails, which for most people was too much. So, we are trying to reduce it. If you still feel you are receiving too many emails for your packages, just let us know. We might try harder. Yes, and the last thing. I don't know. No. You could try to listen on it. But we could discuss this later on. I know some other QE people are in the audience. So, if this is something that would make your life easier, we could look into if it's in our bandwidth to implement it. Yeah, one more thing. We had for some reason, we have for a long time a request for enhancement so that once the update is stable, nobody should be able to comment on it anymore. And then we realized that there are people whose workflows rely on being able to still comment and coordinate, even if from our point of view, it looks like the work is done and you shouldn't be. So, there is shout out to, is there IRC Nick Adam W here? Okay, so Bingu tells you Merry Christmas. We reverted this because of you, so you're welcome. So, now, what do we don't have yet? Because, as I said, never finished. We want to optimize for testing and reporting results for groups. This is on the roadmap. Previously, we are focused more on the single build so that for the single package, it looks kind of nice. It feels nice. Second, we want to have better tests to it for the stuff that actually is testing the packages. For example, running RPM inspect is something that shouldn't be too far in the future. We still don't know, like, how will it work once we will have like large amount of builds at once, especially how GreenWave will really handle it because we were running the code for some time, but there wasn't like, there were no mass rebuilds or anything like that. What we would really like to add is the reverse dependency testing, which was one of the things I was talking about in the beginning, why we would actually want to do that. For example, once your package would break your dependencies so that it's the problem of the person who broke the dependency for people like downstream of him and not for all of the people who use it. If you disagree, shout at us before we try to implement it, I guess. We are starting to discuss if we could enable some sort of either compose or distro level tests as well. Because I'm not as familiar like how far any of these are, if you have better ideas on Sunday, there should be CI special interest group workshop, so there is where you can have your voice heard or ask more questions. The second part that this is actually I think I'm working on right now with Pingu is we are not as certain like how the thing actually works in production, so we are building a camera retesting suit, just a small suit so that we would always see for our really simple dummy packages that we created that have almost no content, but just to see that these are able to go through all of the pipeline that all of the systems communicate each other and respond. So sort of a really lightweight, happy path end to end testing we could run in regular intervals. Pingu already uses this for debugging, but we want to go one step further and have it being run periodically and reporting so that we know before our packages know once this thing is actually broken. Second thing is we'd like to improve the workflow for mass rebuilds. In the first iteration of our single build, what happens is that in the end we would be waiting, all right, the robot signatories sign our package, but if we would do like a mass rebuild that could mean there are a lot of packages in waiting to be signed, and from the point of view of the user it looks like nothing works and everything is stuck. Now once we move the, all right, create update relatively early in the process in Bodey, Bodey can become kind of this source of truth where you would see where in the process the thing is, so you can at least shout at us, it's stuck there and not just it looks like nothing's happening. So with this we could be able to trigger mass rebuilds in a way that wouldn't be just horrible pain for everybody involved, but we are not yet sure what would be the impact, but it's something that is on our roadmap. So this will certainly not be done if we need to do like the next mass rebuild, but it's something we really want to work on to solve. Okay, so the surprise that our thing we was talking about in the beginning was that you can do on-demand site tags. You might have noticed that during the multi-builds we are creating site tags left and right, and the surprise is you can do this for stable releases, not only in raw height, which means that FAT package chain build should work in stable releases like with stable releases as well. The difference is that for raw height the stable tag will be removed when the update goes stable. For stable release site tag will be deleted once you create the update. So if you need to edit the update or you need to add a build then you'll go the same way as you were used to before you had this available. Okay, that was quick. I don't know what Pingu wanted to talk about for 30 more minutes, but if you have questions do ask and there are people in the audience that should be able to answer. Thank you very much. Do you have any idea enough? Okay, so you heard the question, you answer, I will then repeat it for the microphone. Okay, so to repeat the question was all right reverse dependency testing can't come soon enough when we have it and the answer is we really need everybody or most people using regular tests for just their packages first so that we can actually utilize that to create reverse dependency testing that would actually make sense. But I think this is actually the sort of discussion that would be really great for the workshop for CIC. Yeah, no I just want to really advertise it so that more people come. Okay, so just to reiterate we are not actually being blocked on that we still want to progress the reverse dependency testing even if there are like there is no like full coverage or anything but we still are counting on package your involvement so that it actually makes sense. All right, Adam in the back. Okay, so I wonder if I yeah no no I wonder if I should just how many of you like you know your stuff and Nils said something so if I just invite you over so that you can speak to the mic so that we have the safe for posterity. Okay, next question. Okay, good point question is this is something that should be an issue in our github or to highlight like we really come closer microphone is here with a multi-built gating we really like tease you attempt you to put everything you do in one big site tag and just merge it as one thing and then this thing is similar as what you call feature branch development on the source code so you you create a huge branch with all the changes and of course it conflicts with a master when you try to put it there so this conflicts they're going to happen and so please please do not use very many multi-built builds in one site tag unless you really really need it so the smaller the change you do the better you make it for everyone don't over over bundle things which should try to push for the gate because it's going to be complicated but yeah the local mechanism should be a feature request for this whole system so it's I think it could be implemented like when you do fed package request site tag for something and try to submit new package to a site that can say something like you know this package is already in three open site tags and you are going to have a conflict but but really like it's important but not to overuse this multi multi package feature don't don't bundle too much otherwise this locking mechanism will just again it will just stop you from doing your work and the point is like if you are in master build there is like all this impression affected so for me it's no problem to build my package which I'm like I'm doing out there for me it will not be a problem to create or build it in the but master builds are different and for master builds you kind of can can announce it and you say like you know with master build today please don't do new builds until we finish and this is like a dister wide action but if you start using the site tag for semi must rebuilds without announcing it without collaborating with other people when you get into this conflict situation yeah and it's going to be painful yeah but the problem is that there is no way to easily synchronize the activity because that was of Kamalala master build and that was really master build and like we we just rebuild the packages which happens to be using extra providing extension for those for languages so you know I'm not saying feature your question is a bad one it's a good feature I just worried that people like get too excited about multi-package gating and then start to put the whole distribution into one one tag and try to to get it to get it through it like we don't want that one fresh like feedback because when I was submitting the body update I didn't know what is the status of the update you said like there is now I see that pending teams signing the packages but yeah it's not obvious from the body and I cannot see actually if if there is some progress in signing where it's starting to move eventually forward and so on so if there was possible to see like more details like if I saw like there is 50 packages or this time and I'm waiting for the remaining 50 packages I would probably be a comer because I don't know it's moving somewhere but okay that sounds completely like what does it mean maybe one one thing to add to the talk so you saw that there are like 10 services 10 different services participating in the whole gating infrastructure and when you work with this and you have a feedback like this one and you are unsure like to which service this feedback actually needs to to go and like in this whole process who's responsible for what so we have a catchall bug tracker which is in pagore it's fedora ci fedora dash ci project and there's a general sub project and fedora ci project where you can file feedback like this and then we reroute it to the project which is responsible for this whole thing so if you're unsure where to file the issue regarding gating and ci try this fedora ci slash general at pagore and we find a way to to retrieve it people okay more questions one more announcement awesome so you saw this gating process is what happens after you build the package after you merge and and submit a build and it already goes half of the way there and we know that sometimes it's very annoying to like have your package fully packed fully done with all the version bumps you submit the build this build then fails the test and you need to go all the way back and redo the whole thing from from from beginning so to ease this thing there is another possibility to run tests on your packages which is pull request testing and today in this same room there will be zoo talk by fabian who will tell you how you can actually test packages before you merge them into master before you do a full process for this package but like get your feedback early so you can easily change something in the package before it goes from everything in that case any more announcements okay if there are no more questions and no more announcements there is no reason to not let you go for a coffee so thank you for your attention I again apologize for pingu not being here but he's hands his regards and hopefully should get well