 One one Okay, ladies and gentlemen, please take your seats and let's welcome our next speakers for today Tomasz Hosa and Petr Graczek Okay, hello everyone. My name is Tomasz and together with my colleague Peter I'd like to tell you something about the base helper and upstream raise monitoring just out of curiosity How many of you made in some package? Yeah, great we too so Is not working so I Would like to give you a high-level overview of what's absolutely is monitoring in Fedora is talk about Rebase helper if you are not really familiar with this tool It's been here Around for some time, but I'm not sure how used is and then we will talk about integration of rebase helper into Upstream release monitoring service workflow and we'll show you some examples Peter has some demo So hopefully it will work and it will summarize everything in the end So what's upstream release monitoring service? It's a service developed by Fedora infrastructure engineers. It's not by us. We are just like users and it's a service to help package maintainers to keep track of latest upstream versions when upstream releases new version it will it can help you to Like it can notify you that there is new version. So it consists of basically as Ralph said before it's micro service Oriented or how to call it consists of basically two components that are interesting from our point of view And they communicate over Fedora message bus. So the first component is an idea It's like the there's a nice front end where you can add a new project at the URL where the project sources are placed and it basically it then checks all those like latest versions of software and Sends out the message then there is the new hotness that does Some magic and like X upon messages from Anita So let's have a closer look. We have a nice picture where you can see Okay, so so this whole thing is the the service and as I said in Anita You can add a new project provide the URL where the sources are then choose basically What's the type of service where the project is hosted? So it's if it's a github or source for sure that kind of stuff So so I know how to find out if there's a new version and what it does it regularly checks the upstream projects if There's a new version if there's a new version it Basically sends a message on message bus and there is the new hotness which acts upon such messages. So I think as the first thing it Tries to find out is there is existing bugzilla for for the component if it's open and if it's not it will Create a new bugzilla for you for the component in Fedora informing you that There is a new version upstream version and you should You and that you should update the version in your height So after that it tries to trigger it triggers a new Package rebuild using the latest sources in in Koji But this can fail from like different reasons basically Imagine you have a package where you have downstream patches and those Don't apply on the latest sources version then the build will fail also If if your files list in a spec file is like really strict And you don't use wild cards and there is like so name bump or New binary appeared or disappeared the build will fail so when the new hotness Triggers the scratch build or rebuild of the new version it Keeps track of the task ID and waits basically for for message from Koji that the build finished either Either successfully or unsuccessfully in the end it will add the logs and comments into the existing bugzilla informing you that hey, this is the link to the scratch build and Here are the some logs build logs and that kind of stuff So this is an example of how the Bug report from upstream release monitoring looks like I'm sure if you maintain a package you already seen this So what's rebase helper rebase helper. It's a nice project. We started almost two years ago It's got started as part of the RedHack week We had basically it was a week where we could hack the whole week on some new cool ideas and Forget about our regular Responsibilities so rebase helper was one of the ideas to help maintainers to automate and help them with this like boring rebasing of components in Fedora, so it does a Lot of magic and things you should do and maybe not everyone does that so it can download the new sources for you When you run it in the Fedora this git of the component basically in the repository of the component After it downloads the sources it tries to rebase the downstream patches You have for the package on top of the latest version of soft of the sources It uses git rebase for this purpose. So if you had if you have Something like a merge tool merge tool you use for resolving the conflicts with git Rebase helper will use whatever you use with git Every time use use it so after it goes through the process of rebasing the patches with you It will try to rebuild the old version of rpms and the new version of rpms so there are a couple of Like three three different ways how it can rebuild the rpms Either locally or remotely locally using mock or rpm built or remotely using Fed package after Binary rpms are built it will download them or if these were built locally it will just have it already and It will run different Checks on them We support three Different tools that are run basically all the time all three of them because they produce different kinds of output We support package diff rpm diff and a bi package diff basically the idea is that in the To help you notice for example so name bumps and new like new Files of binary header files disappearing and appearing in the new version rebase helper will Compare those two sets of binary rpms and provide you some logs and data So you know you are not surprised and you don't surprise the consumers of your package on the next update So now I would like to give a floor to Peter as he will continue with the new stuff We implemented in a rebase helper and we'll describe the integration more deeply. Yeah. Thanks Thomas well on the last dev conf we introduced the rebase helper to all federal community as Tom I said it is a really awesome project definitely like upstream release monitoring service It helps your packages like It tracks everything. Well, what he did love since last year First of all, we implemented feature called non-interactive mode. What does it mean? It means that if you have a package like PostgreSQL or Systemd The compilation and the rest of the stuff takes a pretty long time one hour two hour days I guess not And we don't want to buffer the developer maintainer user, I guess not and We implemented so that if you enter to the command line rebase helper still does not have a agree Then user is not Interrupted or you cannot take care about the user cannot take care about the stuff like okay, this patch is not working and the idea was first of all to Implement rebase helper to upstream monitor is or upstream release monitoring service. It was an idea But we did it What was changed next was that at the beginning we use a patch command together with melt and after some time we realized that That is not useful Many packages use different mechanisms and we seen that okay. We are using it. Why we should not Use the gateway base command. It works for us. Well and better than patch and we did it and What we implemented next we have a mock We have a RPM build This is local. We need to implement something for Like remote and we have a fat peck pecky g we implemented and the last time or last new issue was We have to find the changes in RPMs like sonam bombs as really great tomah said headers and but We would like to know where sonam bomb was bumped and we have to We have to inform the community. Hey lip PNG was changed take care about it and inform and APP GD does it and I had a several communication with dodgy. He's a clever Smart guy about comparing the AP AP AP checks and he told me hey, we should improve it He provides the new version He provided a new version and we include it in the rebase offer. Well, we have all stuff What is next? upstream release monitoring service is the good place to Inform the user via bugzilla. Hey, you have this change header and so on and I guess that One month ago, we released rebase offer zero point zero like here Which introduced the system for upstream release monitoring service, but it is only for the rebase offer we have to include it in into the upstream release monitoring and I don't know. I did not know who contact Finally, I found the Ralph Bean He's the right person who helped me with a lot of stuff well and Before I will show you what we did Rebase offer can rebase the downstream patches can show you okay. This patches does not work on the new upstreams Rebase offer can inform you and about surname bombs and We were asking hey, we have a very upstream release monitoring service do it and We did it. It is a nice picture from Tomas. It's great Well, this is only the new hotness What it does? It receives from the while fat message the It message that okay, I am here. I am a new upstream and I would like to build but okay This is the same stuff and we say okay We have a rebase offer and we provide the information to them well rebase offer calls is called and we Rebase up a download sources rebase them if it's possible, of course and Create a source rpms old and new and at the end We send wire fat pecky G as I Said before okay, here is and we triggered both Koji IDs and Okay, we have final step. We have a Koji. We have a source rpms. We have And we should write what happened After Koji finish the source rpms task. We does not care if there is a failed start Success state or cancelled state we doesn't care now and fat messages set or sent to the New hotness, okay, I have finished. Okay, we will care In the old version we attach only the build logs Upstream is good, but we need to inform that via bugzilla What's happened and Therefore rebase helper catch old and new triggers. Oh, I'm sorry old and new triggers which regards to the Koji and Download rpms download the build logs and run several checks Which we implemented not only the one we need the both all stuff because each tool provides different informations and At the end if it was success Then we edge to the bug we are we edge to the bugzilla locks Results and other stuff Is it still realistic? Yes well Month ago, I spent a lot of time to integrate it and to redid it and I will show you how it looks like before here You can see who did the this screen from bugzilla new available software systemd yes, cool and it fails from the reason we don't know why and Somebody completed this script build. Okay, but I Already you can see That Sorry, it's my name because I am not running under upstream monitoring service. It is my temporary temporary machine We find okay Over the rebase helper we Check that okay patches were not touched or there are no patches. This is a bug Create the issue to the rebase helper. We include Our rebase helper includes the root lock from Koji of course new only the new sources build lock and What is important for you? That's to the bugzilla picky gd And at the end of course for us Attach the rebase helper debug lock. We need them We need to be sure. Okay, it fails But from which reason and I hope I guess that if this bill be or it will be released officially soon You will send us the issues. Hey, this is not working totally Rebase upper should be redesigned should be Patched or whatever you want and we have to do it and We will do it. Well, we have a demo. I guess that it will work. It's my screen Hey, I Cannot show I see Sorry, sorry No, ah, yes, come on. Yeah, I see This is a bit schizophrenic. I don't know where I have a screen. Yeah Well, this is the real machine Where is implemented the feature rebase helper it is not officially Released because there are still some There are still some bugs Well, we will start a build Yeah, this is that after relative discussion and thank you Ralph for Fixing my problems. It helps me and now I can It was an English mistake. I know This message ID is I guess the message ID from node JS Connie as I are whatever how it's called the component and it sends to the Fed message Information about the new release it is somehow hacked but it will work. Okay, I will send to the Anathia message Okay, I sent and what's happened? You can see that an idea received the message Okay, there is a there is a new upstream release it Calls repo query. It took a time. It takes a time and we will wait Yeah, and what will happen now? Anathia or the new hotness will call rebase helper and initiate a scratch build after a time The new hotness will receive message. Okay, all is success and the states are if old scratch build fails, then we don't care it should not happen because Let's imagine that you have a old federal package and it is not buildable and rebase helper does not care about that It's not my job. It's your maintainer job If it's cancel, then of course, I don't have care. Why you should cancel to the scratch build. Maybe the Koji was down But what in case that old scratch build finished successfully and you knew scratch build How it felt sucks it, let's say then we compare all rpms against old and against the new but what in case that New scratch build failed shall we inform? Yes, of course Because either patches were not success. No, this is a totally because source RPM will not be generated and It failed. Why what happened Built was not correct. Of course, it can be and we should inform the Maintainer. Hey, your package is not buildable from the reason. I don't know that she's not files are missing Licenses missing or you are totally wrong or At the end the scratch build was cancelled and of course rebase helper does not care Okay, from another from any reason you Cancel the scratch build, okay It seems that it took a time. It takes a time. Yeah Yeah, of course, okay Okay, we'll finish and we will see Yes, so also we can use this time for questions if you have any So I'll just summarize I can and then we can jump to questions So I just want to summarize that what's absolutely is monitoring is it just it takes do does nice job by checking upstream Upstream projects for the new releases and notifies Fedora maintenance. It can be not only Fedora I think Anita supports also other distributions, but This time I think only Fedora is implemented. Maybe there's something more. Maybe. Okay. Okay. Yeah, thank you and rebase helper is another great tool that tries to ease the pain of maintainer when doing rebases of component by automating as much as possible and Running some additional checks on the rpm. So you're not surprised and you don't surprise users of your component So rebase helper basically saves time by automating all those boring tasks You have to do again and again and again when new version is released and also The upstream release monitoring Is not Doing all the tasks that rebase helper is doing that can help that can help the new version of rpms to be built successfully and be used so so I on my components, for example, I Regularly see that the scratch builds fired up by Upstream release monitoring fail because those include quite a few downstream patches and these don't apply on the new sources so Using rebase helper to Automatically rebase patches and find out any difficulties is It feels like a natural way to integrate it as natural thing to integrate it into upstream is monitoring Service so I think now we can jump to questions. So that's true. The old one is the current Latest version that's available in fedora and the new one is basically Version that's not yet in fedora, but we we basically download the new sources. We change the version in spec file of the component and Basically, we produce a new source rpm with a new version, but it's just temporary thing and we call this Source rpm and the produced behind binary rpms as the new version basically of rpms or new set Actually, we we don't keep track of we we don't care what's Built in Koji as the scratch build we don't keep track of it from the point of view of rebase helper We just like whatever when you run a rebase helper There is the current version and you want to rebase the component to new version So so the new version is like what we are trying to build with the news Sources we are not like you're using those scratch builds for anything or it's just way how to produce binary rpms So we have something to compare with the old rpms. It's not like we I will And I would like to also note See that when you are doing builds locally, which we cannot do with absolutely is monitoring service When you do builds locally rebase helper can actually try to build the new rpms And it can then if the build fails it will try to inspect the locks before because we can do the rebasing of downstream patches, but It's hard to find out before you build rpms. If there is like new binary or if there is like Soname bump or something like that So if you run rebase helper locally and do the build locally We will analyze also the build log and we'll try to find out if there is like new binary and the build failed when Rpms were composed. So we'll try to add The new file into the file section and then rebuild it again So we'll do this in a loop after couple of times. We may decide that there is going nowhere So but that's not used with upstream is monitoring. Well, sorry. I will show you Here you can see that as we started it seems that there is a magic environment. It works. I don't know why here is that Rebase helper was started from the new upstream release monitoring upstream new upstream release he found the bug in partner bugzilla and He cloned the repo from Fedora download the sources He made the magic via a git rebase and The rebase helper finally built source RPM. I like that program T max And This these are the debug information from us some logs and stuff or stuff Here you can see Pitches were not touched and they are reported to the bugzilla and After a time Again, you can see that there is some status from Koji the task where we're closed from Koji and at the end you see that we downloaded the all debug informations logs and there are the stuff Getting rpms and at the end we are comparing the packages using rpm diff And it fails from another reason. I don't know why I have to check Okay, I am going back. Well, I think we had one question Yeah At the first presentation David Walsh sent this like this Cool Really good point we've we've think about it and Of course, we can automate it so that if all will success then we can directly call git push git build fetch packer build and so on but we No rebase helper made some changes in spec file the rebase helper commands stuff there and if the patches are Somehow modified so that they are applied to the new sources then they are included the patch is included in the this git and push to the Fedora repositories and This you this aim should not be automated because we the rebase are probably include Some garbage and this is not the proper way of course maintainer has the How to say The final decision, okay This is a fine fine final state. This is an okay state The RPM is fine and we can officially build rebase helper is only a tool not the user and He cannot or it cannot Do all automated tasks It can but this is a robot Yeah, sure, I would like just to rephrase But we have all these data. So we have the rebased spec file We have rebase patches and that kind of stuff. So so there should be no problem to make a new repo somewhere With it or new branch or basically create the tar ball including all those files and then you can just Like Okay Yeah, but as Peter said that there is like maybe plan, but we didn't even start yet to Provide you something like now if you just inspect the changes and if you are okay with it just click and you can Automatically we can commit to the This git and create like regular build for you, but definitely user should inspect this before we should not do this You know, we have an issue issue for it on our github really we would like to implement it But I am really scared about it. You are you are right. I think I'll pet question before Generate Yeah, thanks, could you please file issue for it for this people it would be only for tracking poppers and we will see But but the idea with pull request sounds like great. Yeah Yeah, any other question we have At the end, I would like to show some references. We have a rebase out per documents Here is a github. It is all it was already posted as some blog about life journal on the life journal some articles and soon we will have an apple package not over the copper and I hope that during the month not later on we will have a final state and you will receive the hour Modification with all stuff like locks scratch build fails and other You will receive more emails from Godzilla Thank you. I think we have still like five minutes or maybe four questions. So if you have more questions, yeah, yeah, I know about 50 package maintainers Question is really great and I have a discussion with Python Maintainer, you know Sladek Kabada, I guess and he told me a really great idea It does not matter if rebase helper properly merge all But with the non-interactive mode if rebase helper tell me well These three patches fails. This is important for me. I let's say that package has a like rim has a I don't know how many a hundred pages and If rebase helper tells you hey, these three are wrong that it save your time and This is the good good point for you. There are no managers unfortunately, but You know our work is about the time and money and let's say that we have a 500 packages or Sorry, 500 maintainers and you save at least and now then you save a redhead money But there is no any manager. I don't know why the idea is you should be able to To run rebase helper in a window in non-interactive mode or in interactive mode and just keep it running Do your other stuff after half an hour You just open that window and you see if there's something else you need to resolve and you know like there are tasks Those should be for sure automated RDO package Did we we start to rebase helper before they started there? Think so and I'm not sure sure nobody like oh that's her No, no, no currently no nowadays no, but we can share of course basically, we realized they did something like that after They already did it or yeah, like I think we are not looking for another projects like that We didn't see anything for for RPMs and Fedora. So we created one and I guess they did the same thing Yeah, for sure. Also, we are open to you know patches to enable or to add support for for the beyond packages But right now we we heavily like relied on that that's our PM based Distribution and also that you use this get in Fedora because we were trying to solve like our use case But but we are open to You know pull requests for sure Nowadays rebase helper is based or is targeted to the RPMs and we would like to introduce the BM packaging system or something like that But we need a pull request also the beyond packages are like kind of made differently and it's not completely like Mappable from my point of view to to RPMs or to some point. Yeah, and and they also already have Last year we also gave presentation about rebase helper and there was one that be a new user And he told us that they already have kind of like some tools Maybe not single tool but more tools that in combination they do similar thing Maybe also at least for the patches, I guess so they may not be that interested in this as we Yes, we were I Guess that we have a one-minute left the month scarf left. Yeah, okay. Thank you