 All right, we are three minutes past the normal starting time. So let's just get started. Thank you, everyone, for joining this morning, afternoon, evening, depending on where you are. We are going to talk to you about a product that we worked on in the beginning of this year. That's called RPMotorspec. And when I say we, this is going to be me, Jérif Chibon, Sonata Spingu. And Niels, if you want to introduce yourself. Yeah, I'm Niels Philipsen. I'm called Niels. And we're in the same team as Pingu. And what we're talking about, we've worked on in the first quarter of this year, before all the circus with the corona virus started. I think let's get going. So we can directly move to the first slide then. Niels is going to have the ends on the keyboard, and what I'll be doing in the first part of it. So the first thing we wanted to go through was the fact that what we are presenting here, the prototype, as well as the IDs, Niels and I were sitting on the stand here to speak about this. But this is not actually just the two of us. Just not, I'll rephrase. This is not just our baby from the two of us. This is also something that Adam Saleh from our team worked on. And it's also the result of a lot of discussion we've had on the diverse list. A lot of people have given their inputs and thought about this, including the Incompire Climb, and many other people who haven't listed here. So the first thing was, well, thank you for everyone on contributing to this. What we're presenting here is a solution. It's probably not the solution. It's a step forward, hopefully not a step backward. There is already a second proposal that addresses some of the same angles that was pushed forward by Nicolas Mayot as an F34 change, if I remember correctly. So you'll see there a second way of doing something. You'll see here another way of doing what Nicolas is presenting, or you can see Nicolas' proposal and another way of doing what we are presenting here. So let's start with the problem. So the problem is we currently have in spec file, we have two fields, the release field and the change log fields, which are manually maintained. And this leads to a number of problems. One of them being that every time you have a polyquest that touches one of these fields, the polyquests are going to conflict with one another. That also includes outside of polyquests that's doing merches from a branch to another. If you don't do a fast forward merge, then you'll often have a conflict in these fields. That leads to a decision where every automation that you want to build on spec file and automating spec file, cleaning labor will be more fragile because of this field. They will need to be able to handle reconstructions or change log and release. And let's be honest, we currently have three change logs in Fedora. We have one that is described the change of the spec file itself and that is in the spec file. We have one that described the history of the git repositories that stored the spec file and that's the git commit log. And we have one in body, which is meant to be user consumable. That's something which shows in the GUI for updating a system. And that is the body date notes. Now, the truth is that for lots of packages, the RPM change log and the git commit log are just about the same for 90% of the commits. The one potential difference that you won't have in spec file is the usual, oops, I forgot to change the sources file commit that you will see in the git log, but not in the RPM change log. And but for the vast majority of packages out there, outside of this, I forgot to update the sources file, these two change log are going to be about the same. So the idea that you're trying to go to, and you'll set your clue for the next one. There we go. The idea is that we remove these two fields or we don't remove them, but we no longer touch them manually. So we want to automatically bump the release when we do a build. And with this, we actually want to keep in mind to the upgrade pass. So we want to be sure that if you do a rebuild, if you rebuild the package for FN minus one, you can still upgrade to SFN easily, as well as generate the RPM change log. And then since we believe that it's going to be most of them, there will be most of them similar between the git change log and the RPM change log, the idea is to generate the RPM change log from the git commit messages. But then we run into the questions of, yes, but not every commit message should be included. I forgot to update the sources file. It's probably one thing that you don't want to have in the RPM change log, so in the spec file. And we also want to be able to edit and correct. Oops, I made a typo. I referenced the wrong bugzilla bug ID kind of thing. So we want to be able to exclude some commits from the change log, as well as edit past change log entries. And we want to do this in a way that does not impact everyone. We want this to be opt in. So you don't, if you don't do anything, it works as it does today. If you are interested to test, then you can opt in and see how that works. So there is a question already in the chat about the upgrade pass. Is it really important anymore? It is a good question. One of the things is that the DNF system upgrade now does a distro sync rather than an update. So it will downgrade everything that is lower in the new version. So it is technically less important than it was, but it seemed to be, it seemed for us to be a good practice nonetheless. So there are two aspects. It is a good practice nonetheless to try to keep FN plus one at higher version release than FN. And the second thing, it is something which we currently still try to do. And I don't think the packaging guidelines have been amended to say, you don't have to care about the upgrade pass anymore. So we still want to try to adhere to these constraints. So with these problems and this idea in mind, we're starting to look on how we can do the RPM change log. Ta-da! So we looked for a few ideas. One of them that was actually given, I believe Jeremy Klein is the first one that mentioned it on the dev list. It was about using git tags. So the idea is when you believe a comment is ready to be built, you add a tag to it. And in the metadata of the tag, you cannot find a message. And that message could include then your change log, could include the things like also the release field. Some people, the secrecy could be stored in there. But that doesn't actually solve the change log problem, the three change log problem, because well, you will still have the change log in body, so we can ignore that one. You'll still have the commit logs. And now instead of storing the change log in the spec file, you store it in the git tag. So it can be auto-generated, but then you still have to go and check it and clean it up the way it is. It's not actually removing it, it seems potentially helping to create it. But at that point, Fed package build, Fed package commit as an option that just puts what's in the change log, whatever you enter as a git commit message. So technically that option is basically already there. Plus it's gonna be hard. How do we want to retrieve that information? Should we include a thermal-like syntax in the git tag to extract, okay, this is going to be the release field, this is going to be the change log. Then there is the old question, now do you edit such a change log? Do we allow magic keywords that says, well, this tag is actually replacing that tag over here because I made a typo and this is what you should be now using. Do we want to use git push minus force, the dash force for git tags? Do we want to allow people to remove and add git tags as they feel? That seems very potentially problematic because then you may end up having two packages with the same change log that actually come from two different commits and they don't really, you lose the history of what happened or why this build was including that change log because in the meantime, the tag that was included and that change log got moved to somewhere else. So that was a solution that we didn't really like. So we looked at another one. So that was a song I did that was shared on the devil list and that's something which seemed interesting. You generate the change log from the commit log up to the point where the commit touches an external file that is external to the spec file at which point you just include that file. So you have a change log file in this git and just plain the change log just as it is today in the spec file you just copy paste that into an external file in this git and from that commit and all the commits above the more recent ones, this will be auto-generated from the commit message and as soon as it will basically start from the most recent commit and move down but as soon as it faces a commit that touches that file the algorithm is gonna start which means you can edit that change log file and add all the new commits and reward and typo fix everything and then it will just include that file and stop trying to regenerate anything from there and that's the idea that we liked most so that's the one we went for. Then we started looking also to the release fields. So one of the idea and that's I think Neil brought this from the DOBS angle experience so the open build service that open SUSE runs that already has the feature of bumping the release that we built was to use the number of commits and number of builds in the release fields. That was very tempting because it's very easy to you change the version and then you count the number of commit that was done since that version was changed and then you can ask the build system how many builds have there been with that version and that number of commits and then you can just increment. That's very easy to do. Two things that we didn't know exactly how to handle and that were not happening in some ways. One of them was the merge commits. So when you have a single branch linear history that's very straightforward but when you start to consider merge commits that goes that branches and get merged. Well, that's becoming a lot different to handle. How do you count the commits? Which commits do you count? Which commit do you not count? How do you handle that? And then there was the old question of the upgrade pass because that reflects to the merge commits. So if you do a push to a commit on the right branch and so you get number of commits one and then you merge that, you met that branch into federal 32 branch but then there is a merge commit which means suddenly the one commit that you have added on the right becomes two commits on the federal 32 branch which means suddenly the upgrade pass is broken. So that is something that we still wanted to maintain as I explained earlier. And there are a few ways we could do to address this. We could start with the disk tag at the front. So then it becomes version, dist, number of commits, number of builds. It could be one of the one we were looking at most was to try to do version, number of commits, dist, number of builds to try to mimic more of the current behavior of things. So there was a few things and we did not know if we wanted to go in the past of changing the place of the disk tag and the entire distribution and now our tools like DNF package kit although I don't think this one is maintained so much anymore or just would be able to cover them. So this is still something which may be doable. This is something which we didn't, well, we started looking at other things as an alternative. The second alternative we looked at was the ArcPackage Utils approach. That is a utility that Climb wrote which essentially adds a templating engine on the top of the spec file macros. This would work and Climb actually has it working. It does the work, it does the job. It's just we did not like the idea of having a template on the top of macros that seems like a fairly fragile approach. It's adding a template on, it's adding layers of complexity and layers of potentially something that could end up being fragile or hard to debug. So that's also something we could work. It does work, Climb has it working, but it is not the most appealing idea. So we looked for another one again, there we go. Yeah, so the third idea we looked at was basically to try to use the build history and the build target to mimic to the human maintainer. So to reproduce what humans currently do, just using the build history and the build targets. I'm going to touch upon quickly the build history. So there are two ways we can do that. One way would be we just create code G and tell us how to behave. But that means that you need network access every time. It's gonna be hard if you have a slow internet connection and you need to create the entire build history of C or something like the kernel that will take some time. So that's the place where we decided to use the git tags as a mechanism to record which commits was built and then using that information of git tags with the build history together with the build targets which you can provide, which Isaac Koji provides or you can provide as a as a packager when you want to build locally. Well, then we have sufficient information to be able to make our logic works. And so that's the idea we went for. Nils, if you wanna... Yeah, so that's the idea we went for. We are trying to emulate the human, the current packagers experience as good as possible. We do realize that there will be cases and spec files which will not be able to be supported by Appliance spec, Autospec at this time. We do hope that this is a step forward in the right direction. We do realize that it's pregnant going to be the final answer that shall be craving to stone and never to be moved again. So the way we do that is you have very, very little changes in the way we work and how you opt in. Opt out is also fairly easy. We produce reproducible RPMs. So the spec file itself is you build a spec file locally and in Koji you may end up potentially with some differences. But once the SRPM is built in Koji, you can rebuild this SRPM in any system that is RPM based and you'll get the exact same output all the time. So the spec file is no longer the unit of reproducibility but the SRPM become easy. We tried to prevent the upgrade pass as much as possible. That was one of our goal and that's what we are trying to achieve with the emulating the humans. And we do support local builds from the spec file. It's just there are a few tricks to do but basically it is there. So to give you an idea now easy or hard it is to opt in into this. Well, this is a very simple spec file. You have the name, the version and then the release becomes that macro and the changelog becomes another macro. And that's all you have to do. But don't you if you add an existing changelog you will want to save this into a separate file which I believe is simply named changelog and the existing release and you can basically ignore. So how does it work a little bit more in practice? RPM motor spec itself is a CLI tool. It's a Python module library that comes with a CLI tool as well as two Koji plugins. One that runs on the builders and one that runs on the hub. So to give you a little, we have a small graphical representation of the workflow. When you do the Fed package build Koji will start by building the SRPM. The building the SRPM means we're basically doing a Git clone. Then we have, this is the Git checkout. Once the checkout is done, it's going to, Koji is going to call all the Koji plugins which are post SCM checkout that includes the RPM motor spec one. So RPM motor spec will do its magic there. And then we just Koji will just build the SRPM as it does normally, which is the equivalent of RPM build dash BS, I believe. RPM motor spec is going to pull from the Git repositories, the existing tags that we mentioned. Once the SRPM is built, Koji will build the RPM as it normally does. Once the RPM is built, Koji will tag the build as it normally does. And once the tag is done, Koji will call all the plugins that are post tag that includes the second RPM motor spec plugin. And that plugin will call this Git and say tag, add a Git tag to this specific commit which I have just built successfully. And that is the way that the Koji actually manages itself to record into the Git repositories the build history that RPM motor spec then we use in the build SCM spec. And now I'm going to leave the floor to Nils who will run you through a little bit more information on how that works. So I hope you all see the next slide. Okay, the quant line tool is basically the thing doing all the dirty work. It reads Git tags, which get there by some mechanism we'll talk about in a moment. The commit log messages and this to the site changelog file where we put the history in or where we can edit things after the fact. It comes up with the next release value for potential next build and generates a missing changelog entries that aren't in the changelog field yet. And it has code to fill in these values in the spec file like when later this one, the Koji when it creates the SRPM there's where we hook in. We ensure that the latest builds are tagged in this Git. So everything is prepared for the command line tool to run. And then we use that command line in the build route to insert the things in the spec file and then build the SRPM from there. And this one is going to create the SRPM and then build the SRPM from there. And this one is then used to build the final packages. Okay. Then Pingu talked about that before after a build was successful it just tags it in the disk repository. We are implemented a small API endpoint on the Pegasite for that so that Koji doesn't have to have its own Pega user to do that. Okay. So we have a couple of known side effects and shortcomings. So that's always a good topic to talk about. So the order commits came in or the order builds were done is how the change look is sorted, not the commit dates. I mean, anybody who's worked with Git for a time knows you can shuffle these around by rebasing or cherry picking or stuff like that. So the order of dates doesn't hold too much importance there because I could cherry pick a very old commit from some other branch and we don't want to resource that. And we can also at least with a prototype while it was still working in staging we kind of have a couple of problems with staging right now that's related to the color move but when staging is back in order again you'll be able to play around with that. So right now you can just tell Koji to build from some random commit and this will of course influence like let me backtrack that sentence doesn't make too much sense. We can adjust, we have the tip of a branch like let's say Federa 32 right now the current latest stable branch and we can do any number of builds from the same commit with that scheme but we could also build an older commit from that branch and that would then like the change like history would look a bit weird there. I mean, let me just, there's some more to that. I'll post that in the chat. The nitty gritty details about that are here about both of these points. Okay, so right now the prototype has some shortcomings in that it's a normal Python package that is it's in the version Python path it's a normal site package and if you upgrade the Python version on the builders suddenly it wouldn't find the package anymore and the command line tool wouldn't be functioning and stuff like that. So the solution to work around to that is that we do it like in the olden days and make it like have a copy of it and use a share RPM order spec something that's not tied to the Python version and then this should work regardless of the exact Python version. Like if you go at it with a new Python interpreter it won't care that it's a new Python interpreter as long as it's compatible but Python has been good to us in that regard so far. We haven't gotten around to pre-release to implementing pre-releases that we put quite some thought into it and we made some preparations in how the macros work like if you think back to the example spec file the auto rail macro understands a couple of flags and options for these cases but there's no code behind that yet so we would have to implement these use cases before we can use it for these use cases. And in order that the spec files are buildable locally we have to have the macro files available on the packages machine. Right now it's, you would have to install it manually. We could just install the dependency from the Retter RPM config package to our one or integrated with that one that doesn't make too much of a difference. And all the other tools are currently totally unaware of the magic we're doing here so we need to integrate with them a little better. Right now you would have to tell FAT package to skip NVR check because it doesn't really know the right NVR because it would have to have the, it would have to know to talk to our tool for that. Okay and scratch builds from local SRPMs don't work it like if you commit to your repository if you push it to disk it and then tell FAT package to build a scratch build from a known commit hash that works but from an arbitrary locally produced SRPM that doesn't work yet. And we would have to enhance FAT package to create the, to pre-process the spec file accordingly before producing the SRPM from which to produce a scratch build. And with that, I'll hand it back to Pingu. So our current next steps are basically we used to have the staging environment so before the column was started we had deployed this on staging and we were able to test it there. That's also how we got the feedback but for example, the tie to the regular Python packages which makes our PMO to be currently not working with the Python stack. So that was a very good feedback. So we want to deploy this again back to staging once the staging environment is back online so that we can call for testers on this again. After this, we do want to, you know, we have had a few people telling us that this is worth bringing to FASCO. So one of the point of this presentation was to give some more explanation on what the RPM to spec does, how it works but also asking you if you believe that this is worth pushing forward. If so, then we will bring this to FASCO for consideration as them if they believe if they agree that this is worth pushing forward. Then we need to fix the information shortcomings including things like the RPM Dave pump spec change that we need to do. That was discussed in the chat here including making Fed package aware of auto release, macro, including, you know, fixing the pre, what's the, not pre-commits but help me here Nails. Finish the macro, the pre-release and post-release use case. Because where the other use case we haven't addressed yet. So. Okay, so fixing these and then of course to use all the fixes and improvements. But that's basically what's on our roadmap. The first step being today, do you folks think this is good to push forward? And then the next question, the next slide is basically if you have any questions. And I see Yann as one in the scratch wheel section. So there is Fed package has two ways to do scratch builds. You can either do Fed package builds dash dash scratch which is going to do a, which is going to call Koji and ask Koji to build from a Git URL and make this build a scratch build. Or you can do Fed package scratch build dash dash srpm which is going to generate an srpm locally first, upload that one to Koji and have Koji do a scratch build from that. The issue is that the Koji pre-processed the spec files you know, at the srpm level. So if you do a Fed package scratch build dash dash srpm without pre-processing the spec file locally manually yourself first, then you won't have the spec file will not contain the macros that are needed for the RPM to build correctly. So the way you can do it currently is that you will have to call the RPM to build manually on this Git repository before you do the Fed package scratch build command. Or and the proper fix is going to have to teach Fed package to call RPM with a spec on the Git repo, on the Git repo before for spec file that uses these macros called RPM with a spec before when generating the srpm before putting it to Koji. Does that make more sense? Okay, thank you. Another question, what about two levels of macro use auto change log or change log to change log plus previous change log entries? Okay, so the question, Carl is basically wanting us to get rid of the change log macro entirely from the spec file and just have the auto change log instead of it. And have the NILS have an answer to that one. Contrary to looks change log isn't a macro. So yeah, I'm not sure if it's workable if it's workable like have auto change log to expand to change log and then RPM build understands that I wouldn't bet anything on it. So we'd have to try it out. I'm not sure, I guess it's mostly a matter of taste. If it works, if it doesn't, it's out of the question because then we lose the feature of a locally rebuildable package. NIL says that RPM doesn't expand any macros. Like if a macro expanded to change log it wouldn't work, do I get you right? Is packet aware of this change? Yes, the packet team is aware of the RPM auto spec work. We've had a few discussion with Tomasz Tomaszek. They were actually looking at RPM auto spec in the early days. Then they saw the discussion under the release and they were like, okay, there's still a working going but that's basically they are aware of that. And I actually think a little bit like NIL here that RPM auto spec would probably simplify their life to where we knew it was a question. I mean, if not right now you can always get all of us on IRC, wherever we frequent on the devil list, I guess. The project you're seeing on pagoda.io slash dora-infra slash RPM auto spec I've put the link in the chat if you're looking for it. And we do have one question for Jens about comparison to Nicolama use proposal. Do you want to take this one NIL, sorry? I think ours is at least for existing maintenance it's much simpler because it's really, really, really tiny change in your workflow. Your spec file before and after will look almost identical except the difference that the changelog is missing from the spec file. I think Nicolama's approach would you would have to adjust more yourself, your workflow more to that one. It's to be frank, I only got aware of this approach this week so I'll stop here because I hadn't had too much time to look into it myself. There was a question from what happens when Fedora messaging screws up and Koji fails to tag a commit with the build info and the answer is simply Koji doesn't rely on Fedora messaging directly called Spiger CPI. And if that was not available for some reason I don't know whether the build would fail or not I expect. I hope it shouldn't fail at that point. We should just we have to revisit that one. I mean, mind that we worked on that couple months ago so it's not all super present in our heads but the way we should do it is retry it a couple of times. If it doesn't work out, then fail. If you remember back, the builder plugin will ensure that the git tags are present. If that failed at that point then the subsequent build will fail of course but then there's something more serious. So what Neil says here is that basically one of the first thing the builder plugin does is it asks for Koji for the build history of the package and then it checks that history against what's present in the git tags. So it makes sure that git is up to date versus the Koji database basically and that's a second place where we actually are able to tag things in figure on this git from the Koji build history. We have a question from Ifa. Is there likely to be a follow on project that you know work from this work that you have already completed? And the answer is yes. One of the ideas that if Fesco thinks that this is valuable and we want to move forward for that then we will have to schedule a, I don't think this will be a three months initiative but it's maybe a one to two months thing where at least three people push this. And I also remember from earlier on during the talk a question about the actual macro names involved. I think right now this is a prototype so nothing with regard to naming is set in stone yet. So if there are good reasons to name it something else just maybe to also differentiate it from competing approaches we can do that still. It's not in production anywhere right now. So there's the question, could we consider making it so spec files that don't have a changelog change that do not have a changelog section entirely would automatically. We can do that. Change log. I would, the answer to that is everything is possible with time and money. I think it would even be fairly simple. And that probably would be a, we would instead of changing, checking the auto changelog is present. We would just check the change. Some slight change to the conditions involved whether to get active, but yeah. That's maybe a good idea. Do you want, can you open a ticket for this? Thank you. We are a few minutes ahead of schedule. So I'm more than happy to let you guys go to have the more social times guys and girls I should say to have a little bit more social times out there or we can hang around and give answer a question if there are any. All right. I guess we've existed the list of question here. So thank you everyone for attending. Thank you for your questions. I've only seen support in the chat. So I guess people are happy about this. So let's bring it to Fesco and talk to you if I had to schedule some time to finish this.