 Hi, nice to see there are still so many people in talks and awake on Sunday afternoon So I'm flooring on one of the RPM developers and we are talking about Spec files today and packaging. So where are we and why are we talking about this? Like two years ago. We are we are PM upstream rolled out the last two big features that are heavily infecting affecting Spec files there was just a talk about the most prominent about which is scriptlets and file triggers and so For those who have not been to this talk that was just in this room Before this one. So most of the script that's are now replaced new and for yours just about basically to to kick out the last few Few probably doesn't 100 scriptlets and Since then RPM upstream has been doing stuff that has not much impact on the packaging side. So I Say small features, but this was deep again for was changed into Subpackages per package purse. So there's one deep again for sub package per Subpackage and we did a lot of cleanup stuff stuff that packages don't have to care about The one thing we did we added new terms for the rich dependency. So there's new now the new with and without and less operators which work on the same package, which was an issue for especially rust which wave The problem is you can specify like bigger now bigger versions and smaller versions But a match may match different packages. So you're not actually a well not able to match like Version ranges, which is needed for those stuff Anyway So we are now at a point where you think it's time to do something for packaging again with Fedora Hopefully having absorbed the last huge chain chess That's Fedora for those who don't have not seen it yet That's the growth over the last couple of years I don't know if you can see that if you can see that doesn't matter The the blue line at the bottom is the number of source packages and it's growing and growing and growing and growing it's basically a straight line from the humble starts of Fedora core one and So we have to expect that there will be even more packages for the years to come. There's no sign of Fedora bottoming out Or slowing down So the problem is with more and more packages, there's more and more work to do and The question is how how much more work can we actually do and it so we want to take the next Development cycle to look at things that can make packaging actually easier the problem always has been that our PM developers are surprise developers and not packages So we always have a more the focus on the code than on the actual packages because who knows what's out there I mean that 21,000 packages. There's you you couldn't even read them if you wanted to and so we will look into this basically the next cycle and The thing that worked very well with the script that is The question what can we actually remove from the pack from the spec file? So basically every bite that's not in there is a good bite unless it works still That is probably one too much So there are a couple of things that that that we want to look into one is really scriptlets, but it's basically covered by the top before What has come up pretty recently with the I think rust and maybe some of the other new languages is Automatic build dependencies, which I will talk a bit. That's an interesting topic then something that has been discussed Previously, but which is not really which is kind of related to our PM But maybe not a feature in our PM is using this good data within the spec file Which I will talk briefly about there are two big things that might be features for upcoming RPM versions and one is build templates basically Having reusable scripts and help us for for packages and If you know how you build your package you also want to package that into sub packages automatically or semi automatically or at least not by Typing stuff by hand all over again So let's look into this I will go over this only briefly because it fits the topic So we will look at file triggers and how well they work and what's left in the scriptlet area one thing that has come up in the talk previously also That's the only thing I am currently aware of with regards to scriptlets is users and groups and a generation of users and groups and I've been always been on the fence with is this something our PM should do or should not do or is this something that's like Implementation specific for the operating system or distribution your packaging our PM should in my Perspective not care too much how the how the distribution looks like that it packages But the distribution should take care of that but one could argue that if you handling files users and groups are kind of part of the post-extended and maybe it's integral part of files and We should do something about it and we will probably Think about we will think about and probably implement a decorative way This came out better talk previously that may take a while, but it's one of the last Use cases for scriptlets that are kind of legitimate right now and we hope to solve this with that Then there has been talks about using this get data In spec files, this is probably not going to be a feature within our PM because this get is not part of our PM But probably of tooling that wraps around the spec file I've given a Lightning talk on flock about this last year. There's been a lightning talk on this conference yesterday evening That you may want to look up and I'm guessing There are people thinking about this and will probably work on a proposal within fedora so we will from the RPM site be Happy to help if there's any features that make this easier But it's not quite clear if it's anything that really should be done within RPM itself There are two things that that you that will probably yield a house the most benefit one is the change lock Which is always Hathal which could be generated from the from the get change lock The second thing is the release number which could also be just be counting up basically the commit Which would mean you cannot mess it up We have to see if they are What kind of help us you need to to to clean up things that don't work quite well But that's that's a two areas and I think there will be people working on that and we will help with that The main benefit will be will be reducing merge conflicts Which is currently pretty annoying if you try to merge some changes to to one release to another I mean, it's basically yeah, you retype everything or you reshuffle everything in the amount of work that it's saved by the get merges basically zero and the main sources of conflicts are actually Change lock and and the release number and there's a chance of getting rid of that which is probably saving a lot of stupid work Yes, please So so there are a few ideas about us one is basically to have some keywords that you can put into the get commit message and If you want to fix up your get history, you could actually put text on some commits to say well don't show up so that's the two Main mechanisms that can could be used there. So I actually Prototype something like this. It's it's not that Yeah So it's basically it's not really it's basically we need to do the put in the work To to get it done in been pushed in fedora and do it's basically a political thing to actually make it happen It's not it's not like that. They have technical huge issues so now but So that that's just a side. So one thing I When thinking about packages, we'd kind of treat the packages like a huge sea of load a lot of packages They're more or less independent of each other and I think that that's kind of wrong I think there are a lot of packages this cluster together the chair Communalities and but right now there's not really Many mechanisms to actually make this a reality. Yeah, of course We have groups of people that do that, but the only thing that really binds those pictures together are probably shared section of Of packaging guideline and I mean there is some amount of Dealing with this with like common macros and stuff like this, but it's My my understanding is this is on a very low level. There's very little that did That we really share among packages and One Way to get us is basically Pull out common stuff from these packages and put it somewhere and the problem is Where to put it and traditionally there has been basically two Places where to put things are you put it in a package and the package or has full control over it and but also full Responsibility to to make sure everything works All things are handled by rpm upstream Which is great because it's the same for everybody, but it's also not so great because it's very hard to change especially Because most things if you change something just usually you need basically Process in which you change from one way to the other and it's very hard to synchronize this upstream with the distribution and every distribution has its own idea how it should actually look like which is fine from an Operating upstream perspective, but it's not really something That We can have with that So the idea I have which is still very Lofty and and not very much thought through is so I have some templates that can be used by packages for building I found something that some distribution that does something like this already, which is funny enough gentoo Which is very different from rpm, which because they have basically huge they have basically Scripts that build their package and they are there their software. It's a really not really even package But what they do have they have a collection of scripts They can import into packages of a special kind so they have like three different scripts for python they have a script for sea libraries and and All other kinds so they are about a hundred a hundred fifty different scripts They do have which you can import and then can use functions from that so they typically get imported They don't do anything and you as a package job can then just call a function that does stuff like calling make Like like calling configure and make or calling Python setup something and Things like this and it's something that you could do with rpm already if you really wanted hard You could basically put everything in in in in macros and ship a macro file Which is something we kind of already do but it's not really convenient. It's not There's not a real way of doing it Sensibly as a package or so that's something so it's it's it's Also on a borderline where we say well, there are a couple of features that rpm needs to needs to learn But it's probably also a lot of The distribution actually getting together and thinking about what is actually common between which kind of packages And so so my vision is also to get to the point where we have like about a hundred Of these helper scripts that that bind those packages together Of course will be opt-in so it's basically an helper you can use or cannot use and ideas similar with the script That's to to look around for common patterns and replacing them by the global version where possible If this actually gets implemented for a while, which means like ten years It will probably also solved all those special stuff for the distributions because you can basically use the global from this from the release you're actually building on Yeah, of course, that's that's like In ten years, maybe Yeah, but the thing is even if you replace it for the new stuff and keep the old stuff in there It will still yield benefit although if it's not as big as if you just replace the world right now but the thing is the number of of releases of Fedora and well and whatever will only go up. They will not go down So so those lists of ifs will probably get longer and who knows Which distributions will extend their life spans to the decades or centuries in the future so we want to So we want to get rid of that at some point so we will probably Look for some Packages that that are easy targets and we will probably talk to people and see if they have ideas What could actually help them it? That's that that's a two fold thing. There's it's an rpm side. Of course. There's a huge Packature package side and we need to talk about that the thing that's more closer to rpm with regards to features needed is doing sub package templates or It's not quite clear. What name will stick in the end But some packages are also something that you basically copy over and over and over again. They're all basically the same and That could be pulled to a central place and then we reused the nice thing about having having in a central place is That it would also allow the distribution to gain some control over how packages should be So you could make decisions like Do we want to split out all language packages? For example, because if you have a macro that's used throughout you could change this later on if you say Well, now we have 150 sub packages for it for each For each application, that's just too much. Let's come up with a solution that will like squish the most important one into like the the big one and then maybe the exotic one in three other Packages or something as you can then change this in a central place without even bothering the packages Which is also something I think is a huge problem right now whenever you want to do something on a policy level For the whole distribution you basically have to edit 20,000 packages and there are people who have obviously done that and I can only try to to imagine the pain but It's probably not not so much fun and Pulling this out basically does two things at the same time It saves the work for the packages and the responsibility for the packages and also enables the distribution to do more things easy more easily of Course they could be used in combination with the build templates It's something I'm not getting into now too much But of course if you know that supply some package you could basically say well By some packages typically have those sub packages and we just include them there and this happens automatically We don't even have to think what sub packages your type of package actually should have or what the distribution things it should have But that's Left for stage two or five or something The thing that's a bit more complicated here is that these templates work against the way RPM thinks about how spec files should be right now Spec files right now are like to some extent a Quality control for the stuff to get packaged So the idea is you have a file list and it lists all the files and if they are if some file shows up That's not in the list. It's an error and we want to detect that and if some files are missing We also show an error and it's basically you as a package or Looking into the package and making sure they are all fine. It's okay No one got hurt and and and and RPM checks for you that that the things you saw there are still there And of course, it won't work with this kind of templating engine So it requires a change of mind and expectation to say we don't really care. What's in the devil package It's that these files look like they should be there We put them there and if one of them really shouldn't be there Please open a bug and don't rely on RPM telling you this so it's will that requires some like cultural Shift in in how packaging is perceived and we'll see how that works These are some of those basically removed Changes or removed checks that we currently have in place like what to do with the packages empty Of course, you just drop it if it's in template Right now. I probably won't build Oh So don't error out of the missing files You probably need a way to steal files from other packages if you want like a devil if you want this like a sub package You want to split be able to split out like the libraries into a lip package and you disable this you want the lips Go back to the main package and not like looking into what to do then and change the main package to to accommodate for us you We will probably end the way to basically Append to the templates So right now you cannot have like two file sections. That's an error I probably you probably added that that to be an error like a couple of years ago if I remember correctly I might be wrong, but that's something we will probably Change so your bags actually can use the template and say well, but we also have these weird packages Should also go into the devil packages or something like this. So you can append that without copying in the The staff and redoing it There a couple of more complicated things I'm not quite sure how to implement this correctly You want probably at some point have like a sub package per sub package, which sounds like a weird concept and We'll see how we do that the main the major Use case for this is the is the new debug info stuff, which is currently basically hand-coded into rpm code The old one used to be a huge mess of Macro stuff that does things somehow maybe But of course the the way it works like now If in case you don't know we currently have this Debug info sub package for each sub package that does have binaries basically That's something that's more tricky to do because you basically only have to create those packages You also have to put the right files and then they only Have to be the files that match to the sub packages, but they are not the same They are just like related so there needs to be some weird pattern matching thing I'm not sure if that will be in the first iteration but That's the main thing So I just realized I'm much quicker than I thought Which is probably not a bad thing because There's a lot of weird stuff and I'm curious if you have any questions or remarks or things we really shouldn't do They are still faces that are in shock I don't have them here we have We do have some examples on on our wiki page on rpm The for the for the package template stuff My idea is to basically use the normal package syntax and probably add some config options to the package Person package command so you can It's not really spec out yet There are a couple of features that we would add to the to the package command or to the package section basically And that would basically switch all those checks to the right mode and you basically So so that the Template does the right thing Except to the normal packages will of course keep the package sections That we still have the way they are so as always it will package will just continue building Yeah, of course And Yeah, I mean that's that's a point we have to recognize that we are known no longer in the 90s and many many of the concepts that That the build the spec files today have been invented in the 90s And we now have like a lot of more infrastructure that that can cover many things that are the spec file Does no longer have to do with with the two with today's environment more questions I have not thought about this yet the thing is the thing I hope this will work like is that those templates up are Maintained in a central location so not every package has to do their own thing So the thing that's actually going into the package is basically just include this this style or type of package so the Stuff that packages can actually do wrong is hopefully smaller than before Preferably much smaller, but no promises yet that that's a tough question. I think we They will be version somehow But it basically not within the spec file because the spec file also does not really Version the dependencies So there's something that probably needs to be done in tooling around that because you don't want to update your spec files Whenever you change something down there. I Would even argue the opposite is true. You want Those Templates to be the one from the current distribution So if you rebuild a package for distance distribution, it actually applies or Conforms the rules of these distribution So if you would basically rebuild the same package for for an older Fedora release it might have different Subpackages because that's how the distribution should be built But but but we will probably add some hooks to to to to so We'll probably do something like build requires for those templates so they are not completely Unconnected to the package, but but I'm not sure yet how to do it there So it's not it's not like and now there's there's there's a whole topic of of reproducible builds and there's been some work into furthering this course, but we probably will revisit this with this topic in mind, but it is as a first Basic concept. This is the ideas that those templates are not born too closely to the spec file, but it can be swapped out to the right Version that we are building on That doesn't mean that we may say like do some provides or something that that will In the build package signify what was used that's something we might be able to to add But it will not be part of the spec file, but what part of the build package Does it answer the question? there was let's say debugability of rpm is limited and independent of of the of this, but we will probably Need to add more features that will make this easier like spitting out a spec file extended or something like this That's Still not not completely thought through but but I am aware that this will make debugging harder that it is already We will probably do something to to to alleviate this as as much as possible It's something on my list Yeah, I've not yet bought into one Implementation of the users and groups and that's that's basically the reason because I'm kind of hesitant to to call some Some program that's part of the distribution to do such a fundamental thing but The question is the problem is you actually need to create the users during the during the Transaction or before the transaction so it's not it's not enough to just say well It would be nice if we had some users. No, you have to have to actually have them Yeah, and but you have to have someone that actually cares about it And that's probably rpm because rpm is the part that's actually running They will probably make this like a config option like sums macro or two so that that we are operating systems That have shown up upstream actually we got people submitting patches for all this too Yeah, that that that's what's my reaction to So, yeah, but we'll figure it out That's something we need to think about a problem with that right now is that there is no Internal representation of that when rpm currently parses the spec file. It's parses them line by line and explode Expands it Item by item by item. So there's never ever an expanded form of the spec file as a whole Which is yeah, I'm sorry. That's how they did it in the 90s like like many other things but So that makes me hesitant for now to just implement something because you're running the risk of having well The version you show the user which we have expanded and you have to version that the rpm users inside But we need I'm with the template system. We probably need to do something like this The question with the template system is if it really is part of rpm or not and I'm not decided yet as we need for the for the Get stuff We need something. That's not rpm But that's wrapped around the spec file and it may be possible to do all the templates also in the same step So you would rpm wouldn't even see that and that would actually make it easier for us to show to expand It's file because then it would exist as an intermediate step and you would just show that Which may be a nice nicer thing of course the holder all the sub-package templates is all all within rpm, but it would not be and Entered into the spec file in rpm, but it's not Not completely decided yet. I'm kind of hesitant to rely on external tools. I'm also There are also those all those kids This get stuff that can't really be part of rpm. So it's It's yeah We need to have a better understanding what's actually going on to to make the decision Yeah, they're all kind of drawbacks and and and and and funny side side things I'm not even sure if you're really if it's really legal to not ship the original spec file because it's kind of sauce So if you if you expand that and ship only the expenditure you could argue if it's like the GPL Do you have to ship the non expanded spec file because it's the actual resource and stuff like this? There are a couple of interesting things to ponder on One could argue that but it it clearly is it clearly isn't It clearly is not Yeah, if you if you want ponies or other large animals you get you Probably not that Difficult, it's actually no you it's actually probably much quiet. I'm not sure It's probably me so there are a couple of macros which get inserted automatically into your Spec file like like the pre and post of the build script and stuff like this Which is which does some things that you may not be aware of that rpm like Battles with your files without you noticing and stuff like this I'm not I'm not sure if there's one of those macros that gets inserted into the To the preamble probably not but that's something that could be done So you basically have a macro in the macro file which has the version which tickets and rendered into the spec file It might need to let like two or three long change To do that So so so the thing that we're probably going to do first is the dynamic build dependencies It's something I've not said that much on it's The problem with that is it's it's not that difficult to do in rpm You basically just have an built require section which is executed after prep The problem is the result of that and the result of that is of obviously that built breaks at this point Because you don't have the dependencies So it means the build system basically needs to read them reinstalled and the new dependencies and then continue building Which is something now build system in the world actually does right now. So it basically breaks all build systems Which is which kind of Make me doing this a bit slower than Maybe I would have done otherwise So so that's that's the main major issue and it's been thinking about through this a couple of times But there's in the end there's no way around it I think having dynamic build dependencies is a valid use case Especially for all those new languages that ship a file which actually have them You just have to read them out there and you don't we actually don't want our packages to look in the file Copy out the the build dependencies reformat them to the way that rpm needs to have them Then copy them in a spec file and the next build they break because if they change and you didn't look into the file That's that's really not not not a sustainable solution But if you do it you need to actually calculate it and you can only calculate it with the sources So it's basically after during build after after prep Which is not a place right now anybody expects the bill to fail on a default kind of build so But that's probably the first thing we will do and then we'll look into the both the two template stuff my guess is from a from a coding point of view that that Subpackage stuff is more straightforward for us to tackle because it's actually coding effort But we will probably So there are a couple of people that will that will push Get this stuff and that's something we will probably look into first and tight try to to forward this because it will Give a First few on how templating on spec files in general Can work can be done or should be done Which will probably help with the other other templating stuff and And if the world goes towards we are Writing tools around the spec file will probably use that if it turns out to be a bad idea We'll probably do something in our PM So that's the roadmap I have for now Someone else had something which I already forgot other questions More more more wishes back We can yeah, that's something you probably can do easily if there are still bits left in that field They used to be Yeah, right Yeah, we can probably do that It's quite possible that it's quite possible that it did this got removed by accident Never heard of him But If anyone is interested in the history of week and rich dependencies and we can talk about this after the talk Okay, one more. I think you currently can't what about putting in Comment at the first line Yeah, that's that's something that's not related to this all but that's something that has been proposed I think by Michael Schoeder mainly The problem we have right now with the script that says it's impossible to fix a broken script let So if you have an broken uninstalled script let You're basically screwed I mean yeah, you can Fix that by hand by running it without with no scripts But that's not kind of the solution you have if you've just rolled out your broken program your broken package to the world And that's that's something That we need to deal with at some point Okay, what's the time