 Who wants to watch me log into things? Yeah. What fun. Yeah. Yep. There's some water for you if you want. Oh, great. Thank you. It's up out there. You're a lifesaver. Thank you very much. I think that's the one that you're starting, so that's perfect. I'm just going to use the same slides because I don't have new stuff. It's been a while. Okay. So yeah, you're implicated here, Stephen. I'm sorry. Okay. Do you have a question, sir? One minute. Okay. Very good. There are questions on what I wonder about this. Cool. I don't know if you have updates or anything on any of this stuff, but I might... Oh. We'll find out. Sorry. Yeah. Yeah. Yeah. Exactly. All right. Oh, man. You do not want the level here? No, it'll be fine. Thank you, though. Also, the one thing I do want is the different... It's a race. Oh, it's two. Okay. Can I do that? Can I do that? Can I do that? Can I do that? Can I do that? Can I do that? Will it go? Uh-oh. That's good, right? Some definition of good. So, if you're using the company app, then we just have to do that. Yeah, we're just going to try to get it so I have time, because my notes are all in the... Start recording. Oh, okay. Sorry, yeah, I'm... Sorry, you got it all set. I didn't have a little screen flicker. Sorry, I was trying to switch it over so I could have speaker notes instead of having those on screen for other folks that I don't know, but we can do that. I'll make it up. It's fine. Well, unless you're doing it offline, are you... Well, no, it's... It's mirroring, and you can actually... If it's... To extend the screen? No, if you do extend it, it doesn't record. Yeah, once it's having a higher resolution, it actually does. So, you know what I mean? So, we are recording the desktop also. So, if you do extend the screen, it doesn't record. Oh, you can't tell it to record? No, it's really... Well, you have to go with memory. Oh, okay. All right, I'll go by memory. It's fine. Let's see what happens. Yeah, but then if I do that, then that's on the screen. Let's see. Is that going to show that one or this one? Yeah, sure, we'll do that. That'll be fun. Oh, I don't have a Wi-Fi on my laptop yet. I haven't set that up, and I don't know what password it is, so... Oh, boy. You know what? I'm just going to wing it. It's fine. I remember scriptlets. If you all are here to hear me... Ladies and gentlemen, Will Woods. Hey, hi. Absolutely not. Adam, please leave. Hi, I'm Will Woods. I'm a senior software engineer. Hey, I should update that. I'm a senior now at Red Hat. And I work on things involving installations and packaging and saying a lot of cusses about RPM. And half people in this room have heard at me rant about these things before and you're probably tired of it and too bad I'm doing it again. So this is RPM scriptlet reform because I think we need to... Oh. Is it...? Do I just need to actually talk at it? Hello. Oh, fine. Hello. Oh, no. Too much. Too much, Will. Is this an appropriate amount of Will? Am I audible? When you talk into your collar, it works. Hello. Yeah. Okay. If I talk normally... No. Oh, boy. Your leather looks so... Okay. All right, cool. Yes. Good? Yes. Okay. Technical difficulties are now solved. So I'm talking about RPM scriptlets because they're an enormous period in the past and they make everything terrible and I want them all to die. Well... Here, here. If we're being kind, they do stuff. And we all... If you have worked with RPM and the RPM ecosystem in all of a sense that, yes, scriptlets do important things and without them, you know, I don't know exactly what they do but they are really important and if we didn't have them, things wouldn't work. Which is kind of true in that they do stuff and if you don't do them, things break. But we have some problems with that. They do all sorts of things that aren't great. Everything runs its route. That's not my favorite thing about them. Every package has its own unique scriptlets and every package has its own little fiefdom and so the scriptlets for any given package are written by somebody who might not, like, use shell scripts much might not even be... And then Linux programmer at all might not have ever even used Linux. There's at least one package that I can think of where the scriptlets are all written in Lua because the author doesn't know any shell but could figure out Lua a lot easier so just wrote them in Lua. And that's fine under, you know, the guidelines and everything like get it done, I guess. But this makes, doing all of that makes installs and upgrades really, really slow. My background is mostly on the installer team and doing system upgrades. And system upgrades are real slow but slower than you'd think when it's really just like you're making a bunch of small changes to every package in your system. Part of the reason for that is that every time because we don't know what scriptlets do, right, it's a black box. We run some shell script here. Who knows what it does? Magic. There's no magic. There is, but it's dark magic. But we, because we don't know what's going to happen in there and we don't know whether or not the next package is going to need something from it or whatever, so every time a scriptlet runs and there are, what, 11 different ways that they run you can run like pre-transaction, pre-install, post-install, post-post-install, post-rock, gypsy-rock, gypsy-funk. Those are, I don't know, there's a lot. I forget all of, I used to know them off the top of my head. There's like 14 different places that scriptlets run. And so they get run before and after pretty much every package and we have to run like, we have to run F-sync and wait for everything to get run to the desk and then we install the files and then we do an F-sync again and then we run the next scriptlet and we do run another F-sync and then we start the next package and it involves forking and exacting and all that. It makes everything painfully slow compared to how much it could be. Under designs, another thing about scriptlets one of the things that happens a lot during system upgrades and I always feel bad about this because I've written like three upgrade tools at this point is that at some point somebody will be watching very intently, very excitedly watching their upgrade happen and the whole progress meter is going doodly-doodly-do and then it stops and they're like, oh no, why did it stop? And they wait like 30 seconds and it's still sitting there and nothing is happening and they're like, oh no, the upgrade is broken or something bad is happening and they pull the plug. Usually what's happening is like it's relabeling the SE Linux policy on the system or something like that but because there's no progress supporting from scriptlets because they're like, can't be because we never really figured out how to do that there's no way of knowing that's what's happening so your system just kind of sits there. This is why DNF now says like running scriptlets that's something I've made them add like we argued for years, yeah I'm really happy that it's there but yeah, yeah it's been in I'm gonna crash this when it tries to print that message we're so bad at this, oh man but yeah, so there's no progress supporting there's no way of adding it really to the spec I mean we could but we haven't figured out how to do it and it would break all backwards compatibility so like you just don't know what's happening in there and you think it's stuck and you pull the plug and you've completely destroyed your system sorry you should have gone and like, I don't know, gone for a walk so we don't the bottom one is the thing that bothers me like at an existential level we do not know what's happening in there and so for example like, what's that do? that turns out to be a way to just create a file it's a clever way of creating a file at the same time and also making sure you set it to the right mode rather than doing that in two operations so if you want to atomically create a file with the correct mode somebody, I don't remember what file or what package this is even from but you can do that I had no idea that's what that was doing but yeah, you can install Dev know who knew, great oh, here's another fun thing Steven's going to have to remind me what this fixed because it's something absurd it was like, if you installed do you remember so what was happening here was this was actually cleaning up to make sure that we only had one addition Fedora installed so your server addition, your workstation addition or your non-addition Fedora and it was designed to make sure that we didn't that we only had the correct presets for that version and we had reserved the 80- range for that on the file system for that and this was designed that if we were changing the addition or assigning the addition we would remove any others however in Lua that change is adding a percent sign that percent sign means that previously it just treated as effectively anything that starts with the word with eight and and then the range of zero to dot which was all alpha numerics right and then ended in presets so it was deleting everything that started with an eight which included all of the reserved presets whereas only 80 was reserved for the additions and what did this cause, this one character change catastrophe it caused a problem people were basically we had upgrades where none of these services that should be running on the system were running on the system right yeah like this one line in something made it so that people would install or upgrade their systems and then suddenly nothing worked at all because of that and like tracing it back to that was a really shitty week for somebody condolences friend oh here it is here's the big list yeah so these are all the places scriptlets run and remember if you're doing a system upgrade there's 1500 packages so you gotta do all of these steps 1500 times the pre-trans and post-trans those only happen once but the middle parts those happen 1500 times it's slow and bad and the question is do we actually need to do all of that and so well okay by way of a demo, gosh is this actually gonna work we sort of put together a little thing where yes sure I'll let it run so we put together a thing and that is not a spectacular demo but you see some text happening and now it's and now something is booting that six seconds right there was us constructing an entire file system image from like a kickstart and then booting it in KVM and it works fine in six seconds usually it takes like I forgot six to ten minutes was what we were looking at before and the way that we did that was just essentially not do scriptlets that's about it I mean there's some other stuff we skip the decompression of all the package headers and stuff like that but we just take all the package contents dump them into the file system a little bit of tweaking at the end to make sure it actually works I wonder if that's gonna keep running it doesn't smart so here's the thing the way that we did this was combining these two so the way that that magic works is that I went through and read every single scriptlet in every single package in rail seven and oh wait and what I found out was that they're all kind of weird and clumsy and strange but they only do these six things this is everything that happens in every scriptlet in all of rail it's just that which means we really don't need to be run like allowing the system to run arbitrary code 1600, 1500, 1600 times during an upgrade and all of that we just need some stuff that does these things which is I know vague but we already have a solution for a lot of these things and we could have basically equivalently powerful stuff that does what you need to do and isn't, is introspectable we can look at your package and say oh this one's gonna create a user and you know if two packages want to create the same user we can skip one hey we can do it once we can wait till the end that nobody's gonna actually need that user if we're not building a live system let's say if we're building some sort of container image we don't need to create the user in the middle if we're not running scriptlets that assume that the user will be there which they don't need to because why would they but this is basically everything that happens in the scriptlet so we have stuff for a lot of this for users and groups well we have sysusers there's a thing that comes with systemd that just like sysusers for you you can just drop a file into place you don't need to run random code just drop a file into place your user will get created when it is appropriate to do so now there's also a system for control people like to turn services on and off in their packages don't do that man what if we're trying to build like I don't know a container image or a virtual machine image for somebody else don't turn my services don't flip my light switches that's actually forbidden in fedora this is actually forbidden in fedora now but this happens a lot in rel luckily that one's going away so these are all things that are mercifully going away but I'm going to need help and sort of you know discussion on how to get rid of some of them there's some things that are like creating empty or default files moving default configs into place things like that again systemd handles this for us use tempfiles.d it handles everything except I think one case and I can't remember what it was oh that's right there was one part of sysusers.d where you couldn't previously you couldn't give a user a different shell in response to a previous version of the talk they've actually fixed that so hooray people actually want to fix these things join us won't you tempfiles.d snippets for every case I've ever seen if you're trying if you're installing a package and then you're like oh I need to create a file I need to set up some default stuff everything I've ever seen can be handled with a tempfiles snippet and if you have a use case that wouldn't work please come talk to me and we'll try and figure out a better way system specific data and this is something that Steven did if you're generating keys or certificates or machine ID something in a specific to the hardware that you are running on or the system that you're running on if it's not bare hardware there's a specification in the fedora project for how you do that how you handle initial service setup so like don't go handling that in package scriptlets it's not necessary system configuration you just don't mess with this stuff like there's no we have we have presets for some of these things for I mean that's more for services on and off but like you shouldn't be twiddling the firewall you shouldn't be inserting kernel modules when your package get installed I know you have a cool kernel module and you really think it should be installed if applicable but like just don't there's other ways of handling that I'm sure but generally that's not applicable when we're building images one of the big things about the project I'm working on which is well there's composer and then there's welder is the upstream part and that was what was doing the six second image build is that we're building images from the outside in normally when you're building a system we install a bunch of packages into it and I don't know if you were here for my previous talk I basically was like we basically open up your hard drive and it's not like we're laying a bunch of bricks down to make a wall like we've got all these little robots that have like chainsaws and arms and what not attach them we throw them all into this like arena and they battle and they clamp onto each other and eventually they construct like Voltron and that's super cool that it works but it's really insanely complicated for what when you really just want to lay down all the files that are in these packages and then do the tweaks at the end that are necessary the whole point here is to figure out the tweaks that are necessary and do them when necessary this stuff is not necessary especially if you're building your package from the out or building a system from the outside because you're going to be doing it on the outside system and like I don't need your super cool kernel module installed on my laptop then there's things like caches and catalogs this is the this is like running LD config after every library gets installed we're running you know the update desktop or update icon caches and all those update whatever things that people need to run those should be done handled with file triggers we've been transitioning stuff glibc has finally like it took a while for us yeah that's that's happening so like you no longer have gld config and we're getting rid of that stuff on a case by case basis when we can figure out when the people who own this stuff are willing to do it because this is all you know fedora it's volunteer work but it's a good idea if anything I hope that I can convince you all that like I'm not just up here like destroy all software I'm like well let's try and do stuff like smarter and one of the things we could be doing is putting control of tricky things like when do you actually have to run this update script well let the guy who let the other guy handle that the guy who runs the main package that handles the tool that does the update you drop down your files and you just walk away that's how it should be if you maintain any sort of tools or any packages that contain that sort of tooling please come talk to me and we'll figure out some sort of way of making sure that your stuff gets handled automatically that's really sort of my pitch for why we should get rid of scriptlets and that's about it do we have any questions and do we have somebody with a microphone that's okay it's a nice one simple question have we solved the problem yet where you want a file in a package to be owned by this created by that package because I recall that was a problem before oh yeah if memory serves the RPM guys are talking about finally adding I have been talking about adding user and group support because like packages don't really work unless packages have to to install correctly RPMs do they have user names they don't have user IDs they expect you know I'm going to create this user or I'm going to create all these files and I'm going to make them owned by this user so if that user doesn't exist it breaks so obviously if the RPM depends on that being there RPM should be handling that but like 15 years ago it was like ehh so we're trying to get them that on ehh at present RPM itself does not handle this however as one of the bits that we got from the system defolks when we talked to them about sysusersd was that they have now provided upstream an RPM macro that can be used for percent free essentially to do this for us so we have what is effectively a work around but it's a nice handy macro that once we actually figure out how to get rid of the scriptlets proper the macro can just be replaced what backs the macro can just be replaced with that and I can use this in a fedora spec and it will be guideline compliant and could I use this work around mechanism in a fedora spec and that would be sort of guideline compliant as of now no and that is specifically my fault I have been promising to write that spec for FPC to approve for 4 months and I have been dragging my heels on it I owe you that spec ok so that's not fixed but we'll be soon I have a question about the same topic so and my medium size linux environment which has some legacy baggage many groups like post fix exist in the directory service with the same GID and the same name spelling and when a scriptlet tries to run that says add a group post fix it fails with this handle the case of another name service switching service already providing the same group I hope so I don't remember what their solution was going to be my solution is the sort of scorched earth like nobody gets assigned to UIG or GID that's silly they all get assigned dynamically at start up and there's support for that in system D now where if you need a user it can just be created at the time that your service starts or whatever but that's new school if you're dealing with legacy stuff my understanding is that I don't know what the current macros look like the currently approved snippet that you're supposed to use is always like if this user doesn't exist then create it with this UID and GID otherwise leave it alone we're not sure if that check works for non-local groups but thank you I can't answer that because I wrote the fix for that specific problem so I know that that is actually fixed in fedora it used to be that it would just check etsy password the proper approach now is that we'll call getPWUID to see or getPWNAM whichever is appropriate and figure out if it's already if it's available to anything within a switch that's awesome thank you very much there's a no computers were harmed during the rain thing a question on an item that you mentioned towards the end instead of updating caches and catalogs using file triggers instead would that be something like a systemd path unit or something like that there is a capability in rpm I guess it's called percent file trigger where the package where the upstream package that owns the thing that actually maintains the cache or catalog watches a path so it works conceptually the same way I think that the systemd units are actually a little more flexible I think the rpm file triggers can only work for like a directory and a path or in a glob but it should be roughly equivalent where it's basically when rpm sees that a file has appeared here it'll run a script so if you as the packager of something that needs to be injected into that cache or catalog shouldn't have to worry about it and if you are currently worrying about it find the person who owns the tool that you're using and tell them to fix their stuff or come talk to me and I'll tell them to fix their stuff it shouldn't have sounded like a threat so one of the packages I maintain for my work as a third party packager in various ecosystems is a tool that needs to have network communication open so it's a listening demon so it needs a firewall port opened and you said earlier in the talk to just not do it but if in the case where it's running how would I do it and in the case where it's being preloaded like I don't really know how to deal with that particular case all that well I think Stephen might have something to say about that I can take some of that in the case of firewall Fedora's general answer to firewall is firewall D which as of two years ago now can handle it has a the ability to just drop a file and tell it that this should be enabled so it won't take effect immediately but it'll be on next boot so that is the preferred approach for a service that we should do that that being said this is also forbidden in packages that are approved for Fedora because the working groups in the Enfesco make the decisions on what is allowed to be opened by default but for a third party package there is a firewall D upstream feature that allows you to set that with it and thank you for that does that answer your question yeah I think the general sort of lesson there is that activating changes to the system immediately is a policy decision that isn't necessarily up to your package but we have pushed it onto packages historically and we should stop doing that one would hope that all you would do as a package is drop files into place and then things would happen and so every time that you have to as a package or like go crank some knob or flip some switch to make the right thing happen it was probably a good idea if you figure out who owns that knob or switch and talk to them about why it is you had to turn it manually because like that's crappy experience for you and a waste of your time and then you have to maintain that script forever system so that's a general sort of hey if this is a thing that you encounter frequently talk to your upstreams or talk to me and I'll talk to your upstreams or something like that but we can fix all of this it's all open source we have the power any other questions one more oh yeah that was you you can just holler and I'll repeat it back to you that's sort of a theoretical question but as you're rethinking packaging and all the installation stuff what are your thoughts around things like pip and gems and all the other sort of things out there well that's the thing right it is the year of our lure 2018 and every programming language comes with some packaging system so like why do we insist that everybody use our packaging system which is like way harder and gnarlier to use this is a much bigger problem than we solve an rpm especially the way we use rpm today we end up having to wrap every other packaging system which is why we have so many gnarly gnarly things in our dependency database now the most common or one of the most common words if you split apart every package name one of the most common words that comes up is github because like every go module is hosted on github and so the word github shows up like 15 million times it's ridiculous that's a weird abuse of a system that was never designed for that sort of thing and we need to work on designing a system that like handles existing packaging tools without there being that sort of impedance mismatch so I have general big hand wavy ideas about how we should handle that but I don't have anything to tell you about right now what we should do with it like today my big hand wavy talk was earlier in the day and I'm happy to buy me a beer and I'll tell you all about it but yes I probably will anyway yeah if you can't find me yelling about scriptlets at a tree later in the evening then I'll remind you where my hotel is and you can send me home yeah no I think that in the longer term we as a community will need to accept that people do their own packaging and we need a system that works with that instead of against it and what that looks like is a really interesting conversation but it's not one for this moment not because I don't want to have it but because you know scriptlets time so yeah anything else this is part of my whole like practically how do we get from the world we have right now to the glorious future that I someday envision and I'd like us all to share and dance and sing in the fields of wonder but right now we got to dig out from a huge pile of crap that is scriptlets so anyway yeah so during the center west dojo I gave an unplanned lightning talk about how I sped up the prisoning of scientific workstations with a on rile 7 with about 4,000 binary rpms from 6 hours to 1 hour the other change switching from ANSW ANSW's DNF is outside the scope of this conversation it's a limitation of integration between the two as opposed to but the other change was at provision time like before people actually start using the machine or running applications using NoSync NoSync is a small like 100-slot library that suppresses fsync and similar function calls because the idea is at the end of a provision you reboot anyway which forces fsync that's the Debian packets version whereas NoSync is packets for the redhead family that is a clever unsurprising and completely filthy yeah but I mean I'm glad to hear that you're doing these post-transaction scriptlets like to eliminate the amount of syncs but has anybody some people are trying to integrate NoSync with Anaconda because we boot at the end and does sync at the end anyway has anybody like ever considered just like oh I want to do a quick provision to finally run fsync at the end of the of the entire transaction I don't if you, yeah I don't know if you tried to get them to add a flag for that there'd be anger obviously to say about a fault but you know yeah I mean it's a tough question because like it doesn't really those syncs don't help anything or the problem is we can't prove they aren't necessary and that's the whole problem with scriptlets is they're a black box and there might be a case where it is always committed to it working that way and so we can't safely turn it off we can let you safely turn it off and there might be a switch to do that somewhere but we can never be like you should use that it'll make your stuff faster because as soon as somebody's house catches fire then we end up paying the bill Oh yeah I'm sure that people would use adoptate existing systems but for provisioning time it's useful because if you fail in the middle of a prison you just repovision but my goals for provisioning are to have it be that you're just laying down bits and not like dealing with RPM in that process at all you can just have the payloads that you want to lay down ready to go and you can do that with installer as well but that's more long term goal yeah no that's a clever and awful hack I commend you it's worked over 200 times reliably though for controlled conditions alright is there anything else no alright Ben thank you for your time and attention yes it's the libraries are so secure yeah yeah hey here's a can you keep that and then like sure oh yeah the wrong I mean yeah you can't throw anything away