 Good morning. Good afternoon. Good evening. Good night wherever you happen to be So today So I'm Stephen Gallagher. I'm a longtime fedora Contributor and leader Troublemaker and today I'm going to talk to you about Fedora ELN Or as I like to refer to it putting the relish on the beefy miracle For those of you who've been around a while you might remember We had a fedora release once codenamed beefy miracle that was Absolutely not controversial at all So moving ahead first we'll start with a little bit of history The idea behind fedora ELN originally sprang from the rel 8 bootstrap When we were just when we were taking fedora and turning it into the the beginnings of what would be rel 8 it took over a year of Brand of caught you know, cloning the branches built Building the packages figuring out that the that they no longer which things no longer built because they built okay with the pre with a Previous set of dependencies and ran on fine on a new set of dependencies, but they couldn't build with the new one and So on and so forth and oh this this package. You know in fedora We run You know, we turn on every experimental flag in the configure script because hey, it's a fedora I've got to try it try things out. Whereas in red enterprise Linux. Maybe we only want to stick with the production quality features and that can that can result in lots and lots of additional dependencies So we have to also trim those out And in particular My if you've ever heard me give a talk before I'm not a huge fan of Sphinx if only because it pulls in something like a 25% of all the packages in the fedora distro just to Handle writing handle writing docs Because we have all the flags turned on that that sort of stuff takes a lot of time to work through and well Then red hat went ahead and announced that at the release of rel 8 that from that on we were going to be having three year rel releases and Myself and other members the bakery team collectively turned and looked at each other and said Well, that's gonna be fun Because of course we can't If we're gonna be releasing every three years, we can't be spending more than a year every time just Getting the getting the bootstrap into position for people to do development on we can't Realistically that would lead to something like a year maybe a year and a half worth of actual development time, which is Absolutely not going to be enough for a rel release So what did we do? Well, we realized that oh, yeah fedora exists. What can we how can we take advantage of our good relationship with fedora? Actually, I want to rephrase that because it sounds like it sounds like I'm abusing that relationship, but What what we decided to do was see if we could spin up a version of fedora that Was designed specifically around Looking like rel So that we could get a lot of this bootstrapping done In a rolling way from right from the moment that we forked from the previous fedora for rel So Fedora ELN was born now Where does fedora ELN come from? Well? ELN comes from rawhide What we do is we take rawhide and the rawhide disk it We select we have a selection of packages that we know that are going we want to have in rel plus their dependencies and build dependencies We had we set up an automatic rebuild system that will go ahead and build all those packages with relrpm macros rels compiler flags which which affects things like performance tweaks and which hardware is supported and In general, we just try to make it We're trying to make the build process as close to what we would do for Red Hat Enterprise Linux or sent to a stream as possible So that we can so we can see much much earlier on Which things are going to give us problems if we try when we try to rebuild them for rel So how exactly do we go about doing this? There are Three main tools that ELN is using today that to make this all come together The first of those is the content resolver written and primarily maintained by Adam Samalik also at Red Hat Which is a real which was designed for the Fedora minimization effort initially and it's a great great way to visualize How what things are in your distribution? What things you don't want to be there for in the case of rel You know, there are plenty of the packages that we don't want to support So we want to know what is causing those to be pulled in And generally Provides us with a really great way to search and examine the dependency trees in the in Fedora and rel We used this to great effect when boots when putting together rel 9 As it also can do do some interesting heuristic things like figuring out Which who should which group should own which package in rel because Well, they're the only ones that have that are pulling it in if they don't want to own it They have to figure out a way not to depend on it and so on and so forth So from the content resolver we also we get a we get lists and json Data and whatnot that allows us to figure out what packages do we actually want To be in eln or ultimately in rel 10 and then Examine those and go through it go through so we take one of those static files generated by content resolver And we feed it into a tool that I build and built and maintain called the distro build sync It's a broken out part of what was once called the distro baker Its purpose is It it behaves similarly to how uh, koji shadow used to if anybody remembers the the dark times before we had merged the secondary architectures into the main koji in that whenever it detects a A package being tagged into the current rawhide branch, which right now is f 37. Oh, sorry current rawhide tag in koji it then goes ahead and rebuilds that package from Against the eln target with the eln build tags and target and compiler flags and so on Once we have those builds once a day or sometimes more often if i'm hacking on it We use the punji tool written by the fedora release engineering team This allow this Is a very complicated tool that is mostly that mostly just uh gathers up other other tools that we have like Lorax and anaconda and image builder os builds And a variety of other tools And it is essentially the sausage machine It's the it is the thing that grinds all those individual builds that we've done in koji and then takes them in and Turns them into repositories for updates or an install install media boot isos usb isos vm You know virtual machine images and container images So that's where so the eln compose is Pretty much the main output Location for what you're for what you see in eln. It's where it's where you as an individual can consume And and try out and test this this stuff and also you can see whether or not And if you've been going ahead and making changes to the content to try and reduce the dependencies It's where you'll see the the fruits of your labor And then of course this whole process is cyclical because once we have these repositories generated by punji The content resolver can then go and reanalyze them and figure out. Oh, hey, we no longer need these this stuff And just rebuild sync as part of one of its Uh various tools Subtools will also go and untag stuff that is no longer in the required package set. So Then it gets back to punji and hey No, uh, no package that package is gone again our goal of course is to Minimize the amount of unnecessary dependencies without Uh minimizing without reducing fun functionality in a in a significant way But uh, yeah, so I actually had I wrote this slide Today based on a conversation that I had I think I have to double check. I think it might have been with uh, uh, Fabiano in uh fedora developed the other day And I realized we didn't we didn't actually have a good overview Of how the actual process works and I'm so I'm gonna have to Put together a document or maybe a blog post on on how all these things fit together But I thought it would be a useful place uh here in this talk to to just Walk through those three big steps and see where and uh, give people an idea so, um Has eln been successful? I think so. Um, I We in rel 9 When we bootstrapped rel 9 we were able to take advantage of eln and we bootstrapped it by way of sento s stream 9 so Both of those worked and both of those worked considerably faster than they had when we were bootstrapping rel 8 by hand, uh, I think I think we had we had committed to Five months and got it done in three and a half if I'm remembering correctly. It may have been it may have been four months, but Either way, we actually managed to outperform and deliver Our uh bootstrap to to engineering ahead of schedule, which that's not a thing you get to say very often So I'm gonna say that as often as I can until we get to rel 10 and hopefully get to Start saying how we did better in rel 10 than in rel 9 So what are we going to do next? Well eln much like rawhide never stops Uh, once uh, we've got a few more releases of fedora until we get to the uh point where we're going to split off for sento s stream and rel 10 But at that point at the moment when we split, uh fedora 40 off into its own, uh, Finally, you know finalizing uh branch On that same day at that same time rel Fedora eln will suddenly start targeting rel 11 instead of rel 10 uh at that point and we will we will also use that point to Start to switch the bootstrap over to sento s stream and get and as 10 and get that running So it's going to be That's that's going to be a pretty exciting day. I'm looking forward to Hopefully it be running smoother than uh rel 9 and I can't possibly run any less smooth than rel 8 so That's going to be exciting Couple of things that are on the uh the fairly near horizon. Uh, we've been talking with uh with some of the uh The tools team the compiler team and so on And uh as well as red hat product management There's there while not yet finalized there are expectations that we may have that we may Increase the uh cpu baseline for some of the architectures that rel supports uh to to gain to take advantage of some performance gains and uh and also some partner partner uh advantages if we uh by Making sure that people are using recent hardware So that's that's a bit controversial. It's um, it was proposed once previously and uh declined on the grounds that Some of the people some of the people working with eln didn't have hardware that would meet it would match And while that is a concern, uh, it's not that Raw hide won't exist forever. So I think we're probably going to uh approach you know the eln sig about uh taking a look at that for the for the zstream and for the and for The x86 architecture as well, uh in the relatively near future So I've used about half of my time for uh talking at you, uh, but I wanted to leave most Well, the other half of the time To answer any questions that folks have about eln So I would like I'll start with uh looking at the q&a and I haven't been looking following the chat or q&a this whole time uh Okay, I've got an anonymous question. Does eln stand for enterprise linux next and I suspect I may know who this is coming from because they want me to tell the story about uh behind the eln naming so Today we it does mean enterprise linux next it was originally a It was originally a code name that I had come up with which was el nino Which of course, uh, which was a combination of a variety of of different puns to make up one big pun El of course being the spanish word for the but also commonly used to represent enterprise linux uh and The uh, nino of course is that uh eln was meant to be the child that grew up into uh enterprise linux and lastly because uh el nino is a uh weather phenomenon That comes around about every three years and uh tends to leave a mess behind it So I think that should be fairly self-explanatory Uh eln does eln have a logo Uh, it we do, uh, it's a I didn't put it in my slides. Uh, I don't know why I didn't put it in my slides, but you can find it, uh on the uh Well fedora wiki for eln, I believe It It also is a bit of a pun on the the title of this this uh Talk, uh, it is a It is three jars of condiments each with the uh with one of the letters on it To imply The relish All right, alexander Eln rebuilds seem to fail for packages that need to preserve a build order for a bi changes. Is there a plan to have that? uh That should not still be happening Uh, at least in theory that there may be some special cases but For the most part what we what we switched to doing for the rebuilds is first the first thing we do is we tag Whatever things are tagged into rawhide We tag those same things into eln and regenerate the build route before we start a build that way our assumption is We should since Presumably all of the build order stuff has been taken care of in that in the side tag prior to when we merge it We we assume that by using the finalized versions of those packages Uh, we should be able to build against the against the fedora ones to build the eln ones There are some few special cases that I know of for which this doesn't work Uh, alexander if you happen to know any in particular that uh, we aren't catching, uh, I'd Appreciate a a bug ticket a ticket or a uh conversation to let me know what it is Because I haven't caught it yet But uh, thank you for asking that I hope I hope this isn't hasn't been nearly as much of a problem over the last six months or so because that's that's about when we implemented the The new approach How much effort is keeping eln going for example when builds fail? um You may recall that at last nest I had hair But no kidding aside, it's for the most part its background effort it's Generally, uh, we I I generally only jump in and try to deal with issues that have caused the compose process to fail Uh, because generally even if the eln builds fail We most of the time have those fedora versions that have been tagged in that allow us to keep things up and running So the so in general the system the eln tends to stay usable except for when those occasions when we've somehow broken the compose like Yeah, we've or we've broken the build route for the build for other builds it's I would say it's probably taking up But depending on the day between uh 25 and 50 of my time at this point Uh few less when uh others are available to assist of course Uh petter those weird disc tags surely are confusing. Can you explain why they are the way they are? um I might need a little more explanation as to what uh Confused what what me at what confusing means here? Uh We Oh, uh, do you mean I'm not sure if you mean the uh the numbers following eln, you know, like right like right now We're at eln 121 Um, and if the if those are what you mean There are two reasons for that. Uh, we started them at number at the number 100 first Uh, so that we would be very clear that it did not correspond to a fedora release or a rel release or a centOS release, etc That it was expressly a different, uh meaning Because we didn't want people seeing el 10 because it happened to be 10 or eln 10 and thinking it meant enterprise linux 10 so we use, uh We started them at 100 and those essentially represent, uh build root configurations So anytime we make a change to the build root configuration. I bump that uh number So that we can cleanly do rebuilds without having without requiring, uh That we bump that we bump the release of every individual package to do so one of the uh one of the promises we made when we started on this started landing fedora eln into the fedora project was that we would try not to Interfere with existing packaging work as much as possible Uh, we didn't want we didn't want, uh, bode updates to be gated on eln Eln working we didn't want builds being gated, you know the build process requiring eln to succeed in order for fedora to continue working so So some of this was just a Planned convenience on our part to make sure that we would be able to do as many rebuilds as strictly necessary if we had to do a series of them without uh without Unfort without the uh massive filling up the uh get the get history I hope that answers your question, but if it doesn't please ask another one Are you looking for contributions to eln? So what kind of contributions or activities would be helpful? Well, yeah, yes, absolutely. Um For anyone who is interested, uh First of all, if you're a packager, we would love it if you actually Paid attention to and took care of the eln side of your builds I know that a number of a number of you out there probably some people on this Listening to this talk are already doing that You know and but a lot of people are probably just paying attention to their fedora packages and Great, if you're not interested You don't need to be interested. We uh, we try to cover for that But what we would love to see is and and what we're trying to do in inside of red hat And I would like to expand out to fedora and more is if you would Be willing to to start doing some of that minimization some of that trimming of experimental stuff a lot of people a lot of packages out there already conditionalize for Rel in terms of building for apple and similar What we would like to love to see is More of that planning for EL EL n plus 1 as well And so that can easily be tested it done and tested in elend. I'd appreciate a lot of that for other things, um, we could use some people who are Willing to be more vocal and help us Help us communicate our efforts and our plans out to the wider fedora community a little bit more because I Am only one man and I can either be very very visible or I can be very very productive But I can't do both at the same time usually so I would love to have somebody along that would be How it would be willing to help answer some of the questions that we could ask and help and To just help us produce some really decent documentation as well Um, thank you for that question josh Uh, I've got time for a couple more if somebody wanted to create a la niña version of elend that had a different set of flags and packages What would it take to do that? um that Is possible like I said in my previous answer. I think we would Definitely benefit from some and uh Some engineering or some documentation help I would love to try and knowledge dump what I've learned about the composed process uh to someone to someone new and we could No with what we know now about how we created elend it would probably be possible to create other similar Divergent os's from from fedora although Uh, and I'm sure matthew is going to appreciate this. I would prefer if for the most part We only did that when we had a very strong reason to do so You know elend exists because we want to make sure that fedora still if fedora serves the uh, it's its purpose of helping to um I don't want to say bootstrap it here because I've said that to award too many times but rel and so incubate rel there we go Uh, if we wanted if we were going to create another a la niña for example it would It would be a fairly considerable effort but A lot of but a lot of that would be following us. It would us ultimately be just repeating the effort we took for elend so Do you know whether rust library packages will stay in elend or if those should be trimmed down by building against vendor dependencies where needed As far as I know there are no rust package maintainers on the rel side Sorry if this question is too specific No problem if you don't have an answer right now so I think uh I think our goal is absolutely to minimize our set of dependencies and rust is Rust has a really complicated build system build setup in fedora and I'll be honest. I do not understand how it works It is a mystery to me entirely So I guess the answer really is excuse me if you were to speak to If you would go go in the content resolver and take a look at what was pulling those rust packages in I would have a conversation with uh, whatever whatever packages are at the top of that Wanted list and see You know Do they really want this here? And if so can they help? Maintain it in fedora with whether that means vendering it or taking ownership of a you know a dozen or so rust packages uh somebody needs to Be willing to take ownership of that Whom to contact to be an eln packager Uh, there are no special requirements to package for eln You are if you are packaging anything for rawhide That is that is on the list of things that will be bill be in eln You're already a packager for eln if you want to look if you want to package some additional software that Uh, won't go into red hat enterprise linux, but uh might go into something like apple when the next version of apple is released Then we have a tool call. We have a we have a sub project called eln extras, which is For better for lack of a better term, and I know troy will uh correct me in the comments It's essentially a preview release of apple for apple currently for apple 10 and If you want to work there, what you uh, you can contact us. We have we have to write up a process document for this, but ultimately it boils down to Add your package to the content resolver eln extras list Let the content resolver regenerate and it'll just start automatically rebuilding rawhide that rawhide package into the eln extras repository I hope that answered your question And I think I've gotten through all of the questions in the q&a If any if there were any asked in the chat, I have not had a chance to review that at all So if any if there are any unanswered questions in the chat if somebody could uh Replicate them to the q&a section. I'd appreciate it All right. Well, uh, thank you everyone uh for participating in this and I hope I was able to uh enlighten you A little bit about eln Thank you very much Have a wonderful day. Enjoy the rest of nest