 All right, then, without further ado, I'll briefly introduce myself. My name is Stephen Gallagher. I have been at Red Hat for a baker's dozen of years. It has been a wonderful ride and I intend many further years with the company. I've worked on a number of different projects at Red Hat but most recently I've worked on the team that bootstraps Red Hat Enterprise Linux I was on the team that bootstrapped REL8. I was also on the team that produced Modularity for REL8 and coming around to REL9, we wanted to take a quick look at, well, I shouldn't say a quick look, we wanted to take a deep look at what it was that bootstrapping of Red Hat Enterprise Linux really meant because it's a long painful process that happens, let's call it very infrequently. With REL8, I think it was almost five years between REL7 and REL8, but prior to that, at some arbitrary point, what we would do is we would say, okay, this fedora just released, we like that one. We will take that and we will bootstrap it and we'll try to get something minimal that we can use to start building REL atop. And this, to call it painful would be generous. This was an extremely time-consuming and manual effort-driven process. And the problem was that it tended to burn people out and so between REL6, REL7 and REL8, I think there was very little continuity of personnel doing these bootstraps, which of course meant that every new team was going to have to learn it themselves. So that led to bootstraps tended to take six to 12 months after we forked from a fedora to actually get something that could boot at least as far as the installer, which is not good. And that was definitely something that now that we, at Red Hat have announced that there's going to be a three-year schedule for releases, so we're going to be releasing REL9 probably next year and then three years after that, we'll be releasing REL10, three years after that we'll be releasing REL11 and hopefully I'll be retired before we have to release for REL12. But there has to be a better way of building REL from the ground up and taking advantage more of the resources that we have available to us with fedora, not just physical resources like the build system and the infrastructure there, but also the people, the human resources that are excited to help us actually produce something that we can maintain, not just something that we can, something that we use as a hobby is where we start with fedora, but there are definitely a lot of interested parties who want to build something that will be able to last for 10 years and support their applications and we want to be able to do that much more in the public. And that was something I'd been fighting for at Red Hat for north of a decade. And this last couple of years, we've had some huge changes in how we build software at REL at Red Hat, sorry, with the advent of CentOS Stream for REL8, which admittedly had a rocky beginning. But I think people have started to come around to recognizing just how exciting it is to be able to actually publicly contribute directly to the work that we're doing. And similarly, it's beneficial for the Red Hatters because there's a lot more view, a lot more eyes, a lot more code review, a lot more just assistance in all sorts of ways, documentation and whatnot with being able to do their development in the public. So we sat down and we wanted to take stock of that and just figure out, okay, we still have that same problem with CentOS Stream 9 where we need to be able to bootstrap that in such a way that another team can continue to do that for CentOS Stream 10 and so on. So we began a new project and we gave that project the code name El Nino, which it is a pun, as any of you who have seen me give a talk before knows, I am very, very fond of puns. The name of course derives in part from L, the EL portion stands for Enterprise Linux just like it does in Apple or RHEL. The Nino portion is the Spanish word for male child, the implication being, of course, this project should always represent the immature version of what will grow up into Enterprise Linux. And then of course, collectively, El Nino is also the name of a type of weather pattern that occurs about every three to four years in the Northern Hemisphere and brings quite significant storms with it. So I feel like the metaphor there should be relatively obvious. So we embarked on El N and, sorry, what would eventually become El N as a more palatable public name as a mechanism of bootstrapping the next major version of RHEL continuously throughout the Fedora process rather than having them be two completely disjoint projects, we allow our engineers and our packages to be working towards the bootstrap of the next version of RHEL right from the moment that we fork off the current version of RHEL. So what exactly does that mean? What is it that El N does for us? It's an early preview of what RHEL N plus one will look like and it allows us to do some packaging and work and to maintain RHEL style changes upstream in Fedora and test them and prove out that they work well in advance of that moment when we break off in the next two years. We break off and start hardening RHEL. So what do those changes look like? Some of them are very straightforward. We've got things like compiler optimizations or compiler flags. Perhaps we know that the upcoming RHEL is going to drop support for really old hardware. So we wanna make sure that everything can still build if we have on the newer, higher baseline or on some new hardware that we may or may not have yet. Whether or not we wanna turn on some optional optimizations, things like LTO has been a very big topic over the last few years and I'm happy to say that we are nearing the middle of that process. So that's good. Some other things that are fairly common in RHEL but have traditionally not happened in Fedora is in RHEL, we try to minimize the number of packages that we have to maintain, particularly because we're maintaining it for a very, very long time. And that's a lot of work. Even for a relatively minor package, you have to worry about what happens when a CVE comes in, what happens when a major bug fix comes in. And for every package, we have to have somebody who's responsible for it. So naturally in RHEL, our goal is to try and find what is that minimum subset of packages that absolutely must be in the distribution in order for RHEL to be able to deliver on its promises of stability and its promises of being a platform for running stably whatever application you want to put on it. So one of the things that we've done a lot over the years is we've trimmed dependencies out where they may exist in Fedora, but we have, for example, many packages oftentimes have experimental or testing options that are enabled in Fedora because it's often usually built as a sub-package or otherwise made available in Fedora because that's what Fedora is. Fedora is exciting. Fedora is new. Fedora is features and we want our friends to have those features, but we have different needs in an enterprise distribution. So one of the things that Yelen allows us to do is to work further back in the timeline to trim out some of those things that probably aren't going to make it into RHEL, like an experimental feature that we know isn't ready for prime time or maybe we want to eliminate, not have to maintain a whole lot of build time dependencies. So maybe we allow the Yelen version of the package to not have to run its packaging time tests and instead we put those in a test suite that is allowed to pull packages from PyPy or Maven or any of the other packaging services instead of us having to maintain in the RHEL distribution the actual, the package for customers to use for arbitrary purposes for a decade. So in the past in RHEL eight, RHEL seven, RHEL six that tended to be the sort of thing that we started working on after that six to 12 month period of bootstrapping. Once we had a system that was working we then spent another six months generally pulling as many packages out of that as we could and sometimes more successfully than others and sometimes we would still end up with just cruft in the distro that we were now stuck with maintaining because we just couldn't find a way to stop that one package that was using it from pulling it in. With Yelen, we now have the opportunity and our packages especially our Red Hat people have that option now to make those changes add those conditionals or change those build flags six, 12, 18, 24 months before it's actually before we have to be cut down and this is it where at beta everything after this is a bug fix gives them that opportunity and in turn that means significantly less stress and more time spent on doing the things that engineers find interesting and customers love to see which is building features, fixing bugs, adding functionality and generally just making a better product for the user. So we do that and we'll be building that along in the way and Yelen has its own composes. We build every night we attempt to compose they've been broken on and off for the last month for a variety of different reasons generally related to dependencies turning up or disappearing that or changing names because we just branched off REL9 as everyone knows from Fedora 34 and now it's that churny period of Fedora's history where people tend to throw in all of the really big changes tend to go in right after that because they've been holding them back. So we're trying to get that back up and running but in general the goal is to have a preview of Yelen compose every day once a day. So I mentioned before a little bit about how we broke the inheritance in Fedora 34 and then sent to a stream then took over. We wanted to, at this point, all changes are now Yale only. At this point, this is really the point where we're finalizing the bootstrap we're trying to get to what we internally refer to as an alpha, we don't really have a public alpha release but that's the point at which that we used to describe as, okay, we've got a bootable installer. Now, nowadays we're doing that pretty much in sent to a stream, in this case, sent to a stream nine and it seems to, again, seems to be going really well and we've gotten quite a few non-Red Hat contributions which is absolutely delightful and exactly what we wanted to see. We learned a few things in this real nine process, Yale and to send to a stream nine process. First of all, when we initially designed the process we were going to follow, we had plans and we executed on these plans to branch from Fedora 34 at GA freeze, at final freeze. And the reason for that was because we wanted, we needed to have a hard date. We knew that the rail planning needed to know exactly when we could make that branch because there's a lot of moving parts that have to happen internally. And so what we did, so we reasoned that, okay, we wanna get it, pull in as much stuff from Fedora without having to have our packages maintaining, pulling things over manually as long as we can. So the latest fixed date we have is the GA freeze. So great, we broke at the final freeze for Fedora 34. What we discovered, however, was that this introduced a couple of, there's some confusion because there was that period of about a month and a half between the branch date when Fedora 34 became its own branch and the actual final freeze. And it turns out that month, month and a half really confused our packages because they didn't, they didn't, despite as much messaging as we tried to do, they couldn't follow that, okay, so committing it to Rawhide no longer gets it into what's going to be at EL, what's going to be rail nine. They had to, for a month, they had to commit to Fedora 34 and then all of a sudden now that closes off and they're committing to sent to a stream. So that was a bit of a struggle and that was a little confusing and people, and so what we're going to do next time around, which should be around Fedora 40 unless we skip, unless we change our numbering scheme or skip a release, we're gonna actually make that branch right at the Fedora 40 branching date instead of the final freeze date. And similarly, one of the other things that happened when we broke the inheritance at final freeze is, of course, that very moment, ELN suddenly stopped tracking rail nine and became, and started tracking rail 10. So anything that was going into Rawhide at that point, congratulations, that will be in the next version of rail as far as we know from now. And we have started already to bootstrap rail 10 and rail nine isn't even out yet in beta. That was the goal of this project and it's working and what we need from here is as much help as anyone's willing to give to help us get to that next rail 10. And if you have exciting things you wanna do there, you can already start doing that. It's not, you do it in Rawhide, Fedora gets, reaps the benefit and we automatically rebuild it with the ELN flags and compiler flags and optimizations and conditional macros. And we just try to keep that composed up and running and then somewhere down the line around Fedora 40. Rail 10 is spawned and that's how you get a rail. So I am going to move to questions and I'm on a different screen right now. All right, so I will go through the Q&A, Paul Flaherty. What is ELN and where do I get it as a test? Okay, so I hope that I've described what ELN is date and up to this point, but the compose for ELN is available at odcs.FedoraProject.org I believe and it's updated. Well, there's a compose attempted every night and the last successful one is always available under the latest sim link. So you can pull, you can pull an installer image or just install from the tree there anytime you wish. I don't see any other questions in the Q&A. Oh, sorry, can I have a link to the presentation? If you go to the schedule page on the Fedora Wiki for Nest, I put a link into the description on the schedule. It should be easy to find it's the only link on that page as the last I checked. Okay, I'm just scrolling past the back in the chat now because I don't see any, oh wait, this is going to sound silly. Is there any documentation on how to bootstrap something similar to RHEL? Well, not really, that's, again, that was the problem. One of the problems we were trying to solve with ELN was it's too, they were too separated and they were too far apart and the teams tended to change. So we didn't have a whole lot of institutional knowledge and by extension didn't have an easy way to transfer that knowledge and to store it. So at this point, what we're trying to do is we're not going to be just attempting to describe how to start an entirely new operating, an entirely new operating system or an entirely new distribution. That's, well, actually that's not entirely true. We are going to be discussing is how to create a new distribution using Fedora as its base and some of that's covered in material that's available to Fedora remixes, the documentation that we have for that and some of that is still needs to be written. If you are interested in learning how to do that and in turn helping write that documentation, we would love to have you. We have an ELN special interest group. It meets every other Friday, usually. I think we canceled this week or was it, I forget if it was supposed to be this week or next week on IRC. I hope that answers your question. If it doesn't please ask, feel free to ask another. How do we get features included into ELN? So in general, if you make changes to one of the packages that is currently on the ELN content list, those are gonna go in for the next, for well-ten unless the maintainers of that package rejected outright. If you have new features, new functionality, new packages that you really wanna get into ELN and Braille, that is a little trickier and that's going to involve a conversation with Red Hat because we do, again, try to keep the things very small, but if there's a good justification for it then File of Bugzilla, Ask for Inclusion, it will build it in Fedora proper first. File of Bugzilla against Red Hat Enterprise Linux and Ask for Included to be included. On top of that, we are also looking at, and this is gonna touch a little bit on Troy's question about what is ELN extras. We're going to be looking at building a, something like Apple that tracks ELN throughout its life cycle and eventually when ELN forks into a stream have that fork into the next version of Apple as well. I'm actually going to invite, if Troy wants to talk a little bit more about that, he's definitely the expert, so I'd invite him to join as a speaker if he wants to cover a little more. Going to admit I'm not entirely sure how I approve him doing that, or if I can. If Marie is still here, maybe she can help. She is not still here, is she? Okay, well, in that case, I hope that my current explanation serves and if not, please feel free to ask for additional questions or Troy, if you want to just type in the chat, you're completely wrong, that's also valid. Just scrolling back through the chat real quickly to see if there were any questions that didn't make it to the Q and A. Rachel, you always think of Rel as female. I actually tend to agree with you on this, but the Chris Farley joke didn't work, so otherwise it would have gone with La Nina, but then also the EL wouldn't have worked and then it's just all false part. Thank you for the comments about the memes. I was hoping that would go over well. Oh, and David Cavalca, and I apologize if I'm mispronouncing that, has mentioned that they also mirror ELN at mirror.facebook.net slash fedora ELN. And, oh, got one more question. Is there any way to find you guys in IRC? Yes, we are on libera.chat. We were formerly on FreeNode in the pound fedora-ELN channel. You can also find us on the regular fedora-devel channel as I think that is a proper super set of people that hang out in fedora-ELN. So please feel free to join us and we'd love to have you. Oh, I almost forgot. If you also, if you wanna find us by email, we use the regular fedora-devel list. We normally just add to the subject line brackets ELN to make sure that it calls attention to those of us working on ELN. One more question. Is there a matrix room for fedora-ELN? I think, well, I mean, we are, I believe we only just have libera-chat bouncing through matrix. So, ah, here we are. Welcome, Roy. Sorry about that. Three minutes, if you would be so kind. ELN extras for those packages that are not in the red hat set. So red hat says, we want all these packages in the next row, but you want to track it for some other reason. Currently, the only thing right in there right now is the KDE set, which I put in. The reason we are actually not planning on moving these over to ELN and Apple next time, but it is so that KDE wants to make sure that the packages will build properly in the rail setting in Apple. So we are gonna track it. So as in three years, as we make changes to the KDE packages, we know that they will build on rail without any of this. If fedora is greater than 40 problems. When we did rally, that was the biggest thing. If fedora is greater than 30, do this. And it's supposed to do rail. Anyway, the ELN extras, not gonna be part of rail, but isn't automatically moved into Apple next, but it can be. If you want to know about Apple next, see my talk tomorrow. Yes, absolutely. Please, anyone who is interested in Apple next or ELN next, Troy is giving a talk tomorrow at 2.30. Or it depends what time zone you're in. I'm sorry, Eastern, 11, okay. So yes, 2.30 Eastern time, which is the con standard time according to the wiki. But thank you very much. And with that, I believe we have come up to the end of our time. So if you have any further questions, I will be hanging around the events. I'll probably hang around in the social channel for a while. Feel free to come up and ask. Thank you very much, everyone.