 Hello, and welcome to my talk on the new RHEL pipeline. My name is Stephen Gallagher. I'm a Principal Software Engineer at Red Hat. I work on Red Hat Enterprise Linux. I'm also a member of the Fedora Engineering Steering Committee, and you've probably seen my name on most of the mega threads on Fedora Devel. So let's talk a little bit about what are the problems that we're trying to solve here. The big one is this. Red Hat and Fedora and the open source community have always said that we should default to open. We should make sure that everything we do is, in the public eye, can be viewed carefully and inspected for bugs and for security issues. But then we haven't really been living that when developing RHEL. At a certain point in its development, we fork from the Fedora project. And then we do all the development behind closed doors with some interaction with privileged users or partners, software and hardware. But in general, we don't really involve the community in the actual building of RHEL. On the other side, part of this is because Fedora is just too popular. I know, it seems like a good problem to have. But what it means is that we actually have to be cautious in Fedora about making radical changes. Big, huge changes in Fedora affect tens of thousands, if not hundreds of thousands of people. As you certainly saw recently in the last couple of releases, we landed Modularity, which was a major feature of Red Hat Enterprise Linux 8. To say that that didn't go smoothly would be a little bit of an understatement. But the good news is that we have finally gotten that into a place where it's not as disruptive as it had been. And it was kind of a mirror of other projects like that that we've done in the past. Certainly, the switch from upstart to system D was hardly a smooth transition and certainly didn't involve any kind of arguments or flame wars. We could mention GNOME 3.0, but I don't think we'd like to talk about GNOME 3.0 anymore. But some of the things that we really want to do, that we know are going to be necessary for Red Hat Enterprise Linux, are really hard to do in Fedora because it would impact negatively a large subset of the users. For example, a while back we tried to consider the possibility of changing our processor optimizations so that when we built we were required a processor that was more recent. Maybe something built in the last five years would be an absolute minimum instead of something built in the last decade, which is our current state. And what we got for feedback from that was, of course, too many people are using it. Too many people are relying on that stuff. Similarly, we have a conflict with what we call dependency minimization. There are places in the server world and in containers where we really, really want to have an absolutely minimal set of dependencies on packages. And that's not just at runtime, but also at build time. But in Fedora, we have this general policy that we've always gone with of always carrying everything that is necessary to self-host and to build the entire OS and to build every individual package. And we don't always want to carry that into REL. For example, we don't always want to run all of the upstream tests in REL because that may require us to carry a series of test suites that have an enormous dependency chain in that, which we do sometimes, but we don't want to be shipping that because we can't support that. It's unrealistic to support every package that has ever been used in that chain. Similarly, we also oftentimes in REL, we will not build the documentation directly from source. We will build it on Fedora, we'll tar it up, and we'll carry it in REL SRPM because things like Python Sphinx, for example, pull in thousands, literally thousands of packages. And most of those exist solely for the purpose of supporting the documentation generators. Well, that's not something we need to be maintaining and patching constantly. So we want to be able to remove those things. But it's really hard to do in Fedora proper and to make sure that it doesn't get broken today because we can put conditionals in if REL then do that, but if we're not actually building it constantly in that format, we know that we haven't introduced some breakage there. So why don't we just do all this stuff in Rawhide? Rawhide has its uses. Rawhide is definitely the next really cool stuff. It's everything that we want to land in the open source community. It's the best of the best. All new features must go into Rawhide, absolutely. But it's not really the right place to do this because, as I said, you can make sure that you don't break things if you're not always building them in the right in the same way. So what we're going to do is we recently instituted a new program called ELN, or Enterprise Linux Next. So what is ELN? Essentially it is an early preview of the next major release of REL. So right now ELN is going to be developing into what will ultimately become Red Hat Enterprise Linux 9. It's a subset of packages that we rebuild out of Rawhide in a build route that is designed to be more like REL than Fedora. So macros that say if REL will trigger, we may play around with those processor optimizations. When we generate composes of ELN, they will be laid out the same way that a REL release tree would be. So if you're used to using the REL repositories for your businesses and such, it'll look the same. But it'll be just rebuilt Rawhide packages with no other changes. So how does this work? Well, first and foremost, we need to have a package list. Because we don't want to rebuild the entirety of Rawhide in a REL build route because that's unmaintainable. There are probably tens of thousands of packages, binary packages in Fedora that we just are never likely to carry in Red Hat Enterprise Linux. So we want to get a view of what packages we do care about. And of course that also means getting a list of all the things that they require at runtime and that they require to build. So we created this project called Content Resolver. And you can find it at tiny.distro.builders as a web URL. And what it does is it is contributed to by all of the people working inside of Red Hat on various subsystems. Networking, kernel, virtualization, all of the various different teams that work at Red Hat have provided input content into this Content Resolver. And then it churns through, it calculates all the dependencies and it provides output that we are going to be using to provide an automatic list that we will then rebuild from Rawhide every time. To make sure that we have only the content we need so that we don't waste too many resources on Koji. And so that we are not maintaining a separate branch in DistGit. This was a major point of contention when we originally proposed this. A lot of people, a lot of maintainers dislike the idea of conditionalizing their code for REL versus Fedora. And what they wanted was that instead of rebuilding from Rawhide, just create an ELN branch in DistGit and build from that and try to write code that keeps it in sync with Rawhide or merging with Rawhide constantly. That turned out to be a bad idea for a couple of different reasons. Not the least of which that merges very quickly get complicated and manually merging doesn't make a whole lot of sense. And also we want it to be the default that everything that is on that Content Resolved list must end up in ELN at some point. Now, for those packages that don't want to have conditionals, that's fine. What they can do is they can continue to build a Justice Fedora and it will get into ELN Justice Fedora, just as it was packaged in Fedora. And that isn't an ideal situation, but it still gets us a lot closer to what we want REL to look like. And it still allows us to get a feel for what content and what feature set is going to be available to us. Down the road, we'll be able to break that and move into a separated version where we can have individual packages make REL specific changes that are independent from Fedora. But that won't happen in ELN. Where that will happen, I am proud to announce, is in CentOS Stream. Starting with Red Hat Enterprise Linux 9 and ELN, we are going to be building a new form of CentOS Stream. Today, we have a CentOS Stream that allows the public to contribute to the next Y stream release. That would be x.y, so REL 8.3, 8.4, 8.5. Those contributions go through CentOS Stream today. Starting with REL 9, the next evolution of ELN will be a public diskit repository for development towards the beta of REL. All REL features, not including things that must be kept secret for proprietary license agreements or CVEs, all of these things will go through and be committed to CentOS Stream first. For those packages who didn't want REL specific changes in their raw hide packages, they can wait for this phase and they can put those REL specific changes into the CentOS Stream. This will be constantly rebuilding and composing. Those composes will be publicly available and usable and will be working as hard as we can to get as many partners, users, software developers working with those so that when Red Hat Enterprise Linux 9 eventually launches, they'll have certifications all taken care of and an early view of what it is that they'll be able to do. Or if they can't yet do it, they'll have a place to contribute those features that they're going to want to depend on. Ultimately, the point is this all leads to Red Hat Enterprise Linux. A long-term stable platform run for the next 10 years supporting certified platforms, hardware, software, applications. And for the very first time, we are going to be doing the majority of that process publicly in the open with accepting comment and contribution from anyone. I saw a question in the chat from Michael Konekshny. Apologize if I've mangled that. Will CentOS Stream be created from Fedora? Or will it be Fedora to REL to CentOS Stream? The answer is right now we're planning for it to actually come from ELN. We're going to fork Fedora at a certain point and that will become CentOS Stream. So it really is going to be a progression towards REL rather than building necessarily in REL and down. The actual logistics of that are a little funny because REL will also be forking from Fedora and they'll be keeping in parallel for a while, but realistically it's forking from Fedora. Will Composers for Stream 9 be published in the beginning as well? Yes, they will. All the nightlies or periodic Composers that we do will be released as they come out assuming they aren't broken, but we're trying very hard. Thank you. Pingu, did you build that castle? No, my daughter did. My two daughters actually built that. Thank you. Let's see. Some offers to help, which is always great. So there was a question to the equivalent of how would this mean we have a regular cadence for REL releases? Yes, at Summit this year there was a formal announcement that Red Hat Enterprise Linux will be releasing from now on every three years a major release and approximately every six months we'll have a minor release. So this is part of how we're going to accomplish that. So I look forward to any and all help and contribution that you can provide because it will make life easier. So with this ELN thing, will that allow the community to contribute to REL feature and stabilization development? Yes and no. It will allow you to contribute to features and certainly if you contribute a bug fix, I imagine we're going to take it. There will be a point after which we will close contribution to send a stream for 9.0 and then subsequent pull requests will be headed for 9.1. At that point we'll be going through the necessary stabilization to meet enterprise certifications and whatnot and that'll happen internally. Let's see. Yes, COVID did play havoc with our plans. We were of course, you know, that's what we get for saying we're going to do this on a schedule is that the world says ha. But yeah, we are really trying to accomplish this. We should get that info on access.redhat.com. I'm not entirely sure what that means, what that link goes to. David Duncan, are there established community meetings now? Not yet. We're still trying to ramp it up. We're still in a point where we're trying to get ELN to work. We do have composers that are functional now, but for a variety of reasons, including the mass rebuild, things are still in a little bit of a shaky state. As soon as that's done, hopefully in the next week we're going to be trying to open up much wider contribution. Can you clarify what you said in the talk about an ELN-discuit branch? I got a bit confused. So there isn't one. That's the short answer. When we first pitched this process, a lot of people wanted us to make ELN its own separate branch, so instead of having to put conditionals in their code, they could just commit to the ELN branch. But the fact of the matter is that that makes it harder to maintain and it means that the people who aren't directly interested in Enterprise Linux will just ignore it. And that wasn't something we wanted. We wanted to make sure that we were always building from the latest, and so we needed to find our way to work with Rawhide. And so we opted not to create a separate branch. We're building everything from the Rawhide commit. And in those cases where we need something to be different for REL and the maintainer is not willing to do that, we're just going to defer those changes until CentOS stream opens. Which is not ideal, but it is a good compromise for everyone, I think. Let's see. ELN is built from Rawhide branch. Okay, yes. Carl, thank you. What is the scope of the ELN SIG in terms of deliverables? Realistically, our scope is, okay, that is actually a trickier question to answer than expected. At its core, our goal is to make sure that we can always start CentOS stream basically at any moment. So at any time we could arbitrarily just branch off and say, oh, REL N plus one is now getting started over here in CentOS stream. So it's meant to be a continuously bootstrapped OS rather than every couple of years doing a big REL bootstrap to start everything over again. Let's see. Was this influenced by how OCP allowed folks to test nightly builds before release? That is actually the first I've heard of that. No, but I'm glad that they're doing this too. Do the ELN have REL or Fedora branding? It's Fedora. It is Fedora branded. It is under the Fedora umbrella. It will be Fedora. It'll just be Fedora with an Enterprise Linux flavor to it. And yes, as Matthew Miller says, we want to make sure that we keep Red Hat engineering investment in Fedora. Because the more that Fedora has moved towards being a very desktop focused and very successful desktop focused Linux distribution, the more there has been question internally as to whether or not it's particularly useful as a place to develop REL. And so by building ELN and having a very clear spot for that effort to go helps us make sure that we keep the people who are being paid to work on REL to do their work in Fedora so that we get that community contribution and we get that community feedback. All right. So if Dusty says, so if I see an ELN build in Koji, it's not built from an ELN branch. It's built against a different builder with different macros. Yes, exactly right. It's always coming from the RAHIDE branch. It will be coming from the exact same commit that the most recent successful RAHIDE commit built from. And yep, use the Fedora brand. Fedora Enterprise Linux next, yes. There's a story behind the acronym ELN and I'll tell it to anyone who wants to hear privately, but talk about it later. How to join the SIG? Look for information next week, I hope. I'm going to put out a wider call and try to start setting up a regular, probably bi-weekly meeting. Enterprise Liberation Front. I feel like there's a Monty Python skit coming on, but I will restrain myself. Yes, there also may have been some eyes raised over Butterafast, but that really came later. Okay, did earlier REL branch from Fedora Stable or RAHIDE? Is that different now with branching from RAHIDE, since no separate branch? Well, actually, CentOS Stream is probably going to branch from Fedora 34. That's pretty obvious now. But ELN will stay on RAHIDE, but we will also be running some tests against the Fedora 34 branch. Traditionally, REL has always branched from the stable release of a Fedora. Usually we've done an alpha on one release, then rebase to the GA of the next release to start stabilization for beta. But I don't think it's going to have a huge impact because the truth of the matter is that very little happens in RAHIDE during the stabilization branch of a stable Fedora. Some people get early features in, and we'll be monitoring for that. But realistically, I don't think it's going to change much. And let's see. Okay, it's a great server too. Thanks, I'm glad to hear that. How different are the ELN and Fedora build systems? Will they converge over time? Well, they're identical because we are using Koji. The difference between them is entirely in the build root contents. If there's an ELN equivalent build, the builder will prefer that build for the dependencies. It has some different macros set so that you can do if-REL, and it'll build like it was REL. But for the most part, they're pretty identical, and they are using the same exact hardware. We just build with a lower priority so we don't clobber anybody's important build with our rebuild. People were asking if ButterFS in 33 meant REL8 would get it soon. I am not answering that. Not even a little. REL10 has to come from somewhere, so ELN will go on and on. Yes, thank you, Steve. Let's see. Do you think ELN will be consumable for users, like real Fedora server users migrating to it, or more like raw hide in a number of users? ELN is absolutely and unequivocally not intended for production use. Please don't use it on your servers. It is intended entirely to be a place where we try out new functionality, where we have partners working to land hardware support, and things like that. Don't use it for anything you can't afford to lose, please. I am not going to be held responsible for that. How are you building the ELN builds with different macros since they are not defined in raw hide disk branch? Right now, I am surprised to hear that you get that question from Mohan, because you helped me set it up. The answer is currently we are setting them in the Koji build target. You can specify certain options and we are overriding the ones that you are coming from Red Hat, RPM macros, or Red Hat, RPM config, whatever it is called. However, we are actually planning in the next week or so. We are going to be changing the Red Hat RPM config package so that we will bootstrap it with the ELN target overrides. After that, it will be able to check its own build route to see that we are in ELN, so set these macros. That is a bit low level for this talk. I probably should have just said that. After CentOS Stream, we will switch to Miner. Do we expect that the build will go patch CentOS Stream REL miner? Yes, just like it does in CentOS Stream for REL 8. Could CentOS Stream be built in Fedora Koji as well? I would probably say that from a technical perspective it could, but I think that is probably not going to happen in the immediate future just because CentOS has its own build system set up and probably not a good financial reason to merge them at this point. But I don't think that it is technically impossible. Steven Smugin says we are moving the Fedora builders to ELN next month. That is okay. We can afford to lose those, right? He uses Fedora REL hide snapshot kernels to not do that. Could we branch ELN to .ELX in Fedora and distigate instead? I am not sure what you mean by that question, Neil. Sorry. I will move on to the next one. If you put a clarification, I will come back to it. What will be the cadence of CentOS Stream when it is switched to REL miner? I am not sure that is a question for me anyway. Is that a question, James? I am not closely involved with the CentOS Stream 8 folks, so I don't really know what to say there. Post ELN to REL stabilization and CentOS Stream. Oh, I mean, there are lots of things we could do. I am not sure that it makes sense to do that at this point. We are trying to work on some tools to make it easier to operate with the three different distigates, the one for Fedora, CentOS, and the internal one for REL. But I don't think there is any particular need to have... Or as of right now, I am not aware of any hard plans to merge the distigates. I know that it has been talked about, but I don't know that it is happening. Yeah, that is right. Stream does not have a cadence, it is a stream. Contributions can go into stream at any time. It is never closed or never frozen. However, it is branched from and then REL internal stabilization happens. Then when that is released, it merges back into stream and carries on. So, Neil, basically branch it there so that CentOS Stream would operate in Fedora Koji. And then Apple and CentOS would be in one place. It is worth discussing with the CentOS Stream folks, but it is not currently in the plan because it is not a blocker to doing any of the things that we have talked about today. I think I am maybe over my time, so I want to thank everyone who came and asked questions.