 Welcome to our talk. CentOS and Alma makes the dream work. My name is Carl George, and I just now realized that we talked about Jack doing the initial intro and then introducing me, so I already messed it up. I guess we're going to flip it around and I'm going to introduce Jack. I'm Carl, I work on the CPE team at Red Hat, primarily working on CentOS and CentOS Stream, and this is Jack Aboudabal from AlmaLinux. Hey, everybody. Talk about yourself a little bit more too. Yeah, I'm the community manager at AlmaLinux, so you can find me running around like a chicken with its head cut off, most days. And yeah, we're glad to be here. We're thanks also to Fedora for letting us participate, and we've sponsored as well. Not only the conference, but hopefully the community, and we'd like to keep the good energy flowing. Excellent. Where are my slides on advancing? There we go. So, first I'm going to talk about CentOS, which will be my area of specialty. Traditionally for those that... What was the chat saying? Okay, people can hear Jack. All right, so for CentOS, the traditional CentOS distribution, what it used to be was a, what we call a rebuild distribution. We took the sources from Red Hat Enterprise Linux and recompiled them to create what we would call a bug-for-bug compatible distribution. It was kind of like a de-branded white label rail. We are in the process of changing what CentOS is. The old model where the old rebuild model, we've since been calling that CentOS Linux as a distribution. The new model that we're trying to pioneer is called CentOS Stream. That is, in effect, what we're trying to do is move CentOS from just downstream of rail in the development process to just upstream of rail. Fedora is still upstream of rail as well, but a good way to think of it is that Fedora is upstream for new rail major versions, and CentOS will be upstream of rail for new minor versions. Currently, the content in CentOS Stream more or less reflects what's in rail 8.5, which isn't released publicly yet. The current public rail is based on rail 8.4, and in the old models, the classic CentOS Linux still exists, and that is at 8.4 tracking rail as a downstream as well. That's gonna be end of life at the end of the year so we can focus 100% on the new model. Not everyone's a fan of that. It's not the best execution, but that's what's happening with the distribution. Since we're moving it, that's gonna open up opportunity, and it's gonna leave a space in the ecosystem for people that still want the old model. The new model has a lot of benefits that I'll get into later, but that new spot that's being opened up in the ecosystem is where it opens an opportunity for new rebuilds like Amalinix, which I believe Jack wants to tell you about. Yeah, hey, everybody. So if you haven't heard about Amalinix, we're the only 100% community owned and governed, let's call it alternative to the CentOS classic model or what Carl referred to as CentOS Linux. And we're actually quite thankful that things played out the way they did because it gives us a great opportunity to fill in a spot in the ecosystem and do some things that really weren't possible before. And we're also very excited about the ability to finally be able to collaborate, to be able to send things upstream to effect rail, to take care of issues that were brought up or that issues that would have been brought up in the past but weren't necessarily able to be handled because of the model, the way things were set up. So that's us. And again, thanks to Fedora for putting on the conference and thanks for Fedora for having us and thanks for everyone that's been working with us to make sure that we can fill the needs of the community. So the first area of collaboration that we wanna talk about in teamwork is Apple. Obviously we're at a Fedora conference and some people may be wondering, well, what does CentOS and Alma have to do with Fedora? Well, first and foremost, Apple. Apple is a sub-project of Fedora. It stands for Extra Packages for Enterprise Linux. And the real quick summary of that is that every package that isn't in rail itself is eligible to be added to this extra package of repositories, every Fedora package, I should say. These Fedora packages get branched and rebuilt to be compatible with rail and then delivered in this additional repository. It is extremely popular. I remember in Matthew's slides, he normally shows the mirror hits and the DNF count me statistics that show that Apple is highly consumed actually even more than Fedora itself. But the important thing to remember is that Apple is still part of the Fedora project. Apple packages are Fedora packages. So everyone that wants to collaborate and work on extra packages for our entire ecosystem, we all need to become Fedora packages. And then the packages that we care about either submit them ourselves or volunteer to co-maintain the existing ones and maintain Apple branches for them to make them available in rail-based distributions. The big thing to remember is that Apple is a neutral playing field. It doesn't favor any particular rebuild. It doesn't even favor CentOS. Apple is built against rail exactly. That comes into play later. There's a few very, very small amount of incompatibilities since CentOS is now gonna be tracking the next minor version content. Every once in a while, less than 1% of the packages in Apple will need to be rebuilt to be compatible with the next version of rail. We're working on a thing now called Apple Next to allow packages a place to do that to make things work seamlessly. If you wanted to learn more about that, come to mine and Mohan's talk on Saturday about what's next for Apple Next. But that is not what this presentation's about. So I will continue going on. The big, the other thing is that this, that Apple benefits the entire ecosystem. If all engineers or all the community members add a package to Apple, then that benefits rail users, it benefits Rocky users, it benefits everyone. Yep, rising tide lifts all boats. It is a collaboration area. Did you wanna add anything about Apple, Jack? No, basically hit it spot on. I think we can move on to the next slide. Perfect. So rail contributions. Traditionally, rebuild distributions like the old CentOS Linux model and the new ones that are coming about like Alma, they have a mission of being bug for bug compatible with rail. The idea is that if something does, if something is a bug in rail, it should also be a bug in the derivative rebuild distribution so that it's maximum compatibility. That sounds great in theory until you're hitting a bug that you don't wanna hit and you want it fixed. Traditionally in the old model, there was no contribution path for rail. You could file a bug. Ideally, if you also had a rail subscription, you could reproduce the bug on rail and demonstrate that, maybe even open a support case if you paid for it yourself or if your company was paying for rail. But if you just had a bug that you could only reproduce in CentOS and didn't have any way to try and reproduce it on rail, a lot of times your bug report might just get ignored by the rail maintainers. The rail maintainers definitely didn't look at bugs in bugs.centos.org. One of the big changes with the new model is that CentOS StreamBugs are filed in Bugzilla under the rail product. That means that rail maintainers do pay attention to them now. And that was the very first start of how we're making it the contribution path with CentOS Stream 9, we're actually gonna have the package sources be eligible to have pull requests to them or merge requests in GitLab. So that way, not only can you file bugs that the rail maintainers pay attention to you, you can actually demonstrate the fix and send it in yourself. That is a big opportunity that quite frankly, the old CentOS Linux never had that opportunity to do. These new rebuilds are in a very advantageous place because if they find a bug, they can verify that that bug also exists in rail, then they can verify that that bug also exists in CentOS Stream. If it's already fixed in Stream, then they know, okay, this is coming in the next minor version of rail and we just have to wait to rebuild that source. If it's also reproducible in CentOS Stream, then those downstream distributions can contribute that fix and work to get it into rail, which then in turn gets it into their distribution. And that's what's really most exciting to us because it really is it's a direct path to be able to affect rail and to take care of any issues that come up. And as a matter of fact, we already are contributors on that path because we found some issues with Anaconda, which a community member found and then we've submitted that upstream so that now that can get fixed in Stream, in rail. And it's a trickle-down benefit for everybody. Absolutely. And not just for Alma, for every other derivative distribution of rail as well. Yeah, absolutely. And I think that's part of how working in the community model benefits everyone is that not fracturing things and keeping everything centralized in one place and working within the existing systems and with the existing paths is what's gonna let us derive maximum benefit for everybody. I think there's something like eight or something like that rebuilds now, which is kind of crazy. But everyone will benefit from that and the community will benefit from that work too because ultimately it ends up in the community's hands. Absolutely. The next area of collaboration we wanna talk about is special interest groups. A lot of people are familiar with these in Fedora land also. CentOS has their own special interest groups. We're trying to get more of them started now with the ability for some contributions to actually get into rail itself. Our SIGs are growing. One of the real popular new ones is the Hyperscale SIG. We're also looking, there's a K-Mod SIG that's getting started up to provide additional kernel modules. Other rebuild distributions, some of them are considering doing their own special interest groups. That's actually not the best way to collaborate if they're just building something just for their distribution. What we'd like to see with CentOS special interest groups is that all the rebuilds can participate there. We actually got some plans in the works. Nothing final and no promises, but we're hoping that we can make it where CentOS SIGs can rebuild against not just CentOS Stream, but also rail in the same way that Apple builds against rail. That means that a CentOS SIG would be able to create packages that are maximum compatibility with both rail, rail rebuilds, and then the alternate repo for CentOS Stream for any upcoming changes. Yeah, and I think one important point to make here, Carl, is that the model for CentOS is changing, right? The project itself, contrary to popular belief, is not dead. It's just the model which changed. And I think the SIGs are very active. There's a lot going on there. There are a lot of great people involved there. And one of the things that we want to do on AMA Linux aside is work within that current framework so that all that SIG work is still done in CentOS, right? I don't think it does us any good, or really does anyone any good to have our own separate SIGs. I mean, we may have SIGs for different things within the distro, which are maybe not relevant to upstream, but things like this, I think that we do need to work within the current framework and what currently exists in order to make sure that that stuff benefits everybody and that everybody's getting a good look at it. And like Carl said, I actually sent an email out earlier today and we have a pager issue open about creating rel build routes in the CentOS build system so that this way we can actually go ahead and build against rel and then release those as part of CentOS or send the stuff back upstream to CentOS stream. And then again, it'll be available to everybody. And I think that's really the most efficient model that we could probably utilize at this point. And I think that it's an important point to make that there's still a lot of great people involved in the ecosystem. And let's keep it going. I don't think there's any need to create anything of our own really that doesn't benefit everyone else. Right. Having every rebuild have their own special SIGs would just be a lot of times duplication of effort. And then other times it would be, well, this rebuild has this SIG and this rebuild has this SIG and maybe not all of them. No one rebuild has everything that somebody's looking for. In the same way that Apple is a common collaboration area for extra packages, CentOS stream is the common collaboration area to contribute into the core distribution. We'd like CentOS SIGs to be the common contribution area to create extra packages that don't make sense that aren't eligible for Apple but that work across the ecosystem all in one place. Anything else you want to add on SIGs, Jack? No, I think we covered it. And I think, again, your comments were kind of spot on call. Well, that basically wraps up the actual presentation. We can open it up for questions now. I'm guessing there are a lot of questions. If you guys want to ask questions, just send them in the QA thing and then we can, or Q&A thing and then we can pull them out of there. And then we are also having an Alma Linux open office hours right after this on Hoppin as well. So I'll be there. We may have other team members there. If anyone has any questions, please feel free, come in. Let's meet. All right, I'm taking a look at the Q&A now. Miro asked a question. I have trouble understanding what do the SIGs actually do in CentOS? Build content in Apple or build extra content in CentOS that is not in rail? What's the relationship between rail, CentOS, SIGs and Apple? So CentOS SIGs have different rules than what Apple has. Apple is strictly extra packages and also it doesn't have, kernel modules are not eligible for Apple. That's in the Apple guidelines. So CentOS SIGs can provide alternate kernels, alternate kernel modules. They can provide packages that are at, that replace the packages in the base distribution rather than just being extra packages. For example, the Hyperscale SIG, whereas you couldn't add a newer version of, say, SystemD into Apple because that breaks the Apple rules. It's already in the base distribution. The Hyperscale SIG could ship the latest version of SystemD and I believe they do. And users that opt into that repository get that package upgraded. So it's different rules and different content sets. Some of those packages, like I know the, I believe the Cloud SIG, it works with the RDO project and they may upgrade some of the Python packages and libraries that are in the base distribution in order to support the latest version of OpenStack. That's another example of where SIGs are allowed to override packages from the core distribution if necessary. I think that mostly answers the question as far as Apple versus SIGs. Definitely if there's, if somebody goes to a SIG wanting to add an extra package that's not in the distribution, the standard practice is to redirect those people to just add it to Apple instead, as long as it meets all the qualifications and criteria over there. Cool, so- There you go. There's that question. I'm gonna go in a different order because there was an anonymous one right before Miro and it looks like they're stacking at the top. Is Apple to be as trusted as Fedora OS? Is it safe to use without worrying too much about malicious packages? You can trust Apple just as much as you trust Fedora. Now, anyone can become a Fedora Packager, they can follow the rules and then their intentions could change to become malicious. There's no guarantees against that. But like I said, Apple is on the same playing field as Fedora in that sense. Yeah, and I think it has many eyes on it as well. So the likelihood of a supply chain attack there while it exists, I would like to hope is pretty minimal due to the amount of people that are contributing to Apple and reviewing Apple and using Apple. Absolutely, and since Fedora Proven Packagers, since Apple packages are Fedora packages, if something malicious was discovered, any Proven Packager in Fedora could go and undo it and fix it if necessary. So it's on the same level of trust as Fedora itself, I would say. Let's work back. Jack, do you wanna take the Alma question? Yeah, so does Alma Linux work on UEFI hardware and does Alma Linux and RHEL support and video graphics cards? Well, we definitely work on UEFI hardware. The question about the graphic cards, that's a little bit more nuanced, but the general rule is if RHEL supports it, we support it. So I guess I'll leave it at that. I see a question from Ladar. Familiar name, hey Ladar, how you doing? Should I use Alma or Rocky? Why? Okay, so I figured someone would ask this question. Keep it neutral. Okay, so without getting it to flame wars, I think there are a few distinctions between the projects that people should know about and people can make their own decision. Certainly, we're definitely not against Rocky by any stretch or anything like that. I think everyone is doing good work. One main difference between us, I think is the ownership model. So we're a 501C6 nonprofit and we actually just finalized membership guidelines. So basically what that's gonna allow us to do. So number one, the original problem with Sentos was always that it was never independently, Sentos the distribution was never independently owned and governed. What we tried to do was set up this nonprofit so that all the IP everything is owned by the foundation, it is a nonprofit. The foundation has membership guidelines, which basically means community members can join in and they have a vote and they have a say, sponsors can join in, they have votes, they can have a say. And we're really just trying to balance that between all the stakeholders. I think that's really, really important because that was always the problem with the original Sentos and now hopefully, according to the way this was set up, there is no one point of ownership. It's owned by the community. All of it is owned by the community. It's governed by the community. The community votes, the community can propose things. The community can accept changes, they can reject changes. So not all of that stuff is in place yet but it's getting there, right? We're working towards that, the foundation was established. On the rocky end, they decided to go with a for-profit company and it's actually a PBC. And sorry, my kids are wandering downstairs. Wow, hello. I figured that might happen, sorry about that. So what they decided to do was set up a different structure where it's set up more in the way of a traditional corporation. Now that corporation is set up for the public good but again, it's all owned by one entity and there's one single shareholder there. So I mean, I think that's kind of an issue. Things will play out over time and we'll see. It's not anything knocking them but personally as someone who grew up in the open source world and subscribed to open source ideals, I would like to think that we would pick a more open ownership and governance model. So anyway, that's kind of my take on it. And really, which one should you use? Again, that depends on you. I think if you wanna subscribe to that philosophy, you're more than welcome to use us. If you feel like you'd rather use Rocky than go ahead and use Rocky. I think one thing that we do have going for us is that the project started from a group of people that had a tremendous amount of experience doing this. So we've actually been rebuilding RHEL for 10 years. So, we're experienced, we're seasoned, we know what we're doing. We have a lot of knowledge of the ins and outs of what's going on and the testament to that is just how quickly we were able to get releases out and how quickly we've been able to get security updates and Arata out. You know, I think it just speaks for itself. So, I don't- Say something for your office hours, yeah. Yeah. Anyway, I just wanna give people a comprehensive answer so that I feel like, you know, I don't want anyone to feel like we're trying to walk around or anything. Right, I think it'll be great to go into more depth on the governance models and stuff though in the next session. I know people are very interested in that. I'd like to hear more about it as well. Sure. Let's go ahead and get to the rest of these questions for this session. Where did I leave them off? I know that I already answered a question in the text, but from Miro asked, the different content sets from SIGs are delivered as optional repos. Yes, every SIG has a corresponding CentOS Release SIG name package in the Extras repo. It's not installed by default, it's completely opt-in. A big reason for that is what I mentioned earlier that packages in those SIGs are allowed to override packages in the base distribution. So once you enable a SIG repository, it's not going to be 100% compatible with RHEL anymore. That's an important thing to be aware of. It's not necessarily a bad thing that may be exactly what you need to fix a particular problem or to run a particular stack of software you need, whether that's OpenStack or CIF or some of the virtualization things, but they're completely opt-in, they're not default. You wanna take the Alma Plus, Jack? Yeah, will there be an Alma Plus repo? That's a good question. Since we're running short on time, let's take that into the open office hours and we could talk about that there. I think it might not be necessary based on how successful the Kmod SIG is, because a lot of the Kmod packages they were intending to ship were things that the CentOS Plus kernel provided for him. Plus SIG had, yep, exactly. So if that works out correctly, then maybe it won't be necessary, but yet to be determined. Let's see. So I understand how CentOS Stream benefits both REL and Devs. It does replace Fedora's and, it does replace Fedora's and upstream for REL though. First part of that question, not exactly. Fedora is still the upstream for new major versions of REL. REL 9 is gonna be derived from Fedora 34 and we're doing that work, previously that work took place in private whenever REL would branch off from Fedora. Now that work is gonna be taking place in the public as CentOS Stream. So right now CentOS Stream 9, it's still in its early days, but that is Fedora 34 and Fedora itself was the upstream for the new major version. Now we're working on 9.0 of REL in CentOS or CentOS Stream 9. Same part of the question. So how do you convince Fedora packages to maintain Apple back? Apple branches benefiting CentOS Stream. It's the entire ecosystem benefit. A lot of the packages in Apple, they might not make sense for Fedora itself because they're already in Fedora. Whenever, as far as Fedora packages that already maintain Apple packages, nothing changes for them from the REL perspective. If you're in a shop that's using a mix of Fedora and REL or if you're using Fedora workstations with REL and production servers, then it still is the same benefit to maintain your Fedora packages in Apple so that they're available and compatible for your REL machines. Nothing changes there. The Apple Next that I'm gonna talk about more on Saturday, that is just a very small add-on repo for Apple that whenever an incompatibility is found, which right now we're tracking less than 1% of Apple packages even needed. But an example right now is the QT stack. QT is getting rebased from 5.12 to 5.15 in the next REL minor version in 8.5. At least that's the plan right now. And so that means that Apple packages that linked against QT 5.12 need to be rebuilt to be installable. That's taking place in Apple 8 Next right now. And so a lot of people are gonna still use Apple on CentOS Stream, which is totally valid, but because of those slight changes that are coming down the pipeline, Apple Next gives those Fedora packages a way to create compatible packages when necessary. I've been commenting on there. It doesn't really replace Fedora so much as it makes the jump smaller. I would say REL has two upstreams now instead of just one. Yeah, I liked your explanation of major version versus minor version, Carl. I think that's really the most accurate way to look at it. Because previously, before the CentOS Stream model, if REL, I think we're out of time, but before the CentOS Stream model, if REL Red Hat has always practiced an upstream first mentality, if you wanted to get something into REL, the answer was get it upstream first. That meant getting it into Fedora and having to wait three to five or six years to get it in the next major version of REL. That's just unacceptable for a lot of people. It's just way too long of a timeframe to ask for. With CentOS now being just upstream of REL, we can push people to contribute there and it gets into the next REL minor version without having to wait for the next REL major version. All right, so I think there was one question about this Epo work on AMA Linux. Yeah, absolutely, it works very, very well. I think it's time we wrap up in here. Let's head over to the other session just because there may be people waiting in there. So thanks everyone for coming. I see the Kubernetes question. Real quick, I'll answer that one. I don't have any idea what Kubernetes is targeting. That's all up to them, doesn't. I'd like to see them target REL8 and CentOS stream mate, but we're seeing some upstream projects start doing that, but I don't know if Kubernetes is doing that yet or not. So I think that wraps up all the questions and we can go over to your office hours, Jack. I'll see you over there. Yep, all right, we'll see you guys there. Thanks everyone for coming. And if you need to reach us, you can grab us on IRC. Carl is Carl W. George and I'm the mayor. See you guys later, y'all.