 that will come back to the conference. Shreya and Karol, that's good. Good morning, everyone. So it's 10.30. We can get started with the first talk of the day. Let's welcome Brian. So he's going to talk about CentOS Stream today. Thank you. Yeah, thanks. So yeah, my name is Brian Stinson. I work on CentOS Stream full time for my day job at Red Hat. And notice, if you're listening on the recording, there's a number of folks on my team here in the audience. And I told them that they were going to help me give this talk. So but we'll walk through just a couple of things today. We'll talk about kind of the idea behind CentOS Stream, what's it for, what do different people get out of it. But I'm going to spend a big chunk of time just talking about what's available and what you can use those individual artifacts for. Because I get a lot of questions out on mailing lists and IRC and all of that stuff about I'm used to consuming either Fedora or CentOS Linux or stuff like that. What's different about Stream? And we'll talk about maybe some similarities and maybe some strategic differences that we're doing in the project and with Stream itself to kind of highlight what you should be doing. So let's start off with what is Stream really for? Like what are we going to do with it? And if you ask me to put my Red Hat on, we serve a large community within Red Hat itself because Red Hat uses CentOS Stream to make rel. If you look out, I'm sure a number of you may have seen the repositories in gitlab.com. But that's really the place where Red Hat maintainers take code from Fedora, from upstream communities, from other places or patches that they develop themselves. And they use that to actually make it into the Red Hat Enterprise Linux operating system. And so how do they do that? Because it's kind of that balance you got to create between stuff that you show in the public and then all of the things that you have to do in the private. If you've seen someone give a talk about CentOS Stream before, you've probably seen this diagram. And this is how, yeah, Adam is here in the audience. It's actually a really, really good encapsulation of how we actually make this happen at Red Hat. So we start over here on the left. You notice you start with a pull request to gitlab.com. And there's a number of things that are involved with that. Typically, we ask folks to file a bugzilla first, so bugzilla.redhat.com, which we'll talk about here in a minute. You want to add a new feature. You go out, you file a bug, and then it's fixed via a pull request to the actual code itself. And a Red Hat maintainer, a rel maintainer, actually controls that process. They affect the merge and then get the code into different places. But you can kind of see there's two parallel pipelines here. One of them, the top one is out in the public infrastructure that you all can see if you go out to the various properties that we're going to talk about here in just a minute. And then there's a private infrastructure that is actually how we generate the rel bits. And you can see there are a number of stages. We sync and coordinate build activity. We also sync and coordinate what we call gating. And that's a way that rel maintainers can take internal tests and tests that they have. Maybe they pulled some unit tests from upstream or other scenarios. And they put it into this. Some folks are automated testing scenarios. Some folks have manual testing scenarios. But the gating actually controls the status of a package as it makes it through this entire pipeline. And so one of the questions we get quite a bit is what sort of testing is run? How do I know that this is something that I want to put on my machine? And the answer is it's actually controlled by Red Hat gating and a couple of other tests that are run even earlier here in the process. And so a package makes it all the way through. Let's say that the tests go well. We have a good build and all of that stuff. It branches off at the end there. And you all can consume it in what we call a compose. And so a compose you can think of is just a snapshot in time of what CentOS Stream would look like with all the collections of packages that exist at one point or another. And as we move forward, as new packages pass gating, as new builds come in, that compose moves just forward all the time. We don't do, you might be familiar in the past with minor releases or something like that that we've done in the CentOS project. That's not a thing that we do anymore. Stream always moves forward. So what do we use CentOS Stream for? And by we, I mean those of us who have public access to all of the bits. Running interesting workloads, we were actually just talking about a few folks that are using Stream to preview what's coming in the next minor release of REL. Because if you remember, since we're doing all of this stuff in parallel, everything that makes it into Stream is nominally destined for a future REL minor release. It's for finding and filing bugs early. And so if you're familiar with maybe CentOS Linux in the past or maybe even some of the downstream rebuilds that have popped up after the CentOS Linux retirement, typically you find yourself in a situation where you start on a minor release and you run your workload on it and you end up finding a bug. The problem with that is if you're pinned on that one minor version and you open a bug, it may be six months, a year, or something like that before you actually have rebuilt artifacts that you can try things out on. And there's no way for you to influence that process because you're literally taking what RedHouse has released and kind of going from there. There's no way to affect the process in development. And so the easiest way you can think about CentOS Stream, if you want to boil it down to one bullet point, is this starting conversations with package maintainers. And there's a lot of interesting work that you can do and interesting things that you can do via merger quests or something like that if you want to contribute your own patch. But really, it's all about starting a conversation, whether you want to affect a new feature or whether you want to fix a bug or whether you just want to tell us that something isn't quite what you expect. Opening a bugzilla on bugzilla.redhat.com in the public is one of the best ways that you can actually affect change through CentOS Stream by running your interesting workloads on it. So go nuts. The only way you can find bugs is to actually run your services. And so I want to give you an example of how this works. If you were here for the CentOS Dojo a couple of days ago, we were actually looking at an interesting feature in Network Manager. So this is a feature request. It's adding a multi-path TCP to Network Manager. If you're interested in that sort of thing, it's probably something that you want to follow on pretty closely so that you can make sure that your configs work and that maybe we didn't update anything in the configuration files that you wouldn't expect. But you can see, if you can read this entry in bugzilla, the feature was actually logged a while ago. Let's see, that was December of 2021. And there was some back and forth on this ticket to kind of give you an idea of what typically happens is there's some back and forth on the ticket about whether the team wants to do this now or if they want to wait for a new release of Network Manager or maybe they have this planned in other parts. But to give you a little bit of an inside look, it was actually a couple of weeks ago when the team decided that this was ready to actually look at, ready to start the development on. And so they issued a public merge request. So this is an update to the Network Manager package. You can see that if you look on the breadcrumbs here, you can see that this is a merge request under gitlab.com slash red hat. We've got the CentOS stream namespace. That's where all of the package configs go. But this merge request was tied to this bugzilla. And it's an update to 1.39.90, which has this new multi-path TCP case in it. And so yeah, a couple of weeks ago, they had this, they decided to accept this work. This merge request was filed. I'm pretty sure it was on, that would have been probably Wednesday. They affected a build in the CentOS stream build system that's available publicly. And you can see that the version numbers do match up here. 1.39.90. So they affected a build about a day later after they issued the merge request. They got everything merged, affected a build. And then actually it was Friday I was checking on this. It turns out that it had made it all the way through this process. It had made it through real gating, made it to the right place in the build system, and passed verification. And so it was ready for compose. And so I haven't, today is Saturday, I haven't actually run a production compose for the day yet, but you can expect that that package, the new network manager, is going to make it into the repositories probably sometime next week or so. And so that's really tight. You can think of beforehand you would have to wait six months from when someone actually started the work to actually having a build that you can consume. So here it's on the order of weeks. And every package, every update is a little bit different. Teams plan a little bit differently, but I think this is important that you can see change and try things out relatively quickly. And so that sort of brings up the question. You can see this is destined for CentOS Stream 9. But where does Stream itself fit in the whole wide range of the ecosystem that involves Fedora and Red Hat Enterprise Linux as well? And I'm going to throw up a marketing slide because we do this in every Stream presentation as well. But you've probably seen this on a website there, on the Red Hat website. And you can see that CentOS Stream sits here sort of in the middle. And I mentioned this earlier because the CentOS Stream's purpose is really to affect the next minor release of Red Hat Enterprise Linux. So we typically developed the next major version of Red Hat Enterprise Linux in Fedora. And there's some interesting activities. If you're interested in these activities, you can talk to Adam and Troy after the talk here. But there's some interesting activities that happen during the bootstrap period of a new major release. And that's probably if you've looked at the activity so far in Stream 9, up until a few months ago, you were seeing a lot of those major release activities. Starting from now on, you're going to see a little bit less activity. That's OK. It's really features and bug requests that are destined for the next minor release. And so how do you consume it? Like, what's available to you? And what artifacts are out there so that you can run your workloads and file bugs and do all the interesting things? We still publish everything to the Santos Mirror Network. We actually made a new Mirror Network for Stream 9. But mirror.stream.centos.org is a good place to start. We have the nine stream directory in there, plus all the special interest group content stored in that one place. And you can go and download your installer and all of that stuff. That's all located on the mirrors. We've recently consolidated all of this content, including things like the source RPMs and the debug infos. I know some of you who may be familiar with the Santos Project have found those in other places, typically on the vault or the debug info service. With Stream 9, everything is delivered directly to the Mirror Network. So what could you use the Santos Stream Mirror Network for? You should install it on your laptop. I've got Stream 9 on this laptop right here. I actually use it as my daily driver at home. And it helps me, number one, use the thing that we're producing. But number two, it lets me run things like I spend most of my day in Google Chrome, to be honest. But that picks up enough bugs for me to file one or two every now and then. And it makes the whole experience better. So that's how you get, if you're installing Stream on your laptop, I would recommend you go to the mirrors. But what about in your CI systems or maybe some of the, maybe you want to snapshot individual packages or see what's the latest? So typically, we only push to the mirrors once a week. But the composes that we produce, we have a couple of different types. We generate the production composes, which you see right here, a little bit more than weekly. But those are the ones that you'll see out on the Mirror Network. We also have a number of development composes. So if you look back in our diagram, anything that hadn't really past gating yet but was still sitting in the build system waiting for people to test and verify, that's the stuff that we put in the development composes. And those are generated nightly. You can find that on composes.stream.centos.org. And that's really, honestly, a better way to track the very latest and greatest coming out of Stream. The Mirror Network is good for laptops. And it's automatically enabled whenever you have your repositories on the system. But the composes location is great for finding the latest. So put it in your CI systems. And again, it tracks the very latest out of the build system. The container images are another thing that is really important for when you want to consume this in your CI system. If you have a container workload, we do publish tags for both of those cases. Stream9 corresponds to the production composes. Stream9-development is the latest and greatest that hasn't necessarily passed any sort of gating yet. And those are all stored on quated.io. So how many of you actually maintain a set of container files on GitHub somewhere? Do you have Docker files out there? I know I've got a few of them just for other purposes. So I looked on github.com. And there were about 60,000 references to from.centos in the container file type. And so that shows me that there's a lot of folks who have built up a lot of infrastructure and a lot of work around centos containers. And so switching to Stream9 for your container workloads is going to be pretty awesome to do. I want to mention one thing. So this is once you have the bits, you can get involved in one of our special interest groups. A couple of them actually published their documentation to sigs.centos.org. And each one will have different procedures for getting involved and contributing. But the sigs are for building content on top of Centos Stream. So you should join a special interest group. The sigs actually use our community build system. And you'll see in this listing right here, you can see, I'll read this off to you, looks like the hyperscale sig was in the middle of building system D at the time. They end up replacing a number of core components in Stream, one of them being system D. And you can always go out to the community build system to get interesting packages or to find what the latest is from our special interest groups. And I mentioned this before, one of the main purposes is to open a conversation. So if you find trouble either using sig deliverables or stream deliverables, go out to bugzilla.redhat.com and open a bug, request new features. Finally, if you're familiar with the, if you maintain packages in Fedora, you've probably heard of Fedora Copers. I will note that the Centos Stream build routes for both Centos Stream 8 and Stream 9 are available to you in that system. So that's a good way to try out new things, even if you haven't joined one of our special interest groups yet. Go out there and build your packages against Stream 8 and Stream 9. So that was a whirlwind tour of a few things that are available to you. Are there any questions? Yeah, that's a good question. So the question was, are we publishing messages for the Centos Stream Composers? We do publish some messages to the Fedora messaging bus when the Composers are created, but not necessarily when they're pushed to the mirrors. That's something we have yet to do. So yeah, that's right. That's right. Yeah, Trey. Yeah, that's a good question. So the question is, how do I open a bugzilla and how do I route it to the right place? The bugzilla entry that you go under bugzilla.redhat.com, you'll choose the major Red Hat Enterprise Linux version. So in this case, we've got a bug entry up here for REL9. And there's a version dropdown. And you can choose, if you're running REL itself, you can choose one of the minor releases. If you find a bug on 90, 91, whatever, that's part of that too. If you're running Centos Stream, that version is available to you in the bugzilla dropdown. Yeah, Adam. Yes. So the question is, what window managers are available to you? So by default, we publish the GNOME environment, but I know you can look over at Trey and he's doing some work on KDE. And the folks in Apple are looking at other environments as well. XFC is ready already. Awesome. Mate is almost there. Yeah, thanks, Carl. Thank you. Oh, yeah. So the string apps really fast. And if I use it as my daily laptop, like you, how often should I update my system? Yeah, so that's a good question. So I typically just run a DNF update every week or so. It's up to you and your workload about what you, sort of how often you want to consume that. I think I will mention one thing. So security fixes and bug fixes all come in the same stream for Centos Stream, as too many uses of the word stream there. But security fixes and bug fixes all show up whenever they come in. So it's important to make sure that you have a strategy. I don't have a hard recommendation for you, but that's something you should keep in mind for your environment. Thank you. Yeah. Mine is more comments answering others. Yeah, yeah, do it. There's KDE, XFCE, Mate, i3, Lumina, Iced WM. Really? For those are the desktops for desktop managers, which is like GDM, there's SDDM, which has its own issues, and Lite DM. Yeah. For the other, as I was going to say, it's a test that Fermilab and CERN did, and they found that they do once a month updates, and they found that they get the same number of updates whether they're running real well versus the streams. Yep. They're ahead, yes, but as you said, they've all gone through the gating and stuff. They don't make it to that thing unless they've gone through the gating, and they were happy with it once a month. Yep. But I like your answer better. It's up to you what you want for your chat. Yep. Yeah, thanks, Troy. I'm kind of excited that i3 is ready. I need to try that out as soon as I get off this stage here, so with that, I'll hand it over, and thank you all for chatting with me today. Thanks, Brian, for the lovely talk. I think we will have a 15-minute break. The next session will start at 11.15. Thank you for the patience, everyone. I think we are good to go for the second talk of the day. Let's welcome Carl for his presentation on the road to EPEL 9. How are y'all doing today? Show of hands here. Who has used Apple 9 already? Good video. What about Apple 8? Even more? What about Apple 7? Some consistency there. What about 6? Yeah, some long termers. 5? 4? Okay, a few of y'all have been with us since the beginning. Excellent. So who here has delayed upgrading their operating system, their Apple operating system that they're using it on until a package they needed was in the next version of Apple? Pretty much everybody. That sets the stage for what we're gonna talk about today. The road to Apple 9. My name's Carl George. I'm a principal software engineer at Red Hat. I'm Carl W. George on pretty much all the social medias and IRC and Matrix and all those things. And if you wanna email me, it's just Carl at redhat.com. Somehow I was able to get that alias. So based on the show of hands, pretty much everyone here knows what Apple is, but just in case anyone's new or less familiar, Apple stands for extra packages for enterprise Linux. Quick sidebar, enterprise Linux is a shorthand term we use to refer to Red Hat enterprise Linux or RHEL and RHEL compatible distributions. Apple is a yum repo of community maintained packages for use on enterprise Linux. And it's part of the Fedora project. So you might be wondering, where do Apple packages come from? Note, this chart is not to scale. There's more than 90 packages in Fedora. But you can see here where it's kind of divided and RHEL and CentOS are created from a subset of Fedora packages. Everything else in Fedora is eligible to be put into the Apple repository. Not automatically, but it's eligible. That part right there is where the extra in the name and the acronym comes from. It's only additional packages for the base operating system, nothing that overlaps with it. So anything that's part of that subset can't go into Apple. If anything gets added to that subset later, it's gotta come out of Apple. So I mentioned RHEL and CentOS are derived from that subset of Fedora packages. The way that happens has changed in recent years. So you can see the first line up here. These are old logos and kind of represents the old model. RHEL would be derived from Fedora. And then later, the CentOS project would take the published RHEL sources and build another distribution with the goal of being as close to RHEL as possible. We call that a clone distribution, typically. It's not identical, never has been, but the goal was always to be as close as possible. There are a few problems with clone distributions. First off, you can't really fix anything. You can't do much with it. It's gotta be exactly what it's trying to clone. If you get a new bug report, then all the maintainers of the clone can do is, okay, I can reproduce this in the parent distribution. Closed works as expected. That's not really a good user experience and it doesn't really build a good community. You also can't accept any contributions. It's just kind of an as this thing. And even if you have a good idea for something that you know how to fix or a feature you wanna add, anything that you do that, anytime you do that, you'd be deviating from the parent distribution, which is against the mission. So that doesn't really work really well long-term for sustainability. We have a new model now. You see our new fancy logos. Now CentOS is derived from Fedora. RHEL is derived from CentOS. And then we still have other clone distributions like AMALINX there that come along and still have the same mission that CentOS used to have of just reproducing what RHEL has. Now CentOS, since we're no longer, I say we, I used to be on the CentOS team inside Red Hat. Now I'm working on Apple directly. Now CentOS, since we don't have to worry about matching RHEL, can work on actually getting better and improving the operating system. Even though it changes some, it's still very similar to RHEL. It's still bound by RHEL compatibility rules for a major version. And so, and what's in CentOS basically becomes the next RHEL minor version, you know, a few months later, three to six months, something like that. That said, RHEL isn't completely static. And in some circumstances, you'll have changes that are gonna happen in RHEL that happen in CentOS first and they're a little bit different. And in some cases, those changes when they happen in CentOS first affect Apple packages and whether or not they can install and work. In 2020, I proposed a solution for this called Apple Next. We knew that Apple mostly worked with CentOS and we wanted it to keep working, but we were running into small instances where some packages wouldn't install because of one of those library changes. So the idea we had was that, have another repo, Apple itself builds against RHEL, Apple Next would build against CentOS and then it would be optional and maintainers would only use it whenever they'd run into one of those, honestly edge cases that would happen about 1% of the time. We worked on this, got everything stood up for the infrastructure and basically right away had a really good use case for it. RHEL was updating QT from 512 to 515 in RHEL 8.5. I think that landed in November of 2020 and or in 2021, sorry. And we were able to get those Apple updates ready in Apple Next because the same change landed in CentOS a few months earlier that it gave the maintainers opportunity to get those updates right and working not just deliver the fix for CentOS users, but also made sure that they knew exactly what they needed to do a few months later when they did the same update in Apple. So it wasn't just helping CentOS users, it also helped preparation for RHEL users for those packages. So we talked about this earlier during our show of hands and we know that availability of packages in Apple affects what when people upgrade. We get consistent feedback about this, not just from community members, but also from RHEL customers. Eventually that feedback got loud enough that Red Hat decided to take action to support Apple. And I say support, I meant not to say that word, that is a very loaded word that means a lot of different things. So scratch that, I didn't say support. In 2021, Red Hat agreed to give additional headcount to the community platform engineering team or CPE for short in order to work on and sustain Apple and make things happen. The ulterior motive here was really just, we know that people, we want customers to upgrade to the latest version of RHEL and whenever it comes out, we want Apple to be in a place where they feel they can use it. We're tired of hearing customers say, I can't upgrade because something, this community thing isn't ready yet. CPE's whole mission is providing engineering resources for Fedora and CentOS, so Apple naturally kind of fits in with that. Historically, it's the same people working on it. People in CPE would typically be the same ones making Apple happen in the past, but on a volunteer basis. Now with it officially staffed, we can do it as part of our day jobs, we can make it a priority and we can plan for it ahead of time rather than just reacting when things are on fire. The first thing we had on the agenda was Apple nine, of course, and being ready for RHEL nine. So we came up with the plan. It was a little bit hectic, but it made sense to us from an implementation perspective. We already had Apple eight next or Apple next for RHEL eight or for CentOS eight stream eight. We decided that our original plan was we would launch Apple nine next first, built against CentOS stream nine, and leave that as the only Apple nine repo for a little while. And then after the RHEL nine launch, we would do a mass rebuild, all the packages that have been added to Apple nine next, rebuild them against RHEL nine, and then use that to quickly populate the Apple nine repo. That would let us launch Apple nine quickly with the good number of packages very soon after the RHEL nine launch, but not ready at the RHEL nine launch. We were kind of expecting a few days to maybe a week or two. And that was what we came up with, and that was the plan we were sticking with for a little while. It made sense to us from the implementation side, each repo would just get set up once and just left alone and not changed. It would be set up as soon as it was possible to set it up the way it was supposed to be set up long-term, but it made a lot less sense for packages and for users. So as we started talking about this, explaining it to not just the community, but also to internal people inside Red Hat, we realized that it was really confusing. For package maintainers, we were explaining to them that we wanted them to build for Apple nine next only for about six months. And then there would be the mass rebuild that they wouldn't have to do anything for, but then after that happened and after Apple nine launched, they would target Apple nine primarily and only build an Apple nine next when it was needed for library changes. It was also confusing for users. We were telling them, okay, for your CentOS boxes, use Apple nine next at first for about six months only. And then after Apple nine comes out, use Apple nine and Apple nine next. And if you're using rel, then maybe you can use, get some Apple nine next packages installed at the launch if you just can't wait, but don't leave it enabled because eventually you'll see 9.1 compatible packages when you're still on 9.0. So don't leave that enabled and make sure to switch to Apple nine only once that exists. So just talking about it here, like y'all's eyes are glazing over and it's annoying. It's like, yes, this is a bad plan. This is not how we want to explain this. And we're also, there's really dawn on this as we were trying to document it. Like, how do we explain this to people? Not just verbally, but in documentation where you don't have that verbal connotation and feedback where you can help explain and dive into parts of it. On top of all of that, there was also going to be the complexity and added work of doing a mass rebuild. This is something that Fedora Release Engineering does a lot in Fedora, but we hadn't really done it at this scale in Apple before. So it was kind of an unknown thing that, okay, we're pretty sure this will work. There'll probably be a few curve balls. Some of our scripts aren't gonna work right for the mass rebuild work, but we think we can do it. But no matter what, it was gonna be more work and more complexity. The biggest problem of all though, was that Apple nine wouldn't be ready on day one of rail nine. There was a lot of people that wanted to see that, both internally and externally. And all of that combined, we decided, all right, let's look at this plan and see what we could do to do something better. We came up with a new plan. Let's just set up Apple nine. That's what people want. We knew that if we set it up early, building against CentOS stream nine, that the packages almost guaranteed would work on rail nine launch day. So we went ahead and set it up that way. Apple nine next was also set up to build against CentOS stream nine. And we launched them at the same, and the idea was we'd launched them at the same time, but not really focused on Apple nine next. Just have it ready. And then it'll come into play later once it actually matters. Once 9.1 changes start landing in CentOS stream nine. And then of course, after the rail nine launched, once we had rail nine content, we would switch Apple nine to build against rail nine. This was a little bit more complicated for the implementation side, not too bad, but at least we weren't doing a mass rebuild. So I already hit that mass rebuild part. This was much more simple to explain to packages, just target Apple nine later on, you'll, you might run into a case where your package doesn't install on CentOS anymore. So in that case, target Apple nine next, which is the same instructions we've already given them for Apple eight next. It was also going to be a simpler message for users. Just use Apple nine everywhere. Even on CentOS, they don't even have to think about it really because Apple release has a recommends to pull in Apple next release whenever you install it on CentOS. So it was still just super straightforward and what they were used to doing. And the same way it works on CentOS stream eight. The biggest thing was that this led us launch before rail instead of after rail and gave us Apple on day one of the rail nine launch. But first we launched Apple nine. It was about five and a half, six months before rail nine. And this is the first time that's ever happened. And it was really only possible because of the changes in CentOS. We wouldn't have been able to do it otherwise. Right away, we started getting questions like, so is Apple nine really ready or how could you call it ready if this package I care about isn't in there? Like what are you thinking? And it gave us a good opportunity to remind people Apple isn't a specific content set. It's a repository. It's our repository for all of us to add stuff to it. Community maintain packages that we want to use on enterprise Linux distros. It's up to individual package maintainers to add their packages into Apple. A lot of times those packages are primarily focused on Fedora. They don't mind building a compatible build in Apple but they usually won't do it proactively. They'll wait for someone to file a bug and ask for it. They have no problem doing it but they just want to make sure that there's demand for it and that that software is still appropriate to be shipped. That's done by filing bugs. If you saw Brian's talk just before this one, he talked about how CentOS now we're trying to start those conversations and have engagement not just having as is use it and you get what you get community. So, we have a guide now. We actually had it before we launched REL9. Troy over there wrote most of it. And it is kind of a choose your own adventure guide of how you start that bug, the specific buttons in Bugzilla you need to click for people that are less familiar with it and also ways about how to, if you're just a user, ways you can phrase it, templates you can use. If you're already a package maintainer, how you can ask. I'd like you to add this but if you don't want to do it, I'm a maintainer, you can add me and I'll do it. Things like that, really the big thing we wanted to focus on there was that the way CentOS was changing was we wanted to have those conversations and not just be a usage thing. And we wanted to mirror that in Apple 9 or Apple in general and start having a better onboarding process to start those conversations. That worked out pretty well. Fast forward about six months, REL9 launched and Apple 9 being out early allowed the community to populate that repo and have a lot of packages, community maintain packages ready to use on day one of REL9. We got up to about, I checked it that day, had to go back and look at the tweet I did about it to find the exact number because I didn't write it down anywhere else. But we were over 5,700 packages from over 2,600 source packages. That made, obviously that made a lot of community members and a lot of REL customers very happy. It went from, okay, well, I don't really consider the next REL major version until 9.1, 9.2, that's when all the Apple packages I need are available to, hey, I could actually start using this now earlier. And ideally those customers started looking at it even sooner with CentOS Stream 9 and knew what was gonna be in 9.0 and we're already even more ready in filing Apple requests beforehand. So how is Apple 9 doing now? I stole this chart from mine and Troy's state of Apple talk. It's a little bit shrunk down so it's harder to see but you can see the basic trend lines and that's what's important. The first line here is Apple 7 and then Apple 8 and Apple 9 and you can see how the trajectory changes. Also, of note, Apple 7 started about at REL7 launch. Apple 8 actually didn't start till a little bit after the REL8 launch and Apple 9 got to start before the REL9 launch. These dotted lines here show about how many packages were in Apple 9 at the time of the REL launch and if you go back previously, it took Apple 8 about a year to get to the same number of packages. It took Apple 7 about a year and a half to get to the same number of packages. So just tremendous growth and what we're learning is launching early is better. Give people time to get their work done. Oh, and what we're up to now, just checked it this morning. We're up over 7,400 packages built from 3,300, over 3,300 source packages. So fantastic growth there. So you might be wondering, what about Apple 10? Our plan right now is that we're gonna do that the same way we did Apple 9. We'll build it against CentOS initially and then once REL10 is available, we'll build it against that. We're thinking we might be able to have the infrastructure set up a little bit sooner and let early adopters start poking at it and doing builds, but we probably won't publicize it or do an official announcement until about the same time, about six months before REL10. Before that point, there's a lot of churn in CentOS. As it's getting developed off from the Fedora release, there's a lot of things changing. One big example in 9 was we were working to get OpenSSL 3 in there. If we had have launched Apple 9 before that change, pretty much every package builds that links against OpenSSL 3 would have to be rebuilt. So we wanna avoid launching it before large changes like that land. So it's a balancing act. We wanna get it available early, but not too early. So we have a survey going on right now. If you use Apple or contribute to Apple or think you might wanna do either of those things, take our survey. We want your feedback. We wanna hear what we're doing good, what we're doing bad and how you're using Apple. We're gonna use that information to shape how we do things in the future. Those links go to the same spot. One's just easier to remember, tinyurl.com slash Apple survey 2022. The QR code takes you to the same spot if you're scanning it. No rush to do it. It's open through the end of August. We wanted to have it open for this conference and also Fedora's flock two weeks ago. And I went through these slides way faster than I practiced. So, but there's my contact info again. And does anyone have any questions? Are you tracking numbers of like how much churn there is that requires the use of Apple Next from one minor release to another for stuff like that? On going, no. We've done some spot checks on it. This is a little more sensitive. We've done some spot checks on it. And I mentioned the 1% number. I've checked it two or three times. And the library SO names between REL and CentOS are very, very similar. Like library SO name bumps, the things that would require a rebuild in Apple Next don't happen very often, but they can happen. And now we've got a place where we can, as that happens differently, we can get that build done and have it compatible. But it's still really an edge case. It doesn't happen very often. Yep. Yes, QT is the big hairy exception. Wasn't really a question I was just saying. The QT, they're trying to, it's not really the SO name. It's that one of the things is marked private anyway. They're trying to make it so that doesn't happen as often. See what happened with us? Yeah, I won't go into too much details, but they're working on it so that doesn't happen. Excellent. Any other questions? We got one back here. So how long does it take for a new package into Apple? So that depends. It can be pretty quick. If that package is already in Fedora and the maintainer's willing, you can file the bug, they can request a branch, release engineer will create the branch, maintainer will create the build, they'll create a Bode update, and they could publish it. Theoretically, all of this could happen on the same day. So it could happen really quickly. It'll show up in the testing repository first, and then that's in Bode. There's a karma system where you can upvote it. And if enough people upvote it, it'll get shipped out to the stable repository. Like I said, theoretically, this could all happen same day. That's the fastest it could happen. If there's no karma and no feedback on it, it stays in the testing repo for a week. So I would say within a month, probably a realistic expectation a lot of times, just to give people the maintainer time to do the build, things like that. If it doesn't exist in Fedora yet, that's a lot harder. It needs to be submitted to Fedora. Apple, we're not really interested in having packages different than what Fedora has. There's a whole mentality inside Red Hat called upstream first. And so if you wanna add a new package, the right thing to do is get it in Fedora first. Even if you don't even use Fedora, you should get it in that collection of packages, have it maintained with the help from the community, and then maintain the one you care about in Apple, but make sure the Fedora one keeps building, you do the new updates there. There's a package review process there. If you're brand new, you need to get sponsored by a package into the Packager Group. And so yeah, it can take a little bit longer on that side, but we have a lot of helpful people in the community, myself included that we're sponsors and we like packaging, that's what we care about. And so we like getting more people involved and spreading that workload out. Just at the last conference I went to, I was able to, I met someone new in the Alma community. We got to talking about packaging, he got to talking about things he was frustrated with, and I was like, okay, well, why don't you file this bug? Why don't you request that? And now fast forward, he's gotten hooked on it. He, I sponsored him in the Packager Group. He's adding packages left and right, building them for Apple nine, eight, and even seven. He's, his infrastructure spread out between several of those releases. And so that's what we want to see. We want to see more people get involved because Apple is, it's our repository, just like Fedora is our distribution. It's not just for something that somebody gives to you. Thanks. Sure. I don't, I don't mean to comment on all those, I mean, Carl often do code talks. I just wanted to come at one thing so you don't think it's automatically gonna be there a month. There's several packages, several, over a hundred, where the Fedora maintainer has no desire to do it and they just sit there. But that is in our process. We have had, you know, you send an email to Apple Develle and majority of those people have stepped up and grabbed it, but sometimes there is a delay if the maintainer doesn't care. Yes. A month isn't the upper limit. That's, that's my guesstimate, probably the average. Yeah. I, I've definitely done one or two with one day, but that's the rare exception. No, David had a question about you too. I was wondering if there were any, if there was any interest or activity going on around doing some sort of an artificial intelligence review of the use, you know, like any tracking associated with the repositories themselves. Haven't looked into that, but I'd like to pick your brain about more, what you mean by that? We'll talk to Gabriel here. All right. Anybody else? Okay. Sure. Keep them coming. Yeah, so, this may sound like a trick question, but what are some of the, like what are some of the packages that people end up requesting a lot really early? Like, did you notice any of them at the very beginning of Apple nine that were really, really popular? I'm trying to remember mine and Troy's slide from the dojo a few days ago. We had a bunch of the logos on there. People don't really have time to ask for KDE because Troy's on that, like in his sleep before it even can happen. So KDE's out quickly, but I know that's popular. XFCE gets asked about a lot. Mate and Cinnamon get asked about a lot. OpenVPN was a real popular one. I know a lot of headers depend on that to get connected to the work VPN and that's not shipped in real. And so, really, yes, it's in the CSB image, but it's not in real proper. So, screens and other one. That's one of the first ones that shows up. Yeah. Yeah, screen is one of those examples where Red Hat kind of decided, we don't need to maintain two solutions, Tmux and Screen. We're gonna focus our engineering resources on one. So, screen got retired out of RHEL and that means, hey, now it's eligible to be in Apple when it wasn't before. Any others? Well, if not, then that's all folks. Oh, one more, one more comment or that's not all folks. Is there a process for taking packages out of Apple? Like, do packages eventually get retired out? Yes. So, I mentioned how if a package gets added in like a real minor release, it has to come out of Apple Biopolicy. It's extra packages only. We don't wanna have, part of the reason Apple is so popular and so trusted is because it doesn't have problems that some third-party repos have where they add something and it's doing a new version that you didn't expect or you added the repo for package A, but then package B also got a new major version and that changed expectations for what else you wanted to do. So, we try really hard to stick with the extra packages only. That's one example where something would be retired. A less common example, but does happen is that the maintainer just says, I don't really wanna maintain this in Apple anymore. It's an all-volunteer effort, so we can't force anyone to maintain it. Or maybe the software has security vulnerabilities that if it were a real package, the real maintainers doing it full-time, being paid to do it, would develop a backported fix for it for that version, but the upstream only publishes the fix for the latest major version that's incompatible. In that case, the maintainer could say, I don't know how to backport this or maybe it's impossible to backport it, so let's just retire it. Some people aren't using vulnerable software. In Fedora, you can retire a package only in Rawhide and Branch, I believe. Once it's in a stable Fedora release, you have to maintain it for the lifecycle of that Fedora release, but that's only 13 months about, obviously Apple, the lifecycle matches Rawhide 10 years, so we're not gonna say you have to maintain this for 10 years, that's not reasonable. So we don't want people retiring packages early, but we're not gonna force them and it is allowed to retire a package out of Apple for security reasons or if you just don't wanna maintain it anymore. Ideally in that case, someone would post about it to the mailing list or talk about it in an IRC or Matrix and find somebody else that'd be willing to co-maintain it with them. So rather than retiring it and then somebody else asking to bring it back and then they maintaining it, just skip the retirement and transfer the, I don't wanna say ownership, we don't own packages, we maintain them, but transfer like the admin rights so that somebody new can maintain the package. I have a great example of that where I was tightly maintained, like I have a tightly maintained group and so in order for that group of packages to get built because there were several different maintainers, we just basically created administrative associated rights on all of them for everybody. Anybody else? What else have you wondered about Apple? Well, if that's it, then that's all folks. Like I said, my contact info, reach out if you're interested in becoming a packager for Fedora or Apple or both. I can help you get started. Thank you.