 Hey there and welcome to the Fedora podcast. This is episode 26 I'm Eric the IT guy Hendricks and this is the podcast to teach you about the how the Fedora community works We bring you news interviews and more on today's episode. We'll be talking to Carl George buddy mind here at Red Hat and We're going to be talking about extra packages for Red Hat Enterprise or not for Red Hat. I'm sorry. Wow I got Red Hat on the brain As you might imagine so we're going to be talking about extra packages for Enterprise Linux and yes, it does work on rel So before I say anything else stupid, let me bring on Carl and he can he and I can say stupid things together Carl so great to see you so great to connect with you really excited to have you on the show Howdy, how's it going? Not too bad. We had an extra day off yesterday, and I used it to go and get burnt to a crisp out on the lake But it was fantastic great time with my kids Did you take a boat out there or just playing on the shoreline? Yeah, boat tubing and and Lot of a lot of snacks and a lot of Sun Yeah, you look a little red. Yeah, a little bit nice Yeah, I'm I still feel like I'm burning So fun fact Carl and I actually were part of the same New hire orientation at Red Hat, so that means that that Carl and I joined Red Hat at pretty much the same time And and so I think you and I kind of had some communication beforehand But then ended up and wait are you yeah, and and you oh, okay Yeah, cool So, you know Carl and I've known each other for a few years and had the pleasure of connecting at New hire and most recently at Summit so really glad that you you volunteered to come on the show although I'm pretty sure I asked do you want to and you're like yes, I already have the episode planned so Apple is definitely a passion of yours and and a huge Asset to the enterprise Linux community. So why don't we start out with a little bit about yourself? What was your day job? What do you do with Fedora in the open source community and then? My popular demand this question has made it back into the interview. What do you do for fun? All right, well like you said my name is Carl Carl George I Have been at Red Hat same time about same time as you like you said the December to December 2019 Was whenever I was hired some some day in there, and you're probably around the same month For that I worked at a hosting company here in San Antonio where I live for a little while before that I was in the army and Yeah, I've only had three three jobs in my adult life. I tend to stay at jobs for a while but Got one while I was at that host company. I worked on I started working on Apple and Getting involved in the Apple community. That was actually my bridge into the Fedora community in general was trying to figure out how I could get involved with Apple packages contribute fixes things like that and Luckily that made me all the right contacts and eventually got me a job here working at Red Hat Not initially working on Apple initially. I got hired to work on CentOS stream And but after doing that for two years, I think Someone in my leadership chain asked me well, how would you like to start a team just working on Apple? Like primarily focused on Apple and I said, well, yeah, actually, that's that's what I wanted to do in the first place so Started as just a team of one I've grown to a team of two now me and my teammate Diego and we are part of the We're a team within a team so that you might have heard before about the community platform engineering group Inside Red Hat. I tried. I think it's called team, but I try to call it group to differentiate it from the teams within that group Me and Diego were on the Apple sub team within that wider team The rest of the team does interesting things like Fedora infrastructure and release engineering There's other recurring initiatives that come up one of the more recent ones was getting the New account system rolled out for Fedora that CentOS also uses Changes like that will oftentimes fall on the CPE teams, you know Purview to take care of and address. Oh, yeah, and you asked about myself more, huh? Yep. So let's see. So that's work. I seem to like it when they realize that you actually have a life and do thing So for so for a personal life, I like staying cool. I was telling I was telling Eric earlier that my AC actually went when not out completely but stopped working all the way last night Called a repairman this morning and luckily was able to get him out here pretty quickly. Things are cooling off now So it's definitely needed during the Texas summer. I also like you mentioned going out to the lake I like doing a water and lake and river and pool activities as well, especially during the summer days During the fall. I like to get out get out and go deer hunting Or or bird hunting sometimes the most recent hunting trip. I got to go to go to Oklahoma for goose goose and duck That was a lot of fun Let's see Just trying to show show my kids things that I'm interested in I've got both of them using fedora And they like playing Minecraft on their little used thinkpads that I got off of eBay and oh, that's perfect Oh, yeah, oh, yeah, or take them out out get them outdoors also I have a little bit of a weird dichotomy of that I like the outdoors, but I'm also a nerd So like my hunting friends don't really understand likes the stuff that I do at all And sometimes they'll make the mistake of asking like Carl. So what exactly do you do for your job? And I push back and I'm just like computer stuff You don't you don't want to worry about it every once in a while They'll push back and ask like well tell me more like what kind of computer stuff and I don't get but you know Maybe a few sentences in before their eyes start to glaze over and they're like, okay So this is why you don't like want to talk about it. I'm like, no, I love talking about it I just know that most people aren't going to comprehend it Well, especially with the nuance of depth that you have with the open-source community versus Like paid vendor type relationships. I mean you're you're in this very very Shall we say dynamic space in between the community and and companies that charge for Linux? But before we get too too far into that why don't why don't we take a step back and let's talk about what Apple or Apple? What it is obviously that's short for extra packages for enterprise Linux. So what what is it and why do we do why do we need it? So to explain Apple I usually start by explaining a little bit of how Fedora and rel red hat enterprise Linux work together Fedora is a community distribution. It's sponsored by Red Hat has a lot of red hat engineers involved in it and And but Fedora has way more packages than what would be reasonable to support in the enterprise space so the last time I looked I think Fedora was around 60 something thousand packages and rel itself is like five or six thousand packages. So it's You know, don't quote me, but it's like roughly around 10% would be a good rule of thumb for The amount of Fedora that actually makes it into CentOS string first and then becomes the next rel major version Everything that doesn't make it into rel basically the stuff that red hat doesn't want to support for its customers Is eligible to be built in the Apple project, which it's still it's Apple is still part of Fedora It's part of the Fedora project The branches for Apple are maintained alongside the Fedora branches But those packages instead of being built against the corresponding Fedora release They're built against actual rel and so they they're put in that repo and then they're available to the community members So a community member could maintain their own package in Fedora and Apple If that package just does end up getting important to red hat customers and red hat decides to add it to rel Then it would be retired from Apple. We have a very strict policy of not conflicting with rel in any way It's just it's right there in the name extra packages only. I think one of the things to differentiate there is that If if there's a package that you get from say the from the rel repository you can open up a ticket You can get support on that, but then if it's if it's a tool or a package you get out of Apple Red hat's gonna say Sorry, you might look at this resource, but beyond that we can't really do much for you, right? They are community maintained and community supported packages To whatever extent of support you you'll get from the community, right? But I mean the good news is that a lot of packages that are in rel got their start in Apple and got a lot of community support got a lot of Enterprise usage and to the point that someone said we really need this in like the official Rebos or Tories because this is really important for the for the benefit and the use of people that install rel at Let's say an enterprise Absolutely, you know, we hear a lot of feedback like about that with the whole reason my my team and my position my current position exists is because The feedback from customers got loud enough that packages in Apple even though they're not supported by red hat directly They're supported indirectly with my team with engineering resources, but not on a you could you can't open a support case for an Apple package Many customers understand that and they're okay with that, but they still need those packages available They need them to exist for the corresponding Apple repository for the version of rel they're on and so we would see that come up in feedback from customers about I Can't upgrade say from rel 7 to rel 8 until these Apple packages are available or become or get added to rel directly so they'll block major version upgrades and Customers that get all newer versions of rel itself are easier to support So we want to remove that as a roadblock for those customers as much as we can But one thing I definitely have noticed because I've been in my current role for a couple of years now and Actually, it's right at two years now I believe I think I started with the I think I started in the real business unit in the beginning of July a couple of years ago But I was I was a part of our team during the rel 9 launch and I was curious if you had any thoughts about Apple 9 and Comparing that to to older releases because it seemed like across the board Between Apple between our partners We were in a position particularly things to send to a stream that we had a lot more packages available for rel 9 on day 1 Or within a couple of days after Yeah, you just summarized it pretty well That's exactly what we did we I mentioned that being a you know being a blocker for customers that want to upgrade their major rel you know to the next major version of rel and Wanting those packages available so we got to thinking about it We had just started the team and we knew that you know Apple 9 is the first thing that we need to work on How can we do it better than we have before? I don't remember the exact timing But Apple 8 didn't really it sort of launched is like a beta thing and then it just had a real slow uptake We didn't really officially launch it and get it all the little details sorted out I think about six months in it took a long time to get it get it going so around the around the 8.1 Release I think And because of that because we didn't open it officially open it till then a lot of main Apple maintainers Didn't add their packages until then and that takes time as well So for you know a large portion of those packages, you know customers We're looking at maybe real 8.2 8.3 or even later some of them some of those packages still aren't there We still have requests come in where somebody says the package I want is in Apple 7 But it's not an Apple 8 or Apple 9 So we looked at that for nine and thought how can we do this better and what jumped out of me was just the obvious of Start it sooner open up the floodgates. Let the maintainers do do their thing We're not because our you know, we were a two-person team There was no way we were just going to build everything ourselves and even why not did that would be That would be a band a short-term Band-Aid fix. That's not a sustainable ongoing maintenance Approach to it so we wanted to get it open up as soon as possible And you mentioned Cento a stream because Cento a stream launched about six months before Cento a stream nine launched about six months before rail nine We started with that as our base to build against started building packages launched Apple And the maintain just like we were hoping the maintainers came in droves and we got lots and lots of packages built I want to say the final number was like 2300 or 2600 packages That we got built and that's that the community built not me That the community rallied around and got those stood up And so we were able to launch rail with you know thousands rail nine with thousands of community maintain packages available Right on day one, which was a huge huge improvement over what we've done in years past That's fantastic. I mean that's that's exactly what what the prediction and the the assumption was when when Cento a stream moved between fedora and rel and I mean even even from the enterprise side working on the rail team. It's it is amazing how much support we had day one and Just the sheer number of packages and I think you were sending me stats like every few days and and it was increasing not by not by Single digits, but by hundreds of packages every few days So I mean it was just an amazing opportunity to see Rail engineers and community developers all working together on this Let me see if I can run those numbers again I do know that earlier this year Apple 9 surpassed Apple 8 for number of packages available, which was really awesome to see and Well, well Carl's pulling that up. Sorry. I should have suggested that During the pre-show, but we got talking about air conditioning in Texas summers But while Carl's looking that up now is a great time to join us in chat. This is a This is an interactive forum. So if you have any questions for Carl or any questions about Apple, feel free to throw those into chat But once you're ready, go ahead and share your screen and we can or if you just want to read those out I can throw those in chat. Oh, it's not it's nothing worth looking at It's just a little script that I run that it downloads the repo data and then does some XML magic on it and gets the total number It's running right now wait for that So well, well, uh, what we're looking at the numbers for for eight and nine Might divert our conversation a little bit, but it sounds like you've got Now that you've gotten a major release or two under your belt Especially as the the sub team for Apple as part of the community platform engineering will come up with some really long obnoxious acronym for for your for your team slash sub team But in the meantime, it sounds like you've got a proposal for Apple 10 You really want to not change the direction for for Apple but really want to change what Apple does and and how it's How you interface with with Apple you want to talk about that? Absolutely, so I mentioned that before Apple 9 when we first started the team We started thinking what can we do differently for for Apple 9 to make it better? And I think we did that and Yeah, I see the link right there. You just dropped the discussion for our discussion link in there in the chat So that'll be great the So I started thinking ahead to Apple 10, which is what's next on our radar, right? rel's on a three-year major version Rough cadence now. So that means that you know rel rel 8 came out in 2019 rel 9 came out in 2022 So everyone, you know, it's no promise, but everyone should expect rel 10 in 2025 based on that three-year cadence It's the same reason rel's also on a six-month minor version cadence in case anyone wasn't aware It's why those things have been getting out more regularly So I know that we know that in 2025 rel 10 is coming and I'm guessing that in 2024 CentOS stream 10 will be available Just based on the way that the development of the projects going the way the discussions that have been happening in fedora eln Which is a whole nother sidebar that I think I can summarize that fairly quickly So I mentioned that so that CentOS is derived from fedora and then rel is derived from CentOS Fedora eln is fedora packages It doesn't have its own it doesn't have separate sources But it is the fedora rawhide sources rebuilt with all of the flags that they'll have once they become rel So different compiler flags certain macros defined differently That's really useful for packages to find out that the conditional statement they put in their spec file Doesn't account for you know rel the rel macro being higher than 9 so they need to adjust it slightly things like that So fedora eln They've been discussing about when they're gonna start doing their first composes for CentOS stream 10 and that's that's gonna start happening this year That's not going to be like the launch of CentOS stream 10 But it is the start of it that when they're gonna start doing their composes to make sure all the compose tooling works correctly It's still gonna be a sync from fedora eln I believe that the the plan is that the fedora 40 branching point Which we're not there yet 38 was the most recent release Fedora 40 is whenever it's going to the development of stream 10 is gonna branch off and Gonna start getting unique changes until then it's just gonna be Pipe build pipeline things making sure that all the builds work correctly and all the composes work correctly things like that So we know this is coming up Everything about 10 and I got to thinking about how we could do things better We also to talk about that. I also want to talk about another thing we did Called apple next. Sorry, all of these things are like interconnected and it's like everything's a side conversation So we started apple 9 early, but we'd already known in Before the apple team actually got started inside cpe We started apple next and what that is CentOS now that it leads rel by about six months When there's a library change coming to rel Then that happens in CentOS about six months early Rather than like a month later when CentOS was downstream of rel When CentOS is a month late later than real downstream Most apple maintainers if there was a library that changed they just left it broken for about a month and just said, okay Well, I'll I'll fix it once the changes in CentOS And then we'll fix the fix the packages then and then it'll be fixed for everyone and just left it broken for rel rel users Now with it being more than a month that started getting less tenable We would see not a lot, but a small handful of apple packages that wouldn't install on CentOS stream 8 Because of a library change that was incoming for the next rel 8 minor version But hadn't happened yet it wouldn't for you know, three four five six more months That wasn't really appealing at the time. I was still on the CentOS stream team inside seat still inside cpe But what I wanted was that for an apple package not working I didn't want that to be a blocker for anyone trying out CentOS CentOS stream specifically So we came up with apple next. It's an extra repo that adds on to apple And it is a build target for maintainers when they have one of those libraries that are different A good example is qt and the kate plasma qt gets rebased occasionally in rel and when that happens Every everything that links against qt the biggest one being kate plasma in apple needs to be rebuilt against the new version of qt It's qt is fairly backwards compatible, but it still needs to be rebuilt to get all the right symbol versions linked to it correctly Because of that we need we couldn't just leave it broken for six months We wanted to do something better and that's what apple next is maintainers can take their apple packages that are Working on rel and not working on centOS Rebuild them on centOS publish a slightly newer release in this apple next repo Rel users wouldn't won't look at it. They just look at the main apple repo But centOS users would look at both apple and apple next and they would get the higher release and get the compatible build We knew we couldn't just do it in place because then if we rebuilt it for the new libraries And it would start working on centOS, but stop working on rel and we didn't want that either So we have this for just those those handful of things. It's usually less than one percent of packages that are affected by this We had we had that for maintainers to target and build against So we started doing that But what we noticed from that it was a solution it worked it got people compatible packages and it it worked pretty well It solved the problem for the most part From a we were bolting this on to the side of apple type approach Um So we start for ten I started thinking about how we could do that better. We had a couple of drawbacks with the apple next approach um The first one was that it was kind of confusing for users and maintainers We we had a lot of people think that apple next was a standalone repo by itself like a replacement for apple For centOS users and it's not that you use it with apple. It's only got a handful of packages in it at any given time Um, we also have maintainers confused about when they needed to do apple next builds For the most part maintainers could just do their main apple build and it'll work on both rel and centOS But we saw a lot of maintainers doing builds for both thinking that they always had to do a build in both And that wasn't the case either So we had that confusion and then the another pain point for the maintainers was that Whenever something was rebuilt in apple next for a library change When that same library change happened in rel those maintainers had to do those builds all over again That's not that big a deal if it's like one or two packages But like in the case of kde plasma you've got hundreds of packages that need to be rebuilt in a certain order So that's double a massive amount of work and that wasn't really appealing So with that all of that context, I know that's a lot I started thinking about how we could do things better for apple tin rel itself is developed has minor versions and is developed with minor version branches And I started thinking about that and and then there's some parallels there between rel minor versions and each new version of fedora each new version of fedora Rather the rawhide branch of fedora, which is a rolling release that keeps going through time It doesn't really have a version it technically sort of has the version of the next version of fedora So right now fedora rawhide is you could say it reflects fedora 39 content Even though fedora 39 doesn't exist yet at some point fedora 39 will branch off of rawhide Get its last finishing touches put on it and then it'll be released as fedora 39 We sort of have something similar in in the rel development space CentOS stream is sort of the rawhide for rel It's not rawhide like fedora's rawhide where it's getting breaking changes for the next mate for a major version But it's got the changes that are that are due and appropriate for the next minor version of rel And then of course there's stream 8 stream 9 stream and eventually we'll have stream 10 There are major version branches So think of it like a major version rawhide. So you'll get some changes But not the big breaking ones that you'll see in a new fedora release So that that brand that development goes on like stream 9 right now It goes on through time until it's in the life date But the rel minor versions like 9.3 will be the next one It'll branch off from centOS stream 9 get its finishing touches put on it and then be released as rel 9.3 So there's some similarities there with the fedora workflow in that maintainers can do their work in Either in rawhide or in centOS stream And then stage them for the next release in fedora's case the next major version release in CentOS streams case and rel's case It's stage for the next minor version I really was envious of that type of workflow because apple for all of its history only only targeted the current minor version of rel that was published Apple next made it where we actually Even though we didn't really admit it or talk about it at the time It made it where we were targeting two minor versions of rel like 9.2 and 9.3 that wasn't released yet at the same time Um, so I thought well if we're already targeting multiple minor versions of rel Let's just lean into it because having apple and apple next is confusing and has double work If we just have minor versions of apple Maintainers can stage their work do all of their builds in the leading branch. It'll be compatible with centOS stream CentOS stream 10 and then whenever the next rel 10 minor version happens They'll have a repo ready to go with all compatible packages And it's basically going to prove what we've told people which is that The content in centOS stream is what you can expect in the next minor version of rel So we're going to put our money where our mouth is I mean, we're already doing it a little bit with apple next But we're really going to lean into it and say the package is built again CentOS stream 10 for this six months month period Are going to be what you can install on rel 10.0 on day one Which is sort of what we did with apple 9, but we only did it for the 9.0 release We did it do it ongoing for every minor version after that And that's what we're trying to do now is actually have a minor version branch for every release of rel Following through into apple apple 10 Sorry, that was a nice long winded answer that answer most of your questions around it it does and It's a it's a long thing to explain it in the in the discussion thread that you linked I actually have some some markdown charts that I think actually help a lot explaining it Um but If you want to see those I would definitely encourage people to go read that discussion thread and comment with your feedback Or even just give it, you know a reaction a thumbs up or whatever you think about it There's already been a lot of good discussion there There's there's several things we need to still sort out how it's going to work One of the one of the big ones that we had a lot of discussion around kind of a minor detail We had a few different ways that would have worked, but what we called the disc tag That is a subset of the release field Normally for apple packages. That's just e dot el a dot el nine Similar for rel rel packages use the same pattern Sometimes you'll see rel packages that have the minor version there, too You'll see like e dot el nine underscore one in some packages And it's a much longer story to talk about why some rel packages have that and some don't I won't get into that But for apple 10 what I proposed originally was just To lean into that lean into the minor version thing and just use el 10 underscore zero underscore one whatever it is for every apple minor version branch and for every package just to make it clear Which which release it was built to target Um, I think that'll be a lot more sustainable approach Um than what we've done in the past There was some discussion around that exact format. We talked about ways We could do have most packages be just dot el 10 and other ones have that but then there's some sorting problems Um, there was another solution that was proposed that had There's a special character a till day character that will do some interesting things with rpm sorting where Normally el 10 underscore like one sorts later than el 10 by itself because of the underscore one Even if the leading el 10 thing is that a newer was built against the newer minor version So we had to have some way to make that sort correctly There was a way to do it with by having the minor version ones use a till day But unfortunately the till day is not still not really widely understood by a lot of our community community members and people like that I still see questions about it whenever a fedora package takes advantage of it's it's a valid character in rpm It's just not widely known about So eventually we uh, we did a vote within the apple steering committee And we decided not to do the till day approach and just did underscore minor version for every branch Is how we're going to implement it. So little details like that are still being finalized We're talking about how we're going to structure the release packages Um, originally I thought we were going to have to have a separate release package for rel and a separate one for centos But we're actually experimenting with an approach with dna variables where We won't actually have to do that. We'll have one apple release package like we do now But we will get rid of the apple next release package, which is what centos users need to use currently So a lot of little fine details that we're still sorting out about how to implement it That's what we're discussing in those threads on fedora disc for the fedora discussions But we have lots of time to figure it out We're we're probably going to start doing some kind of proof of concept in the in the staging environment Just to make sure that all the build targets worked the way we expect them to And the branching especially is what I'm wondering if it's gonna I want to make sure that it works the way we're expecting it to but in the end we're hoping that Users don't notice anything and maintainers just have a more comprehensive Understandable approach to building their packages and staging them for the next minor version And not having to worry about multiple branches quite as much not having the confusion around the apple versus apple next branches Hmm That is a a glorious and wonderful explanation of this proposal. Uh, it I expect nothing less from No, I I I expect nothing less from you Especially when we'll have a conversation on matrix and and I get the occasional response. Can we just chat about this? It's like all right. Let me grab my popcorn. Carl's going to get on a soapbox I have a few of those But rather than getting on a soapbox my my script did finish running about counting up the apple packages. So Yeah, so there there's a difference between source packages and Like source rpms versus rpms. The rpms are what you would install The source rpms are what builds. It's like the it's the overall package One way to think of it. You may have One source rpm foo that builds foo and foo devil and foo docs and so you'd have three pack three rpms generated from one source rpm For source rpms. Apple seven has 7200 Apple eight has 4900 and apple nine has 5800 So that's from a that's from the maintainers perspective of the packages they're maintaining It's even more impressive when you look at the actual built rpms We've actually had a big influx influx of rust packages Thanks to fabio. I believe I'm saying his name right He's on the fedora packaging committee with me. He does a lot of stuff with rust and fedora. He's great He's been bringing a lot of rust packages to apple to support Build dependencies for other other packages that are trying to be brought there So on the rpm front apple sevens at 13,000 Apple eights at 9,000 and apple nine apple nine is at 16,000 So coming up on double the rpms and apple nine Then are in apple eight and it's apple nine's already surpassed apple seven on rpms Just not on the source rpms yet the input But we'll get there we'll get there before too long. I think Yeah, that's awesome And I mean it really kind of confirms what the fedora community and what the rel engineering teams all saw Uh with this move just sent to us stream it really forgive the pun streamlines the uh the build efforts. I won't forgive that fun. That was terrible It was it was um, but I'm a dad and I'm entitled to some dad jokes. So All right, all right, I guess it's forgiven when you phrase it like that So You mentioned uh maintainers and the community a lot So if if there's someone in our audience who's new to apple and new to uh contributing to software How could someone get involved with some of the packages or with apple itself? So I mentioned that apple is part of fedora and so the answer to get involved in apple is to get involved in fedora We'll have that question come up in in like our public chats or in fedora discussions Someone someone will ask about hey, I want to maintain this package in apple And our first question is are you already in the fedora packages group? And if they're not then we'll kind of not to pump them, but we'll kind of redirect them there The fedora joins sig has some excellent documentation about how to become an active maintainer in fedora And so it explains it better than we can so we we'll send them over there and tell them You know if you have any questions, you know feel free to ping us will help help you through it But this is the documentation you need to follow submit a package for review You'll find a sponsor Once you become once you get sponsored you're in the fedora package or group and then any of the packages you maintain You can add apple branches to them Or you could also once you're in the package or group you can be added to other packages You're interested in co-maintaining We'll have that sometimes where there's somebody wants to bring a package to say apple mine and it's not there yet The fedora maintainer could do it themselves But sometimes they're not interested in maintaining fedora branches that they only use fedora and they don't want to think about apple But they they're not blocking the package being added. They're just not wanting to do the work themselves So if another package says hey, I can do this I can do this for you You know aside from them getting becoming a package you're in the first place But once they are a package they can say if you add me as a co-maintainer I'll maintain the apple branches because I want to use it on rel 9 or centOS stream mate, whatever it is And so we've had a lot a lot of a maintainer increase with that where maintainers Join the fedora project because they want to maintain apple packages because they have packages They want to see brought to these newer apple branches. They'll get involved that way. They'll become fedora packages If they're adding new packages then like said, they're also a fedora maintainer So then they're maintaining their package that they got reviewed in fedora in across, you know, say fedora 37 38 Raw hide later on the new fedora releases And they'll add it to as many apple branches as they feel like maybe they only need it on apple 9 Maybe they want to provide it on apple 7 8 and 9 and just provide it in every place possible It's all just what they're willing to do volunteer wise and what they're willing to maintain and take bug reports for So it's not a matter of just getting involved with with the apple community It's it's really getting involved in the fedora community and then moving into apple specifically if that's That's where your your package would would receive the most benefit I'd say apple will be in that from the maintainer perspective apple would be like a focus area of the fedora project Not just its own thing I know that matthew miller, uh, he talks about the fedora project leader Whenever he's talking about apple internally. He tries to always say fedora apple to remind people People inside red hat that apple is part of the fedora project that it's not just this important thing That's off on its own on an island. Uh, it's it's part of fedora right Now I think I think one of the reasons he wants to make sure to uh Reiterate that is that we look at some of the download numbers for apple packages versus fedora packages Um, and apple packages are hugely popular fedora is great A lot of us red hatters use it on you know, everywhere we can And advocate for it talk about it at conferences all those things But the usage numbers for rel and its ecosystem are much higher Uh, and so apple packages are much more widely used than most fedora branches So it's it's a it's the probably the most popular artifact, uh from the fedora project So it's important to not forget that we're part of fedora whenever people say, oh, yeah, apple's great I use it everywhere. Then you're using uh something from fedora right So as we kind of wrap up is there anything that you would like to to share with people any any thoughts any any burning ideas You want to get out into the open? Hmm RPMs aren't dead We see that uh We see that happen a lot conversations about that about there's new universal packaging formats Um flat pack and snap and things like that And I don't personally have anything against those. Um, I don't really use those myself mainly because I understand rpm really well Um, I actually just got back from uh, and I saw saw you there also a red hat summit I did an rpm packaging workshop there Uh, actually erics boss asked me if I could come and do that workshop with him at red hat summit So that was fun Um, so I like rpm a lot. Uh, it's a very it's very old technology But it's very mature and I and uh, I don't know it it scratches something in my brain that I like There's nothing I don't have anything against flat packs or snaps or app images or any of those other things but I think that there's there's certain advantages to To using times you would want to use those and other times it's sticking with our the mature rpm technology is a much better approach Other people can talk about and advocate for like say flat pack But for me, I like helping other people understand how rpm works just because I'm I've been doing it for a long time and So the more rpm package maintainers we have the more people can help share the maintenance burden and maintain what they care about Infador, exactly And eppel as well. Eppel doesn't do anything with flat packs. It's just rpm packages so and I think in a lot of times For flat packs at least I think snaps are a little bit different But flat packs work seem to work really well for desktop applications And not so well for server command line oriented applications So for a lot of things rpm's still a lot better fit And for like something that like red hat enterprise Linux server that's targeting server environments You'd still want to use rpm's there And even the advocates for flat packs if they wanted to contribute to or change something in the base distribution They still need to understand that native packaging format. So it's still a useful skill. It's not going away Even if some even if some workloads work better as a flat pack not everything will So it's it's funny you mentioned scott and the summit session you guys did together because he's going to be doing a We're actually adapting that uh rpm build workshop into a 30 minute live stream on friday So those of you that know me on the from the rel channel Uh, uh, we host a show every friday afternoon called into the terminal and this coming episode is about building applications using rpm so carl you'll have to drop by and And uh, first i've heard give them hard a hard time in chat. We just had the planning meeting this morning. So Who's going to be the the winning victim of going through the lab as everyone watches them? So scott and nate are our hosts for into the terminal I've I've taken a little bit of a step back into just a producer role because trying to get live demos and Monitor the manage the stream monitor chat all that kind of stuff is a little much for one or two people to do So it's easier to have a couple of people on camera doing the doing the thing and i'm kind of behind the scenes right now but uh, you'll you'll have to drop by and and uh throw throw some Throw some flak in in chat for for scott That sounds fun. So I noticed a an interesting comment scrolled by Yes, a couple of them. I saw a flat pack for the win, but rpm's the oldie, but goody um, oh, yeah, you just posted it up there on the screen uh Edward had a question about uh, he saw that whenever redhead decided to drop katie, which i believe that was in rail seven or it might have been rail eight um, I think it was i think it was relate Uh, I believe katie is in rail rail seven and in rail eight redhead decided like we're not maintaining two desktops. We're just going to maintain gnome gnome by itself um But because apple is extra packages, uh, it was very quickly added to apple eight and it's maintained there um, that there there is an apple packages sig um We talk a lot about what apple actually is as far as the people working on it and we've gone back and forth between Is apple a sig is apple a uh, is it an initiative? Is it a it's not really a spin of fedora? There's also um, what's the other term working group is another one another entity within fedora that it could be Um, and we're kind of a weird hybrid. We don't really fit into any of the existing buckets What what I what I tend to lean towards as I say it's a it's an ongoing initiative Not as opposed most of the initiatives are short term So it doesn't initiative doesn't even work perfectly either um, but I would say it's an ongoing initiative. We also have a sig um, that is That's not the apple that's different from the apple team within cpe That's also different from the apple packaging. Um, sorry the apple steering committee Where we talk about like policy stuff around apple and how that's how that works things like, you know, not replacing rail packages Um, so the apple sig that is you know, people that want to help each other with apple packages and their dependencies um, some of those some of those uh Uh, kde dependencies are are part are maintained by the apple sig Um, but there's also a kde sig in fedora and that's where most of the maintenance actually happens They take care of kde in fedora and an apple A lot of nuance and a little bit of a discussion going on in chat about uh, About flat pack versus rpm. But uh, it seems to be a hot button issue. So yes, when you asked me about burning things I wanted to get off my chest. I was like, yeah, well Here's a good one. Everyone likes discussing this Um, I know that it's come up recently in the news too um, there was some uh, a mail on this post that's got into the the uh, the linux news circuit around labor office and what I've told people about that is that um Rel deprecates packages all the time if you look at the rel release notes Usually every minor version something is deprecated where it's decided You know, we're just we're going to keep the lights on for this and maintain it for the versions of rel that it Still is in but we're going to remove it from a future version of rel Not always the next major version of rel sometimes it hangs around for a couple of releases um But definitely at some point in the future people should plan for it not to be there and that's what happened The maintain the red hat desktop team indicated on the fedora mailing list that They they're they don't want to maintain labor office in rel 10 and so Because of that they're going to pull back their participation in maintaining the labor office rpms in fedora itself For rel in the future they're expecting that the labor office flat pack is good enough And if people want to use that they can and like I said flat pack works fine for desktop applications um But they they withdrew from the fedora packages Which is labor office is a big name in this space But this actual event happening is not uncommon. It's pretty regular. Something gets deprecated the rel maintain I mentioned that uh rel is only like roughly 10 of the fedora rpms actually make it into rel So there's a huge amount of rpms that are just community maintained in fedora And so when something when rel maintainers draw, you know withdrawal from a fedora package Uh, then those those are up. They they become what's called an orphan other fedora maintainers can adopt them um Sometimes existing co-maintainers of the package that aren't that aren't working on rel Can just keep those packages going and just take over main maintenance of them directly um And even sometimes it happens where the rel maintainers decide that Even though they're it's not part of their job anymore They may still want to keep maintaining that package in fedora just on a volunteer basis because Maybe they've been in that upstream community for a long time and they still want it to work correctly on fedora So they'll they'll take it from being there their day job to their side project Um, those sort of things happen regularly. Uh, it's it's a bit of a non-event Yes, labor office is a big deal and a big name that people recognize But stuff gets deprecated all the time maintainers change And that happens in fedora itself too aside from any kind of red hat business decisions Somebody a maintainer may get busy with life. Uh, they may not have time for fedora anymore They may they may have a kid they may get a new job that takes up all of their time All kinds of things happen We have processes in place for that where You know, if there's if there's some kind of critical thing There's a group called proven packages in fedora that can step in and fix things in a package Like get a get a cve back ported or update to a version a new version that fixes a cve So we have some crossover there Those aren't the primary maintainers of the package. So long term What ends up happening will like the orphan process the package will get orphan a new maintainer will say hey I'll take that I can I can take care of that package. I want to make sure it stays around If nobody adopts the orphan package then after I believe six weeks it gets retired from fedora rawhide It stays in current fedora releases and current apple releases Just until until those go into life. So That happens sometimes where we'll have like an apple package that the maintainers have withdrawn from it may be orphaned It may even be retired completely from fedora But we try to keep it keep it building and keep it secure as well as we can I say we I guess the context for that we would be The overall everyone participating in apple not just the apple team and cpe Although we do look at those things or the apple steering committee where we also talk about those things There's a lot of we in different contexts, you know the wearing mini hats thing for sure well, that kind of brings us to the end of the end of our agenda, but I can attest that carls very active out on mesadon out on the fedora's discussions page And a lot of those different special interest groups so if you have questions or if you want to get involved carls always available and and In the fedora team is amazing at Bringing in new new maintainers and helping young developers get get started with something like fedora packages So there's plenty of links that we pulled together for this episode If you're watching us live, give me about 30 minutes or so I'll get the episode or I'll get the show notes taken care of if you're watching this after the fact It should be we should have all the links and everything ready to go for you as well as chapter markers. So definitely check that out So carl, I really really want to thank you for for joining us today It was it was a great catching up with you and great chatting about one more thing. I forgot about Sure tying it tying up the thread about liber office If if everything goes through like I'm not making any kind of product promises for rel itself But the expectation that the the rel workstation team talked about on the fedora mailing list was that rel tends not going to have liber office If that's what comes to pass then anyone that wants to maintain the the main the liber office maintainers in fedora Can branch that for eppleton and maintain it there as rpms. So That's that just gets back to the the extra part the community part Red hat product decisions other than not overlapping it staying true to that extra part Doesn't really affect what's what's in apple, you know, as long as it's not in rel and conflicting We can put whatever we want in apple that follows the package guidelines. That's you know, that's free software right and and I was I was not directly involved, but I was made aware of the change earlier on and I mean, honestly, it's I don't want to say it's a non-event, but Uh, I mean liber office is supported by is it the open open document foundation? There's so many foundations now. I can't keep them all straight as well as an amazing community in fedora and apple So the idea the red hat desktop team doesn't have the resources to maintain liber office Is only half the story the other half was that even even in the in the fedora develop mailing lists it was pointed out that The the red hat desktop engineering team is very very much focused on finishing this This move to weyland trying to get some of the functionality trying to get some of the feature parity So that x11 hdr, so that's all that specifically Right exactly. So trying to get those things Fixed and working in weyland so that x11, which is now like 300 years old can finally be completely retired And we don't have to maintain both of those both those build structures So that's that's what rel engineering is is focused on so why not let someone who's paid to do that? focus on fixing those issues And then taking a well maintained and well supported project like libre office and entrusting the community to manage that Luckily, I don't think I don't think that particular story was was too terribly blown out of proportion. I know I saw some Less than friendly comments out on social media and read it about that but It was a lot less than I was anticipating, but I think a lot of folks Actually took time to read that that thread and realized that there's a lot of things that need to be knocked out on the desktop And you know, unfortunately red hat only has so many engineering cycles And the the power of the community is so much greater than what what a small desktop engineering team can manage So I felt like it Long term it's going to be a good move for for the community and I I would not be the least bit surprised if libre office was available in evelton Yeah, that's what I've noticed a lot with community discussions is that uh nuance is hard and You may want to go into a long detailed nuance take about how how this thing is this way and then you know Everyone else gets a bumper sticker slogan of oh, well this thing is terrible. This thing is that And that's what gets traction not the long detailed nuance take that takes facts takes in all the factors Ask me how I know about this after working on the centOS stream team Yeah, we we both went through that transition But before carl has a chance to get me to say something stupid that I'll regret I think we'll wrap this episode and uh so quick announcement around the next episode, uh, it's It's up in the air right now I'm going I know this is ridiculous. I just started working on the podcast, but I'm going to take a step back for just a couple of months Summer has proved to be very very Well insane So between kids being home and and work life balance and very busy season at work I'm going to be taking a bit of a pause I'm not going to say that the podcast is going to take a pause because I've been in conversations with a few folks that want to went to host So this is your chance if you want to be part of the fedora podcast join us on the join us in the matrix channel and And you know, let us know What what kind of setup you have what let us know if you've got any materials you can we can we can listen to And we'll try and get you set up with the topic and a guest and and keep the podcast going But as for me, I need to take a step back for my own sanity a little bit In fact, there's this isn't the only the only sacrifice. There's there's some dnd campaigns where where particular gunslinger or wizard won't be won't be participating for a couple months, but It's it's not the end for sure. But just know that Just know that I plan to be back this fall when things settle down namely when my kids are back in school Don't tell them But so Carl if you want to host by all means I'll set you up I think I'm going to pass on that but uh, I would uh, I'd be open to being a guest again in the future For sure. Yeah, you've been on royal presents. You and I've collaborated on a few other things So always a pleasure to talk to with you. So on behalf of my guest Carl George and the entire fedora podcast Uh, geez the entire fedora community. Thank you all for joining us whether live or deferred Make sure to ask any questions in the comments below. We try and answer those as quickly as possible and stay tuned to uh this youtube channel as well as Social media mastodon twitter all the things uh to see when the next episode will get scheduled Hopefully it won't be too much of a of a of a break. But uh, anyway, thank you all for joining us And we look forward to seeing you again real soon Take care