 Holy cow It's a good crowd. Honestly, I can say that when I do this talk, I do a similar talk a clock I get six people in the usual suspects. I Want to say thank you all for coming to this talk Can you not hear me? I apologize. Can you hear me now? Okay, we will now peace to this tone for the rest of the meeting The front office are now deaf. I am sorry They will need water All right, well, this is a this is about a 24-slide thing However, the first couple slides are just The sorry All right, well, and I'll go over the quick part fast and then people who are coming in setting up This is a talk on the future of Apple We have a new function of Apple coming in Because rel 8 is coming to be coming out in the next month Years soon my name is Steven Smudgen This is Kevin Fenzie. Hello Today's topics we'll be going over Sort of a history of Apple in five slides Where we are going and then I would like to have the good portion of the meeting towards talk with us So if I I know it's 50 but at 20 minutes before that Okay Apple in five slides Okay, this is the presenters Kevin Fenzie is a rock star He also wrote East for Rush I've been stuck here on this planet for quite a while and try and get off but We have a higher heckler Just to keep me honest if there's any time I am not honest and he doesn't catch me anyone in the audience can catch me Okay, Apple is a set of packages that are branched off of Fedora It has a really bad name Called Apple because the fact is being all its system administrators We had a very hard time coming up with a naming scheme So first naming scheme of what it was After about six months of arguing over the other name schemes. We just stuck to it. Everyone else knew about Apple versus The next gen or the future We normally do packages That people want on rail But well for various reasons red hat would not want to Support because well It's gonna be very expensive to deal with NGIX and six other different HTTP servers We started a while ago and Well, our normal path is we take from Fedora of the release that rail was based off of We well these days it would be a Branch, but originally it was sort of like a CVS fork of those packages. We branch them into Apple Branches and then we rebuild them in the Fedora Koji servers Currently we we started off with rail five well, well for rail five was our main starting point and then we did rail six and Seven or our current current name once Who uses Apple? I Hope many of you use that Raise your hand if you use Apple daily or if you use it with some system that you're dealing with We branch everything Fedora would not be built without Apple that is actually one of the reasons Apple was started inside of Fedora We had a ton of packages we needed to build the entire our own rep infrastructure with Rolling it out just as we were about to turn it off And it's we mostly are most of the community members who are focused on Apple are more towards the slower paced Wanting to get it into a long-scale production system, which leads us to our next Graph The lower levels here are your fedora users over time Each of these as you can see they branch down And they slowly die off Branch down slow down. This is the Apple curtain Down here is Apple five It's still going slowly off Apple six Was kind of slowing down for a bit and then it's all sudden started picking off again Because people are finally realizing that red hat is not turning on Apple five a rail five releases again So they're going those fist boxes are getting to six seven is picking up here We're looking at sort of site data For the future that's rawhide So it's a little bit here. We'll hide actually was used quite a lot back here then it kind of went away and then kind of It's just at the very top here It doesn't really show up enough to in my picture to be because the red why So yeah, each of these are different sort of eras of fedora usage Anyway, it's used quite a lot and it's continuing to grow and I'm expecting That when eight comes out there will be a yet another And six will start doing this sort of 20-year draw One thing to note from the last slide Just to mention very recently Apple seven past Apple six, right? Yeah, okay, so that might be Stacked stacked graph in if these were separate ones Apple seven has finally after five years reached the number of users that are it and it was almost a race where it didn't Because this jump up If this is still going down it would have happened in I was thinking it was going to happen about here in January, but also in a whole bunch of rail six systems came online using Apple and Apple seven More to get there But it's finally more users Systems out there there now this doesn't mean that we know that this is not every system and there's a Low weight on this if there's a whole bunch of systems behind a firewall. There's still counting one So this is sort of the minimum Size of these communities not the actual size or a guess of the maximum size So who makes Apple happen Apple's run by volunteers both from centos and fedora communities The build system is fedora Koji and this is really just Sort of a historical accident because that's that what's what was there. That's what what we had and At this point, you know, if you were designing things logically, I don't know how many of you were here for the last talk the Penrose discussion but Right now everything kind of flows very strangely if you thought of it logically you might think that add-on packages would be something on top of Centos rather than on top of Rel in fedora, so it's all Sort of what we had at the time and historical Basis There's a steering committee That has five to seven people from the various communities on it make decisions about overriding guidelines coming up with How things will be done, etc. etc. and The release engineering and things like that are kind of done as a ad hoc thing When as people have time so there's no Employed people who work on Apple Apple is just a side gig for people who do fedora release engineering and Centos Community work and Centos build system and things like that. So It's really kind of amazing how Well, we've done with the resources that we have All right, so where are we going? There's obviously lots of things to discuss especially with regards to new rel but Let's start out with complaints and issues that we've had Kind of for the existence of Apple One of the problems we run into is that rel supports things for a very very very very very very very very long time and When you're a package maintainer and you say oh Sure, I'll maintain my package for Apple. It'll be great and then five years pass and you're like well I don't want to do that anymore or this package is now not working anymore I don't want to commit to the fact that I'm going to be maintaining this thing for the rest of my life So we've run into that problem We've run into the problem of not keeping old packages around Occasionally there will be problems and people will be like oh, no There's new update of this package broke a bunch of things, you know And in the meantime before it gets fixed we'd just like to roll back to the previous one Which was working fairly well And of course there's ways to do this the fedora koji keeps all the old builds around But it's not convenient. It's it's something that people have to go in and Hunt down the package figure out what what's what download it the selves manually downgrade it, etc So we've run into that We've run into the divergence problem from fedora So we're taking fedora packaging components specs and sources and so forth Building them against rel and at the start of the cycle. That's great They're sort of in sync. That's it's the version that that works with those libraries and that Package set but as time goes on things diverge So the fedora version will bump, you know 20 versions But you don't want to do that to the Apple version you want to keep there an older stable one and then there's new macros in the fedora side That cause it not to build on the rel and you run into a version skew problem there We also run into the problem for our centos users When a new rel version comes out There's a lag between when that version is released by red hat and released by sent the centos version if we move along and Pull in that new rel version and build stuff against it Then the centos users might not have compatibility against that So we run into that sort of version skew problem there as well for for short periods of time And then there's there's other issues that we've run into over time as well Anybody have one that isn't up here? Problems you've run into issues Or does that cover most of them? Yeah Yep, yep, that's another one that we didn't have up here is architecture support So there's a certain Certain architectures supported in rel certain ones that are only in centos We build against rel so we run into problems trying to build against those secondary architectures With fedora koji so yep So here's a little reminder for our Apple 6 users Even though Apple 7 just barely passed Apple 6 Yel 6 is going to end of life or at least high maintenance or whatever the term is In 2020 so we're just not going to do much to Yel 6 There's it's going to be basically the way it is now and just keep puttering along until until the end of life so for 7 We've been talking a long time about Doing changing a little bit of the procedure we use for when new releases come out right now Each Apple repository is just that collection of Apple packages that gets created each time we push out new updates But we're talking about maybe moving to a system where we have a base repository Which is created at the time of that release and then have an updates repository on top of that What this would give us is the ability to actually downgrade packages again Because there will be a package that's in the base repository when the release first came out It'll also give us the ability to Have people to Centos users could point to the previous release until the new centos release comes out So they could not run into this incompatibility with the new releases It's a little bit more work, but we're looking at probably doing that to Save some of these problems. I made a mistake on the slide. It's everything I Forgot to change that everything stays static was an everything path. It should be say stuff stays static Yes, right. Yep Yep So this also would give us an ability to archive old releases so if right now Apple doesn't support any of the older point releases, so if you have a Rel 6.8 box and for whatever reason you never want to update to 6.11 Too bad. Apple only supports 611, but if we move to this scheme, although it won't help Apple 6 but For 7 if we move to the scheme you can actually continue to point at the archive version You won't get updates, but you'll at least have those still have those packages Let's see Okay, now comes the meat of this argument Apple 8 beta came out this November October and there's no Apple or sorry EL 8 route red hat Enterprise Linux 8 came out in November There is no Apple 8 yet, and there's been a lot of people wondering when is Apple 8 coming out Apple 8 is been delayed because we've got to work out a lot of different things Apple is not built the same way as Fedora is the Koji system takes Has to know has to sort of built a package to know about the package to use it and We didn't build any of the rail packages and so we cheat by having it Koji look at an outside repository That gives you enough information for how we've done things with rel 6 5 6 and 7 but it doesn't work well with modules because There are times where modules may need things that we don't know exactly through an outside repository without extra work And it without extra logic that we needed to do There are that means we have to we're gonna have to break a lot of things and as somebody said recently I was in a meeting. It's better to break everything and have everybody scream at you at once Then break it in small bits over time, and they go away because they're tired of dealing with all your death of south thousand cuts so this is a good time to break everything and We're trying to come up with all the things we could break in one time to get it done Our current pathway looking forward to try and get an Apple 8 out is we will not be building modules for Apple 8 beta The beta Apple will be in a different path so that people aren't you know, it'll have a different path so that we can Work through all the different changes we want and we're not affecting the main Apple It'll be pub alt Apple versus pub Apple once we get to 8 general release or whatever. It's called GA GR GQ We will go with putting it in The regular pathway It's a beta for on our part as much as it's a beta for anything else, but it will be there. I'm hoping by March It will have packages that are non modular and We may Be able at we will work our butts off to try and get modular in there is when we can EL8 again comes with modules, so we were going to have to put in later rules to deal with all this If you want to replace the rail package currently in Apple We don't allow that if it Red hat ships a broken aws cli If Apple take if rail takes Ansible inside and then sit down and then takes it in Just for example not not not that was picked Purely as a random one because it's an a No You'll be able to replace it with a module and a module stream that'll have a naming scheme that will make sure that we don't replace the rail at Ansible, but we will be able to Have You'll have your module of Ansible and it will put the packages in there and it will know not to you won't get broken ish Again if you want to support multiple versions of something you will have to use a module Modules do not give you Parallel Installability they give you parallel availability if you want parallel install ability There's a lovely project called SEL, and I would love you to support them If you want and to deal with the other programmer problem package your problem of God I Don't want I want to upgrade this to something that's supportably You can put a lifetime now as a policy in on your module There's a thing called the product PDC FPDC Fedora project definition center or some other an acronym that fits those letters You'll be able to put a poll I want this to be released supported for this one release and Then at the next release you can pull it out. I don't care. I'm not good. I may say I want it back in But if I don't want to support it anymore, I'm taking it out and Because we'll be using a path system Sorry, you will be able to Got my slides mixed up paths were supposed to be here. Okay, I Had to rewrite most several of these slides this morning because we came to several agreements of what we were going to do with epa late this morning Packages in EL8 are shipped through two channels and A third channel that is not a two-channel system The base OS are packages That will not majorly change or this is how it was explained to me because I'm a two-year-old So I may not have the exact right definitions and somebody who actually knows them like Josh or Mike back there who's not there can tell me Appstream are things that are more likely to add and remove over a half-time. There's going to be some sort of cadence That they will upgrade things and change things and fix things and make things and go faster And then there's a bunch of stuff that you might need to build stuff, but most users of a package don't need and Having that in there always ends up with well, I installed this and it's I'm pretty sure this broke my system So you have a grub problem. No, I'm pretty sure that GCC develop broken my system Those sort of issues that support has to deal with The putting it in a channel that's not by default where it's going to be a unless you really needed it That's what the code ready Linux builder is. I missed one again. I was supposed to put this as an acronym We will build all Apple packages against these channels they will all be in a some similar way To be decided as we learn this but that's what we'll be building against I'm saying this because there are some other channels that are in rail. There are what do we call those like HA high availability and Add-ons we will not be building against us. We're only going to be building against these three. It's in an add-on We can replace it, you know in the old-fashioned way that we're doing stuff One of the big problems other big problems is well, which add-ons are real but is Apple building against? Well, we're building against this in this. Okay. We didn't know that please take it out. Well, we've been building against it for six years We're gonna start off with just three things if somebody finally says oh, we really want you to build against it from now on We'll add it in but only after they have signed in blood Is that? All right The past we showed in the earlier with seven you might have noticed that there was this slash stuff a Suck in that it didn't make any sense The stuff is going to be here because in fedora. We call it everything, but people said it's not everything anymore because we have modules so I came up with the Replacement which is stuff? We have stuff We have modules if it's not a module. It's stuff Those packages will be called stuff things anyway The pathways will allow things to be and it'll be again as Kevin went on it will allow us to go through and do both archiving regular archiving Regular building outs of things and have a two-channel system for backups and pullouts and pull-ins We will rotate these at regular intervals to be decided because we don't know what rel eights Rotation cycle is if rotate it rel decided it was going to rotate every month We couldn't keep up with that because we're all volunteers There's nobody here who's going to have a full-time job of okay I'm updating things if it's a six-month cycle we may do that But it'll be some cycle that we can keep up with as volunteers Any other proposals from the audience that we the Brian can catch down he's got something to do All right We have a lot of questions We know we have a lot of questions about how late it's going to happen I'm not sure that's a big enough number for the number of questions we have We think we know how that some of these things are going but we haven't run it yet, so it's and it's not that we're a special case and It's not a statement about modules and thing the way we build things has been always of a special case and so we Are making things harder for ourselves because we can't do certain things Oh Module we're not filled service It's sort of a thing that sits next to koji or on top of koji and whips koji into doing what it wants to do Pretty much yeah What modules in it can be so again some of these are policy and some of these are we're going to do it in this Fair in this all area we're going to find out it breaks We'll figure out why it breaks and then we'll go okay. How can we make it not break? But we would we won't break regular Apple while we're doing that's the main thing I want to do because if you're already on Apple 6 and you have or Apple 7 and you have stuff that you need doing You don't want us Futs in with your stuff while we're trying to fix something else again Directory layout changes we think we have been a proposal Somebody might say oh, that's gonna cause this problem, and I need to really fix that okay Composed changes again Currently what happens is every night? Apple is not so when you look at for door you see an 18 you see pub Linux releases 829 slash and all that is composed Once and stuck on the mirrors Because that's how and then updates gets composed every night and move stuff gets pulled in Apple It's like raw hide every night It gets composed and a new thing put out So that means that it's rolling and stuff comes in comes out and sometimes it doesn't happen at all because something else broke but it there's no It that's sort of the way we've done it because it was a quick hack and we've done it now for 10 years So we they're going to be composed changes to make it so that we have this once a semester I'll call semester semesters can be any length of time My son has now had a semester a fall semester. It'll be longer than a spring semester. So yes Composed changes can for door modules get auto-composed when a commit happens. How will we deal with this? How will we deal with new modules and dead modules and modules that have come in in the middle of a stream and The combinatorical number of module Interactions that can happen because you can have module a stream be compiled module B stream a Compile module C stream Z and they all have to know each other and But some of that we don't know because it's done from somewhere else There were a lot more questions, but I knew I had a very short time to get this thing done So I wanted to make sure that you guys had a lot of time to talk to us There are about six slides. I deleted here and a trash fire and a really cool image that Matt Mike had given of a tornado fire tornado All right Jumping to the last of the set Which you can't really do but we'll do anyway this is computer and we just not a number What do you think what are your plan? What are your wishes for? rel Apple 8 to meet your needs because We need your input and your health because we are done as like I said, we're volunteers most of the people who work on this are Fedora well it started off with mostly Fedora people and now it's mostly sent us people who come and help us do stuff My primary job is anything but Apple Although I so We need your input to know when not I'm not Going down a rabbit hole thinking that I'm helping you and I'm really not helping you that makes sense you sir So Apple packages are Going to be in the same spot as all those Apple specs are still in that same spot They're just so what'll happen is because they didn't show this on here. I will try and mind this out Kevin stand here No, no, we're here. Sure Kevin There is a there is a pack in the get there is Kevin Under Kevin there are several branches Floor your hand up This is Fedora rawhide This is for door 30. This is for door 29. Sure. This is 27 blah blah blah blah blah blah Apple 7 EL 6 and then there will be on this hand same Kevin C6 C7 C8 and potentially 5 or Stable or whatever you want to call the stream branch for the thing is for the module which could be built for fedora sent us Apple everything so what you would do is you would go into a Source That fedora project org or source that sent us that org or whatever the name it will be You will do a get clone after you or Done a what is it to do a fork and a fork and then PR PR it back And then the PR will go in the person who is maintaining it can then go Yes Where it needs work, you know the usual BS And then that's how those things will done. So if you want to fix an apple package, you will fork the apple You'll fork Kevin to Steve Steven will then have all those branches you will Make the thing and then you'll do a PR and then Kevin will accept it or not onto the right branch That sound other than just yeah We don't care So the question was a very long statement about wanting stream branches to do everything Yeah, okay, the answer was Yes, sorry the question was what do you want the answer is I want stream branches to be able to have lots of branches So that I can pull stuff into rel or out of rel and it a person can build against the stream branch and have Solution that they need Whether it is supported by red hat or not certain things will be supported by red hat and Will have been signed by red hat so they'll know it's not supported and then certain things won't And the other big thing that stream branches give is is the timeline Support timeline issue because we have lots of people who are like They get a request. Hey support for Create a branch of your package for Apple and they say no I don't support Apple because I don't want to be here for ten years running this thing Whereas if we have a stream branch we tell them okay Just do a stream branch and put whatever you're willing to support You know if your upstream is only going to have this thing for two years You could say two years and then I'm out so one of the other proposals that I didn't put up here and I took off the slide Came up because I didn't want to be thrown out of the room or and Burnt at a stake afterwards was Apple 8 will all be modules Nothing but modules in Apple 8 Because it solves all the problems by creating new ones Everything is a problem and you just have to find the next solution The problem was that we don't have a lot of things as modules open VPN is not currently a module Clam AV is not currently a module and that's a lot of extra work to put on to Packagers Who may not be interested in going down modular path at this time? Also had put a lot of work on us trying to convince them that we needed to do it that way and that would be a very easy no And I for them There's also a lot of stuff that's in Apple 6 and 7 now that are like really small add-on packages So they're like you know PHP blah blah blah and it's one library That does one niche thing and it seems to me like we'll have a lot of difficulty getting maintainers to say Oh, I'll build a module for this one little tiny teeny tiny thing Whereas building it as a traditional package is makes more sense But I think it gets back to Also, how difficult is it to make a module how difficult is it to maintain a module and maybe at some point down the road We make that easier to the point where those people are Satisfied to do a module for that kind of thing. So I guess the goal could be Apple 8.9 Is all modular picking a number that I'm not going to stick to Because Apple is not supported Like Very strong preference Those three years Yeah, there's a lot of upstreams that behave that way that are like Here's our our current thing and we're planning to you know, totally redo it so if you are able to get those plans or know those plans you can set the life's appropriately and Provide some kind of upgrade path to where they you know get the next stream or at least they know that there's another stream of The next thing. Yeah Another proposal that was on the table which comes up is that but it was I It is a lot of work on our part is That at every those dot releases when rail comes out We tell the maintainers at that point they can pull stuff in or out We're gonna unofficially do that if you're a maintainer and you've got something really old and you're like When six dot whatever the next one is Comes out There's stuff you want or not. We're not doing anything six when seven dot Seven Starts coming out. We're gonna let the maintainers know that a if you need to update or you need to do stuff We're gonna make a fast-track process Do this send this email out say that you're gonna do it and then when we branch to seven seven You can go do that We'll pull that in and then you'll have the seven six tree there Which has all the old stuff you need that old stuff you got it seven seven will have the new stuff So I'm hoping as I'm hoping most of you guys are ops people Whatever that cool word is these days Sis admins Do are you guys question or? Okay, oh Yeah, sorry, I asked for raised hands and then I didn't so I'm hoping you're mostly this because I'm trying to make sure so that your jobs are not broken the last thing you need is October 31st, you've gone to the beer Fest that because it's the last day of October Fest or whatever the last day of October Fest is and Your pager is melting in your pocket because Apple just upgraded a bunch of stuff and you weren't ready for it But on the other hand for the ops people in the room. I also want to make it so that you're not spending the next 8 to 2024 worrying about your rel 7 package Without being able to move forward in a way that you have a way to do so Yes, sir I That's my plan it's my plan for this the problem and the reason why I'm trying to set expectations in this room Is that in the past? Whatever came out on EL 6 first day. We did this We in infrastructure have only had enough cycles and such to do that Till now till 2020 1120 and we're going to move to an incremental improvement structure Right Yes, sorry 10 minutes We will So yes, that's like I said module solve a lot of problems and us moving to modules in a incremental as We fix things and get things going way which should help you solve your user problems Better without having to use a hammer But it will we won't be able to have everything on the first day and it may be slow that's Yes, sir So, yes, I know I that There are ways. Yes, you could have that happen and That I'm going to expect to be the as you said the new RPM hell It'll be some other thing. We will have to figure that out We'll have to figure that out because the the issue is is currently the solution everyone is going to is Not using packaging at all and just dumping a tar ball on a bunch of disc drives People have gone back to that. I have run into so many systems where I couldn't get it in Apple because the Apple version was too Old so I just I kept the package on Apple because I didn't want to do this But I just did a tar ball over the other stuff and it's broken Because now I have two different versions running on the same system We're going to run through that It's going to happen. We'll have to solve it as we go. It is not And it may be unsolvable. There are problems that are unsolvable. We all live with that Resist admins. We have users We'll work with it as best we can So I want to mention one other quick thing about modules that might be a big win For the Yale side of things in that say say Python 5 comes out or something like that and Some Apple maintainer makes a package or a module of it builds it up and it's really popular and people start using it and then say maybe It's something that is Desired for support. So it maybe gets pulled in and there's a module that comes out in a later Yale release. That's a Python 5 or whatever Having the ability to see what things are succeeding and popular in Apple may be a means for determining what things are worth Supporting commercially or putting more resources behind, you know down the road. So it may be a win in that direction Thank you, sir. We got two minutes. Sorry. Can I track what packages are most popular? No, I No, no, what? by popularity What how support would know is support would get a lot of things saying hey, I'm getting this from Apple You're a red hat a customer. You got your you have a way with our hend to get that we do not have a way of knowing this all I know is This IP address asked for this repository Everything else is just best guesses Right any other questions? Yes, sir So, okay Our pressure on the builders mirrors I'm archiving off stuff and Nobody mirrors archive So very few people mirror archives. So only those mirrors that are archiving the old so 7.6 will be here 7.7 will be out 7 that's when 7.8 comes out 7.6 will move off the archives So the disk space will just be a little bit more, but it should not be greatly more than what it is and if it turns out that it is turning into greatly more then we will work out some other solution because the mirrors are the hidden lifeline on here that I didn't put up here, which is There are Packagers in Apple. There are sent us ops Who do the other stuff and then there's these whole big room of people who aren't in here? Who run mirrors that you get your packages from? They do it off mostly off their own time. They they are not paid by it usually their university and they just have free bandwidth and they they haven't told their boss that they're using it for this So if you run into a mirror of a manager somewhere or somebody's running a mirror thank them buy them the lunch someday and Anyway, I think that's about it for us up here if you don't have any of our talks I Will let you out early so you can get to the coffee place before anyone else