 Hi, my name is Steven Gallagher and this is my compatriot, Mohan Bhattu, the International Man of Mystery. And we are here to talk to you today about Apple. So what is Apple? Apple is a business tool used to create spreadsheets and to work with... What are you talking about? That's Excel. Wow. It's Apple. Okay. Hold on, hold on. All right. So Apple is a prize awarded every year by an international panel of scientists. No. That's noble. Noble prize. You already changed the title of the talk. Come on. All right, all right, all right. Okay. So Apple is a member of the resistance against the fascist organizations of the world. No, no, no, no. That's rebel. Ah! Did you even prepare for this talk? Okay. I'm not going to answer that. All right, so Apple actually is all of these things in a way. There are many, many businesses out there that rely on the extra packages that the Fedora Project provides on top of Red Hat Enterprise and Linux and CentOS and the like. And in many ways, we could say that we deserve the Nobel Prize for all of the advances that we've helped enable in the scientific fields, the scientific Linux and CERN, all our users, very heavy users of Apple. And lastly, of course, being part of Fedora, Apple is naturally part of the rebellion against the proprietary empires of the world. Pause for applause. So how many people here have heard of this thing, REL8? Anybody? Nobody? Okay. We'll talk about that down the hall in a little while. You can go to that. But unfortunately, you're going to be lost. So last spring, Red Hat, you know, it was part of the daily news cycle, revealed this new thing, REL8, and, you know, it was just REL7 with the number increased, right? Right? No? Oh, that's right. It was really different in just about every way from REL7 that, honestly, we probably should have talked, we should have considered changing the name. So why don't we talk, why don't you talk a little bit about the differences here? Yeah. So, as Stephen here is saying, the REL7 and REL8 are really different. And even the Apple 7 and Apple 8 are based upon REL7 and REL8 respectively. So when Apple 8 was set up, which was like five years ago, the people who did the set up is not actually working in this area. So it was not very much documented. We need to figure out on our own, like how it was set up and how can we do it or replicate it in a way so that we can set it up for Apple 8. So that was a big challenge to start with. And then REL has modules. So yeah, I'm sure whoever came up with that idea knew what they were doing. Absolutely without question. Moving on. Then, so obviously modules are different. They are like sort of a virtual repose. So we never had a set up for Apple, which has modules has in it and like separate virtual repose because Apple always had a quantity to one repo as it will do it. So to figure it out, it was a challenge. And then we wanted to add Apple Playground. So it's a place where maintainers can play with Apple. You know, you're going to explain more on that later on. But this is another thing that we added. It's sort of a new release. And then there is, when you try to start working on Apple in staging, it was not actually as good as production. And there are a lot of things that are missing, a lot of services that are running the whole versions and a lot of data is missing. So we have... It had not been synced in what, like, five months. So it did not look anything like the production repo and the environment. And it was... Yeah, that was fun. Fun. Yeah. So we need to tweak around and finally get to a stage where we can just build something in staging. And then, as I initially said, there was not much documentation. The people are not around, so it's like a lot of trial and errors. Let's try this thing and see if it works. It didn't. Let's try another thing. Let's flip this switch over here. Oh, no, no, that electrocuted smooch. Try this one. No, no, no. That was Kevin. A lot of trial and errors. And, you know, it took some time to figure it out. So and now I want to talk about Apple Playground first because how we did the setup and the changes that we made, which we are going to explain in the next slide, is going to apply for the Playground as well. Apple Playground is sort of raw hide for Apple 8. So why we did this? There are like two reasons. One is we wanted to provide a place where maintenance can test things just like as in raw hide and don't want, don't have to wait like 14 years or so to get it into the build route and start building other packages on top of it. So this will help them test their stuff right away and that's one of the reason. And the other reason is Apple, Apple, to start with, it doesn't have any RPMs to build against. And it will be harder for people who are maintaining Apple packages to test their things because none of the packages are available in the build route. So and again, the 14 days turn around time will kick in. So we decided to have something like a playground where people can test their things and their builds will be ideally available and be prepared whenever the other dependency packages are available in the stable report. And another major part of the playground is it's meant to be a staging area too for big changes, which in Apple 8 we're going to allow large changes to land where we haven't in the past, but they'll have to land at a minor release boundary. So if you want to bring in the newest version of KDE, for example, you might build it in Playground and then when RHEL 8.2 lands, then you can merge it into the stable repo at that time. So Apple 8 Playground is in its own release and it has its own build route. We run Composers Nightly, which will be shipped and nobody can back to you. So among the set of things that we needed to deal with RHEL 8 was the fact that we now have this app stream repository or application streams. And unlike RHEL 7, which basically had one big pile of RPMs and you always just took whatever the highest number was, with RHEL 8 we now carry multiple different versions of things, sometimes higher versions, sometimes lower versions. And our tooling in Koji was not equipped to deal with that. And so what we needed to do was we needed to find a way to take the content that was published by Red Hat and then break it up because modules themselves are basically virtual repos sitting inside a single repo. So what we did was we took that repo, all of its metadata, and then broke it up into individual repos, one for every module and then one for all the stuff that wasn't in a module. And then we collected the set of things that we knew that needed to be in the build root, squished them back together, and stuffed that into a specialized build root for Apple. And the tool that we did that was called Grubby Splitter. Then another piece that we needed to make some changes to Koji to be able to handle generating a build root that may contain, let me start over, traditionally Koji would, when generating a build root, perform a bunch of sanity checks and make sure and deduplication and remove more than one copy of a package with a given name and all of these things interfere with the idea of actually sticking modules into the content. So we had to create what we called Koji Bear Mode to just say, this is your input, it all needs to be there. Just don't touch it. So we sent Mike McLean these nice warm arm warmers and he got right to work. The last part of the puzzle for dealing with the AppStream data was that once we had constructed this build root, we had to be able to apply it on top of a CH root in Mock or a container image that was in fact using the default module streams of this content. And because this new Koji build root repo was not modular and DNF has failsafe capabilities to protect you from accidentally installing a higher version of a non-modular RPM on top of a module that you have selected, such as if you pulled in a third party repo and it happens to be offering a package that has a higher version or you have a partial repo sync and you're missing the metadata, this failsafe prevented that from happening. So we also have a feature called module hot fixes, which you can set on a particular repo that says this repo is specifically allowed to override modular content and it's intended for in rel production environments to be able to ship out an emergency fix to a customer without having to do a full module generation until the next scheduled update cycle. So hot fixes. And once that is figured out, it's much easier to set up the Koji set of things with Koji tags and build roots and targets. It's easier. But then there is this question about the playground, how the branches are requested for that. So we decided to use a thing called packages.config and test grid branches. How many of you have ever used or known about packages.config in test grid? No one. Nobody. That's great. So I know a lot of you, how many people are maintaining packages in Fedora? Okay, yes. So one of the pain points in maintaining packages in Fedora is like you have to build package builds for each and every release going into different branches, right? So there is this packages.cfg file which will allow you to mention the releases that you want to build against. And once you submit for package build, it will look at this package.config file and it will build for all the releases. Whatever you specify in the package.config. So you don't have to switch branches and do changes and stuff like that. As long as they're all going to plan to build from the same grid commit, you might as well just build just one command. So we wanted to use that feature to build for Appalach Playground. So we have to make the changes in Fed package. Whenever they request Appalach branch, it should automatically request Appalach Playground and as well as it should handle creating the package.config file in Appalach branch because we want people to build against Appalach Playground as well when they submit it into Appalach because not many people will be concentrating on Playground. And the other thing to note is that when you want to separate them, when something is going to go into Playground for experimental purposes, all you have to do is remove that file and then you can commit to each of them separately. And to handle that part of things, we have to tweak FedS commitment, which actually processes the ticket. So we tweak that to create all these files and as well as create the branches. And so in Apple 8.0, we were able to build against modular content from the relapse streams, but we weren't able to actually have people build modules themselves for Apple 8. So now in 8.1, we've finally gotten through some of these hurdles and Mohan will talk a little bit about the configuration issues we had to face with that in bed. But at this point, it is possible to build modules for Apple and in fact it is now the default if you are building a package against Fedora, it will also build it against Apple 8. So you get a module and you get a module and you get a module but you don't deserve one and then you get a module and everybody gets a module. Thank you. Thank you. Stop. What about Apple 8.0 Playground modules? That's still a bit of a work in progress. It turns out that there's a little bit of difficulty in generating modules specifically for Apple 8.0 Playground. But it's also probably less important because you can always generate a stream for Apple 8.0 Standard that is more less stable shall we say and play with that and just make it and just allow people to opt into that. So it's probably not that big a deal but we are working to try and figure out how we can get it to also generate Playground. Probably by Raleigh, Dr. Dane. Yeah, probably around Raleigh 3. In conclusion, computers were a mistake. Also, don't trust a remote runner. If you have questions, you may direct them at this man. But seriously, we actually, we had planned this, we had put in this talk for a half hour segment and we were told we were getting half hour segment and then the schedule came out and we had an hour segment. So we're going to just blame him that his 15 minute portion of this talk was supposed to be now. Yeah, he was supposed to talk about more about Grobb's platter and he did actual set up on that thing. So he has more pain points on that. But you were going to go into a bit of detail about the configuration changes that we needed for... Oh, it's actually completed that. Yeah, we talked about that. A lot of changes. That was very vague. The configuration changes that I had to make in the problem side of things is the changes related to both the and how it works for Apple 8 and as well as the modules. Because as we were saying, we never actually enable modules from Apple side of things. So those are the biggest changes that we have to make and then how we sync them for the mirror manager side of things. That's another challenge. And the Composers. We never actually did Apple Composers nightly. So initially we started with Apple 8 Composers just with our RPMs but then we added Apple 8 modules so we have to add that as well. But those are something that are related to relevant side of things. Unless if you have more questions regarding technicality into that, we can... Why don't we open the floor up to general questions? I'm sure that you have some. I thought you have some. Yes. Sorry, the question was can I opt out of making Apple 8 modules? Yes, you can. You have to specify in your module medical. Sorry? It's funny that the person who asked that question was in fact one of the module team leads. I believe he is just being a bit wise. Sure. It's a common question he says. He was asking it by proxy. Come on, there's got to be a question out there somewhere. What about switching from module back to RPMs? Okay, starting with the hard questions. I like this guy. The question was what about switching back to RPMs from modules? The short answer is that right now that is a limitation in the technology that is being worked on. We are probably going to need new metadata for it and so we're going to need to make some changes to DNF to accommodate it. This is something that we don't just want. It is an absolute critical requirement and it will be resolved, but it isn't today. Yes. That's why I was not asking. I remember that when I was working with Apple, there used to be a problem that sometimes there was a newer version of a software that was in route. Somebody actually knew that. How do you know that situation now with modules? How do you make sure you have questions? Okay, so the question was sometimes there would be periods where Apple would have a newer version of an RPM than was available in RHEL. Usually that only happened over a short period and it was usually a result of miscommunication. What would end up happening is that RHEL would freeze at a certain point not let the maintainer in Apple know and then they would do a release that ended up having a higher one. Then usually what would happen is that we would end up just blocking and removing it from the Apple repositories. Today... If something like that happens, they will come back to release engineering and ask us like, hey, this is wrong. Then we will go back. Then we will block it in Koji. But that doesn't happen often. Usually communication is better than that. Also, one of the things that modularity is going to give us that's really nice is the fact that Apple can now start shipping module streams that are not available in RHEL of the same software. They can ship it even if it's in a RHEL RPM as opposed to a RHEL module. We now finally have the ability in Apple to say, yeah, I want to ship something newer than RHEL. Before we just had to say, you can't do that because you might break their system. Now they will at least have the opportunity to opt into that if they are willing to do so. The comment from Steven Tweedy was that modules override non-modular content. So you can build a module to replace anything that is in RHEL whether or not it was in a module in RHEL. You see that there will be a situation where migration of RHEL that Apple can ship possible. That means you can start something from Apple that you are unsupported. And then they will support migrations of system version of RHEL. I'm going to try to rephrase your question. So are you talking about system upgrades? Okay, so the upgrade process is still to be determined. Upgrades from something like RHEL 8 to a presumed RHEL 9 or 10 or whatever they decide to call it. That will largely be done in the realm of LEAP, which may in fact, that's one thing we should keep in mind is maybe we can have it just say, you know, yum disable any module that is in our whitelist. Do the upgrade and then see if you can turn them back on. If you can, great, if you can't, they can deal with it then. But that's a hypothetical. I haven't given that any thought more than that question, than since you asked that question. System. And they hurt, because the content streams for the patients and the models are not the same life cycle as your major release. So it just changes the nature of the problem slightly. It doesn't really introduce some substantially new problems, but we are looking to try to make it more seamless to do. So do I get that assumption wouldn't be that you would have like one person to turn it over, like you can relate to the real version itself. Yes. The source code is not related to the real version, but the boundaries are still built for this version. Yeah. I imagine probably none of that got onto the mic and I apologize, but I can't, I don't think I can re-create it all, so. Ideally, we would like that. Okay, I'm going to try and do a very quick summary. Steven said a lot of things that were right. Which honestly, everybody already knew. But, okay. So one of the things that modules help with is that we are going to be able, down the road, to be able to say, to build the same module for Fedora 29 and 31 and for REL 8 and REL 9 from the same git commit essentially, same exact source starting point. And then on upgrades, it should be able to just say, okay, give me the version of the same stream I have selected that matches the system to which I am upgrading. Doing that for REL is a major requirement and it's absolutely going to be done in advance of whatever the next new major release is called. With Apple, that'll probably have to be on Apple's side to make sure it works. And that will probably almost certainly trail the REL release by some months. Did I see a hand over there? We got time. Nothing but. You sir look like you have a question to ask me. Do you want me to ask a question? I do. I actually did have a question. So, do you have any policy guidance or guidelines so that if an Apple contributor says, oh, I'm going to module the rise of my stuff to kind of guide them to make sure that they understand the benefits of why and when to use modularity versus I'm just giving myself a little extra work and there's no real value. So, and also clarifying expectations, if you build a module, here's what we expect of you to maintain or compatibility and true upgrades, stuff like that. So, to get that in a nutshell, do we have packaging guidelines and advice to give potential packages of modules in Apple for when is building it as a module the right choice? When is it not optimal? That's okay. And if you do so, what are the trade-offs? What are you going to have to do to maintain it? What are your guarantees regarding its life cycle and so on? Some of that exists today. We're adding more as questions get asked because being engineers and experts of the technology, we're not always the right people to figure out what do you need to know to get it done. So, every time somebody comes to us with a question, we're adding to the docs at docs.fadoraproject.org under the modularity sections. Brian, the question was if I want to build modules in Apple, when do I get the tools to build it locally in my personal machine before I build it in Apple? Do you have them? They exist today. We have the ability to run the module build service in offline local only mode. You have to pre-cache the build-root stuff, but you can build it locally. It is not yet, MBS is not yet available in Apple itself, but you can build it from Fedora. And also, if anyone's interested, the set of steps necessary to accomplish that is actually not that high. I wrote a blog post about how you can actually fake it without any special tools, just the traditional RPM tools, and create repo stuff. So, I can link that to you later if you're interested. On this network, I'm not doing a live demo. We are lucky enough that Slides.com worked. Well, if there are no questions, Mike, you look like you have several, actually. Thank you. We have a question. Are there any plans for CentOS Stream and Apple? Plans is a strong word. My best guess is that CentOS Stream is going to end up looking fairly similar to what REL does. So, once we figure out the playground situation, it'll probably be the same solution that would allow us to just add a new platform that builds against CentOS Stream as well.