 My name is Jaco Selkowicz. Thank you for joining today. I am an engineer with Red Hat, and I'm involved in Fedora in a number of different capacities. My main responsibilities are in ELN, to help us get ready for REL10, as well as helping with flat packs, which I'll discuss today, and that has taken me into a bunch of different areas as well, and to probably, if you were maintaining a package you may have seen me file a PR sometime in the last while. I started with Fedora about 12 years ago. I think it was Fedora 14, coming from the SIGWIN community, where I then joined Red Hat a couple years later. That's about nine and a half years now. And today we're going to talk about flat packs without modules. This is a change that was introduced in Fedora 39, and I'll go ahead and get started. A quick recap of flat packs in general. Flat packs are containerized desktop applications. So they are intended for graphical applications around any desktop. And while they are of particular importance for immutable or atomic desktops that we have nowadays, they are also just as usable on your traditional workstation or KDE's Plasm-Spin or the other traditional RPM DNF installations as well. They provide a number of advantages in both cases. They work across distributions and versions, so because they come with their own runtimes, they are containerized. So they're not dependent on the exact distribution and version that you're running underneath, because they come with their own runtime that provides all that. The compatibility needs to be there at the kernel level, but generally speaking, that's not often as you can run, for example, a modern flat pack on RAL, even though there's a couple of years' distance between them in terms of versions and so on. So it isolates things as that as well. It also provides a sandboxing and permission model that you cannot get with your traditional installation, and that while the default permissions of each app are defined by the app when it's built, they are actually adjustable, and there are apps that allow you to adjust those permissions, add or take away those permissions, and do those as well. So that provides a very important model that we see also in nowadays in mobile apps and so on. This brings that kind of permission RAL to all the next desktops in a way that was not possible before. It also allows multiple forms of installation. You can have a system installation, as you typically see with a distribution package, and also allows a user to install their own packages or their own versions of a flat pack separate from those on the system, either parallel to those on the system or instead of those on the system, there isn't one even. It gives you that flexibility to adjust how you install as well, and that's again something that you don't get from an RPM. You can't really locally install it. If you install an RPM, you've got to install it to the system, you have to have the race to install it to the system, and so on. This gives you some flexibility in terms of how and where to install it as well. So flatbacks consist of two parts. There's a, as I mentioned, the runtime, which are the base dependencies that every, that most apps of that type will use, and that provides that infrastructure, and then there is the parts that are specific to that individual application, and they're tied together. An application says, I depend on this particular runtime, this particular branch of a runtime, so right now most of our Fedora flatbacks are using a Fedora 39 runtime, and we have a few because we have those that are primarily tentative for NoMaps, and then those for KDE apps, and so we have a couple of runtimes. If I'm a KDE app, I'm saying I'm going to depend on the KDE Fedora 39 runtime, and that will update itself periodically because we keep that up to date, but that app is a separate thing, and goes on top of that on that runtime. The way flatbacks work is there's an isolation between the two images, that the runtime images, which Fedora, they're built from the same RPMs that you would install on your system, those go into the user prefix, and the application packages are rebuilt to go into the app prefix, and those can only be in the app prefix, that keeps them to separate. That doesn't mean we have to rebuild the packages that go into apps that are not part of runtimes, but this is the model that flatbacks use to provide the isolation between the application and the runtime, and every so every flatback only includes the pieces that are specific to that application that are not part of runtimes. That's flatbacks in general and a little bit of how we do in Fedora. Now, Fedora flatbacks have a long history, started about five years ago. In Fedora 29, I was not involved then. Owen Taylor and Helen Lambert have been rebuilt for quite a while. I'm a recent newcomer, relatively speaking. I used to go back about five years from Fedora 29, and when they were originally created, they were based on modularity. We'll give it, get back to that in a minute. You can see on the right on this chart shows the progression of flatbacks, the number of flatbacks we provided in Fedora from the beginning until the last release. Now, in Fedora 39, we already have most of the flatbacks we had going into by the end of 38. Most of those are already available for 39. There are more coming. We do have about a dozen or so that we need that are still going to be caught up to Fedora 39, but these numbers actually are kind of the end of the release. We finished building flatbacks for Fedora 38, and we switched focus to 39. We left off about 300 flatbacks, and we're picking up from there. By the time we're done with Fedora 39, I expect that number to be much, much higher. We're still there at this point, but you see there's a big progression. I joined during the flatback effort during Fedora 37, and you'll see with extra hands on deck that helped us jump that number considerably, and it's gone up where it's since then. I expect that by the time we're doing Fedora 39, that number will be much, much higher. We mentioned that when flatbacks started and then Fedora 38, it was all based on modularity. A lot of this predates my time, but the question is, why did we do modularity? Well, back in 2018, modularity was a big thing. We thought that modularity was going to have a major impact on packaging going forward. Obviously, they didn't come to fruition, and so we're changing course. It did allow for certain things that typically an RPM builds didn't, wasn't necessarily out of the box, but we found other ways to do those things. So modularity became less important to the effort and never really caught on in the first place. So we have changed course, and the reason we've changed course is the reasons why we're moving away. One of the big things against modularity was, frankly, it required a lot of infrastructure. And so we had to have a whole service, the modular build service, just to build the packages that would go into our flat packs. And even to build the runtimes, we had to build a module. We didn't have to, we don't rebuild the packages, but just to build, we had to build a module just to build a runtime, even though we weren't building RPMs for that. And so that, that's a lot of overhead. And then we, the way we're building the containers, because that's how we ship flat packs in Fedora, as we make them into a container, we ship the container, and then flat pack imports those, those OCI containers into the system and, and integrates them into the flat pack system there. So we had to build those containers, we were using OSBS and OSBS version one, which is what we've been using, is based on Obershift 3.11, that is coming towards end of support, if it hasn't already. And to, you know, the amount of, again, that's, that requires a whole cluster per architecture. And so adding architectures, it was difficult to maintain that a lot of overhead and infrastructure required to maintain this path that we've kind of been able, we're going to be able to get rid of, through the new, the new way we're doing things. For packages also, it, well, there was overhead as well for them, because you had to write a module empty. And there was a, there, there is a tool that allows you to do that kind of speeds up the process. But you had it, you still had to know how to deal with it. You also had to know how to order your dependencies yourself, because that the tool that wrote the module empty for you would give you the packages that you'd require, but it wouldn't tell you how to, the dependency order of them. And so you'd have to go figure out that yourself. And that was sometimes a trial and error, an error effort. You'd build the module, then you try to building the flat pack container and find out that now it didn't work because of something else. So then you have to go back and read, try rebuild, rebuilding the module with some adjustments and then rebuild the container and so on it. Again, a lot of overhead for both this engineering infrastructure as well as for developers. It also was hard to build locally for testing purposes, because you had, you had that module build service in the way, and there, there was a way to do it, but it was a little hackish, tricky. So eventually we found out that, I found out, but we discovered that there are ways to do some of the things that we need a modular to do in terms of handling, handling options and separating from the regular RPM builds that we probably don't want to interrupt those. We were able to find ways of doing this with some creative creation of Koji tags and tag options and so on. So we found there was an easier, simpler way to do this. And that brings us to this, the change in Fedora 39. What, what, what did we do in Fedora 39? We, but it's really, it comes down to a few things. Flat pack module tools was rewritten. That was, that was a, that was a tool that we were building with until now, and we're still using it. And yes, it's still called module. It's still in the name. We didn't rename it. Naming things is hard. There's also some common code in between them, because we couldn't, we're not quite ready to, to get rid of build stuff yet until we've closed it out. So we didn't, we didn't make a whole new thing. We just kind of rewrote what we had, and I said we were mostly Owen. And so we, we, yeah, it's the same name, but it was, there's a considerable jump in version number and to reflect the change in the code that involves all of these changes. So you, Fedora, Flatpack module tools, which has the Flatpack module command line, and it is the kind of the main driver for developers and also is used on the infrastructure as well. The other main addition to the R39 is the Koji Flatpack, which is a Koji plugin. It's not something you run on your system, but that's used in Fedora infrastructure to do all of the buildings in Koji. Therefore, a no more module build service, no more OSBS, everything happens in Koji and the Koji Flatpack plugin is what enables that. And then we, we've added a whole bunch of Koji tags, targets and disc tags. These are the hierarchies, it's just the base names of all of them. There's obviously sub tags and so on. It's all structured to support this, but in Koji, that's not a big deal. So we have Fedora 39, that name will presumably continue in future versions. We have our runtime tag, tag hierarchy, a app, tag hierarchy, and then the Flatpack hierarchy itself is where the actual Flatpack builds go. The runtime hierarchy is for certain things we need to add to the runtimes and to the app builder as well. Things go there. It's just really a handful of packages. Those packages have a specific disc tag. The bulk of what you'll see is the Flatpack app builds. And those are builds of Fedora content, same spec files that used to build RPMs, but then built in these tags with a special disc tag and they're built into the app prefix. And when you use the proper, when you properly use installation path macros in your spec file, it's very easy for the prefix to be changed in that way and for the builds to continue. And when you don't, you usually get a PR for me saying, fix Flatpack build, I've set up for you. And then we have the Flatpack itself go into the final tag and that's where, and those are the ones that actually get shipped as updates and released through our Flatpack channels. So this is the kind of what we've, what we accomplished in Fedora 39 and what's, what would actually changed if you will. What does that gain? What do we gain from this all? A lot of change, what do we gain from it all? If you're just using Fedora Flatpacks, you're not going to see a whole lot of difference because in the end of the day, yeah, your Flatpacks are Fedora 39 content, you would expect that that anyway, that would have happened either way. Not a huge amount of change. One minor benefit that you do get though is duplication. Well, with Flatpacks, you have to, you download the entire thing because we ship Flatpacks as containers in Fedora. But when you, when Flatpack downloads it, it then duplicates on disk. So when you have a commonly used dependency that's not part of a runtime, for example, boost, commonly used library, various things use it, it's not included in the runtimes. And our runtimes are based on communities, Flatpack runtimes, try to have a certain degree of equivalency there, although it's not exact for various reasons. But boost is an example of something that's widely used, but, but not part of a runtime. In previously, you would have to have a separate build for every module or, or in some case, we think like boost, we actually had common modules, but every common module had had to have a match the runtime. So we ended up still ended up having two or three builds of boost at a minimum. And sometimes we had many more than that of the same package built different modules. Well, now every, every package is built once for Flatpacks and all Flatpacks use that same build. And so when you download multiple apps that use the same dependency like boost or like the game support libraries that both know and KDE have, you get the exact same bits literally. It is the same package built on Koji into the, into the Flatpack app tags, and they all include it. So they will do duplicate, they are literally the same bits. And so that is one thing that you'll see you may see as a user is that it takes things, take a little, a little less space. Another thing we were able to add that we weren't able, well, it would have been difficult to do before because of the requirements of OSBS is to add an architecture. We were able to add Power Little Endian to match the, so that we have now X8664, AR64 and now Power Little Endian that matches the architectures which we provide Silver Blue and Kino White. So we have matching, matching architecture support there. That would have, you know, on Koji, which is now in door 39, we're just using Koji, that's going forward. Before that would have required having the infrastructure to set up an entire openshift cluster with OSBS. That would have, that was difficult. So that is something that was, it was an easy win for us as part of this switch. For our friends in release engineering and infrastructure, you already know this, but this is, this is so much easier. And for them, and this is much less overhead. And we're, we're even going to be able to start decommissioning some stuff that we were the main or last remaining users of. In fact, OSBS is on its last days probably until we just got to wrap up some things. And they're already looking to pull a plug on that. So for them, they are, I thought, not to speak on them. For everything I heard from them, they are very much enjoying this as well. For packages, this is where you see the biggest gains is it is so much simpler to package flat packs now that we, that we have this, the modularity out of the way. You have one file. It's a container-diamol file. It is small. You'll see it in a moment. Very straightforward, self-included. You don't have the whole module in D. Dependencies are taken care of for you by the flat-pack module tools. You don't have to list out every single dependency and figure out the dependency or yourself. We have a depth chaser that includes, that takes care of that for you. We also don't have to deal with having to make empty commits just to bump the modules that module builds over small and rebuild it. You don't have to do that. Container-diamol, you literally could leave the container-diamol untouched for six months between, you know, for each release. And even though you're updating the RPM all the time, container-diamol will still pick up the newest packages. You don't have to bump commits or have empty commits in this case. Just keep using it. In Koji, things are separated better. They're list packages, sort of a packages, flat-pack packages, if you will, which are objects. Those are separately there. And so, for example, if you have GX as an app, just take a simple example. Before I used to have a GX RPM, a GX modular RPM, then you had a GX module. And then you had a GX flat-pack, all of which showed up in the GX quote-unquote package in Koji. Now, you don't have half of that. You have the regular RPM and the modular RPM. They're both part of that. They're both builds of the same package. And then you have a GX-flat-pack quote-unquote package in Koji. That's where the flat-pack builds go. So, clutters, things last. Also, those builds have meaningful version numbers now, rather than having a modular version number of 38, 20, 20, 301, whatever, a whole D-stamp appended to it. It shows you which version of Fedora is based on, which window is built, but it doesn't tell you what version of the app is in that flat-pack build. Now, the version of the quote-unquote main package in the flat-pack is the version of the flat-pack. And so, you can actually see, oh, this flat-pack has this version of the app in it. But it's a little more visible and so on. Things are faster because we don't have so much overhead. We've talked about shared dependencies already. That also makes things easier for packages. Local builds are also easier because we don't have modular builds in the way. And when we need to intervene and kind of build things step-by-step or build packages manually, we can do that. We're not restrained by having to build things through the modules, while flat-pack module tools does make it a lot simpler. Sometimes some hand-holding is necessary bootstrapping things like Perl, for instance. For bootstrap, Perl for our flat-packs definitely required some intervention, like most bootstrapping things do. We were able to do that and then everyone can use that content, which is what we get from that later. So, not a lot of time left, but let me do a quick demonstration of what this looks like in action. I'm going to switch screens in a moment here. Bear with me for a moment. So, what does it look like a flat-pack? For the sake of the flat-packs, I'm going to take this example of an app called Minder. And I just discovered this yesterday. I'll make a directory. It doesn't matter what the directory is called, by the way. At this stage, later you'll want to have a sensible name, but unlike with modules, the directory name doesn't actually matter at this stage. So, I'm going to do a thing here and I'm going to let FlatHub, you know, this is the app that's on FlatHub. And I want to say, I want to say we have, but we also have it on Fedora. I want to have a Fedora flatpack out of it, so I can use the information from FlatHub, including things like permissions and so on. So, to automatically create the container.yaml for Fedora. Now, you'll see here that it says, oh, you've got to be a little more specific about your app name, because that wasn't quite enough. Now, that's not a surprise because I've looked through this. Try this again and container.yaml. I will say that, by the way, that the flat-pack module ended command to use currently in testing. So, you don't have access to it today unless you're using the pull request, but hopefully it will be something that we get to everybody soon, because it makes this step so much simpler when you already have an application on FlatHub. Let me use it as a model. So, this is the container.yaml. This defines, and this is all that we need to define, the creation of the flat-pack for Fedora. And it has the ID, which is how all flatpacks are referred to by an ID in that form. So, what runtime we're using, which packages that we're including in the flat-pack, and then we tell how to start the flat-pack and how to run it into the words and the permissions that this flat-pack needs. Everything looks good. So, I'm going to go ahead and get this started. Pack module, build RPMs, local auto. I'll get this started and then I'll explain what's going on in the countries of time. So, what does it do? And it was mentioned before that it takes care of dependencies for you. All I need to say is, I want the Minder package to go into my flat-pack. Flat-pack module tools includes a depth chaser that says, okay, you want the Minder package. Well, the Minder package has all these dependencies. Now, the dependencies that are in the runtime, I don't need to worry about the runtime provides this. I've already specified which runtime I'm using that provides these. There are, however, in this particular package, and in many cases, a few dependencies that are not part of a runtime. They're not used widely, but this package, or a few packages will use it. It doesn't justify inclusion in the runtime, but it is used. You will see already, however, that these have already been built for the benefit of other runtimes. We already have these packages built, and therefore I don't need to build them again. It will use the builds that are already there in Koji. What I'm doing here is a local build of just the piece that's missing, and right now is the Minder app. It's the flat-pack module tools when you do local builds. Like it does on Koji, it uses mock builds to create the RPM builds, but the RPM builds with the app prefix, and certain other things modified with macros to make sure that they conform to flat-pack requirements. And so it's right now doing a local mock build of Minder with an app prefix and so on, with all the requirements of the flat-pack app. Doing that will just take a minute or two. See, it's flat-pack module tools is doing a lot of the work for me already, and this will just take one more minute, hopefully. Oh, I think we're just about done. And of course, when I was preparing this, it seemed to be a little shorter. Always takes a little longer when it's live, right? This is not a large package, which should be done fairly quickly. I do see that we're running short on time, so I'm going to try to cut this really short for this. And there we go. The RPM build is finishing up and it's cleaning up the builds. Those of you who have done live demos before will know how this is like. I'm going to build a container. I don't know about time to finish this, but now it's going to do a local install of the container. Basically, it takes up those packages, installs it into a mock route, takes only the stuff that's an app, and then packages it up into a container. Then the dash-install will actually go ahead and do a separate step to install it on my system. I could then run that, see that it's working, submit it. I'm going to have to cut that out of the demo because I'm running out of time. But this is really how simple it can be to make a flatback for Fedora. In the interest of time, I'm going to have to cut this short and go back to the slide deck. I'm just about done here. Again, I'm not going to read it out of time, but there's a lot more apps that can be built now in Fedora 39 that we weren't able to do before. And there's more stuff coming, so stay tuned and we're going to keep lowering the bar to make this easier and easier for people to contribute to Fedora Flatbacks to Fedora and to really grow that base so that we can provide these to all of you with what we expect for Fedora content. So I don't know if there's much time left, but I'm over, but thank you for listening. I don't know if we have time. I'll take questions, but I do realize we're running a little late. Thanks so much, Yakov. I think we can squeeze in a little bit of time for some questions. We have three in the Q&A tab, so I think we can just do a quick run-through on those. The first one is from Neil Gampa. Do we now have good tooling for building flatback artifacts locally before submitting to Koji? Yes, and that's the demo was basically maybe that question went up before the demo, but yes, it is very easy to do a local build. If you can do a mock build locally, you can do a Fedora Flatback module build. With a flatback module tool, do local builds is very straightforward. No hacks required. I do this all the time. And the one caveat at the moment is that the package doesn't already need to be imported into Koji. It has already kind of been Fedora as an RPM, and then you can do it. Right now there is a dependency on having that there. Being able to do this before the package exists in Fedora and RPMs is something we'll need to work on, but the package has already existed in Fedora, as you see that it's very racer for local builds. And a follow-up question from Neil, would we be able to build RISC-5 flatbacks? We don't have the infrastructure for that at this time, because RISC-5 is not part of Koji, I guess. I would have to defer that question to those who are dealing with RISC-5, Koji in terms of that, in terms of local builds of an architecture that are not supported. I hadn't thought of that question. It's a good question. I would have to have a decision in terms of what's possible to do locally when on an architecture that isn't supported. It's a good question. I don't have an answer at this time, but I'd be happy to discuss it in the channel. Sure. And then our last question comes from Michael Gruber. How can we deal with different library versions, which are not in the runtime, such as is there a way to bundle in the flat pack without bundling in the RPM to different library versions? Not quite sure I got the question, but Fedora flatbacks are built from RPM content. As you see that there are no means to build something only in a flat pack like a flatback builder. We can see that as an advantage. It brings quality to for a flatback, so you don't necessarily get on flat hub, but in terms of picking a specific version of a dependency for a flat pack, we can do that on across the versions, across all the flatbacks in a particular version of Fedora. We could do that. I don't know that we have the means of this time to say this flatback needs this version of the same package, and this flatback needs a different version of the same package. If you have multiple versions of a package that need to be supported in parallel, they should be separate packages in the Koji terms in Fedora. They should be separate, and then you can mix and match like that. But it is one trade-off that we made when giving up modularity is that you don't have quite that flexibility, but generally speaking, we didn't use it. So we can pick a version of something that we're using for all the flatpacks in Fedora, for that version of Fedora, but not for individual flatpacks unless they are separate packages in Fedora, which in some cases maybe they should be. But if there's a specific issue, again, I'd love to discuss it in the channel and see if we can figure out what needs to be solved here.