 Well, this is the talk about packaging Pantheon for Fedora, and I hope there's some interesting lessons for people to take home from this. Yeah, there's some more people. Do you want a previous slide for a photograph? Okay, first a few words about me. I'm Fabio, and you can't find me probably everywhere by my nickname, it's Deckerthorpe. I'm a package, an improvement package for Fedora, a member of packaging committee, Goal, SIG and Stewardship SIG, and a blog about the things that I do on Deckerthorpe.com. So elementary in Fedora, can this work? What do you think? History is based on Ubuntu LTS, and that on Debian-based system in Fedora is something entirely else, so maybe, not so sure about that. So short history of how this came to be. So the first thing I did was to build some manual bazaar snapshot builds of the Granite GTK plus toolkit extension library in copper, and it was in 2015, so it's almost four years ago that I started working on this. And I started with Granite because almost every other elementary package depends on it, so that seemed like a good starting point. And bazaar snapshots are not nice because bazaar is terrible, and elementary used launch pad at the beginning, and launch pad is also terrible, but that's another topic. One year later, the first official Fedora package of Granite happened, and that was also my first Fedora package overall, so that was at the end of 2016, or beginning 2017. Okay, that's a good start, yeah, 2016. Okay, then 35 new packages, and about two months later, most important things were packaged for Fedora, and Pentium session did technically work. There were a lot of rough edges around the whole thing, but technically it did work, and the caveats will be mentioned later. And then I had to add about 25 more additional, not really essential packages in 2017, 2018, and now almost the whole Pentium stack elementary packages are available on Fedora. And I wanted to make a shout out to Neil because he helped me with most of the package reviews, and that was really helpful because otherwise this would have taken not four years, but eight years. And how did I do this? Because it's not so easy to package 80 packages from upstream and not have things break and make this doable for one person, basically. I started building snapshots builds of upstream code in copper to catch issues early and tell upstream, hey, you broke something again, please fix it, or I will be very sad. And then I started building stable releases of projects that were not ready for inclusion in Fedora in another copper, first it was elementary stable, then I moved that stuff to elementary staging. And there's only, I think, five packages left in there, so everything else is in proper Fedora repositories. Then some services that are really helpful, Koshay, where I added all the packages. So even if something in raw height happens, which breaks my packages, I will know about this only a few hours after it happens, and then I have plenty of time to fix that before the next release. And all the packages are also tracked in release monitoring.org. So I will get notified about new releases every time. So I don't fall behind upstream here. So I think with the exception of one package, everything is up to date to the current upstream version right now. And the one package I can't update because it depends on unreleased downstream patches to matter. So I can't do anything about that because I can't patch matter in GNOME without screwing up GNOME workstation and I don't want to do that. Right. I also wrote a small helper program which runs automatic builds of Git snapshots now. It checks upstream every hour for new commits and builds these in copper. And I don't have to be involved with that at all. That runs on a small server and it submits those builds to copper and they build. And once in a while I check the copper repo and look in there, just did something break? Did upstream change something again or did they do something stupid? And left much conflict indicators in a translation file which breaks everything. Yeah. Those things happened a few times. Right. Now on to the interesting stuff. What are the awful issues that you have to deal with and build systems are really awful? Although things have been getting better. First elementary packages almost all used CMake build system and they didn't only use CMake, they also used some custom CMake modules that they took from somewhere else and bundled them in every package except in some packages where they didn't bundle them. And one of the issues the CMake build systems had in a few packages is that they just discarded all the C flags that were coming from the environment and overwrote them with dash G, which isn't really helpful for Fedora packages because that results in some hilarious build failures when the linker tells you, eh, you need to compile your objects with position independent code options and all that stuff. And I looked at the build logs and didn't know what was going wrong. Then I looked at CMake and saw, ah, it's discarding all the C flags. Okay. I submitted a fixed upstream and that was merged quickly. So that's all fixed now. Another issue in the build systems was that they hardcoded a lot of installation parts instead of detecting Liptier and Lipexectier automatically. And that was all tailored to Ubuntu specific paths which obviously don't work in Fedora because they hardcoded something as Liptier for, even for 64-bit systems so that's a whole other mess. And they didn't install things to Lipexectier because that's not really a thing in Ubuntu which is another issue. But those things have also been fixed. So things are getting better. Eh, yeah, the projects that relied on the external CMake modules didn't bundle them and they were only available as a Bazaar repository on Launchpad at an address that is here. And I didn't really feel comfortable with packaging this at first because I mean it's a bit, eh, looks like junk. But, yeah, that's the correct repository. It's been moved to GitHub now and it's been retired a few months ago because it's no longer used, but, yeah, that was necessary for one or two years, yeah. But upstream elementary has moved almost all of the projects to Mason now and that really helped with packaging and also it made it a lot easier for developers to submit fixes upstream because CMake isn't really that approachable, but Mason is really easy to work on and I also submitted some fixes for Mason build systems upstream. And yeah, that made these third-party CMake modules also obsolete. Right. Let's talk a bit about Bazaar and Launchpad. Eh, yeah, issue tracking and merge requests on Launchpad weren't really nice. I mean we're all used to the GitHub merge flow, workflow now, which is really easy compared to this and the Launchpad website was also a bit terrible. Well it's still terrible, but they don't use it anymore, eh, yeah. And sometimes the discoverability of projects and determining which package that's available on elementary OS comes from which source repository wasn't all that obvious because you get a PPA, the Personal Package Archive from Launchpad, but you can't tell which sources is this package built from. You have to do some really awkward digging in Launchpad to find out where does this come from, where's the source for this package? Eh, yeah, but that's better too now because all the projects are hosted on GitHub. Yeah, eh, right, it took some time, but I think now everything's on GitHub and it also makes contributing to upstream very easy compared to before and elementary developers have also noted that contributions to upstream have really spiked since they moved from Launchpad to GitHub just because people are more familiar to the workflow on GitHub. So it's very easy to fix issues now and to report bugs. Okay, take this with a grain of salt, but it's not too far from the truth I think. Yeah, it doesn't help that elementary is built on Ubuntu LTS because there are some issues with it. For example, the session settings for the Pentium session are always tailored to the latest Ubuntu LTS and that obviously isn't compatible with Fedora because Fedora has a whole lot of other changes on top of that, primarily a new agnome stack where all the components have different names or there are different components, agnome settings statement was split up into 25 binaries. I don't know, eh, I had to adapt to that and that's why I maintain a fork of this repository for Fedora and I maintain support for different Fedora releases in different branches and I try to merge upstream changes and changes that are necessary for Fedora into those different branches to make everything work. Then there are other issues that come from elementary hours being based on Ubuntu and that's because that's primarily system settings applications. For example, the date and time plug in the switch bot doesn't work at all in Fedora. I didn't really have time to look into it why it doesn't work, but I guess date and time zone handling works really differently on Debian and RPM based distributions so I don't want to touch that because I don't want to ship something that breaks people's machines. That's going to be a recurring thing and the locale settings also doesn't work on Fedora because it requires app statement to query installed language packs and to install language packs and that obviously doesn't work. I felt an issue to make this modular so there can be other backends to install language packs, but that issue hasn't moved anywhere so I'm not holding my breath on that one. Then the mouse and touchpad settings required patches for recent GNOME releases for years because Ubuntu shipped some ancient GNOME settings statement, G settings schema that elementary OS was relying on and they didn't want to patch it to use the new version so I had to carry downstream patch to use the version from current GNOME releases, which was a bit of a pain when you always have to rebase that patch to current upstream, but that's been fixed now because the latest Ubuntu LTS doesn't have that problem anymore. The user account settings are also not available yet because they depend on Ubuntu specific downstream modifications in account service, which is also interesting, but that's not really something I can do about. Then there's also some issues with GNOME that makes things a bit harder than they need to be, but not too many. The only thing that's really noteworthy here is that the elementary theme was broken for almost a year I think because of CSS styling changes that came with GTK 3.18, so it looked like crap for a year and I couldn't do anything about that. I just had to pester upstream, please port to the newer GTK 3 releases because I don't know what to do with this. That's been fixed sometime last year or two years ago. Then another thing that's interesting is that the Valor compiler introduces new checks and new errors every release, which makes at least three or five packages fail each time. I always have to backport upstream patches fixing these compiler issues, which it isn't really hard to do because at least there is upstream activity to fix those compiler warnings and errors, but it's additional work for me and I don't really like additional work. Then there's something that's really annoying that sometimes GNOME components change their D-Bus interfaces and sometimes they change G-Settings schemas and that produces some issues that are really hard to debug because either the application just crashes, then at least you know something's wrong, or something just stops working and it will take sometimes to find out why and there's an example about that later. Yeah, G-Settings I just mentioned and the window manager that's used for Pantheon, which is Gala, builds on top of libmatter, but Mata often breaks its API and ABI and breaks Pantheon in the process and Gala still only supports Mata versions up to 3.28 and the current Mata version in Fedora 30 is 3.32 and 3.34 is going to be released with Fedora 31, so I have to carry a compatibility package for older Mata libraries in Fedora 29 and 30, which is a bit interesting because it's not that easy to do and I have to carry a lot of patches to fix stuff and the packaging is also not really pretty, but it works now without too many issues, or with one maybe issue. Yeah and then that's a whole other topic, elementary really isn't used to being an upstream project because originally they only developed for their own operating system and they didn't really expect anyone to package this for another OS or they really didn't support it, but they've been really helpful with getting me through some issues and helping me fix stuff, so it's really getting better all the time. Some remaining issues is that, for example, Granite's library lacks support for parallel installability of different versions because even though the library does ABI bumps and bumps the library version, there is geobject introspection related files that don't bump their version every time, so those conflict and so I can't introduce a compatibility package for Granite, which leads to other problems because elementary apps usually always require the latest version of the Granite library because they soon require the new functions and functionality that's added with new Granite versions, so that means I can only update these applications in Rohite, even though the changes themselves to the applications might be small, they just depend on the version of Granite that I can't ship to all the Fedora releases, so sometimes stable Fedora releases don't get all the application updates, but that's not something it can really help with because they require Granite release that I can only package for Rohite, but things will trickle down the stable releases eventually. That's related to the last point, they don't really care about ABI's stability at all because they just push everything to their elementary stable release and don't really support any other use cases, so they just push new Granite versions even though it bumps the library version and they just rebuild everything and they push everything to stable and hope nothing breaks and most of the time it works, sometimes it doesn't, sometimes it breaks third party applications that are built for elementary and that's not really nice towards these developers as well, but yeah, it's getting better, maybe I can get them to properly version the libraries so I can at least install them in parallel for Fedora. Another interesting point is that there's no releases for the Gala window manager, so I always have to package Git snapshots and there's no release on the horizon, they claim master is stable, but it's always a bit of an educated guess whether I should include that commit for Fedora or should I not include it or which branches for Fedora should I build this, so that's a bit of guesswork that's involved here and I hope I don't break things too often, but it looks pretty good so far. Yeah, now I wanted to mention the current state on top of Fedora 30, there's still some minor issues left. Locking the session does only work on top of LightDM because the session locking and screenshot component they use for Pantheon which is LightLocker doesn't support running on GDM which is kind of obvious, but there's not really an alternative to LightLocker for Pantheon that supports GDM, so it just doesn't work if you start Pantheon from GDM. Then I had to, after some numerous issues I had to remove Upsenter from Fedora 30, for example because it crashed Deeper's broker and completely bricked Fedora if you had it installed and by bricked, I mean bricked, you can't, keyboard didn't accept key presses, you can't get to a terminal, you can't get to a virtual, to a DTY, nothing worked, just setting a system D target to single user mode worked and then uninstalling and installing Upsenter brought the system back to life, but that's not really something I want users to experience and that's one of the reasons why I think Upsenter isn't really ready for Fedora yet, but yeah, so maybe it will come back in the future, maybe it will have better support for DNF back end in package git, but yeah, at the moment it's gone. Then another issue that occurred last minute before the Fedora 30 release was that media key is no longer work, that was a bit of a surprise to me, but that's caused by norm setting statement 332 changing its debars interface, so Gader can no longer talk with the G setting statement, so they can't exchange media key events anymore, so they don't work anymore, but this still works in Fedora 29, at least that, then some components are still not packaged for Fedora, some due to the issues I mentioned and the elementary mail application isn't packaged because it still depends on WebKit GTK 3 and that's been retired from Fedora I think three years ago due to security concerns because it's obsolete and outdated and everything, but yeah, there's an ongoing port to WebKit 2 GTK 3, but that's not released as stable yet, so maybe it will come at some point to Fedora, there's nightly builds happening in the couple repository for everyone who wants to try it out, but it's not been released as stable yet. Yeah, that might sound like there's a lot of issues, but honestly basically everything should just work except those minor issues and they even get some feedback from users, for example this one, thank you for great work, Francion on Fedora feels much more stable even than on elementary OS, which is really nice feedback to get when you're working on something alone instead of a team producing elementary OS. Yeah, so that's everything I wanted to say, just two small points for possible conclusions. Ubuntu sometimes doesn't do its downstreams any big favors by porting ancient stuff to current releases just for maintaining compatibility, for example they often patch things to include old G settings schemas in new versions just so they don't break existing applications, but applications continue relying on the old locations for those keys and so nothing gets ported to the new versions and things keep relying on the deprecated ones, which doesn't really help and maybe elementary should start forking some GNOME components instead of piggybacking up on them, for example GNOME settings statement would be a good candidate for that because sharing that with GNOME creates all sort of weird issues and it will also resolve the issue that it often breaks debuts and G settings interfaces, but yeah, I don't know how feasible that is, but it would be a good candidate to resolve some issues. How late is it? Not too late, well almost too late. Are there any questions? So the question was if elementary upstream was helpful with getting me things fixed for other downstream distributions and I have to say at the beginning they were a bit hesitant because I pastored them with a lot of issues, but they've also been really helpful and we've managed to fix almost everything as you can see and I've also contributed some things back upstream, so for example I ported the granite library to Mason, which also made it easier for me and so yeah, there's contributions flowing both ways. Other questions? No. Okay, now I have 30 seconds for a short demo. I'm already gone. This is how it looks like and everything works. Oops, it doesn't work. Okay, thanks for coming to my talk.