 Thank you. Well, welcome to my talk on set of set topic upstream and software distribution So let's jump right into it. What is actually the problem? We want to solve with the upstream project at free desktop the main issue is that the software we currently display on package managers and Well, these advanced tools we use for package management and Linux are displayed as packages Which sounds weird at first is pretty simple because a package is a technical item And it's not very discoverable for the user for the end average end user who is a non-technical person What a package actually means there at least in Debian tons of packages which are containing developer and symbols or Documentation or something which your user is not directly interested in because he usually or she usually wants to display Display and install an application or something else, which is more simple to discover for for the average non-technical user Another thing is that the applications if they are presented as applications are not presented well currently This means that we have different levels of localization of the descriptions of them That the quality of screenshots if there are any is quite bad because they are usually submitted by users Some are also very good, of course, but yeah generally It's not under the control of the application author who wants this application presented in a way He wants it wants others to look at it Then they are missing URLs and especially lots of differences between distributions because for each package every distributor currently writes It's its own description of the content. So on Debian We have some some descriptions the door does their does their own and so there's no Unification in that regard and the lots of duplication on efforts Also software centers at distribution specific. This means that on each distribution we currently write a new software center which which open problems and Well, not perfect integration into the desk of environment So in order to fix this we would need as common standard on how we handle applications and how we display them in In order to even make an attempt to create a distribution agnostic software center So the solution is upstream After him is an XML metadata specification of software components and I'm not talking just of applications here So for components means anything from a generic any kind of software software component to firmware Which is something currently in development with the firmware update tool Which also uses upstream to desktop applications, which is what this talk is mainly about to add ons which are extensions to To existing software or other components to codex to inputs methods and even to fonts Which is also feature which is currently in development and which GNOME already uses Those metadata Files are created by upstream projects But distributions are still required to review them and to pack them together in order to make it possible to ship them to the to the user and to display them in the final software centers, however, all the work in Translating the data and writing the data is done by upstream and by the people who know the application They want to be presented in the downstream software center best Yeah, and also the upstream specification itself is not Completely self-contained. We don't want to duplicate efforts. Therefore it Extends existing metadata for example from desktop files This means if you have a summary in your desktop file already, you don't need to write the second summary in your upstream metadata file so To get a bit more concrete on this how does it look? Oh, sorry. Yeah, and it's installed into user share up data if you want to ship it on the application You should drop the XML in there. So let's get a bit more concrete. How does it look like and what does it contain? For example, first of all and most importantly every upstream upstream XML files kind of tricky there Contains an ID which is in case of in case of a desktop application the desktop file name of this application and this ID makes it Makes it possible to find an application on any distribution out there This means if you have an ID you can if you are in any distribution get to the package name and install it this is especially important if you want to install an add-on to your software, which We just need to query the ID of this of this add-on you want to install for a software and Then call the distribution package manager to install the package which matches which contains this software component Defined by the upstream XML. This is this is really nice because you then need one only one code path to target or existing distributions Which might ship upstream data? So yeah, and this is also used to associate the desktop file with the up data upstream up data and So it's important to keep this in line with with a desktop file in case you're packaging Packaging an application which is the desktop application Then we have a long description, which is even translatable upstream so the effort of translating long descriptions of software is not It's not read done in every distribution anymore, but it's translated once for every upstream project and it contains lots of other stuff like Project links to something you use as my fight interesting and browsing the application in in a software center like noon or Apple or Gnome software for example a home page a place where you can report bugs or and help your L Where you can get to some more extensive documentation on how on the problems you might discover when working with the software? You also can define your own screenshots and multiple of them and even with captions so You can display your software the way you wanted to be shown in the software center with your own screenshots instead of relying on some user submitted screenshots This is still possible though. If you don't provide screenshots the distributor might Just ship something which was provided by users, which is what we do at Debian for example And there's also an option to to add technical items which are provided by this component This is useful if you want for example command line help if you are searching for a binary and want to know in which In which application it is in then you can then this information can be used by the system to determine Yeah, which Which software component needs to be installed in order to provide this this yeah this technical items This is for example the case with mine types if you need to have a new file type And I know which with which application we open it you can carry the upstream metadata And you will tell you a list of applications that are available which Can handle this my mind type the upstream upstream upstream xml. Sorry. It's a really bad name there There's also distribution xml with this basically The component metadata described in the upstream xml fused together in one big file This file is a general file on Debian because of reasons or xml in any other distribution Yeah, this that is a general file in Debian has mainly political reasons and yeah reasons of Specific to Debian so you don't need to care about that much Yeah, and this distribution xml file makes the data accessible for software centers and provides said component package name mapping Which is essential in order to fetch data fetch packages from the package manager which measures component Yeah, and what distributions also do is merging metadata from desktop files and other metadata They might find for your application into that xml so everything is available in one file So how does it look like as a diagram you have this compose server at the distribution which has this package where you As an upstream where your project is packaged containing desktop files and maybe the xml metadata what they compose survey the distributor does is to extract that metadata and Construct new xml out of it and also extract icons which are used to display this application Then cash screenshots and do a lot of other stuff This is the mirrored on the standard distribution mirror and shipped to the client where it is transformed into a safety and database In some cases or just read directly by an upstream supporting library Then you have your software center which takes this information and communicates with a package manager in this case through the through package kit which is a Distribution independent package management abstraction layer which contains back ends for I think almost all Distributions out there so it's really a convenient way if you want to stop packages in a disorganistic way and Yeah, makes it possible to itself and move applications by that and also pulls in some existing metadata in case there's for example No long description from the upstream file. We can still fetch the trust description from the package itself There's also a feedback server plan for ratings and comments from users But as I know web developer so far there has the specification does not contain anything about that So in case this will happen later, I'm happy to add something saying to the specification, of course If you want to access upstream, there are different options for you And different libraries what I would strongly recommend against is to implement a parser for this XML or jammer by yourself Because it can grow pretty big pretty quickly Especially because you need to support Kubuntu Ubuntu and Debian you need to you need to pass the Jammer files to so there's lip upstream which uses the sapien cache and Can also access the XML and jammer direct directly in case you need it for some reason Sapien is of course faster than the XML jammer implementation. There's lip after you cut cute, which is I think most relevant here which uses the sapien cache access and there's also which you might run across the lip upstream glib library developed and mainly you said no, but also many other projects which That's direct XML and jamming parsing and which is this thing GNOME software is based on This is written by which to use why those two are maintained at free desktop Yeah, and so don't be afraid for the next slide because there's a GNOME screenshot on it This is how it looks like in GNOME software with a long description just to get an impression What the final result is multi paragraph translated description multiple Multiple screenshots and there's also a way to display the add-ons you can install in a very user-friendly way and licensing information and Yeah, of course some technical data like category categories or sources from which repository came Also the other items I mentioned before like input methods or codecs or even fonts can be represented in a software center That way even with Examples for how the font might look like which you won't get in a package manager where you need to know what you're looking for in order to install the front So that's it for the upstream specification So but there's a different issue this talk is also about and that is the big field of distributing applications But yeah traditionally you I'm just looking at the time here traditionally a Linux distribution is an operating system and applications and An operating system is meant to be stable and doesn't shouldn't change much because you don't want surprises and Accidental breakage while applications are something different where you want the latest application the latest and greatest and Newest application shipped to your users directly without going to distribution packaging Currently the way is that you as a upstream wide software Which is packed where just a package or which at the end Goes into a distribution and remains their stable software and you can't tell your users if you for example get it back Yeah, I fixed that in the latest release. Can you please try it out because users might need to upgrade their distribution in order to do that? or use a PPA and all these different entities have different motivations and different Different things they want and all everything is saying in that way, but yeah this whole way of having an upstream which Which produces software having a packed by a package or having it in a distribution is quite takes quite a long time and makes trying new applications Much harder a rolling release distribution is of course an option, but rolling releases have their own issues like Yeah breaking sometimes and usually users want their old as stable and also be able to roll back to an older software Application version in case the new one doesn't well doesn't work as they wanted to so Short why not use PPAs? PPAs have a large security risk because they are installed with repermission and especially annoying for for people who are developing a distribution is that some people develop PPAs which interfere with distribution upgrades So they they mess with the in the namespace of the normal distribution with the packages And if the user wants to upgrade for example to a new we're going to release and there are plenty of Plenty of things where it can where this upgrade can fail in case some PPA contains a package Which is not in line with the distribution packaging So a solution to that is limba, which is a software and solution solution I I manage a develop which Subdivides the current distribution into multiple layers, which is the operating system layer Including the kernel xorg and wayland debas system defaults audio and all that stuff you would expect in a generic desktop Linux But not more so not For example, no toolkits. It's just the stable interfaces which are not expected to change then there are run times Which continue which are shared between applications, which contain all this toolkit libraries and and Different pieces one an application needs to in order to run Yeah, limba itself contains Produces bundles, which are very similar to packages which contains said upstream metadata dependency information. So which Which runtime components they need signature and payload Which is just the stuff you would usually ship in a package The main difference is that those those runtime those bundles are created by upstreams So in this case in this scenario to the Qt project would provide bundles limba bundles for Qt core Qt UI and Another stuff and you are send an upstream application developer would also create a bundle depending on those parts So I know if you want a newer version you just need to depend on a higher version of that You can simply imagine it if you are a python developer as a spy pie, which is not python specific um Yeah, in order to in order to make this work limba uses some some new kernel features like shared subtrees and recently added overlay FS and uses those features to to glue the runtime and the application together and to remove duplication between layers So if you have have one copy of Qt already, which is sufficient You don't when you will not get another one installed and he will also not get static linkage If you don't just explicitly do it because it's not needed in this scenario. You just need to depend on a different on a different limba bundle with a higher region So this whole layer is is glued together by an overlay FS mount when running the application It will also be that sandbox in future. Maybe but yeah, this is something another project is working on This is the illustration again the OS is debian then you have this individual components STL For example Qt G stream or whatever library your software once and you have the application on top of it You're making use of those bundles those ones are easily created You just need absolute metadata and make a few changes on your build system Yeah, I will go quickly over that. I think I've said most of it already One thing is which is in development is limba hub is a central place to upload your limba bundles, too But this is because I'm not adapt developer developing slowly. So maybe you will hear of it in future And yeah limba produces absolute metadata which is integrated in your software center So your software center will display not only at software, which is available in the in the distribution package manager But also which is available in limba package sources and for the user The user won't know if it's from the from the software of one notice if it's from a limba bundle or from the native distribution package manager simply because it integrates really well into the system So there's another solution developed by people at redhead and by Alexander Alexander Larson Which I will just briefly mention because there's a talk tomorrow about it by Daniel Roteal who will be going into detail about it Which centers around an OS and a static runtime. So it's not composed of different parts but it's a large piece of stuff and Yeah, this application is sandboxed it sandboxed why simply to protect the user's privacy and Don't make it access parts of the system which Which are not allowed for access. This is especially important in the mobile space, of course Yeah, those parts are always three repositories. I will just refer to the talk talk tomorrow We're about that so simply because of time reasons Yeah, and exegia parts are provided by up streams like in limba, but They are built using an SDK which is provided by larger projects like KDE or GNOME Which contains some pre-built software pieces which the application can make use of if it wants more it needs to statically link it or include it in In its own bundle. So there's no sharing of those resources anymore so comparing those solutions Limba is very flexible, but it also is very susceptible to ABI breaks in case some parts some bundle breaks ABI You will You are screwed. So this is needs to be tested by upstream stem serves They don't have have the distributors doing the work for them anymore This can be worked around of course by continuous integration solutions and other things to To catch those problems, but yeah, it's an inherent issue of this approach Also, of course limba does duplicate some things it does a good job at Avoiding lots of duplication, but it cannot remove it all because the distributor will always also ship its own copy of Qt for example Yeah, and exegia provides exactly that ABI stability if you build something with an exegia SDK you can be certain that it will work with the exegia run time and Will not change at all. We'll just maybe receive security updates, but that's it and So you must care if you want another dependency, which is not in the SDK You need to bundle it with your With your application and ensure that it stays up to date and security issue free on your own So that's it and are there any questions? And thank you for your attention, of course Yeah You're suggesting that Katie should release their own binary Yes, that would be the ultimate result of it that Katie builds For example with the framers components as bundles and those are then used by upstream projects to build their stuff on top of them So Katie you would be responsible for building it and not have it done by the distributors Yeah, I've tried it works Yeah, I tried some key components and it's working quite well As in and socially and generally we will see we need to try it Sorry Yeah, those dependencies are handled by limba limba has some some nice properties like Conflict free installation of software versions So you can install multiple versions of cute on the same system and they won't interfere They each application due to the kernel tricks will just see the cube version Which is which it depends on and which it wants to use so and the other versions are hidden from it So in order to turn the dependencies, you will just declare dependency on the bundle you You refer which contains for example cute or whatever other component you want and limba will do the rest for you in case the upstream didn't break a big Yes, in case cute doesn't Doesn't ship fixed versions on its own You would need to do that and maybe namespace those cute copies in order to well distinguish them from the upstream components That's also feasible Or Yes, it's similar to that what steam is doing and to some extent it's a Duplication of work by the district packages But I would I'm careful to say that because as a district package or you need to care for a lot more things like Upgrade paths and how everything interacts in the system, which you don't need to do with them But you just need to ensure that your application works and also limba is not designed to install things like Core system parts like the kernel or system D which it simply can't do because of technical limitations It's just designed for for applications and toolkits and libraries and not for for system parts So that's to some extent. It's a duplication, but it's not the same Yeah, that would be possible I think The in the end for example in case this takes off Distributors will ship their own limba Repositories enabled by default, which they have control over to some extent so they can ship on the stable base For example updated copies of Kate or whatever or when you or digicam or whatever application They want to be up when to have updated without interfering with the usual process of Upgrading packages in the distribution Is Exactly that's the point of it. Thank you