 Welcome to the Distribution Stab Room. Our first speaker this morning is, no, don't undue talking. Our first speaker this morning is going to be Andres Shedora. Andres Shedora, who will be presenting Managing Build Infrastructure of the Debian Derivative. Yes, the talk is being recorded. The video has been set up and his microphone has been undue. It requires no human interaction. You show about this. It looks like it's only... Point another wall. It's supposed to be pointed correctly. Well, wall is also quite nice thing to see. You are in fine, sir. Thank you very much. You can hear the t-shirt over there. It speaks to the fine process of the project. It can't be listened to. Okay, so I probably, I didn't need to present myself once again. But I will do this anyway. So I'm Andres and I'm a Debian developer. And, well, I started using Debian around 2006. And very soon I understood that, well, Debian is great because of the contributors. So I started contributing myself at first by sending my reports, then patches, and then finally I became... Well, I started maintaining some packages on my own. And in 2013 I became a Debian developer and started maintaining even more things in Debian. And then two years later I started working for Collabora, an open source consultancy. And which is great in a way that we were doing lots of fun things with open source in our work time. And at Collabora I started working on a project which is called Apertis, about which I will talk today. And just some background. I never run any Debian infrastructure in my life. Even though I've been part of a Debian project for quite some time. And so I've got no idea how to run proper things like BuildD or Deinstall and other things. And because I started working on Apertis I had to manage quite similar things, but not based on normal Debian software at all. Right, so Apertis is a Debian derivative distribution, which is based not just on Debian but on Ubuntu and a bunch of other packages we've developed ourselves. It's been originally developed for automotive purposes in particular entertainment systems in the cars. But since then the focus has been shifted to all sorts of other onboard computers in cars. And as a part of the project we provide lots of infrastructure for development based on Apertis, which means code hosting at the moment is Cegit, but we're planning to migrate to GitLab soon. Backtracking system, code review at the moment is Fabricator. And of course package build services, image generation services and automated testing based on Lava. But which I was going to talk as well at Forza, but unfortunately my talk wasn't accepted. So Apertis is mostly based on Ubuntu at the moment, so we are a derivative of it. We take a bunch of packages directly from Debian, most notably SystemD, DBS and some other things. And we've developed a few software frameworks of our own APIs and other software packages. So as I mentioned we use SystemD for process tracking and application life cycle tracking. We started using it many years ago when it was just a new thing in Ubuntu only. So this was one of the reasons we used Ubuntu because they were upstream for up armor. About a year ago I think we've migrated from Xlog to Wayland for graphics and we've been using GStreamer for multimedia playback since the beginning. In Apertis we use Flatpak in OS 3. So the application bundle format is based on Flatpak which itself is based on OS 3. It's not quite the same thing as Flatpak's bundles but we plan to migrate to actual Flatpak and not the solution based on Flatpak soon. We have an update system based on OS 3. Originally we had our own system based on beta butterfests snapshots to make sure that updates are atomic and we've got recovery modes and so on but because of certain drawbacks of butterfests we've developed our own system based on OS 3. So couple of more details. First of all the butterfests system had significant maintenance effort because we had quite a lot of custom code based around working with butterfests snapshots. Then our bootloader didn't support butterfests so we couldn't use it on Slash Boot for example and other file systems which had to be something else. With butterfests we needed to use InetromoFest at all times which isn't the best thing sometimes in a better system. And because the updated directly manipulated butterfests volumes it was unsuitable for for example unprivileged containers. And then with butterfests testing was a bit difficult and butterfests itself has quite a lot of issues which are being tackled but still like it's not I myself don't consider it a really production ready system. So in comparison OS 3 works with any file system at all any file system which supports Unix semantics. You can store multiple trees in the same repository which is definitely a benefit and you don't need extra partitions for upgrades so it saves quite some space. Because it's a more complete solution on its own we have less custom code so the maintenance is much easier. And because it's a user space solution it works better with containers and we can use it with unprivileged containers as well. So some people may ask question why are we using this crazy mix of Ubuntu and Debian and not just Debian because Debian is great universal and obviously it's free software and it's a community not a company. So despite all of those great things about Debian at the moment when Apertis was created there were certain issues with Debian. Most importantly if there are people here who are here who ran Saad for example they may remember that it took I think four years or about that to get it released because in Debian the policies the release is when the release is ready. Until recently there wasn't any particular timing employed so if there were too many release critical bugs the release might be delayed even years. Since then it's changed but when Apertis was created it was still an issue. And then stable it's a bit too stable for many people and you know with the frequency of releases it means that you get updates only once in two years and it's not always what we want. And then with unstable things may break too often and well I myself broke unstable two weeks ago when I dated one of the key bits of it which was Excel. Unfortunately I didn't do much testing, didn't do enough testing and I uploaded something which was broken for a couple of hours until I uploaded it fixed and a couple of people, well quite a few people ran into this issue and well it's been resolved since then but this is an example of what may happen in stable we cannot use it directly, you can pull all of updates from there. So Ubuntu on the other hand has also quite a large install base because there were lots of people who are too lazy to run Debian they just run Ubuntu for example and so there's quite a lot of testing despite more frequent releases. So Ubuntu is an upstream for up armor which was quite important when the apparatus was created because we heavily rely on up armor for privilege separation and security. And well they've got regular time to releases and they also had an LTS when apparatus was created so these days we've got LTS in Debian as well but back then it wasn't yet like that. So there are a couple of downsides. Obviously our apparatus is a derivative of a derivative. Oh what's going on? Oh yeah. So this complicates upgrades and rebates because you know in every release of Ubuntu there's quite a lot of updates coming from Debian originally and when we're rebating to a new version of Ubuntu it just this impact amplifies. And then we don't really need all of the things we have on Ubuntu like mayor for example and other things so it doesn't add any value for us because we use Wayland we don't even use Xorg anymore. I think there's a slight missing. Anyway so we are currently reconsidering this decision and we might change it back to Debian because lots of issues Debian had when the decision was taken are no longer issues and it might make sense to rebate to Debian LTS for example. Right so now a bit more about the infrastructure we have. So the core of it all is open build service which builds packages in controlled environments and maintains repositories from which we create images and the updates and so on. So this code hosting and review tool at the moment it's a combination of Cigit and Fabricator but we will be using GitLab quite soon for code hosting and probably for reviews as well. So we pull a few packages directly into our Git like systemd and debas so that we can apply our patches to them if needed or we can use newer versions from Debian and the rest goes like the normal packages from Ubuntu Debian they go directly into OBS but I'll tell you more about what OBS is and how we use it in a moment. So OBS every time it needs to build a package it creates a fresh route of operatives in this case builds a package there and publishes the binary packages the devs in internal repository. So all the source packages are version controlled. The system resembles subversion in a way because it doesn't handle proper branches like in Git you can create branches but it's basically just a copy with some version information which you can build packages in their own space they are not published in the normal repositories so you can test how things work in the branch and it has quite some fine-grained access control tools so you can let your users some users can just read packages, some can commit, some can do reviews and so on but with more information OBS was in the talk of Android yesterday you can watch it online. So I purchased a split in multiple components which are target, development, helper lives and so on important is snapshots. Each of them is an OBS project or so on so it's more or less independent from others in OBS project can use each other for builds. So for example here development can depend on packages from target and SDK can depend on packages from development and this means that if the packages in development require dependencies to build from target they are installed in the truth by OBS and used but normally not other way around so an exception of this is basically target because target needs some packages in development to build but basically this split is mostly for images we have separate target images and we don't want development stuff like GCC in them so while GCC can be used for builds it's not installed in the images so it naturally goes to development not target but Linux kernel is obviously in the target so when packages get built they are published in internal repositories in OBS which are not in apt formats but they use for OBS itself to build other things basically so when your package is dependent on something else OBS internally fetches the their package and pre-installs it in the truth but for apt we use reprepro which takes packages from OBS internal repositories and publishes them in format apt can understand so in OBS it's quite easy to do full distribution rebuilds by just adding one more repository to the project OBS starts rebuilding all packages in this repository and they go nowhere they stay in the repository but they are not published in the actual apt repositories until you explicitly ask OBS to do this and well it's quite a tiny screenshot on the screen but basically you can add as many repositories in OBS you like and you can select like all like you can tell OBS which other package projects it depends on what architectures to build for as you can see we don't support 586 for example or ARMv6 so as I mentioned the packages to which we apply no changes relatively to Ubuntu or Debian they go direct to OBS we just import them with no changes some packages where the patches are trivial like you know fixing minor tiny things we maintain them directly in the OBS using the OBS version control system and we just apply patches top of them in the Debian source package and increment the versions so that we can see where we have local changes and we use the fork of Ubuntu's Mergematic tool which pulls new updates from Ubuntu and does automatic merges when it's possible and then packages where this approach doesn't work we keep them in Git and use Git for merges and other things so that we don't rely on Mergematic at all those packages like I mentioned SystemD, Debas and Mesa and so on so for non-operative packages packages from Debian and Ubuntu we use the 14 format from Debian so we have upstream and apertest namespaces for tags and branches obviously we try to reuse the Git repositories from Debian so we also have Debian slash something tags and branches in repositories normally and we use Gitvill package and GBP PQ to manage the patches in the repository you can read about them in one pages for Gitvill package basically I won't go into details and for native apertest package we have a similar scheme so the master is the version in the current release and there are branches for previous releases like 17.12 and we normally it's an approach which is not accepted in Debian usually so we keep the Debian metadata the package and stuff in the master not in the separate branch as normally is done in Debian because it just adds some maintenance burden for no reason and we normally just put tags on those branches for the releases the version numbers are normally like 0 and then the release number and the just increment so Jenkins builds every commit on those branches in a controlled environment and if the build exceeds it commits the result to OBS and OBS creates major quests and builds it in the proper truth which we then review and merge we use a tool called build snapshot written by Simon which just creates releases out of Git branches and if it's work in progress not tagged as a proper release it just builds a package which has this funny Git revision number in the version so that we can see its development version and every time new packages are submitted at Fabricator they also get applied and build by Jenkins so until recently we used linear image tools to build images it was split in multiple stages I will talk about this later but the linear image tools proved to be quite difficult to use and maintain so we've changed it to our custom tool called DevOps which does the process much better and reliable so this was the original scheme we used so we had this separation between hardware dependent and independent parts so OSPAC was basically a table with the root file system not including any hardware dependent things and hardware part was the kernel, firmware and other things and we would combine them into images like this so with DevOps DevOps is a tool written in Go which is basically a Debin OS image builder with DevOps we split the build process into stages in a YAML file and every stage does its own thing and you can use templates similar to Ginger 2 basically and select parts of it depending on the type of the image it's a really cool tool written by Shorz, Simons and Denise Pinkin basically and here you can read more about it so in this tool we've got rid of the hardware parts as such we still use the term but it's actually just packages and files installed into the images when they're built we also generate SysRoot for a software development kit and then the Jenkins jobs they trigger tests in Lava and when the images are built it also goes through the fabricator and closes the bugs which were fixed in the images so this is how it looks in Jenkins when it's all green and happy so there are quite some challenges working with all these because obviously a merge-o-matic cannot handle anything more complicated than just a couple of patches applied locally Git does help but we cannot put all things in Git so we need to decide what can we do what can we put in Git and what have we maintained manually removing all packages is quite tricky sometimes because you need to check what's gone in Ubuntu and Debian and so on and draw a couple more issues with OBS basically because OBS is not Debian stuff and it doesn't do the same things with building well as I mentioned we're going to migrate to GitLab soon and become a common platform for automotive systems and not just infotainment and to learn more go to aperty.org and there's a wiki where you can read those things and learn more about the system this is it