 Okay, so welcome, this is a talk about line a key, a system developed to make building Debian derivatives easier. So first starting with what Debian derivative I'm actually building, the first thing that the first actual derivative I made was Tanglu, which was created in 2013 to embrace technologies and also processes that weren't possible in Debian at that time. It uses system D, which is a, right now it's not, it's a really boring fact, but back then in 2013 when we had this large system D framework, this was actually something remarkable. And we thought we would just build a system based around system D in order to show how it works and give us a data point, an example for Debian on, on how to, how it can work in a Debian based system. Because back then many people were asking for examples and weren't sure how, whether this is the right path to forge, so yeah, we, we did that. We also had aimed for timed releases, which is difficult with a strong, with a small team as a later turnout. And we wanted to focus on workstations and desktops mainly. Also one idea was to be as close to upstream and Debian as possible, which also meant that we basically tried to replicate Debian's infrastructure inside Tanglu itself, which did lead to some problems. Where's the next slide? Oh, well, sorry. I don't know why the, why the slideshow is slow at this point. Okay, I will just talk about the next slide until it appears. There it is. What you need for building a derivative is mainly repository management, which is the Debian archive kit. In this case for Tanglu, we need built infrastructure to make packages. We originally thought that we wouldn't need to do transition tracking, but if you base on Debian testing, you actually require that in order to do uploads in the right order, in order to rebuild things in the right order, because if you just sync them in batch, you will rebuild them in any arbitrary orders. And actually testing turned out to be not as easily buildable than we originally thought. This is also the reason why we need BritNate to do package migration, and we also need to synchronize packages with Debian, which also means that, which not only means that we need to fetch stuff from Debian and get it into our distribution, but we also need to merge patches. We apply it downstream. We also need to build ISO images, track bugs, have a wiki, generate upstream metadata so stuff shows up in GNOME software and other tools. And also we need a lot of misaligned tools to do various other smaller tasks. So a short excursion into building things, we are a distribution which rebuilds all packages that Debian has, and it bases upon testing. So we had the task of building 28,000 source packages on two architectures, which is quite a lot of stuff. So we originally thought that we should use Wannable, because that's what Debian uses, and yeah, this turned out to be a really hard task, because Wannable was developed in the origins of Debian, and it's almost impossible to set up for any derivative, which basically brings us to the first and only cat picture in this talk. Yeah, I would just quickly go through this if you have questions on why Wannable is difficult, ask me later about it. The next thing we thought about using open build servers, back then this wasn't possible, especially not if we also want to use the Debian archive kit, because the open build servers by OpenSUSE wants to take over also repository management, which we explicitly didn't want. Meanwhile, Colabora did put a lot of work into making it work for Debian and for derivatives, which is awesome, but back then this unfortunately wasn't an option. So we went with Jenkins and created 28,000 jobs with a matrix in it, which led to a, I don't know, I think it was about a gigabyte XML configuration file for Jenkins, which was actually pretty terrible. It worked for a small while, but even just rebooting Jenkins took about 40 minutes. So while Jenkins is great for CI, it's not great for building a massive amount of packages. And yeah. So in the end, we went with Debal, which is a system written by Paul Tagliamonte and made some custom glue for Tangloo in order to build a lot of these packages. But this also had its own problems. So actually package management is something we weren't really happy with initially. We also had other problems with actually replicating Debian infrastructure, which is, for example, that there are tons of Chrome jobs, which are run on a time-based, which means that as soon as we upload the package, we generate packages files and then something else which might actually immediately run afterwards, because this package first has to be rendered and generated, just runs later. And this causes huge delays, for example, to get one package into the distribution. A lot of steps need to happen. And yeah, this is not ideal if you want to do fast development. Of course, you can tune the Chrome jobs to run in the right order to make it less annoying, but this is some work. And it's a bit, yeah, it actually doesn't really solve the problem. It just makes it just mitigate it slightly. Also, if you aren't part of the Debian infrastructure team, it's unclear how these tools work together. For example, I didn't know that a Britnay throws out a text file, which is then directly imported into the Duck database, which Duck just overrides tables and testing with contents from this text file. And I asked the FTP masters, and nobody knew how they actually, how stuff actually migrates from testing. So it took a while to figure this out, and it's actually a well-known thing, but yeah, if you're new, it's not really something you know. So in general, it's really hard to set something up that resembles the infrastructure Debian uses. Fortunately, Duck, and especially Britnay, got a lot better. Britnay is now really easy to use, and Duck is also in the process of being easier to use for Debian derivatives. For example, while writing Tanglu, we got rid of a lot of hard coding of the unstable name and Devinisms in it. So yeah, this is pretty great by now. There are also maintenance issues, because as much as teams complain in said Debian that they don't have enough manpower, they actually have a lot more manpower compared to a derivative that's just started and has about four members. So they have a lot of teams which maintain separate parts of the infrastructure. So a RIS team which is dedicated to maintaining Britnay and doing transition tracking. And the FTP masters who only deal with Duck. So they are experts in their field, while you, as if you start a derivative, need to basically know everything and also maintain everything, which is difficult, especially since every tool is configured in its own way. And since Tanglu was doing a lot of Dutch releases, we had to basically overwrite things, basically adjust every configuration for every single time, which is annoying. So since I'm running out of time, I just go through a few solutions. We tried to mitigate this, which is FAT message, which is a system to make parts communicate with each other, as is developed by Fedoro. But it didn't solve the problem completely because we still had the configuration management problem. We also had Rapidumo, which is a large collection of Python scripts we developed in order to glue the different parts of the infrastructure together, which was kind of, well, it worked, but it wasn't great. So we had something working in the end based on these things and also Git repository with all the configuration files, but it wasn't awesome. Then PRS happened where Purism approached me to help with developing their PRS, and I thought, do we really want to use this for this new thing? And meanwhile, I was already working on some integrated solution to resolve the problems we were facing, which, yeah, I'll list it here again. And surprisingly also, the tools for Tanglo didn't actually work for PRS because we thought we would have made them generic enough to be used by other derivatives, but in fact, this was not the case. So I developed Lenekea, which has a massive scope, hence the name of the Lenekea supercluster, which is basically the home of our galaxy. It's a galaxy cluster. And yeah, I could go on and explain the name, but I will skip that now. So the basic fundamentals of this, that we have one source for all configuration, which is a central database. We have tightly integrated components, which only talk to the database. We want to minimize human intervention. So as soon as something can be done by a module inside Lenekea, we don't want any human to upload a package for rebuilding. But for example, we want to upload the package of Lenekea or detect what needs to be rebuilt and have it done without having to intervene. We also want a web interface because that's actually how things are managed by now and not having users to log in into or SSH into machines. And it's also nice for permission control so we can have users give individual permissions to users to perform certain tasks, which was a problem for Tanglo because they always had to go through me or someone else who had permission if they wanted something migrated or something done. And as soon as you had permission, you were basically able to do a lot of things which we didn't want to give to new comments to the distribution already. We also want to support Flapag and OSG natively and also maybe Snappy, which is emerging technology. I'm playing around with that for a while. So yeah, and also share work with other distributions. Yeah, share scripts are terrible. So this shouldn't happen because it's basically a sign that we did something wrong. This is the basic architecture of the thing. I would quickly go through it. Some things aren't ready yet. I will talk about, I will mention that when I talk about them. So what we have in line here is adapters to existing projects, which is, for example, for the Britney package migration, the Spears module, and for a Germanator, which is generating meta packages and for checking the installability of packages, which is those. We have these things which basically set up these tools, so you don't need to care about that when you make a new derivative. They write the configuration for them based on information they found in the center database. They read data that's generated by these tools into the database, so it is available for other tools immediately. And they receive triggers via 0MQ communication in order to run. So we also have modules which are doing tasks like synchronizing packages, building ISO images, managing build queues in order to build things. And they share, in general, share a lot of code to execute actions faster. The build stuff isn't quite there yet. It's currently a work in progress. For long-running tasks, we also have a job runner, which is a pool of stuff, a pool of machines that can perform arbitrary tasks ranging from building images for the distribution of building packages, to importing Git repositories and doing all kinds of things. They use encrypted connections because, at least in Tangle, we don't have, we don't have a central build form where only trusted communication is happening, but Tangle packages are built sometimes on service someone has in their basement. So, yeah, we also have user interfaces which contain CLI tools to administer the services right now, so we still have SSH into a machine in order to get things done, but there's also web front-ins and development. Some of them are already working in order to view QA information on the central location instead of going through many places in order to view what's happening in the distribution. There's also an IRC bot in development in order to announce certain changes that are interesting, like a new image was built, please test it in the IRC channel. Most of the user interfaces, though, are very early work because I'm actually not a web developer, so this is a kind of new territory for me. Yeah, in general, these things are really easy to set up in a self-assembling because they communicate via the database or via zero MQ in order to pass messages amongst these other. So, as soon as you have one and connected to the central instance, it will immediately detect the other modules and work with them. Technology uses basically as boring, I guess, except for the fact that we use a deep programming language for most things and Python for the rest and also Mongo at the moment, but I'm pretty sure that this will change, at least in terms of using the database. Yeah, quickly, for future plans, first thing's first priority is to make package build work properly and integrate the transition tracking module and basically write a lot more modules to perform more tasks in CYLINE Ikea. Also, one pretty important point is to use OpenQA, which is a service by a developer OpenSUSE in order to do testing on the final image and to test to automatically check whether the image we ship to our users is working and installation is working and things like that. And, yeah, packaging line Ikea for Debian is something I'm working on. It's, except for the few JavaScript things, it's relatively simple. And as soon as it has more documentation, it's actually easy to set up for other derivatives. You can just install it from Debian and use it. At the moment, if you are developing a derivative and have existing infrastructure, I wouldn't recommend using Ikea just yet, but wait a bit because it's still a very early development and there are a lot of things that will change and a lot of modules still in development. So, yeah, do you have questions? One of the problems that I've run into, like dealing with like many derivatives and stuff like that, is what is the best tool for things like a replacement for packages.debian.org, like a web interface that actually lets users take a look at what's in your repositories? At the moment, we are entangling using packages of Debian.org. So, the packages web application for this, but linear KIA has, because we have this information already and we can make use of it, it has this web SW view thing, which is not only showing packages, but will also in some future be able to show flap packs alongside. So, this is the thing I would replace it with long term. Short term, I think packages web is the thing to go to. Also. To take the packages.debian.org sources and actually go set up your own instance? Yeah, it's open source. No, I'm sure it's open source. I hadn't looked at how easy it was to do. It's a bit tricky, because you need to regenerate the database for it internally and it also runs with Apache. So, if your system is around about engines, yeah, you need to, well, work around that or write something to run the Perl script without Apache. But it's possible, it's actually one of the nicer things to set up. Yeah. Is the intention to run it only for derivative distributions or can you run it for a partial distribution as well where you have, you know, you use stretch and then you have your own repository with a hundred packages that you need to build and sync and continue working in blackboard, maybe. Right now, most modules assume that the repository is complete. So, if you could mirror it, which is what we do in pure S right now, we just mirror all binary package and don't rebuild them. So, that would work. But actually, the modules that actually assume that the distribution is complete, which is the depth check thing, you could just not use them or fix them in order to work for a partial distribution. So, right now, I'm not focusing on partial distributions because I have no partial distribution to work with, but it's definitely possible to use it. Also questions. So, the depth check modules will run into problems, I think. I have a question for myself because we are here in the blends track and the blends are actually not creating derivatives. How often do you think we are too used and for what purpose? When I created Tangle, I thought I would only do this once and actually nobody would ever create Debian derivatives and I actually meanwhile learned that a lot of people do create Debian derivatives, mostly internally, so they can be used inside the company, but also pure as happened. So, I think creating Debian derivatives is a quite common task meanwhile. There was also recently a talk by the endless people who made a derivative dedicated for people who have less stable internet connections and basically in the third world. So, I think for these specialized niches, creating Debian derivatives is quite common because you can ship one ISO image for users to just install instead of telling them install Debian and then perform these additional steps in order to get this thing working. I always think that tweaking Debian in itself is more sustainable because these people are creating the derivatives and all other places of the world and reinventing the wheel as Michael just told us before. To make the case for derivatives in case of Tanglu, just saying let's go for system D wasn't possible and also saying let's go for timed releases isn't possible. So, there are some things which Debian can't do and Debian shouldn't do even, I would say, and these are the niches that can be filled by derivatives. Oh yeah, for example, pure as is 100% free software and is trying to get endorsed by the Free Software Foundation, which is something Debian, which has the non-free branch, also cannot do. So, those are cases, for example. So, you mentioned OBS and the endless guys were using OBS, talking about OBS. So, for a new project starting today, how would you contrast what you're doing with what OBS is capable of doing today? That's a good question. So, I haven't used OBS, so I can't really comment on this because I used OBS about three years ago, the last time, and back then it wasn't really, really nice to work with. And, but meanwhile, a collaborator did a ton of work and it's even packaged in Debian, meanwhile. So, I would need to see how well it works. So, I think if your objective is to just build things and have some basic repository management, then using OBS is a good idea. If you want more, like the whole transition tracking and automatic rebuilds, then using this is a better idea. I think one of the ideas is to maybe get rid of Duc and to absorb the repository management inside Linear Kaia, which is something I think might need to happen at some point because Duc is a real bottleneck at the moment. But, prior to that, I would also see if I can somehow interface Linear Kaia with the open build service instead, which would have some synergies in that area. Or just make it plugable so you can use Duc and open build service and anything else you want. So, yeah, this is just the fabric where you can put in other services and have them share code and information. That's the basic idea behind it. So, okay, thanks for the talk and, yeah.