 Hi everyone, welcome to the Debian derivatives panel. I'm Paul Wise. I started the Debian derivative Census and have been working on the derivatives front desk for a while. We have Colin Watson representing Ubuntu, Rafael Herzog representing Kali Linux, Jean, I don't know how to pronounce this. Jean-Michel representing Do-Do Linux, and Wookie representing M. Debian, and Holger Levson representing Debian, Edu, a blend. Okay, so we'll start by, maybe you can give us an introduction to your derivatives and blends and your role in those derivatives. Okay, hi. As Paul said, I'm Colin Watson. I work in Ubuntu, where if anybody doesn't know what Ubuntu is, it's a general purpose mass market Linux distribution based on Debian unstable aiming at user friendliness and a number of specific products. I specifically work on it, we're sponsored by Canonical. I specifically work on mostly the best layer of user space, so I work on installers, bootloaders, package management tools. I also do a fair bit of release engineering works, so things like our package management, sorry, things like our archive management and image building tools, that sort of thing. If you want to hear more about the latter part of that, Steve Langesek and I are running a talk on Thursday afternoon, which I think is entitled Ubuntu Daily Quality Improvements, and we'll be going into a lot more depth on it there. I'm Rapha Letzog and I'm working part-time for Kali Linux, which is the derivative dedicated to penetration testing with lots of security tools. It's baked by offensive security, a company which sells training on security training and they use Kali Linux as a product to give them a good image and to find fine customers and stuff like that. We're really small, a few part-time people because they work in the security field and they also work to package the tools that they use. I'm mainly doing all the infrastructure part and everything related to the installer and building the image and keeping up with the build-demons and stuff like that for them. I'm Jean-Michel Philippe. I'm the founder of Doodoo Linux. This is a computer system targeted at children from two. We try to make a computer system as easy to use as gaming consoles. We are currently in use in several nursery schools in France and we have 43 language teams on Transifex, so it's quite used around the world. One of our objectives is to stimulate children's fulfillment using digital technologies. We also want them to master technology for real. If you feel interested in this topic, I will speak in a lightning talk on Saturday. Hi, I'm Wookie. Again, we've had various MDB and talks over the last 10 years, so hopefully most of you have worked out what it is by now, but if not, it's an attempt to make MDB and smaller fundamentally. It's evolved somewhat over the years. Initially, we made quite a lot of changes to make things run on small machines and that produced an enormous maintenance load, which fundamentally was too hard, so we stopped doing that. Now we just use exactly the same binaries as Debian produces, but with all the crap removed and a reduced package set. It's about 3,000 packages rather than 17,000, which of course reduces your metadata size as well. The nice thing about that is it can be entirely automated, so it is. We have stable releases synchronized with Debian, and now, in fact, MDB is available from the Debian FTP servers, so we've got quite closely integrated with Debian. It is a derivative, but we don't really change anything, in a sense, from the point of view of binaries. We've also provided tool chains, cross tool chains, for a long time to enable cross building. A lot of the cross building stuff that has been made to work has been tested in MDB in first and increasingly in Ubuntu first, and we badly need to stop doing that and get it all into Debian. There's various other bits and bobs, but that's basically it. Hello, I'm Holger Liefsson, developer, FTP master and documentation maintainer for Debian Edo, which is a Debian sub-project, exists since 12 years. We started as a CDD, which is now called Blend. For VZ7.2, we might be 100% Debian. This still needs to be discussed with the stable release masters, but there's five packages, which where we have the version in Jesse, we would like them in VZ, and they just only apply to all packages. So, what we are is a streamlined installer for AMD64 and i386, where there are four profiles to choose, standalone, then the main server, which needs to exist, which then an LTSP server profile, which you can add on the main server or on other servers, a workstation, installation and LTSP clients via PXE. Therefore, we have documentation in six languages, which describes the setup from a school admin point of view. It's out of the box configured for schools and universities, but really it can be used in offices or wherever, because basically it's just out of the box working setup with some education applications, which you can just remove with UpGaT, or you can add your own. Debian Edo is basically a different kind of installer for networks setup. So now we'll open it up for questions from the audience, and if there are none, then I have some pre-prepared questions. I'll start with one that I'm semi-interested in. Some of the derivatives have pre-installed images. Those of you who are creating such images, could you say something about how you do it and how we could integrate that into Debian? So, you're talking about the sort of thing that you would copy onto a system in OEM Yeah, we've been doing this kind of thing since about Ubuntu 5.10 or thereabouts. Although it took a while for any actual OEMs to pick it up after that. So for a while, all of the Ubuntu installation images have been live images by default, or at least the desktop ones have been live file system style. So you have a file system tree, you're installer copies it all onto disk and fiddles about until it works with the system that you're installing onto. And this sort of thing works quite well with an OEM model, although there are obvious differences. At the moment, we build everything using live build for the last couple of releases. We have a live boot system that's similar to the one used in Debian, although for historical reasons, they're a bit divergent. Regarding the sort of things that you need to change when you copy onto the target system, you obviously have to go around and make sure that you replace any components of the system that are unique, that are supposed to be machine unique. So, host keys, the popularity contest ID, that sort of thing. At the moment, we just have a hard coded list in the installer, unfortunately, that's where we go around and re-configure various packages. I'd much prefer to see some kind of system of, a directory of hooks that people could drop, that package containers could drop in to declare that sort of thing. So I had a thought on that. Maybe we could have support in the package for systems that are not yet configured for the hardware that they're installed on. So, the open SSH keys, for example, would be generated on first boot. Well, I'm the open SSH container conveniently, but I think that, well, indeed, I don't know what the DeepCudge container is going to say. I feel kind of like that's an unnecessary thing to have in DeepCudge, that it's a sort of thing that we can declare using much more lightweight hooks than that, and achieve much the same goal. Well, I don't know if you, I think there's a Google Summer of Code project on it, at least there was an idea of doing one. And I had a discussion once with Didier Abu on this topic. I believe that just a shared depth of question would be enough. So, maybe the DI in OEM mode could set up a precede, and the posting stuff, the various services would not do what they are supposed to do on first boot, and only do it later, in some way. Maybe we saw, maybe looking into a startup script, which is generating enough, or something like that. But I don't think you need to change DPKG to handle that. Yeah, coming back to, do you want to answer on the OEM question, maybe? Kali doesn't have really OEM image, except while we provide live image, so live build is doing all the fancy stuff for us. Right now, for Judo Linux, we are not really concerned by cryptographic needs, of course, it's for children. But in quite near future, we may need a cryptographic key, because we have currently an issue with web content filtering. We are using Dance Guardian, and it is not able to analyze encrypted HTTP traffic. So, a possible solution. I understood is to introduce a local proxy with HTTP, secured HTTP, and in this case, we would need a cryptographic key for the local proxy. In this case, we would need on our live CD something to regenerate the key each time the system is booted. And, of course, when it is pre-installed, the same issue is occurring. Yeah, so, making images. Yeah, this is a subject dear to my heart. I could give you a whole talk about this. So, we supply lots of architectures, MG64, i386, RME L, RMHF, MIPSOL, PowerPC, MIPS. In practice, mostly people using Mdebian are installing on non-X86 small systems. So, you have to cross-make images. And, Debian's stuff basically doesn't do that very well, because we expect to be able to run configure scripts at uninstalled and unpacked time. And that's a problem if you can't run it, because it's the wrong architecture. So, we invented a thing called multi-strap, which uses apt and d-package, rather than doing what debut-strap does, which is to explicitly avoid using apt and d-package, because it's intended to run on non-debian boxes as well. And that actually works. So, the problem with that is that there is no way to say, I've unpacked this, but I can't run the configure script. You can't run pre-inst scripts before you unpack, because you can't run the scripts now, and you've got to unpack it. So, what we just do is install everything without running pre-inst scripts. Now, actually, they nearly always are intended to fix previous cockups. So, this actually nearly always works, except where things create their user in the pre-inst script and so on. Yeah, this is exactly what the first stage of debut-strap does. And packages do have to, it's an extra constraint in packages that they have to behave in some quite specific ways. We've had some amazingly difficult debug problems with it. Yeah, but debut-strap has another, so it does do that, but that means, because it unpacks them again using d-package later, you have to have the devs in the image. So, now the image is twice the size, because you've got everything unpacked and all the devs, so it can unpack them again afterwards, which seems slightly crap. And debut-strap can only take anything from one source, which in the real world of embedded distros is a serious limitation, because in practice, everybody always wants the stuff out of stable and some other packages out of their local repo, which might even be proprietary stuff. So, one of the things, so an app, of course, can take stuff from anywhere. So, using apps to get whatever's needed to satisfy your dependencies is actually much nicer, but we always have this problem with the pre-inst scripts. Now, in practice, it works well enough. If you're making a specific image for a particular product, you just write some hacky scripts to do anything important in the pre-inst scripts, which works. So, I just did it for the ARM64 port, because, so, you can work around this by using QEMU, except where you haven't got a QEMU, because it's a new architecture. So, it's a useful tool, but it does have its issues, as Colin says, as weird things go wrong if you don't run your pre-inst scripts. So, yes, that's what we do. And that tool is already in Debian and has been for years. So, if you want to use multi-strap, you just app get it, use it there. Yeah, Debian Edo mostly uses the Debian tool, so we built CDs with CD image. We have modified DI to ask these profile questions and also to have everything asked in the beginning, which I think now went back into DI, I'm not fully sure. We lack the resource that the people doing a building a life system, even though it would be trivial. So, if somebody wants to help Debian Edo, building a live CD would be great for VZ. And what we mostly need is packages which support configuration better, like with ETC profile D, so that we can drop in our configuration without modifying the package and breaking upgrade passes. That's really what Debian Edo mostly needs and this is from a few selected packages. So, any questions from the audience? No? Okay. So, have a more general question for you. What can Debian do to make the lives of your derivatives easier? I got to go first again. When Paul asked me this before the session, the first thing I thought of was a plier of patches. I've got about something like 80 unapplied patches in the BTS that have been sitting there for ages from my work account. But if you multiply it up by all the people working in the Venture, it's actually quite a lot. In general, doing a good job at keeping your package building cleanly causes us to spend a lot less time going around. We make a change. We find that actually the package blows up due to this several package deep chain of something and we end up spinning wheels quite a lot and going around and going back and fixing things that really all two have just worked in Debian. The vast majority of packages do build extraordinarily cleanly across a wide sphere of time, but we generally have a few hundred build failures active at any one time. Keeping your package building cleanly makes a huge difference to derivatives who are trying to make changes that are really incidental to that. Things like cross-building and multi-arch improvements. These are projects that, as Wookiee said, we've been working with in Debian and other folks in Debian who are interested in those. Those sorts of things make a huge difference to some of the more exotic things that derivatives sometimes have to do. There are certain things that require changes to many many Debian packages across the board and that is really helpful if Debian developers can be proactive about enabling their own packages. For Kali Linux, you're based on stable for most packages for all the infrastructure, but we tend to import a newer kernel. We patch it a little bit, but kernel are very well maintained on the Debian side. Unfortunately, we don't have a DA release that goes with them always. Either we keep an older kernel or we have to patch DI over self temporarily. It would be really great if Debian could have a more regular kernel and DI release combined. That would make it easier for us. Otherwise, I tend to agree with Colin. Of course, when we tend to submit patch, we like when it's integrated quite quickly. We're lacking also a team which is dedicated to security tools in Debian. There is a team around forensic tools, but not generic maintenance team for security tools. I guess we will try to create one because really we have a lot of package to contribute. They are all team maintained by two or three people on the Kali side. We really want to share the load a bit with Debian on this. This is something that I'm going to work on in the next few months. For Judo Linux, we have our own package repository with tens of packages. Most of them are our own packages that we are building to set up the environment. We also have patch packages from the official Debian Archive. We have upstream versions that are newer than the Debian Archive. Finally, we also have some software that is not available at all in Debian. In the end, we have tens of packages, three releases and four or five architecture. This is a lot of packages to build. Currently, we are unfortunately doing this manually. This takes a lot of time. This is an error prone. What we are dreaming of is something like the Ubuntu Launchpad, something that anybody can use and which would be based on the Debian infrastructure so that we get all our packages and our repositories built automatically. This would also help us test new architectures. Recently, someone asked me if Judo Linux is available for PowerPC. Unfortunately, it is not because the developers cannot build on PowerPC. Just jumping on this topic, I agree as well. There are many Debian infrastructures which are not really easy to use for derivatives. In Kali Linux, we opted to use RebuildD. We used Repropro. We are not using BuildD and we are not using DAC. There are many parts that really should be better documented, more easy to use and to set up because it doesn't make sense for us to have to discover every time or to build our own solution based on tools which are easier to use. I am fixing this for the package tracking system. I expect other people in Debian to fix it for other parts. Yeah, Raphael just beat me to it. I was about to complain about the fact that it is really hard to make BuildD. Everybody needs a BuildD, even if you only maintain one package, you want to build over and over again, if you want to build a whole distro. The one part we make difficult to reproduce is that the rest of it is great. Repropro is brilliant. SBuild is brilliant. Deep package builds everything. It is all great. BuildDs are a problem. There is a thing called PiBit, which is what MW and people ended up writing, which uses zero MQ and does building and cross building. It might well be something we should pick up and use. RebuildD is okay, but it is a bit thick and a bit broken in various ways. Roger keeps threatening to make the BuildD thing in SBuild actually work, but it is not quite finished, I think. That is a thing somebody needs to work on. It will be lovely. Other things before I forget what I was going to say. Most of what we have wanted has gone into Debian. Generally we have no complaints over how it has gone. It has taken a long time, but some of it was difficult. Deep package vendor and cross building support and multi-arch support. I would say that, again, accepting packages quickly is great. Sometimes a multi-arch package takes a day. Some of them are still mouldering after a year and a half. That is annoying. Sometimes the freeze does not help. That is always true. That is mostly it. Somebody actually wanted to say something to the audience. We should probably let him. Yes. Wasn't Paul Tagg working on a BuildD system called dbuild.me or something like that? I don't know much about it other than that it exists. He is generally very excited about things. I am sure if you talk to him, he will. Paul Tagg. For Debian Edo, we only use packages from Debian main. That is not an issue. We do have our own archive, mostly for being able to test faster. For squeeze, it is also needed to build the images. The final ones for Weezy, hopefully not. We use a duck install for that, which is clearly overkill, because it is 10 packages in there. We hope to switch to the Debian form of PPAs, which were set probably to arrive maybe this year later. That would be quite cool for Debian Edo. We can just use FTP, Debian org for everything. I could ask the opposite of that question. Is there one or two things achievable or maybe aspirational that you think your derivatives could do better by Debian for? The previous question was, what could Debian do for you? Is there some things that you could do for Debian that you are not doing now that you could do? Or maybe things that are more pipe dreams that you should do? Obviously, we have lots of packages that are not yet in Debian, so the first thing to do is to contribute them all to Debian. Some of them are mostly really, because they are quite clean, but some of them are really ugly, because sometimes they want the software quickly. I don't know if you have tried to package some security tools, which would be with lots of games, which are downloaded in a specific version, which is not the current one. You have the opposite problem as well. Debian is sometimes quite an older package, so it's really a headache to package them properly for Debian. We have quite a few packages which do burn the multiple gems in it. You benefit from not having to abide by Debian's stricter quality standards. We do benefit, but we do suffer from it as well, because we... It makes your life easier in some ways. It would be nice to be able to contribute more tools, but that is the main value that we can bring to Debian. We can go further, maybe integration, but I'm not sure it's going to be useful. Obviously, the stack of patches that we have constitutes technical debt from our point of view, and we clearly need to pay that down for both for our own good and for for Debian's. The more we do with trying to improve our own distribution's quality, the more it becomes obvious that the best way to do this is often helping with particularly big transitions in Debian and shifting unstable forward at the fastest possible rate. I think the more people from our side that we can encourage to join Debian, the relevant Debian teams to help out with those, the better, I think. We have quite a bit of manpower that can be put to use for a variety of things. Again, I'm probably going to be talking more about this on Thursday afternoon, so I won't go on too far now. We worked out probably about nine years ago that the easiest way to make our lives easier was to put stuff back upstream. Basically, all the cross-building and multi-arch stuff has not been entirely driven by in Debian, but that was part of it. That's actually been quite a significant contribution. I think it's fair to say Ubuntu has helped a lot because, to be honest, a transition like that is easier to get done in Ubuntu than it is in Debian because you've only got to persuade a relatively small set of people with commit access to everything rather than thousands of developers who go, what is this crazy talk? We have done some things, and there's this PIBIT tool. Yeah, that's this really more in an earlier question, but one thing that I very much noticed when trying to push particularly multi-arch changes back into Debian is that we spend an awful lot of time explaining over and over and over and over again what we're doing. It would be helpful if more people kept up with that kind of thing, I think. It's not directly related to derivatives, but I would be interested to hear your ideas, if you have ideas, how Debian could become more current to keep being stable but also to have more current software. What would your advice be for Debian in this regard? We have an awful lot of places at the moment where we rely on we rely on human effort to do a lot of our testing in Debian. So the whole process of migrating from unstable to testing is completely reliant on humans telling it, well not completely, but it's very extensively reliant on humans telling us whether it worked or not. In order to make that work, in order to allow enough time for this whole process to operate, we have to artificially slow ourselves down to allow ourselves time for humans to give us feedback. The more that we can move towards automated testing and responding quickly to, you know, promoting packages quickly in response to those automated tests rather than having this very human driven process, the more I think we'd be able to move forward much more quickly and with many fewer incredibly complicated tangles than we have at the moment. On Kali we have a repository named Bleeding Edge and where we basically use our package and we grab the latest step to inversion when we detect one and we put back the Debian directory from the formula and we try to build it and make it available to advanced users who want to look it up. So this gives us early feedback whether new versions are working or not without actually having to do lots of work. Maybe Debian could have some similar infrastructure as well and it would help maintainers and because quite often our problem is Debian is taking too long. Things tend to get down but there are many maintainers who are not interested in doing it right away. They want the latest version and a good version for the next table but they don't care about the current version in unstable until a few days before the freeze. That's the problem. Yeah, I'm not sure. I don't know how to fix this problem but you notice it particularly when my Leonardo hat instead of my Mdebian hat, they're building the very latest everything all the time and there's quite a big impedance mismatch with trying to get that stuff into Debian at all, even running it on the system. So yeah, faster would be better but I don't really have anything useful to say about, a bit on what Colin said about trying to automate things because it's the only way to make things go faster I think. Well one thing clearly the main problem is under release management and it's a difficult issue but I do believe that the PBA could really help because we could have transition prepared in parallel and instead of the release team having to say so next one is this one and we'll take the take and it needs to go through. We could have multiple transition prepared and the first one which is really is going to be the next one instead of having a human to only some guesses from humans to decide. There's one potential problem there is that transitions get entangled with each other because of the dependency trades and stuff. It's not a silver bullet by any means. I entirely agree with Raphael Rotelp. One thing we often notice in Ubuntu I've driven several transitions there which we've started in our development release about the same time Debian does often because we've just auto synced something from unstable and then have to finish it. It's very noticeable if when you're following this in Ubuntu that there's a handful of transitions that are managed exceptionally well. The parallel transitions are wonderful. We generally find that the vast majority has been pre-prepared by the parallel team and there's absolutely no problem. There are various other transitions where we find that there are a couple where I find that I basically had to do all the work because somebody had started in Debian and hadn't properly tried to build everything that was going to need to be rebuilt. And preparing this sort of thing in a PPAs, in a PPA, you would still have to effectively redo the work in unstable. You'd have to rebuild everything because partly because of this entanglement problem. But you might at least have some chance of ensuring that people have built everything before they start, which is not currently the case. Jörg, just that I see that he plans to start on this week, so on this PPA bike set plan. I think bike set was the name we agreed on. But I'll leave it to him. Cool. So a similar question. So is the release cycle working for you? And for the derivatives that aren't based on Debian stable, how is your release process different or better than Debian and how can we integrate some of those changes? I'm saying this a lot, but we'll talk about most of this in theory. But briefly, we branch from unstable. The main effects that the release cycle has in us is that essentially the flow of incoming changes sometimes slows down. And we have to start sinking from experimental instead, which is a lot more ad hoc. We'd generally be a lot happier with shorter freezes. I think everybody would. Kelly is based Debian stable, but there are no really no real policy. We, well, when we started, we worked on the wheezy because we knew it was going to be the next table. We had a number of problems because while all the tools were not really ready during the freeze and we contributed quite a few fix to a live build to even to GDM for accessibility issues and to multiple detail. Well, important things, but we do have two versions in parallel, Kelly and Kelly Dev, but we mainly use it for a few tests. It's not really a continuously maintained as a Kelly Dev one. We have a proposed unit area, so we're preparing this new stuff for the current version always. So we're not really set on a specific base. It might be that in one year we're going to use Debian testing because we need to hold for too many of our packages. It depends. Currently, the stable release of the latest stable release of GDM is still based on all stable, but our objective is to be based on stable, of course. The release cycle is quite good for us because we don't have so much time to spend on setting up the environment for a new version of Debian. That said, the main issue we could have with the current release cycle is freshness of drivers for people who want to use a printer that just they've just bought at the supermarket, for example. And so regularly we see people coming and telling us well, I can't make this work with Julia Linux and our answer is unfortunately, yes, we know we are still using software that is two or three years old and we cannot do anything for you. So our issue is not of a release cycle itself, but of the availability of fresher packages for drivers. And I'm not sure there is really a simple solution for this problem. Yeah, that was mentioned in the kernel BOF and we're going to try some stuff because the same problem is buying any new hardware. May or may not work with old kernels. So because our thing is entirely automated, we have no problem whatsoever with the release cycle. Everything in the table gets processed as it comes out and when there's a stable release, we just do a stable release exactly like Debian's and copy the install scripts and change them a bit. Sorry, the install read me thing. So yeah, it works fine because most people want to work in unstable except for some people who actually want to ship stable stuff. And in practice, they have a load of their own packages. You know, other people use our stuff to make actual other distros rather than us being anything other than exactly what Debian currently puts out. And that's fine. Debian either relies a lot on the Debian stable release process itself. So we use Debian, that's Debian. So what we do on top of this is manual testing of our features and since half a year, we also have Jenkins jobs which test the actual installation which helped us to find some bugs there. We don't yet test the functionality or the network services combined with a main server and another system but that we plan to do this during development really daily, automatically also. So we have two minutes left. Is there anything any of you any of the panelists want to say about Debian or your derivative or something else? I don't really. Debian provides a great best for us. It's an astonishing effort to that's kept things working over this massive period of 20 years and notwithstanding Joey's talk I say no reason why it can't continue for 20 more. Well, Kali is only starting so we are planning to be for multiple years so obviously we we want Debian to strive for a long time and we're going to do our little share not much but a little bit. Yeah, nothing special. Well, if we have chosen Debian for do the nukes it's just because it's great. It works very well. It's full of various software and of course this is driven by a community. So yeah, you could argue that Debian is a crappy base for an embedded distribution but actually it works a lot better than people expect it to. I think as Colin says you're doing a brilliant job. The only thing is please take our patches about weird cross-building crap. Okay, it's important. It doesn't matter if you don't understand just put it in. Debian Edo is really happy Debian user stable user so thank you for your for supporting your packages and stable and also for providing stable back ports that's very useful. Thank you.