 So today we have Joshua Powers from H-Linux, Raphael Herzog from Kali Linux, Solvegg from Tails, Nolan, sorry, I forgot your last leak, from Cumulus Linux, Mathias Klump from Tanglue, and Adrian from Rescatux. Okay, so we'll start with introductions. Do you wanna say a little bit about your distribution and your involvement in the distribution? All right, so again I work for HP, working on H-Linux. We basically produce a derivative for HP's use, particularly on HP Helian, which is our cloud software. My role is I work on the distribution in QA team. The QA side is obvious. We have lots of testing with both HP hardware, and making sure that our custom kernel works there, as well as making sure we're producing a stable system. And the distribution side is working with our partners on using Linux, and working with external partners as well, on getting their drivers into our distribution, and working with them to add some advanced features. Hi, I'm Raphael Herzog. I'm working on Kali Linux, which is a penetration tasting distribution. It's built on top of the event testing. I've been doing packaging and infrastructure stuff to help them run their distribution. Hi, I'm Salvegg. So I contribute to Tails. We gave a talk a few days ago, so it's about, it's a Debian Debit based on Debian, focused on privacy and security, and it's a live system, so you don't install it. Hello, I'm Nolan. I work on Akema Linux, which is a derivative of Debian that is used on network switches and routers. So those kind of when you pizza boxes with a bunch of ethernet ports on the front. So it's a little unusual in that respect. Hi, I'm Matthias. I work for Tanglu, which is best described, unfortunately as very similar to Ubuntu. We have a strong focus on the non-technical Linux unexperienced end user and desktop user. So we ship latest KDE and GNOME in a time-based six to eight month release cycle. And we also want to try out some new things like atomic upgrades, up for radical simplified installer. So yeah, that's what Tanglu is about, new technology and end user focus. Hi, I'm Adrian. So Rescatus is a table life based distribution. We use LXDE as a desktop environment. So its main feature is that it has a point and click with it, which is meant for rescue tasks. So for example, in only three steps you can recolor a graph and then you can boot into a new Linux again. So I did the line in talk on the 15th. It's already online, so you can watch it for more details so that we don't get a longer presentation. So in the audience, we have Zego from Laurentius. Yes. So you want to introduce it, Ivan? Okay. So I'm working on the Debian side of MOS, which is Mirantis OpenStack. And that's a derivative of both currently Ubuntu trustee and CentOS, which is based, which is focused on deploying OpenStack on medium to large scale clouds. Also in the audience, we have Gardens from Lernstick. We have some grimoire people here and Emmanuel Casper for Proxmox. I see John Vert from SteamOS and there are several people from Ubuntu. So you can ask questions of any of the derivatives in the room. So anyone to kick off with a question? I'm not sure which camera is pointing at me, but... Oh, I'm good. Excellent. Fairly simple question, I guess. Why Debian? Why choose Debian out of all the other releases or even making your own? What sort of advantages are there for you? What's the importance of what Debian does for helping you to produce your distributions? For Tanglu, one more important point is that I am a Debian developer, so it was kind of the obvious choice to base on Debian. But also Debian has a very strong focus and quality assurance and the packages we usually get from Debian are really well tested and are working well. So, yeah, that's definitely a thing. Also, creating another distribution on top of Debian is not that hard and there's lots of support in the Debian community for derivatives. So it has a tradition of supporting derivatives. So that's something which definitely helps us and Tanglu. So on the HP side of things, strong community seeing thousands of people kind of contribute and being able to contribute ourselves directly to you, not having to go through any kind of commercial entity, but being able to go directly to the community and work with the community. Also the strong sense of denoting between free software and non-free software, right? And that's a big thing for us in giving to our customers and being able to say, look, these licenses have been checked out, we haven't violated any of them, we're not gonna cause you any pain or cause any legal issues. And so working with Debian there as well has just been awesome. First off, I'm just a long time Debian user, so I couldn't really imagine using anything else, but from a practical standpoint, there's several key things. I mean, Switch is one of sorts of weird CPU architectures and Debian supports them all. Like Debian supports ones that no one's ever made a switch on. So that was a huge advantage in that we didn't have to set up a bunch of cross infrastructure and figure out how to recompile packages on some architecture they've never been piled on before. And also, I mean, de-package is cool and all, but the actual, I think, true value is the kind of care and effort that people put into making consistent packages that work well together and kind of work together as a piece of whole, which you often don't always see with other distributions regardless of what kind of technology choices they make for packaging. For Kali Linux, the choice was rather ob-voiced. Obvious, sorry. It's a successor of Backtrack, and Backtrack at that time was mainly handmade with scripts and stuff like that, and they really wanted something cleaner and more reliable. So they picked Debian also because, well, it's a quality, because of the quality, and they can concentrate on what matters for them, which is a large, or not so large set of package that they add on top of Debian. So I don't want to spend time too much on what already works, and then I can concentrate on the new packages. We were based on Debian with it, Debian stable, but we switched recently to Debian testing, so there's no more quicker feedback loop, which is useful, but it's also at a cost for us because, well, we have problems more regularly, and maybe we can talk about this later in the talk. Okay, so one reason about choosing Debian is because you can choose, I mean, Debian is stable. So when you want to build this live distribution that fixes your system in order to boot again, or in order to be in a good state, you want it to be stable. And the other reason, you know, the user target for Scatax is mainly Ubuntu people. It's new-byte people that it's beginning in this new Linux world, and do not very much how to use the common line interface in order to fix their systems. So why we are not developing with Ubuntu? The reason is because if we develop in Debian, then every Debian derivative can benefit of our improvements. So that's another reason. Debian has a base system for other distributions. So there is this base on Debian also because we have Debian contributors already and developers. It works, it's fabulous, we love it. We don't know any other distribution that well, so, and it's good. Okay, we have a question from DKG. Hi, I wanted to ask sort of the counterpoint to Neil's question, which is why not Debian? That is, you're doing a derivative distro, which by definition isn't Debian, and in some cases that seems like that makes lots of sense, and in other cases it seems like maybe your changes could go into Debian directly. And I wanted to know what you see as the advantage of being a derivative instead of working directly contributing to Debian and how you see your relationship with Debian as an upstream. I don't mean this as a challenge, like everything should be in Debian. I'm just, I wanna think about what the different advantages are. We're not a Debian pure blend or something like that, mostly because we are a live system and it's part of the design goals of Tails to be a live system because you cannot ensure the same level of privacy and security if the system is installed and the user can keep changing it. So, it had to be a derivative in our case. Also, we have a very specific use case, so most of our changes, well, we bring back most of our changes in Debian, but there are a few changes that will never be in Debian, probably, so. For Kali and Nix, they're both technical and non-technical reasons. And technical level, well, quite a few software are not really suitable for Debian main, at least, because in security fields, there are many authors who say don't use this software to attack service and this is kind of licensed, it is not in line with DFHG. There's also the fact that we have to ship some software on short notice and it tends to be, for example, a large Ruby underrated application with lots of Ruby gems which are not yet available, so we tend to do some acts like integrating a bundle of those Ruby gems in the proper binary package which you could not do within Debian. And also, on the non-technical level, Kali Nix is sponsored by Offensive Security, so they really benefit from the image and its marketing tool for them, so it would be harder to have the same level of, well, it would be harder to create such a brand entirely within Debian. I guess that's the two main reasons on our side, but that said, as I said, we're based on testing and we're trying to push more and more stuff within Debian because now it starts to make sense because the feedback, the time to, between when I uploaded to Debian and when it integrates, when it comes back to, into Kali is much shorter. So for each Nix, our objective is to work directly with Debian and just consume Debian entirely. There are times where that just isn't possible, but going back to that, if the community is working and producing a great distribution that we want to consume and use, and likewise we should be getting back to that distribution. So any of these bug fixes, any of these enhancements that we're finding, we want to contribute those back. When we were still on Jesse testing, I think we were doing a pretty good job of that, finding issues, getting them fixed and not carrying fixes on the side. Now that Stable is here, we're going off and doing some more advanced feature development and stuff like that. And then this is where we see ourselves kind of having to carry, like in my talk, I called them foreign packages. These are extra packages with even newer features in them that probably wouldn't make their way into Stable. Another example of that would be like the kernel. We're carrying newer drivers, like I mentioned earlier, with the more advanced features and those are things that we like to see down the road into Debian, but not immediately. So those are kind of the things that stop us from just consuming Debian entirely. So we're based on Stable and we put out a release every three months, so the release cycles don't exactly line up on that score. So, but for the most part, we tend to contribute up to upstream upstream with the intention that we will then pick up the changes that we contributed in the next stable release of Debian when they pull down those changes from their upstream. Yeah, for Tanglu, there are a few changes which we do like doing a time-based release cycle or changing the installer or shipping newer versions of KDE and GNOME on this release cycle which we could not easily do in Debian. Since if I would propose let's do a time-based release cycle it would not be in the scope of Debian because Debian wants to do stable and wants to create a stable release while we want to create a smaller and shorter release cycle. So that wouldn't work and also we do some changes on packages which would turn out to be controversial because Debian has also the desktop users and more advanced users as target group. So all the default choices we do for the user might be a problem for Debian additionally to the release cycle issue which would be a problem to establish that in Debian. Also because we advance faster, it's more difficult to get everything done in Debian since the maintainer reaction times are sometimes slower and we also sometimes do not upgrade the copyright files so doing the same stuff in Debian would be more complicated and would take much longer. So that's why it's a derivative. So in Roscattox the main reason why not use Debian directly is because it's faster to develop so that you don't have to care about Debian specific stuff. Then we currently use some Ubuntu PPAs for boot repair and some patches packages. So we would have to migrate these packages into Debian itself to send the patches and so on. And then there's the thing about the release cycle because we don't always have time to match with release cycle. But the Rescap package, the main app in which it's the method in Roscattox is called Rescap. So it is not packaged yet. But I see Roscattox being converted into our pure Debian blend in a few years because I don't see why not. So that's it. So actually two questions. Which architectures do your derivatives support? The second one, when you upgrade certain packages like OpenStack, RKDE, OrgNome, do you know what kind of packages you break? So usually you care just about the subset of packages when you ship new features. So do you at least know what you break in your distributions by these upgrades? Okay, so in Roscattox we support 386 or whatever the kernel chips here with the current Debian distribution because we see it was, I think it was 486 and now it's 586. But the distribution has also an AMD64 kernel so that you can use the 386 packages so that it works in both 386 and AMD64. Well, we cannot test more architectures. And the thing about packages upgrading and breaking things, it's just testing again the use cases that Roscattox must fix and see if it works everything as intended or not. So Tanglue supports AMD64 and I586 and we are currently thinking about also supporting ARM but that requires hardware and some more resources so it's to bootstrap it. So yeah, it's planned but not yet in action. And regarding what do we know what we break, we pretty much know exactly what we break because we are rebuilding all packages coming from Debian and sync with experimental, unstable and testing. So yeah, and we have QA tools in place which tell us if we broke something and what we broke and so we can fix it in the development suite in time. So yeah, we are aware of that. So for architectures, let's see, we support AMD64, PowerPC32 and ARM and no doubt ARM64 will be coming at some point but it's not here yet. We're more or less at the mercy of what chips people choose to put in these switches so it's kind of not entirely up to us. And as for breaking packages, it's never been a huge problem for us because as I said, we're based on stable and there's only a handful of packages that we patch aggressively, mainly Quagga and the kernel and STP and things like that that are very networking centric. And so we have a constrained set of things to test so we just test them. For Kali Linux, we have four architectures, AMD64, E3, and I586 ARML and ARM outflow because actually a lot of Kali Linux is embedded in small devices on phones and stuff like that which is quite useful in penetration testing cases. Typical case asking someone to let you recharge your phone, keep plug your phone and it actually enables a network device on the, which will bridge your traffic over and intercept everything. So that's that kind of stuff. And for the second question about, do we know what we break? Not always, we have quite a lot of bug ports. We used to have quite a lot of bug ports and it's sort of manual verification but now since we are based on testing, we have our own Britney instance. So at least we detect when we break dependencies. Dependency do not cover all problems that can arrive but at least we ensure that all package are in stay label, dependency-wise. We also have a Jenkins instance that installs all of our meta packages once a day so we discover such problems, also problems in post-test and stuff like that quite soon. And if you follow my monthly reports, you will see that I have filed quite a few bugs the last month about such problems that we have been introduced in Kali through the ambient testing. So we originally started shipping with, or supporting I-36 and AMB64. We dropped I-386 after lots of discussion and we're kind of investigating RM64 right now and trying to get it going as well. As far as breaking things, we do plenty of QA, making sure that everything installs properly and builds properly, still building some of that work out. But again, just to be honest, and I'm going to continue to work on that and have some more features that will be useful by others, I hope. But that's hopefully the main points I see for the future right now. So for HLINIX, I think as I said earlier, working with the community and making sure that we're not entirely becoming almost a fork of Debian, but making sure that we're working, contributing directly into Debian and getting them pulling directly from there. In five years, hopefully that becomes even more and more natural to us. It's something that's a little new and still kind of getting the hang of what that process looks like. I certainly hope in five years that we've figured that out. Also, I think some of the trademark discussions we've had earlier this week around getting Debian's name out there, getting it known to our customers and what we're doing and how we're using Debian so that they're more aware of it, as well as getting rid of other distributions throughout HP potentially that people are using Red Hat or CentOS throughout things. Maybe they could be using us instead. And again, adding value back to whatever needs that they have working with them and then working with the community and then seeing those things put into Debian. I hope that in five years, the idea of running a vendor proprietary network operating system on a switch or a router sounds about as insane as running a mainframe. Yeah, for Tanglu in five years, I have no idea what happens, but I want to see it as a bigger community of awesome people working together on the distribution in close collaboration with Debian so we can do the things which Debian with this longer release cycle can't do and complement it on the desktop while Debian is running on servers or on more conservative desktops. And yeah, this is one thing and also I would like to see Tanglu running on mobile phones. There are some faint ideas in the future of how to do that, maybe, because now Canonical and Ubuntu have paved the way of making it possible to do it, but yeah, this is a longer discussion, but yeah, in five years it might be possible. So, Rescatax, in five years I see no future. I mean, I prefer Rescatax to be a Debian rescue blend so that I can focus better on the Rescap program and well, so that I don't have to care about default rescue application that should be on the live CD. And so, for the new future, there's the, we have to release Rescatax based on Jessie because it's currently based on Wisi and Beta so it has to be stable. And we have to also sort out the secure and aced Linux support so if anyone knows how to deal with secure and aced Linux from a Debian live, please contact me. So, that's it, in your future release, it's going to be released soon. So, in the future of Tales, that would be very, very fabulous. If we didn't need Tales anymore, if the government stopped spying on their people and other people, if the company stopped wanting to sell your information, but that's not gonna. But that's not gonna happen. So, let's hope there are more privacy-focused distribution so that the different use cases are covered and for Tales specifically, more hardening for sure, and maybe ARM and whatever you want to contribute to make exist in it. Any more questions? Probably would sound more like derivatives one-on-one quiz but did you consult to the page on wiki-debian.org derivatives guidelines? There is such a page, by the way. Raise the hand who did. Yes. Oh, nice. So, how do you deal with the problem that you have modified or uploaded new packages? How do you channel off the bug reports to your derivative? Because so far I think we didn't improve with ReportBug to support, let's say, alternative bug reporting systems, right? We have just primitive email or the system we use in Debian itself, right? But ideally we could support like multiple backends for the ReportBug. So, in Tales, we have a small subset of Debian packages and we don't have ReportBug. So, we ship WhisperBug, which is a way for users to send encrypted emails. So, we get those bug reports and we open bugs either on our bug tracker if it's state-specific or upstream, sometimes in Debian, sometimes in other upstream. So, we have kind of a filter between the user bug report and the... So, our users bug report never happen directly on a bug tracker. They go through us and we dispatch them. So, this is one of the problems that we tried tackling right away with HLinux. It was a big concern of ours that all sorts of questions would be coming to the Debian community, which shouldn't be going to the Debian community. So, our original thought process, and I talked about this a little bit during my derivatives talk, but we were taking every package and actually inserting the word HLinux into the version name so that they would know who's where it came from. They would know that it's coming directly from HLinux, even if it was a package that we weren't even changing. They would come to us so that, even if the community saw it, they'd go, this is HLinux, this is HLinux's problem, go deal with them. After kind of doing this for a first release and kind of thinking about it a little bit, we said, you know, this does not make sense for the packages that we're not changing. Those packages are just the same as upstream. If they want to go upstream to report an issue, go for it. That's totally fine. Ideally, they should be coming to us anyway, but that didn't make a lot of sense. For any package that we are modifying source or making any kind of changes at all, it still gets the HLinux string added into the version so that it's very clear that it is still coming from us so that, you know, if the community sees it, they'll know they came from HLinux. It's not from us. So that's kind of how we've been handling it. On our side, we have no special solution either. We got a few users who misreported bugs to the Debian bug tracker. That's why I still have it on my to-do list somewhere. I'm very down to fix report back because, well, it's easy to detect when you're running a derivative and it should really not allow filing bugs to live in directly and instead have some way to redirect it somewhere to some other email or have some special instruction. I think there are, instead of the dpkgvounder file, there's already a link for where to report bugs so actually report back should just display this link instead. So it should not be too hard to implement. There might even be a wishlist bug already on report back but nobody did it yet. Instead, we just advertise widely that we have a separate bug tracker and usually know about it so they fight bug on our side except for a few beginners who don't read stuff. So we don't ship report bug. Instead, we have actually a support team that people would come to with their problems and then we have our own internal bug tracking system. So we don't tend to file bugs upstream. We tend to fix them and send the patches upstream and then let it kind of filter back down to us and we'll carry back ports of those patches as necessary. Yeah, our users like to file bugs at the forums but we are managing to teach them to file them at our bug tracker which is independent from Debian and which is the place where bugs should go and in case the issue affects Debian as well, we are reporting it to Debian again or we are just linking the bug because we have a field in our bug tracker called upstream bug where we then connected with the Debian bug so we can track both and yeah, don't get lost in the bug flood. So yeah, that's what we do. Okay, so Roskatax, if we find any bug, so we report to upstream but sometimes you also need to report to Debian package itself because sometimes upstream do not reply and maybe the fix is important. Then there's about gathering bugs. Mainly it's done within the internal chat that the users can use inside the distribution and mainly the bugs that we file into Debian are Debian live improvements so that we can make our users life easier and Debian life even better and another problem about gathering bugs in Roskatax because Roskatax is mainly a tool for fixing your computer is that when you try to fix your computer you try several tools and you want your computer to be fixed as soon as possible so if Roskatax does not work for you, you are going to search another tool so that it works again and you can use it so we might lose some bugs because of that. And finally what I would like to see somehow is the book tracker tools and web-based things so that they connect one with it's one another so that you can go into Ubuntu and say, into Ubuntu book tracker and say that Debian is the parent book tracker and somehow the links, the bugs are linked and forwarded, yeah. I think that's something like that already exists, yeah, that's it. So I wonder if there are things that Debian could do differently that would make your lives better as derivatives and I want to encourage you in answering this question if you have things maybe if you don't but if you have any ideas, be bold, right? Debian is a place where large changes can happen across the infrastructure and if there are things that Debian doesn't currently do that would make it good for you just imagine what are those things? They might need a team of people it might need years of work but what are the things that if we could do it in Debian would make your lives easier as derivatives? We'll start with Rigo. So what happens to me is that I maintain OpenStack in Debian, okay, and some packages, I don't own them and we have strong package ownership in Debian so a package maintainer can decide to upload whenever he wants into CID and break all of my world. So that happened recently with Python Mock which created eight FTBFS in my packages. The maintainer of Python Mock maybe didn't even know it would do that to me and so it happened also with SQL Alchemy as well. So one thing that could be really improved is to better know when and where to upload like if the Mock maintainer has uploaded to experiments all instead of CID then that was the correct thing to do for me but he didn't know. So if we had objectives in Debian to say okay we are going to support this software and do what it takes to do it in a better way so that we don't break things when we upload then that would be a huge improvement at least for me. So I think for us, just quickly thinking through it immediately the reproducible builds would be great and getting the trademark stuff so that we can again start using Debian kind of in our name or in our product so that people realize oh it's just Debian with a little extra, that would be great. Big pie in the sky kind of stuff just thinking about it. I do think we're kind of running into an issue with cadence, right, with Stable. Stable's great, it is as the name implies stable and it definitely is that but two years down the road we will be carrying all sorts of different foreign packages and all different stuff to get newer features to our customers since we're working on cloud software. What does that look like? Is there a way to get that even faster? It should be we'll be using tested testing builds something to that extent. So I think the Debian solution for that sort of thing is back ports. You can upload back ports. You don't have to be the maintainer to do that as well. On our side, what would be nice on Debian side is to raise awareness that some package in Debian actually used by derivatives and maybe considered twice before removing them. I mean that's the topic possibly for the QA team or for all the people who are doing or teams who are cleaning up their list of packages. I guess I will bring back this topic in one of the later both because that's really something that can be automated. Well, since we're based on testing any effort to improve testing would be nice as well because we wanted to be really working every time. So in tales we are of course interested in the reproducible builds that some people are working on so that when it's in Debian, it will allow us to have it in tales. And there are probably a few things but in general Debian is a great upstream. So thank you. So this is more a personal opinion than Rescatus one but anyway, so one thing I think Debian lacks of is communication or I mean Debian communication should be improved. And for example, in the Debian review this week there's work to be done. We are going to do some work with Paul later and the thing is that Debian is not only a distribution but meta distribution that all other distributions use as a base. And sometimes in Debian people say too often bring it into Debian. So bring your changes, bring your packages into Debian and this is right, okay? So we want everyone want every package and every improvement into Debian, that's right. But we should improve our way of dealing with derivatives. Yeah, on the tanglo side it would be really great if the tools which are used in the Debian infrastructure would be more tailored to also work for the better for derivatives because for example in Duck there are still some hard coded distribution or suit names like unstable or testing which we patched out to make use of Duck. So by getting the idea that other derivatives might want to use the same stuff which is running on the Debian infrastructure which would be a great thing. And also while switching to the 3.0 source format standard it's a nice thing because it means we can run automatic rebuilds of packages of source packages. And another thing which we really love in tanglo is that in case we do a change and it can't be directly integrated into Debian because it might be a controversial change or you might not want it in there that you still include it and make it a conditional option that it's included in case the package is compiled in tanglo. So if you recompile the package on a tanglo system you will get this patch while you compile it on Debian you will get the previous behavior like it is on Debian. So this means we could drop the difference and basically have the patch upstream and in the end just recompile it to have the different behavior. So yeah this is something awesome and it would be good if more people could make use of it. Also for Ubuntu I think this would be of great help. So but in general Debian is an awesome upstream and there's not much to complain about. So I think we're out of time. Later today there's going to be some work on derivatives related infrastructure in Debian and I think tomorrow or the next day there's a buff between derivatives and there's also another derivatives work session as well. I thank you everyone for coming and see you around.