 Hi everybody. Welcome to the Fedora track, I guess, at DefConf. And so to start the day properly, we'll be talking about Debian. So just to introduce myself, my name is Stephen Kitt. I'm a Debian developer, but I also work for Red Hat. It's not incompatible. My day-to-day job doesn't involve packaging, but my day-to-day job does involve running Fedora on my laptop. This is one of the things I like, and it's part of the freedoms we have in free software is to be able to modify stuff. And so I started looking at Fedora packaging. So this talk is going to be about my thoughts as I discovered the Fedora packaging stuff. I'm going to be talking about Debian as well, because I imagine lots of you aren't familiar with what goes on in Debian. So if we want to compare, it helps to know what I think is important in Debian and what I like about packaging there, and then look at how things are done in Fedora, at least how I understand things are done in Fedora. I don't have answers. If you think I'm saying I have answers, I'm probably wrong. See the quote up there. Lots of this stuff is quite complicated, but the underlying thought behind my talk is really that distros are getting more and more complicated. We've got more and more work to do inside distributions, but distributions aren't exciting as they once were. So it's harder to get people to join the effort. And so I think what we need to think about focusing our efforts more, doing things in the right place so that we avoid duplicating. I think this perhaps is an extension of the keynote, extending collaboration beyond the Red Hat distributions to other distributions. And Debian is just the example because that's what I know, but I think it applies more generally. Some caveats. So you can assume I'm wrong about Fedora. This perhaps is my way of getting into the community by saying stuff that's wrong and being corrected. If I point something out in Debian, don't assume that I'm saying that I think it's missing in Fedora. It's just that I think it's interesting in Debian. And of course Stockholm Syndrome, I've been using Debian for over 20 years, packaging for quite a long time. So if I think something is nice, maybe it's just because I'm used to it. How I came to packaging helped set the scene a bit. So I went to university in the mid-90s in the UK. And back then that meant Solaris on spark stations to a large extent. But this was just at the time when Linux was becoming usable vaguely on the desktop. So I was running Slackware 3. And it was great to be able to run or at least try to run some of the software we used for our studies on my own computer because it meant not having to go to the labs all the time. Being able to work from my student room and so on. And so that's how I got into sort of porting package, porting software at least. This was also a time when software development in the Unix world re-exploded in some way. And so lots of students, me included, were interested in running new software like MUT or SLRN or JED or Lix. And this was also a time when storage was expensive still. So we had small quotas. And if you wanted to run MUT on a university computer to read your university email, the admins weren't going to compile it for you. So you had to compile it in your own home directory. But that meant that you were eating into your quotas. And so with some friends at university, we set up with the agreement of the university as well. We set up the system where we had a shared home directory for hosting software. And so I started packaging software there using GNU Sto. And then I joined the administration team for the university, and the student hosted computing services which are still called TARDIS at Edinburgh University. And so I did more Sto-based packaging there. Then in the late 90s, I won a competition and won a bunch of CDs with Debian on them. And I started using Debian and never looked back. Then I became a Debian contributor. So in Debian's definition, a contributor is anybody who interacts with the project in some way. So I started filing bugs and that made me officially a Debian contributor. Then I started maintaining a package. The usual story, some software got orphaned that I cared about. In this case, it was the joystick calibration tools because I like playing games on my computers. And so I became the package for the joystick packages and actually became the new upstream for that as well because it had been abandoned upstream. So I now maintain Linux console tools. So that's input attached, JSCal, JSTest and so on. And then eventually I graduated onto the next level, which is Debian maintainer. That's when somebody in Debian entrusts you with packaging your software with supervision. If you're just a Debian packageer, every time you upload a package you need to get sponsored. There's the same kind of mechanics in Fedora, as I understand it. You need somebody to check your package and upload it for you. Once you're a maintainer, there's no oversight on your small selection of packages. This also gives you access to some of the benefits of being in Debian, like for instance access to an LWN subscription, email from Facebook and Google recruiters, and access to all of Valve's catalog on Steam. Which I find quite ironic given the freedom values in Debian, but never mind. As a gamer it was quite interesting. And then I became a Debian developer because I think that's a bit like a trusted packageer in Fedora. Once you're a developer you can upload anything you want to the archive for any package. So if you try and do it for something that's not yours, there are social barriers to that, but technically you can. And you also get to vote on all the major decisions in Debian. What's it like when you're actually developing in Debian? So to cover that I thought I would go over the dashboard that we have for packages in Debian because if something appears in the dashboard it means that somebody took the time to write the code to put it in the dashboard which means that somebody cares about it. And cares about it enough to do all the social engineering and so on that goes into getting stuff changed in Debian. So you can see here this is the top of the screen with my name and then the list of packages, source packages on the left. So I have 57 source packages in Debian main. I'll explain what that means just a bit later. But going through all the columns shows what's important in Debian. So obviously bugs, I think that's the same everywhere. And we have two big categories of bugs, RC bugs and the rest. RC bugs are ones where if your package is important enough you'll delay the release, otherwise your package will just be chucked out of the distribution. So you really need to care about those, keep that at zero, otherwise your package is liable to go away. And the rest is, well, stuff you haven't got around to fixing yet. Then we have version information. So this is the versions of the packages that we have in Debian and in a favored downstream Ubuntu for whatever reason. And so we have the stable version which is the one that we still support and provide security fixes for and important bug fixes for. And then testing which is the next release that's coming along. So we are in the process of freezing it, so it'll be coming out soon. Unstable which is the staging grind for everything. And then experimental which is where you upload things that aren't supposed to be part of the release yet. And so version numbers progress from the right to the left and then from the left to the right into Ubuntu again. We have also another type of versions which are the ones underlined there on the left. Backports, so for some software where the release cycle of Debian is just too slow we provide backports that are a bit like Apple in a way. So users of Debian stable, the last release can install a newer version of the package quite easily that's been built for stable. We don't have any dependency issues but they get access to new features and that package there in particular is the gog.com downloader which needs to be updated every once in a while. And the excuse thing, that's a linked page that tells you why your package hasn't migrated from unstable to testing yet. Then we have a VCS column. This is a link to where the packaging is stored and this being Debian there are a number of different ways to use your package. I tend to use Git. With one specific workflow, this being Debian again, there are different workflows even if you use Git. And the tick mark means that your VCS is up to date with what's in the archive because sometimes people forget to push their repos when they upload something to the archive. Then we have the buildDs, buildDemons and this links to a page like that which shows the state of your package on all the architectures in Debian. And this is a big difference with Fedora in Debian. We have tons of architectures. The white ones at the top are release architectures so if your package doesn't build on one of these then you might get kicked out of the distribution. So there's a bunch of stuff that we care about there that can cause pain sometimes like RML at the top for old ARM systems. These are so old that they don't support certain C++11 atomic operations and so there's software now which will never build anymore for these machines. We have new architectures like ARM64, MIPS64, YELL, PPC64, YELL and then older ones that are slowly dying well like RML, perhaps MIPS I guess. And then the gray ones, they are either upcoming architectures that aren't ready yet like Spark64, PPC64 or old architectures that used to be release architectures and have been demoted like alpha or architectures that have never really made it like SH4. And so it's nice to know what's happening on these but it's not bad if your software doesn't build on them. But if you care about an upcoming architecture you can keep an eye on it there and the portiers are always glad for help. Going back to the dashboard, the next set of columns are quality metrics and a bunch of different tools that run automatically on the whole archive every so often and check things for us. So Dev Check is a set of basic tests for package dependencies. Lintian is a bit like RPM Lint. It does lots and lots of different checks of varying types. It could be spelling fixes, updates or licensing issues or whether your software uses Bind Now or Pi, things like that. And in Debian it's become the way of driving change. So we have this weird thing in Debian where Debian policy, which is the equivalent of the Fedora guidelines, isn't normative. It's descriptive. So things end up in policy once they're the consensus. So if you change behavior, if you want to get a change into Debian, you convince enough people that it's a good thing and then once it becomes the norm, it's codified in policy. You don't change policy first to drive change. We used to drive change through the release team and release goals. That's not happened over the last few releases. People are thinking about it again and there was discussion of the last Debian conference. But in practice, what happens now is if you want to drive change or introduce a new check, you add it to LinkedIn and then people see these warnings on their packages and so if they care about their package, then they tend to fix them. This isn't address packages that have dormant maintainers or maintainers who don't care so much, but that's how change has... Well, it's the emerging way of doing changes in Debian now. Then few parts. That's a tool which installs and uninstalls your package and makes sure that that happens cleanly so you don't leave anything behind. Then we have CI. I think this is probably fairly similar to the CentOS CI that was discussed in the keynote. The idea here is that you can add tests in your package, what we call auto package tests, that are run externally to your package. So these aren't tests that run during the build. These are tests that run after the build. We take the artifacts that the build produces, install them and then run your tests, along with any other package you want. This is really useful because, for a library, you can write an auto package test that checks that once your package is installed, you can link code that uses your library and this is a better test than a build test because during the build, the build knows where stuff is, but you might still have messed up your package and users can't use your package. So I maintain tool chains in Debian and this is one of the things I've been meaning to do for the last several years is add CI tests for my stuff, but I haven't got around to it. Rep, that's the reproducible build stuff that Hogar talked about yesterday. So a tick mark here means that your package is reproducible. If it's in brackets, it means that the current version of your package hasn't been tested yet, but that the previous one was. I think this is really important in Debian and go and watch Hogar's talk if you didn't see it yesterday. Then we have another column, which is pop-con, the popularity contest. So this is a fairly meaningless number, really, which measures how many people have installed your package and are willing to tell the world about it. Even installed doesn't mean all that much because lots of people have got tons of packages installed that they don't use and we sort of try and measure that, but it's imperfect. Things like Rella time and so on mean that we can't measure things very well. And so it ends up a bit like the pile of books that you've got at home that you mean to read one of these days. But it's still useful to get some measure C trends if your package is going up or down and then when we decide whether to remove something, obviously if something's got a high pop-con, we'll think about fixing it rather than removing it. But if it's got a low pop-con, lots of RC bugs, then it's nice. Just to give you some measure, the highest pop-cons are around 200,000. And then finally, the last column in the dashboard is the watch column and so this is what upstream is on. So we have a flag in the package. Well, a piece of information in the package that tells the tools that look at this how to check what the current upstream version is. And you can see here whether you need to update your package or not because same as in Fedora, we try to keep up with upstream as far as possible during Debian development. So that's all the columns I've covered and if you scroll down these dashboards, there are a number of different sections that reflect different things in Debian as well. So the first one is the main section. This is the real Debian distribution. What is DFSG free? So free according to the Debian free software guidelines. It matches our licensing, all the dependencies are free software as well. All the build dependencies are free software too. Then you have Contrib, which is not officially part of Debian GNU Linux but is hosted on the repository. So this is free software but that has either a non-free run time dependency or a non-free build time dependency. So for instance here, this is Bazalisk 2 which is a Macintosh emulator. To run it, you need Macintosh ROM which is obviously non-free so it can't be in the main section. Then we have... So that covers the different types of software. There's also non-free which is completely non-free stuff like the NVIDIA drivers and so on. Or firmware that you unfortunately need to run lots of computers nowadays. Then we have different sections that cover different ways software gets into Debian. So non-maintainer uploads. This is software that I've uploaded even though I'm not the maintainer of the package. So usually that's to fix an RC bug when the maintainer isn't responding for software that I care about. Then we have sponsored and other uploads. So that's software where someone else has packaged the software and has asked either me specifically or the community in general to review their package and sponsor it for them. And this is a great way of getting new contributors into the community slowly without the sort of administration that's involved in joining Debian properly. They can start packaging and get sponsored. And you tend to sponsor the same packages several times. And that creates relationships between maintainers. And eventually you hope that the maintainers will graduate to become Debian maintainers or Debian developers. Then we have another section which doesn't correspond to anything in the archive. This is intense. Well, this is more about the future of packaging. So own WNPP bugs. And just now I can't remember what WNPP stands for. But basically it's the list of bugs that say I intend to package this and I want this to be packaged for me. I have an ITP for Piper because I maintain libratbag and ratbagd so it makes sense to have Piper in there as well. But Piper isn't really ready yet. So it's still sitting there. And then finally, the last one is packages for which I, as a Debian developer, have given others Debian maintainer rights over. And that pretty much covers the dashboard. So let's talk about Fedora now. As I said, I work for Red Hat. So when I joined Red Hat, I got a laptop provided for me with CSB. But my colleagues told me I'd be better off running Fedora for the kind of work I'm doing. Well, they were right. Fedora turns out is a fantastic developer distribution. And I find it absolutely great. You go and look for Fedora on the internet and you end up at getfedora.org and it's a very, very focused website and a punchy message, choose freedom, choose Fedora. That makes me think of train spotting. And lots of people choose not to choose freedom. They choose something else. But who needs reasons when you've got windows or something like that? Look at the quotes on Wikipedia. And there's just three big squares there and you go over them and they change to download links. So you're just a couple of clicks away from an ISO. You stick that on a USB key and later you're running Fedora, which is fantastic. Discovering Fedora as a user, like I said, in focus and you can see that in the distribution as well, it doesn't claim to do as many things as Debian does, although Fedora now has a huge number of source packages as well. But it works really well. So that's testament really to lots of people. But distributions, Fedora specifically, other distributions, upstreams, it's not hard to make a Linux system that sort of works. It's still an awful lot of effort to get a distribution that works really well and Fedora does work really well. Fast updates. And of course at home I tend to use Debian testing, so I'm on a rolling distribution. It's a bit more like Arch Linux than Fedora. But still getting a stable distribution out every six months and keeping it running all the time with stable kernel updates and so on is no mean feat. And it's a great development environment. I find it quite interesting that Fedora has Eclipse packaged, Debian has NetBeans packaged and has been trying to have Eclipse packaged for years, but it's a really painful thing, packaging Java in general, but Eclipse in particular. DNF, so that takes some getting used to compared to apt. There's some good things about it, like rollbacks. I find that quite fantastic when you need to use it. The way it doesn't recover well when it crashes, that's not so good, but it's been discussed on the mailing list, so it'll improve, I guess. And then RPM Fusion, so that's Fedora's part of hypocrisy, I guess, a bit like Debian's part of hypocrisy with contrib and non-free that aren't part of the distribution but that lots of people end up needing anyway. Fedora has RPM Fusion for that. That's out of scope really. And then like I said, I like being able to use my freedoms and that means packaging stuff. There's stuff I package in Debian that's not in Fedora that I use. And so I started looking at packaging. And this is another area where you can see Fedora's focus and the effort that's been spent on websites and so on. You go to the Fedora website and there's a website dedicated to what can you do for Fedora. And it's not about just packaging, there's a whole bunch of different things. So when I was preparing this talk, I went back to the site to take some screen grabs and the one that came up for me that time was design. So I would like to be able to design, but I can't. So nope, nope, nope. And the second one that came up was packaging. Yes, that's exactly what I was looking for. So you click on that and you want to become a package maintainer. Yes. And that takes you to a wiki page. So the design changes a bit, but it's still... Right, okay, so somebody in the audience points out that there's a new version of wiki media that's being set up. So this should improve. But even despite the deficiencies of the wiki medium here, I still find it interesting that the first thing you land on is a flow chart that shows you the whole process from start to finish. So you know what you're getting into. And then it goes into detail and the detail is this. And I thought it was interesting to go over that because it shows what's important to the community. How you get people involved in your community shows what's important for your community. And the first thing when you want to get involved as a package is read the guidelines. And you get links to two pages, one that shows you the basic workflow for building a package and then another one that goes into a lot of detail. And you really need to read both. And I think that's very good that if you want to... It says that you want to drive the level of quality, at least for packages, if you recommend that they start by reading all the guidelines first. And another thing I like about the guidelines is that they focus on one way of doing things. So when I started looking at Fedora packaging, this was RPM build. Now it's Fed package. So that means that things change in Fedora as well. But you don't end up with the same kind of situation as you do in Debian where newcomers come along and they say, how do I package this stuff? And they get a dozen different answers. And Fedora, there's one answer, Fed package. Then you start doing the administrative stuff. Create accounts. Join mailing lists. And this is an important part of being a member of a community on the mailing list. So if you join them at this point, you've got nothing to say, so you'll start receiving email and you start reading stuff and getting a feel for the community. Ensure suitability. So this is about licensing and stuff like that. Before you start work on your package, make sure that it's not going to be for nothing. This is quite complicated. I saw some reviews just recently where somebody had gone through the effort of doing all the packaging and in the review, it was pointed out to them that there was a file somewhere that had a bad license and so the package couldn't go into Fedora. Understand responsibilities. This is important as well. If you become a package maintainer, it's not just about getting your software, you're interested in, uploaded into the archive and into Fedora. It's about providing at least some level of support for afterwards. You might receive bug reports and stuff like that. You need to react to them. Then lurk. So it's not stated explicitly like that on the Wiki page, but that's the intent really. Stick around. Have a look at other reviews. See what other people are saying before you pipe up and introduce yourself and stuff. Then set tools up. So that's easy enough. Well, for somebody with a technical background, I guess at least. And only then make your package, upload it, request a review. And at this point, you've put in all the effort and so there's a fair amount of chance that your package will end up in Fedora, I imagine. And so at this point, it makes sense to tell upstream. And this is something very important, I think, as well. And a focus I've perceived from reading the Fedora documentation is stick close to upstream. And I like that message because it means less duplication of effort. If you can get stuff done upstream, then all distros benefit from it. And it means you won't have to do it again and again and again every time upstream release a new version. And it's also useful for upstream to know that their package is going to be in the new distribution. So they're going to end up with a whole pile of new users potentially and they might be getting new bug reports, possibly for a new version that's whatever version has ended up in whatever release people are using. And so it changes expectations for some upstreams. And I guess perhaps in some cases at this point you'll get a reaction from upstream saying, no, please don't. Then next step, introduce yourself. So at this point you pipe up on the mailing list and say, hi, this is who I am. Again, part of community. And I guess this is part of the friends thing in Fedora. And then the last two steps are actually get your package in the distribution. So get sponsored. And finally, you've done all this. So you add your package to the SCM. And then it goes in. From all this you might have gathered, I think community feel is very important. So during this process you get a chance to measure it. So I was hoping to be further along in the process than I actually am by the time I gave this talk. So I do have some packages that I'm working on but I'm trying to stick close to upstream thing. And there are changes that need to be made upstream before the packages are suitable for Fedora. So I actually haven't gotten round to the request to a review part or introduce myself on the mailing list part. But I am working on it. And I have been lurking a lot. So the mailing lists, you get told to join Fedora Devel, the announce list. And if you want, if you're keen, the packages list. So not many mailing lists, but they're fairly high volume. And if you compare that with Debian, Debian has Devel which has a reputation, a certain reputation which it doesn't really live up or down to depending on the way you look at it. But people say that Debian Devel is a very high volume mailing list. It isn't anymore. It's much less than Fedora Devel. It's really, I think the SystemD stuff killed it, to be honest. Which is a shame. But we have lots of other mailing lists with more specific focuses. First is there's a Debian mentors list which is dedicated to helping people with packaging. And what I find very interesting in the Debian community is that you don't just get newbies asking for help on that list. You also get experienced Debian developers who are discovering new packaging techniques or packaging things that are in new areas. First is they might have been packaging lots of C stuff and they need to package a Python module at some point and they don't know much about it so they'll go and ask there. And it's a place where you can be vulnerable without it mattering. You can say, yeah, I've got 20 years experience in Debian but I don't know this stuff and nobody will look down on you for it. And the feeling I got from the Fedora mailing lists is that they're all very polite, even in the flame wars which happen every now and again. And you do see the difference with lots of people on the mailing list who do this professionally. Well, like Matthew Miller said in the keynote, there's only a third of Fedora contributors who are employed by right-hand but I do think that makes a difference and you don't get all the snipers that you get on Debian mailing lists, people who are not actually contributing to the community but who will take any chance they can get to flame on the mailing lists. And that makes for a more welcoming atmosphere. IRC, so I haven't spent all that much time on hash Fedora but I thought it was very, very, very interesting that if you try and join or if your IRC client rejoins and you haven't actually had time to register, do the register stuff or give the password for your nick, you end up on Fedora unregistered. And I guess it shows the importance that is placed on having stable nicks, owning your nick, owning your reputation or whatever. And then the review queue as well, so I've been looking at lots of other reviews and again that shows the kind of community feel and the reviews tend to be mostly constructive. It's really, well, the impression I get, there's not much, you know, if you make a mistake, it's just a mistake. It doesn't mean you're a rubbish package and you never will be. And I also find it very interesting that Fedora does quit pro quo reviews. In Debian, the reviews that are done before a package goes into the distribution, they aren't technical, they're just for licensing and it's the FTP master suite, well, the FTP team. So a small number of people get to review all the licenses, it's a nightmare for them. Whereas in Fedora, there's a technical review as well as the licensing stuff and lots more people do it and it means that you get more cross-pollination between people. People can discover techniques more readily and of course it shares the workload. One thing I did notice is the difference in attitude towards teams, well, in Debian we call them teams in Fedora, I think it's more six in the working groups. So a bunch of people who are interested on the same kind of packages and get together to do stuff. We both, Debian and Fedora have the same problem of wiki pages for old teams that have tumbleweeds. But I thought it was interesting that there's a very different focus. In Debian, we use teams a lot to welcome new contributors. So if you're interested in packaging Java, we'll tell you to go and join the Java packaging team and that's where you'll learn the packaging practices. You'll also find sponsors and the feedback you'll get will be very relevant because these are people who encounter the same problems that you'll be encountering. In Fedora, at least on the wiki pages they explain how to join the project, there's not a single mention of SIGs or teams or whatever. I find that interesting. I don't know if it means much, but on to the actual packaging itself. The big difference I saw is what constitutes your reference, source packages. In Debian, the reference is, as far as possible, the upstream tarball, or a tarball that's built from a Git tag or whatever. And on top of that, the Debian packaging. In Fedora, the reference isn't what upstream provides, at least the way I understand it. It's the source RPM that's produced at the end of the build. There are good things about that and bad things. I won't go into the details, but I thought it was an interesting difference. Then there's more technical stuff about the distribution layout. The stuff that I've been having trouble with is Slashlib versus Slashlib64. In Debian, we use Multiarch. To be honest, I think Multiarch is a much better approach. I think it makes sense in all cases, but it's only really evident that it makes a lot of sense when you have as many architectures as we do in Debian. Then there's the LibExec thing, which I guess makes sense as well, but causes some issues with packaging that we never encounter in Debian because we don't have LibExec. And then helpers. So these are all the tools that go around the packaging. Basically, RPM macros in Fedora. And in Debian, it's DebHelper and the Galaxy of stuff that goes along there. Over the last few years in Debian, there's been an awful lot of effort put into these tools to just make packaging as simple as possible. So handle all the complexity in the helpers, which means you can get it right once, fix everything, and if there's a bug, you can fix it for all the packages that use them in a single place. I don't think the RPM macros are quite there yet. I've been working on a CMake package, and I should really be filing sending patches for the CMake RPM macros, I guess. But I think it's interesting to invest time in the tools because you reap the benefits over all the packages. And that's something that historically, people have said RPMs are nice to package because there's just one spec file, you write everything in there and that's it, it's done. Whereas in Debian, there's all these different files and there's a rules file, which is really complicated. And now there's a rules file in lots of cases, there's just two lines to delegate to DH. Another big difference I saw is in the way you handle, well, the two projects handle licensing information. And I thought this was very interesting because Fedora has the patent issue, the stuff that means that we have RPM Fusion because our MP3s and so on can't go in. The reasoning that... Sorry? MP3 playback is okay. MP3 playback is okay, right. But encoding I guess still isn't, like lame and so on. And the reason that I've read at least for this, I don't know if it's true or not, is that because Fedora is backed by Red Hat, there are deep pockets that people could go after in lawsuits. And so the patent stuff is important. And I guess this means that it's a sad reflection really on the state of licensing and the open source world that licensing issues aren't as scary. And so in Debian, we've got the opposite problem. We focus an awful lot of effort on licensing and this here is the top of the Winegecko Debian copyright file which I maintain. And the reason Debian doesn't have up-to-date versions of Winegecko is that it takes me months to go through new versions of the package and do all the licensing review to write this thing. This file is 4,000 lines long and it's machine readable but not machine writable. And you can imagine it takes me months to write and then it takes the FTP team months to review it. It's really, really bad. And this is Firefox. If you don't know the connection between Winegecko and Firefox. This is actually Firefox. Firefox is too big to fail. It's one of these packages that's too big to fail. And so it goes into distributions without much scrutiny on the licensing but there are some actual licensing gotchas in there, you know, stuff that's got the Apple license or MSPL or whatever. And Winegecko isn't too big to fail and so I have to handle this. Winegecko in Fedora says this. So we trust upstream to do their work. What do upstream say? Not much and they get it wrong. So I'm not saying Fedora should put the same, you know, go to the same lens as Debian does. I think Debian goes too far, at least the FTP. The working practices around the Debian copyright file have gone too far but there's probably something to look at there. And then trends of what does it tell me? What does my experience tell me about distro trends and what are the impressions I get from Fedora? I think all distros in the Linux world suffer from the same kind of problems nowadays. Old timers are moving on. Like I said earlier, distributions aren't exciting anymore so it's harder to renew the community. I get the impression, at least in Debian, we no longer have any really sort of big figures that are doing an awful lot of work. There are some people who do lots of work quietly. Some people who make lots of noise but don't do much work. At the same time, Open has one, Open Source has one, Free Software has one and so we have more and more upstreams, alternative package management systems, you know, like NPM, RubyGems and so on, that mean that it's not unusual now for somebody to come along and say, hey, it would be great if we could package this. Oh, and by the way, to get this in, you also need to package 250 new dependencies. I had to look at Keybase a while back when it was still a Node.js thing and they had around 200 dependencies that needed packaged, so Keybase isn't in Debian yet. Alternative distribution models as well. This is all the stuff around containers and flat packs and so on that they don't make distributions irrelevant, but they do mean that more people are doing the kind of work that distributions do. Perhaps there's an opportunity there. Quality, quality, all right, all distributions have put in a lot of effort in getting packaging better, but there's not that many people who look at what actually goes inside the packages, you know, until a security bug comes along. And in Debian, there's been an awful lot of discussion on the mailing list about packaging Node stuff and there's a guy called Pirate Proven who started packaging GitLab and you might think of GitLab as a Ruby thing and yes, it is a Ruby thing, but it also uses an awful lot of Node and so he's been packaging lots and lots of Node packages. He's got over 400 packages, Node and Ruby already and sometimes somebody goes and has a look, you know, when he files an intent package bug, somebody goes and looks what's in the package and says, this is silly, there's no point in packaging this, what we should be doing is patching the reverse dependencies so that we don't need this and you end up with Node packages that look like that, so you can't read the details there, but it's one package that only exists to check if another package is installed, right? What's the point of having dependency management in your distributions if you end up packaging stuff like this? What are we good at? So going back to the main Fedora website, Freedom, Debian I think and Fedora, we're both good at ensuring that our users have the four freedoms. So licensing, what are we good at? And this is stuff that I think we should think about, you know, pushing up. Fedora says stick close to upstream, but there's stuff that all distributions do and perhaps we'd be better off if we were telling upstreams how to do them. Check your licensing, how to build software, how to build distributions, which is taking all these moving things stable out of them. And you need to do that for distros, you also need to do that for flat packs and containers and so on. And lots of people are going to be learning the same lessons that we've been learning over the last 20 years. Handling fixes, getting security fixes to your users without breaking their production and fixing important bugs as well without breaking their production. We know how to do that in distributions. Lots of people are going to be learning how to do that again and again and again. User-oriented quality, this is getting the configuration right, getting something that's usable quickly, but that can be adjusted to the user's requirements easily. We do that an awful lot in distributions. And other stuff, more technical stuff, like understanding what it means to be stable, so ABI stability. This is something, you know, lots of people in the Debian world they see unstable versus testing versus stable and they don't understand what stable means and they think that unstable means that it's very buggy and it's going to crash a lot. It's not about that, it's about ABI stability. And there's stuff going on in Fedora that's very interesting on this front. Stuff that Dojie Cicadilly is doing with Libby Gale and stuff like that. There are tasks on Taskotron that will tell you if your RPMs are breaking ABI contracts. I think that's very, very interesting. But there's stuff that doesn't mean much if we don't educate upstreams about it because we can't cope with changing ABIs if the upstreams don't change their SO names and so on or, you know, their node versions, etc. What are we not so good at? Well, getting upstream involved. I don't know if it's something that we're not good at or if it's just that we get lost in the day-to-day grind of getting distributions out and so that gets put on the back burner. And maintaining quality, this is something that we all struggle with. Packages that are old and un-maintained but where there's just enough maintainer activity to keep things ticking along and it just gets old and crafty and... But in all that, there's lots that's not specific to any given distro. So there are lessons that we learn that are valid across distributions. There's also lots that's not specific to the fact of being in a distribution at all. It's equally valid just for general software engineering practices. So in conclusion, let's define the future but define it together. And continuing on from the keynote this morning about collaboration between REL and CentOS and Fedora, I think there are lots of opportunities for collaboration beyond that. And that's it from me. If you have any questions, the floor is open. So do you see some space for collaboration between Fedora and Debian on some level? Yes, the question is, do I see space for collaboration between Debian and Fedora? Yes, definitely. And I was quite interested to see that Hans de Goethe, or sorry if I mispronounce the name, has given a talk just afterwards because he's one of the Fedora people that I saw collaborate with Debian a while ago in the games teams. So I'm a member of the games team in Debian and there's a games sig in Fedora. And at one point, there was some collaboration between the two and I think there's space for more of that. I think there's also room for not necessarily collaborating directly on the distributions themselves but coming up with ways of explaining to upstream how to do things better. And I think that would have more weight if it wasn't just Debian telling people that they're doing, well, not necessarily doing things wrong but they could do things better, but if all distros somehow came up with another standard. There's another question. Yeah. We do work together and talk to you. Licensing is an interesting area because I agree with you that Debian is much more systematic about licensing. But in order for those of you who haven't been there, there are some people who are interested in licensing and take it seriously. And we do often find ourselves working to maintain some of the distributions because like I said, you can't just be slapdash with your license if you don't take care of it. So I think it goes on that it's not very visible necessarily. Yeah. Yeah. So I'll repeat the comment. Well, summarise the comment for the recording. So basically Adam was saying that there is collaboration that goes on in the background between individual packages, individual maintainers. And there's also an awful lot of collaboration that goes on just because Fedora Packagers are also upstreams and likewise Debian developers are also upstreams on the same projects. And so there are people involved in GCC and so on, yeah. And there's collaboration that goes on. And he was also saying that licensing stuff does have more weight if comments on the license come from multiple different distributions, yes. And so instead of dismissing one distribution, they just dismiss two distributions. Does everyone else just package it without complaining about the problem? Yeah. So the comment was that in practice it's just Debian and Fedora talking about stuff like licensing. And other distros, I guess, care less about it. Yeah, for AURs, for instance, in Arch Linux. And so people just, instead of dismissing one distribution, they dismiss two. Yeah. Yeah? Yeah, so I wanted to ask, I work on upstream software and also I think it's in Fedora because I work on Fedora. But I'd also like to see it in Debian. I don't feel like you're really covering the spectrum of users in this in Debian. I'm wondering, is it in Debian, what's the ratio of packages available for packaging versus people who want to package? Do something good and wait until the developer comes along and sees this tractor and does it or should I become a Debian developer? Or what are you going to say? Yeah, so the question is about getting upstream software into Debian and if there's a way of measuring the ratio of stuff that's waiting to be packaged in Debian versus available workforce, I guess, and whether you need to become a Debian developer if you want to get your upstream software into Debian. So I'd say, to answer the last question, becoming a Debian developer is probably always a good idea if you're interested in that. There is one way of measuring the ratio of interest to available workforce which is the WNPP page which lists all the packages that people have pledged to package and more importantly all the packages that people have asked for and that's, it's huge the HTML for that page is over a megabyte in size, it takes ages to load in a browser so yeah, there's an awful lot of stuff that people want in Debian and there aren't enough maintainers so, and yeah it brings me to another thought I had when I was preparing this going back to where I came from in my packaging history, what's important is actually getting software into the hands of end users, packaging isn't an end in itself it's a means to an end and so, I hold lots of hope for stuff like flatpack and so on there will be another way of getting software into the hands of end users and perhaps more driven by upstreams because upstreams will be able to, well hopefully write flatbacks that will work across distributions but yeah, there's no miracle answer really, you can ask for a package try to get users who are also Debian developers interested in your software, usually once a Debian developer starts using your software a lot appears in Debian but if you just file an RFP request for a package it's liable to just sit there, yeah so we have a percentage of people who are end users, where end users can upload and this is a place where a lot of interested people can review packages they usually just do initial package off because they don't really know anything, but if you do a package I'm doing it, I'm showing it to the developer, they need to be triggered and they will give you stuff and give you all sorts of comments that you can express or this can be easier, just make it easier to trigger yeah, so for the recording there's another way of getting a package into Debian which is to start the packaging work yourself and upload it to mentors.debian.net or .org a camera which it is now and that's a way of getting your package up for review and then you send an RFS and you're more likely to get interest that way because you've actually put in some work and people will go and look at your package and tell you how to improve it and once you've got people hooked like that it's a bit like when you start in an auction they're liable to go all the way to get your package into Debian and that's also a first step for you to becoming a packageer and maintainer and so on, yeah Yeah, so the question is Yeah, right, yeah, so yeah so the question is how can upstreams get changes in Debian packages that's a really, really tough question so the last point you made about getting newer versions that's easy enough in theory you file a bug asking for a new version if the maintainer is not responsive it's not likely to move all that much but if you disagree with the way the Debian package is done that's a very, very tough situation because there isn't really any conflict resolution process in Debian that works and not something well, not something that you can invoke as an external someone external to the Debian community there's a TSC well, the technical committee, yeah, thanks technical committee but that's more for conflict between Debian developers they might step in for upstream versus Debian, I don't know probably not it's not something you want to try I'd say, you really need to find some Debian maintainer who's interested in your package and is willing to be your advocate in the project and that's, yeah, last question yeah there are as many well, so the question is so Debian, does Debian perceive Ubuntu as a derivative so yes and what is the relationship between Ubuntu and Debian basically and the answer to that is there are as many answers to that as Debian developers I think and probably more so it's a point of contention some people in Debian consider that Ubuntu sort of leeches off Debian others consider it as a good thing, I think it's probably overall a good thing the more users there are of Debian stuff the better things are it's a complicated relationship if you look at some of the core stuff that happens in Debian it is driven by Ubuntu people like Docko who's the GCC maintainer he's a canonical employee so this complicated, I think things have changed recently well, over the last few years as well with the change of focus in canonical so it's perhaps less of an issue now than it was at one point they appear in the dashboard but there are other derivatives that are important as well like Mint and so on so yeah, that doesn't really answer the question but I don't really have an answer, sorry well, thank you very much for your time