 Linus Kernel, in Debian, Ben Hachins. Okay. Okay, first I'll just briefly introduce myself to those of you who don't know me. I've been working professionally on software development since I left university in 1998. I started contributing to Debian in about 2003 at bugs washing parties and so on. I've been a regular contributor to the Linux kernel since 2008 as part of my day job. And around the same time I started contributing to the maintenance of the kernel in Debian as well. I've been officially a member of the kernel team and uploaded since 2009. Initially I was trying to fix the issue of non-free firmware packaged in the kernel. More about that later. Now I'm triaging bugs, fixing bugs, backporting features, updating the packaging, and really dealing with everything involved in the maintenance of the various packages that we have. I suspect you probably are aware of the Linux kernel. Test your as a reminder. It starts in 1991, 5, 3, 8, 6 only. I adopted as the kernel of the Debian system in 1993 when we started. 20 years later it now supports more hardware architectures and devices than anything else. If you look in the kernel source tree there are 24 separate architectures and that's even before you consider the 32 bit versus 64 bit variant or little Indian versus big Indian. If you were to cancel those there could be 40 or 50 architectures. Over 2,000 device drivers, each of them supporting potentially very many devices. At this point users demand Linux support for hardware. The hardware vendors have to provide Linux drivers. This doesn't always make it into the official kernel releases but there's always going to be some sort of driver. Today it's still the default kernel for the Debian system although there's some competition now from KFreeBSD. Just a reminder about the Linux release model. Some people are still familiar with the old way things were done. Each stable release had an even second component. So for example 2.0, 2.2, 2.4, 2.6. The third component was then incremented for our stable releases which were just fixing bugs and adding some smaller features. The major development was then done in odd numbered series which would go on for a year or two and then you'd have the next stable release. There was a big problem with this because users could wait for many years to get all these features that they were probably hearing about being implemented in 2.5. The users still on 2.4 were getting pretty frustrated and then when 2.6 finally came out there were a huge number of features and a huge number of changes to the way they were used to doing things and it was quite a painful transition there. So since early in the 2.6 series there's been a new model has been adopted by Linux and the other kernel developers. We have a stable release, fairly stable every two or three months with about five stable releases per year now. The use of the Git version control system and before that BitKeeper made it a lot easier to do distributed development and to integrate and to stabilize all these many changes over such a short period. Even so there are still going to be bugs in each of these stable releases. There's also a branch from each of these which is then number 2.6.x.y. Well it has been up till now. That usually lasts until just a little after the next stable release but that's not always the case. Distributions like Debian Lite to support a single base kernel version for the entire lifetime of their stable releases which are two or more years in the case of Red Hat and Suze is even longer. So we now have the long term branches for various base upstream versions including 2.6.32 which is the basis for squeeze. They were now up to I think 2.6.32.43 with a huge number of bug fixes. So you may be aware that Linux 3.0 was just released. Unlike previous major changes in the first two version components that actually doesn't make a huge change functionally. Really it's just the version numbering changes. So x is now the second component and the y for stable bug fixes is the third. So the first bug fix release will be after 3.0 will be 3.0.1. And the first and the next new stable release from Linux will be 3.1. So a bit about the Debian kernel team. There are about five, there are five general maintainers. Max Atems, Bastion Blank, Dan Frazier, Lawrence Mullenhoff who's here in the audience and me. But we have many more contributors who are working on specific areas, architecture supports. For example Arm and MIPS needs some quite specialized, they have specialized issues and need attention from an expert. I know very little about Arm and MIPS for example. And then we have people looking after specific features that they're interested in. For example we have Ian Campbell who's one of the upstream Zen developers who's done a lot of work on making Zen work well in squeeze and subsequently. And we have people who just do bug triage and that's incredibly helpful. We could still do with more help. Bug triage as I said. And PowerPC is one of the architectures that we still have quite a few users for but we don't really have an expert on the kernel team. So it's been somewhat neglected. So the duties of the kernel team not so different from those of any packaging team but there are a few unusual things. Bug triage, we get a lot of bugs. We have about a thousand open bugs at the moment. It takes quite a lot of time to deal with incoming bugs. Sometimes without a whole lot of information for example the system crashed and the user doesn't have any log. We've got to somehow help them to get a log or some sort of diagnostic information to understand what went wrong before we can make any progress at all. We've got a back port, bug fixes and features in particular for stable releases. We actually have to add features. Unlike most other packages we do have to add support for new hardware because otherwise users can't install stable on their new hardware. At the same time with these bug fixes we want to avoid changing the kernel API because that's disruptive for anyone who's using modules from... using out-of-tree third party modules. For every new upstream release we need to look again at the build configuration. There are a huge number of build options for the kernel to enable drivers, file systems and other features. Sometimes these get reorganized and renamed between releases and then if we're not careful we drop a driver which is not very nice. And we also have to ensure smooth upgrades. There are some implementation changes that affect other bits of the system. For example, the kernel mode switching transition a whole lot of code for managing video devices was moved from drivers that were part of the X server into the drivers in the kernel. The old X drivers definitely would not work with kernel drivers that expected to do mode switching. You can't have both of them doing it. So we had to do a bit of thinking about how we ensure that the user is never left with a system where the drivers are fighting over their video devices. LibATO was another fun transition for boring technical reasons. A whole lot of the kernel device names for hard drives and CD-ROMs and drives all got changed. So we had to try to fix up all the configuration files that might mention those names. And sometimes we integrate features that were not accepted upstream but we don't like doing that. The official Linux kernel packages, the main source package for these is Linux 2.6. At some point we should be renaming that to just Linux. The binary package names produced by this change with each upstream version and sometimes more frequently. This is because the kernel is kind of like a shared library in its relations to out-of-tree modules. If you package out-of-tree modules, they will need to depend on kernel versions that provide specific API. So whenever the API changes, we need to change our package name. So you have a huge number of binary packages whose names follow this format. These have the kernel and all the modules we build for it. Then we have the development packages corresponding to that. There are a few more Linux headers packages which have this common headers because in fact most of those don't vary. We have headers for userland. The C library defines most, basically wraps much of the system called interface, but not all of it. So there are some header files that get included by userland programs as well. We package the source with all our patches so that people can build custom kernels. We package the upstream documentation. We package some useful tools. Currently the main tool there is PERF, which is performance analysis for both kernel and userland. If you've not seen that, check it out. It's quite useful. And then this rather uninteresting package which is used basically to support the Linux latest 2.6, which I'm going to talk about next. From this we get meta packages, and those make it easy to do automatic upgrades across upstream versions, despite the fact that the binary package names keep changing. These ones don't. So Linux image and the flavor. I should mention that the flavor here distinguishes the various different configurations that are available for an architecture. So for the i386, for example, we have the 4.8.6 flavor, which basically runs on any processor from the 4.8.6 up. We have the 6.8.6 PAE flavor. That runs on anything from the Pentium Pro up that supports the physical address extension. And we have MB64, which runs on 64-bit processors. Other architectures, and on other architectures there are some major differences in the basic hardware, which mean we need different flavors to do with, basically for different kinds of motherboard. Linux headers flavor similarly for the development package, Linux source, Linux dock, Linux tools, and so on. So if you install these, you might have the latest version of the source, the documentation, Perf. The installer will, by default, it will install one of these meta packages. So you always get the latest kernel version from the suite that you installed. There are a few more packages that are built from separate source packages. Firmware 3 has a very small set, unfortunately, of firmware images for which we have source and licenses that comply with the Debian Free Software Guidelines. Linux base contains some scripts that are shared by the various kernel images. There's also a Perf wrapper there, which will call the right version of Perf. And that has scripts to handle transitions, for example, the libata transition. And finally, Linux kbuild 2.6. This is separated out from the Linux 2.6 package because everything in Linux 2.6 can easily be cross-built without a... without libc. And Linux kbuild 2.6 cannot. But Linux kbuild 2.6 actually supports cross-building. So yes, that's the kernel build system, and that's a dependency of Linux headers' files. So while these official packages work for most of our users, I think, I hope, some people need to do custom builds. For example, some ARM platforms need different configurations. They're not compatible. You can't... We can't come up with a build configuration which will work on a whole variety of ARM platforms. We need a separate one for each, and we just can't... We can't cover them all. So then you would need a custom build for some. Then somebody just wants to enable new features that are added by upstream, but we're a little concerned that they're going to cause regressions, and we don't... Basically, we wait a few releases to see what the fallout is, to see if they get bug-fixed. Some people are eager to try them straight away, so they can do that. Simple as way is using the usual make and make and still. Well, there's also a configuration step in there, but anyway. There's also an upstream target called dev package, which is maintained by various dev and developers who have contributed to that. There's also the old kernel package, which provides the makeK package command that enables a little more customization. I don't know how unnecessary that is now, but that, I should point out, is not maintained by the kernel team. So going back to those extra features that we sometimes integrate, as with any package, we don't really want to be carrying large patches that haven't been accepted upstream, because then we're left with the burden of merging those with upstream every time we take a new upstream version. But sometimes there are big features that are really quite important, and we feel we have to include those. So previously we had OpenVz and Vserver, which were kind of kernel-level virtualization systems. They let you have virtual machines without their own kernels. Most of the functionality of this has now been re-implemented in upstream Linux through these C groups and namespaces features. So these... Although those aren't complete replacements, we've dropped these features after squeeze. There will be no more releases with these extra features. Xen, the story is much better. Xen basically got merged upstream. Linux 3.0 will run in DOM 0 and can provide networking and block backends, which I think is basically where they're right, Ian. Debian Live really needs a union file system, and so far there's no union file system upstream. There's been a lot of talk and a lot of proposals, and it just hasn't happened yet. So for now we're carrying the AUFS patches. Thankfully the AUFS developer is keeping those up-to-date with upstream, so that's not too much of a headache. Then there's the real-time patches for preempt RT. The effect of the real-time patches is to limit the maximum latency for responding to events, which is useful for industrial control. It's useful for audio mixing and creation. And there are also some financial services companies that are doing some very clever tricks, and they also want real-time features. These features are gradually being merged upstream, and the remaining features that are not upstream are actually quite a small patch set. So for the Debian package of Linux 3.0, we've actually added an RT feature set, currently for the MB64 architecture only, but if people are interested in RT or other architectures, we may be able to add that. So I have a question from IRC. Could you comment on the status of LXC and Debian in upstream compared to OpenVZ? He said that some functional analogies are missing. Which ones? I don't know. It's not my area. Sorry. Someone said memory groups. Yes, that's missing in squeeze, but it will be there in future. There's another thing which didn't quite fit on this slide, which is GR Security. I can see there are useful features there, but trouble is that upstream really doesn't like it. There are a few features going in from GR Security, but mostly it's just a huge patch set, and it's quite intrusive. So we'll have to carry on thinking about that. So another thing that people like that is not in true, out of tree modules, are necessary even for some users. So we provide the development packages as I described before, and we try to avoid changing API during a stable release so that there shouldn't be a need to rebuild these during the lifetime of a stable release. We don't always succeed in that, but mostly. There are basically two ways these might be packaged. There's the dynamic kernel module system, originally created by Dell, and as a result, it's supported by a lot of Dell suppliers. DKMS is also integrated into Ubuntu and SUS distributions, so quite a few out of tree modules are packaged this way. Possibly not as a Dell, possibly as an RPM, but you can actually install those with Alien, and it seems to work. That will rebuild modules automatically, so if you install the development packages for a new kernel, a new kernel version, or ABI, then it will try to rebuild. You probably don't want to install and do rebuilding on a production machine, but it does support manually creating depth packages of the modules. There's also module assistance, which always builds packaged modules, doesn't do automatic rebuilding. One plus point, though, is it changes the package name for each kernel ABI. DKMS doesn't do that, which can make it hard to do parallel installation for multiple kernels. One more question from IRC. Dilex asks, he proposed a patch set to enable real-time features for E3686. AMD64 is already enabled with Linux 3.0 package. Corsak proposed a GRSEC feature set. What's up with them? Well, as I said, it's somewhat contentious. Yeah. You've mentioned DKMS and module assistant. Would you care to recommend one over the other? I would basically favor DKMS. That's kind of the consensus when the kernel team discussed this. Returning to firmware files, most peripheral devices have a microcontroller. It's running non-free firmware. Even if it doesn't require a firmware file, it's still running non-free firmware. Sorry. Some of them require the host to load that. Several drivers in the kernel used to include the firmware, which made the kernel non-free, which means it shouldn't be part of the Debian system. We sort of fudged this with general resolutions, making an exception for Edge and Lenny and finally fixed this in squeeze. But the bad news, of course, is a lot of users still have these devices that need non-free firmware. Pretty much any Wi-Fi card you can buy, some network controllers, and all the AMD Radeon GPUs. So the kernel team looks after this package, which, of course, is not part of the Debian system. The firmware non-free covers most of the firmware files that you're likely to need and that have clear licenses to redistribute them. Unfortunately, some of the firmware that was previously included inside drivers, it was in source files that said this is released under GPL version 2, and we don't have the source for the firmware, so that means we can't comply with the license. The firmware files were also collected in a Git repository where the conditions for inclusion are less strict, let's say. Documentation, you need documentation for the kernel. The system caller API is documented in Manpages Dev, which is extremely high quality documentation. You also have miscellaneous documentation of other features, particularly what's in PROC FS, CIS FS, and so on. That's documented somewhat with somewhat variable quality in the Linux doc plus version number package. The internal API of the kernel is also somewhat documented in a format called kernel doc. From that, we build manual pages in the Linux manual-upstream version. There's also a package called the Debian kernel handbook that's maintained by some members of the kernel team, and it covers a lot of the Debian specific information, the same sort of things that I've talked about here. Currently, it's all about Linux. It might reasonably be extensible to cover K3BSD and Herd. And on the wiki, the page name is Debian kernel. Despite the name, I believe that's all Linux specific so far. So, more questions? So, I'm experiencing a number of kernel bugs in Squeeze, and obviously, I realize that the kernel team are obviously overloaded, and that my kernel isn't the one that I'm running, isn't very close to Linux upstream. And although I could go ahead on and debug my kernel, I'm worried that if I do so, I might produce a patch that was no longer relevant to anybody. What would you suggest as a general approach? Should I just live with my bugs? What should I do? Well, actually, the kernel in Squeeze is fairly close to the long-term branch 2.6.3.2, not why. There are some bug fixes we've applied, and a few features that haven't gone into there. Nevertheless, all of the bug fixes and features we've applied are upstream in a later version. Of course, there may be upstream changes that mean that you can't fix a bug in the same way upstream and in Adobe and in stable. But as a general rule of thumb, you would think it would be worth wildly bugging problems myself and reporting patches. I take it you'd prefer that than test bug reports. Yes, if you want to investigate and produce the patch against our stable version, then you could send it first to ours, and then we can look at how or if this applies to upstream. That's great. Thank you. A short question based on my experiences from the University of Oslo. With Red Hat Enterprise, you will always be in the situation where you can go back to a previous version of the kernel when you upgrade, and I really like this non-bridge-burning approach to kernel upgrades. It's really hoped that the Debian kernel will get the same feature. You want us to change the package names more often? Every time there's a new kernel, I want a new version, a new package name, so I can keep it in parallel with the old one. Yes, it's something I thought about. It's quite problematic if you're getting these pulled in automatically through the meta package, because app will never auto-remove kernel packages. Which is maybe was a sensible heuristic? Maybe still is, but really there's nothing removing those, which means it would be pretty easy to fill up slash boot if it's a separate partition. Then you also have the problem of, well, if you live out of tree modules, how do you make sure that those are available after the upgrade, which may well be an urgent security fix where you insist people must install right away? We don't want them to break their out of tree modules. Now, I know that RET has a slightly different... Well, I encourage people to do, I think, kernel module packages, which are a different thing again, and they I think will sim link modules across if it looks like the ABI hasn't changed. So we could maybe think about doing something like that, but we'd have to look very carefully at how this would impact on the... on DKMS and module assistant and whether it would get in that way. I do not have a question but a comment. I would like to encourage the kernel team to get all the WN.org machines on kernels that are in our archive. So we more or less eat our own dog food, which we produce. We are doing quite, you did quite well with the Squeezy release, but there are still some machines that are running custom-built kernels. I'm aware of that. Part of the problem, though, is that, as I said, you need different configurations for each ARM platform and for a lot of different minutes platforms, and the build-dies currently, well, at least some of the build-dies are not fast enough to be able to build all those different flavors. So get us faster build-dies and we can add more flavors. I wanted to say, to ask something about the last question as well. In the RMAFS tools, you can already specify that you get the backup of the RMAFS when you create a new one. Would it make sense to create the backup of the kernel as well when it updates it to a compatible AV? That's... yeah, maybe, maybe. So I've been using 2632 and you're doing an amazing job of maintaining it and providing bug fixes and that's very great. At some points, I need features of newer kernels, so... I'd like to have backports. So far, what I was doing was installing the kernel from unstable instead of rebuilding it, but it seems that recently there have been some updates to RMAFS tools and some increased dependencies. So do you plan to provide backports of kernels for a squeeze? There are other people providing backports in squeeze backports. Thank you. One more question from IRC. Dilex asks if there is documentation of the own build system from the Debian kernel team. Sorry, I didn't quite catch that. Is there documentation of the own build system from the Debian kernel team? Of the old... what? Of your own build system. Of our own build system. Use the source. Sorry. Okay. Matthias from Linux magazine. What will the Linux kernel version for Debian VZB be? Is it going to be... Oh, this is dangerous. 3.0 base? It will be 3.0 something. Okay. At this stage, I would guess maybe 3.2. Okay. Thank you. Okay. I have one wish and one question. The wish is could you please build the Linux K build packages also for the experimental packages? So it's possible to install them and still have the take EMS like virtual box compiling? And... Maybe. And the question is do you still plan to convert your packaging to Git or is this plan postponed? We do have it as a long-term goal and I really hope it would have been done by now. We will do it. Any more questions? Do you already have an idea of how much is going to change if you remove the 2.6 from the package name and everything that it entails? Changing the source package names doesn't really mean anything for users. The only disruption I can see would be the way bugs are tracked. Currently we have bugs tracked against Linux 2.6. Any bugs reported against existing binary packages will end up besides the Linux 2.6. But of course if the bug hasn't been fixed, then it will still apply to the new Linux package. So I'm not sure quite what we'll do there. Hello. Do you have any plans for Ksplice, even though Oracle just spotted? I think maybe as a part of our contribution back to the free software community it might be something that we as a project could look into to rescue that project. For those who don't know Ksplice it is basically a way to patch it up. So I think it would be nice if we could use it. Any more questions? This is not so much a question as just thank you. I really appreciate, I know you guys are doing a really big job and it's a critical job and there are lots of machines that I can maintain thanks to you. So I know it's hard and I know that there are problems and it sounds like there have been things that we did, everything that you guys are doing. So thank you. Thank you.