 Welcome everybody. Ben Hutchings, also known as BWH, will present this buff about destabilising the Linux kernel API. So the basic problem is explained in the description of this talk, which I've brought up here. The kernel, the upstream developers for Linux kernel do not attempt to maintain a stable API. Even for bug fixes, we kind of try to keep the API stable during a single stable release, so security fixes and point updates generally don't change the module API, except sometimes in very specific areas we don't think tree modules will be affected. Even then it takes a lot of work to do that. So the question is, is that actually necessary? Could we just keep changing the API? The package names for Linux kernel builds all incorporate and identify for the API, similarly to shared library packages, and that's because if you build an extra modules package using module assistants, for example, that will have a dependency on a specific API, and therefore we need to have a package name that identifies that API. So if we start changing the package names during a stable release, there are some problems that could cause, and the purpose of this meeting then is to try to figure out can we solve those problems? As it turns out, since I proposed this event, one of those problems has been solved, which is very nice, apt, no longer, apt will now let you auto remove older kernel packages if they were installed as a result of upgrading the meta packages. So that's good. But we still have two other problems, basically which are... Ben, could I interject at that point? I'm an app developer, so I can comment on that. It's not completely solved. I mean, it's just really basically working, because as soon as you have a firmware package installed or an out-of-three model or something like that, which depends on the Linux image, it's going to be keep back. So most of these users will still have it around. Well, the firmware packages never depend on the Linux image packages. They are depending on a Linux image, the vital package, and as each Linux image, a specific package provides this Linux image virtual package. Really? So if you have a dependency on a virtual package that prevents any of the... Yes. Yeah, I suppose it would, because how would you know which one could be auto removed? Yes. Okay. So we need to get rid of those somehow. It's not even a dependency, it's a suggestion, but yeah. So what is a suggestion prevents auto removal? Yeah, because it's a suggestion, so it provides a feature, so you can't out-remove it, because the user might depend on this. So I mean, we need a bit more metadata for this, actually, because we don't know that the kernel is actually, yeah, that these are not completely different packages, but just version updates actually were, yeah, just a different name. So we need a bit of logic or metadata to distinguish these, and yeah, that will be... Well, I can't understand why suggests would prevent auto removal, unless you say that... I mean, for example... That's right, that's what it should. I mean, you have, for example, you have a text editor, and it suggests a package to print something, print your... Right, but suggestions aren't auto-installed. Yeah, they're not installed by default. So then you would be manually installed, and then for that it wouldn't... There would never be auto-removed, anyway. Yeah, sure, but you could have it installed by an other package. I mean, another package could depend on it, and you are removing this package, but you have grown used to the suggestion established, so it's kind of tricky to auto-remove it because you could lose something, or the user could lose something. I mean, it's a bit complicated, whereas... Yeah, I could. But I could also forget it. Yeah, it's a complicated trade-off, and we are usually defaulting to not breaking user experience, because, yeah, that's bad. Actually, if you're depending on something and it gets auto-removed, so, yeah. Okay, so we are going to have to maybe get rid of those suggests Linux image, which I think is a stupid suggestion in the first place, because... But only the suggestion. Also, the out-of-three modules are... Sure, but that's kind of covered by point B. Yeah, still more work. That's just what I was trying to say. Yeah, not completely solved. Okay, so any bright ideas? So this is in... I've created this document on Gobi under the apicot13 boff linux-module-abi. So... I can see a few people found it already. Yeah. So the other problems, yeah, out-of-three modules need to be rebuilt, which... DKMS can do, module assistant does not do, and practically can't because that would involve installing packages while installing another package, which doesn't work. So I don't know if there's a... can something be done to make it easier to build and deploy updated packages on... build on one machine and then deploy it throughout your organization using either DKMS or module assistant. No ideas? Is that a hand? I'm not very familiar with DKMS. Does it already have the possibility to build some meta package, which depends on the version you generate? That way, people could just get installed this meta package and if they have some local mirror where they push the newer versions, then at least these versions would get cycled out automatically. Well, see, DKMS can build packages and then they're organized at the moment. They're built for a specific version of the module and then they can contain binary builds of that module for one or more kernel versions. So then what you could potentially have is a single package that gets updated to a new version for each kernel version and the package name doesn't change. But then that wouldn't have any dependency information. That's how DKMS works at present. It might be possible to improve on that to build packages more like what module assistant does, where they are specific to a kernel ABI version and then they have the correct package dependency information. Have you ever tried maintaining a kind of a private repo with modules packages within the organization? Do you want to mic? Yeah, well sort of. I've got a private repo for my four packages built from NVIDIA drivers which I'll share between my computers at home. And as far as I recall they are built by DKMS. Yes, they are. I'm mainly installing them anywhere I need them, so I think it doesn't really fit the problem here. Okay. Also, I maintain my own repository for our company of Dadi modules, which are out of three. DKMS of course has a problem of not maintaining the architecture. Of not what? Not keeping the architecture, so we use module assistant. When we build for CentOS there are, I forgot the name of the model building framework there. It's KMP or Kmod. Kmod, yeah, right. It's been called different names by different vendors even though it's more or less the same thing. Yeah, basically they put the model in updates and they seem to have some... Oh, you think of the week updates. Yeah, right, week updates. And they seem to have some sort of... Yeah, I kind of forgot about that. The way the week updates works is that if you have a module installed in, I think in the updates or extra subdirectories, and you get a new, install a new kernel version where the ABI might have changed, but none of the symbols that that module depends on have changed. So it's still loadable in the new kernel version. Then it gets linked into the week updates directory for the new kernel version. And that way, if the ABI is mostly stable, then... which is the case for RETAS and SUSE kernels, then Altree modules keep working. That sort of depends on their having... They have an official white list of symbols which they promise not to break. So they keep a subset of the kernel ABI separate and then Altree module vendors are expected to stick to that if they can't use anything outside of that list. Potentially we could try to maintain something like that in Debian, but it requires a lot of thought. And I can be sure that some of the Altree module packages that we have are going to depend on things where we wouldn't really want to white list. If we listed, if we said every symbol that was currently used by an Altree module package in Debian, all of those symbols are going to be stable, then we would be pretty much back to where we are now. We would have to make the same sort of effort to avoid changing ABI. I think, I mean, I haven't gone through and tried to list all these symbols, I don't think it would make the problem a lot easier. Just to give some information, the DATI modules I think were broken. I'm not sure, I don't remember exactly, but were broken I think between 6.2 and 6.3 and maybe between 6.3 and 6.4. Maybe the... I guess it would be less disruptive if we said, we promised that security updates won't change the ABI because then stable point releases could because there's less urgency of installing a point release and therefore more opportunity to prepare and rebuild your module package, whatever Altree module packages your organization might depend on. What do you think of that? Something also I didn't really check. How does it work with different kernel flavors? What do you mean? Different kernel flavors on i36. Well, okay, each flavor obviously has its own ABI N away, but aside from that, we can make... No, I mean with weak updates. Okay, so no problem, sorry. So, yeah, each... we kind of have... every flavor has its own ABI, but then every... let me have ABI versions for changes that affect all of those at once, or most of those, does that make sense? Or am I just being confusing? Regarding the point release and security updates, when you do a point release that has an ABI change, then the security updates after that... Yes, that's true, yes. So if there's a security update immediately after a point release, then yes, it does become a version to update. Hadn't we security updates in the past that needed to update... to increase the ABI anyway? I think there has been, but not for a long time. There was during the Lenny release cycle. But I think it's... it could also... I think I can get... imagine many kinds of security updates that are only possible with an ABI change. Hopefully this kind is very... it doesn't happen very often. But I guess it will happen someday. Something like this will happen anyway. So the security update immediately after a point release is also something that should happen, hopefully not that often. So maybe that would be just a special case of... usually it should not be increased in security updates, but in certain cases it can happen. So we're putting inside security updates that are forced to change the ABI. I was thinking if it's perhaps manageable to tell people that after a point release if there is a current security update, we will also update the old kernel for a while, like three months, half a year, whatever, after the point release. Yeah, I don't know whether that's possible because we then have two different versions of Linux in the same suite, in two different versions of the source package with separate... that would have separate binaries. No, in fact, we're even producing some of the same binary packages in the same suite. I don't know whether that can support that. Actually, we did, but Ubuntu has multiple kernel releases in the same distribution. And it's usually that the meta package just points to the newest one, like it's done with our releases, but the older release is usually still kept. So it might be possible. There's also the question if we can afford to have the additional maintenance boot actually attaching two kernels instead of one. I'm not seeing anyone else writing in the... probably, no. Is this just my machine not updating? As far as I know, the Ubuntu kernels are separate packages. So a new version is another package. But regarding Duck, it is possible to have that, but it's definitely not so easy to do. Right. Which would be more work for both release teams and FTP team to make sure that it works and keeps working. Yeah, I don't want to. Definitely don't want to make a lot more work for other teams. So does anyone know about how Ubuntu deals with out-of-tree modules? They attempt to support. I know they've tends to favor DKMS. Do they have anything to make... Do they have anything to assist in rebuilding modules with DKMS on the machines other than where they're going to be installed? Or it's the assumption that wherever you use these out-of-tree modules, you do have the build toolchain installed to make auto-updates work? So I'm not actually working on our kernel stuff, so I'm not 100% sure. But as far as I know, they actually are just using DKMS and relying on build tools on the machine where you install the packages and modules. Right. Okay, so it looks like we don't really have any... don't have any solutions here, really. Maybe at least some ideas. Anyone else got anything to say before I wrap this up? I haven't been looking. Did anyone come up with a point on IRC? No, don't see anyone there. All right, there. Well, thanks, everyone, for coming. And maybe we'll solve this later. I guess we're not going to change anything right now.