 I didn't prepare any slides, but I'm going through the process of how we do point releases from a user's perspective, but also from the release team's perspective. So first, I want to have a look at, well, the announcements users get when a point release has happened. So this is the last example, the latest point release. So on Debian announced, there is a mail sent by the press team, and they explain what has changed and what the point release is about. So as you can see, the update mainly adds corrections for security problems and some serious problems. I come back to that later, but so it clearly states that it's still stable. It's not that there are features changed or something like that. It's just to make sure that the systems stay secure and that packages can still be installed and things like that. So if we have a look at what changed, there are a couple of categories. So first, the non-security fixes, and then you can see that even if they are not mentioned at the security fixes, there are no Debian security advisories for it, there are still security fixes that get applied and accepted in stable through a point release. The reason is actually very simple. It's that the security team thinks that the fixes for these are not important enough or not critical enough to include in the security archive. The reason they prioritize that in that way is that they are kind of understaffed to make sure all the security fixes get handled with. And especially when fixes are not that critical, they advise maintainers to contact the release team to have a look if it can still be included in the point release. If you look at the second one with the ASCII doc, I hope everyone can read it by the way, I don't know. So there is stated replace with DB Latech, and this is actually an example of something where the functionality of the package was just broken and by changing the dependencies, the functionality was restored. As you also can see over here, there are many packages that were rebuilt against the latest kernel. That means that there was a security fix or a really serious fix in the kernel, which of course needed other packages to adjust by rebuilding or sometimes even by slightly patching. You can see over here Linux 2.6, several fixes. That doesn't say much, of course, but the things that get fixed in the kernel are mainly regressions or smaller security fixes. Another thing that got changed quite recently is that we now, for every point release, make sure that the Debian version in ETC is updated accordingly so that users know which point release they're on and that it's also easier to see if something breaks and what point release did break their system. Another important part of point releases is when we fix something in the Debian installer. There are various reasons to update the Debian installer, but as you can see here, some of the things are not always clear from the description, but there were some issues in the installer that you couldn't read all the text that was displayed, things like that. For instance, also what we fixed in another point release was that you could install old stable with stable. Then you have the security updates. As you can see, they are ordered by Debian's security advisory number so that you can easily find them back on the web to see what the details are about the security issue. When you follow the security link, you can then go to the list of DSAs and click on the issue to see and stuff like that. Then, of course, the usual end of the email where there is an explanation about what Debian is about because people are subscribed to Debian Announce, but there are not many messages so when people get a message on Debian Announce it's really useful to tell them or to remind them what Debian is about. Then, of course, maybe one of the most important things here that if there are issues that you can mail the release to. Then if you go to the developers view, so if you think there is something that should go into stable, we use the proposed updates. Proposed updates is a mechanism to make sure that when you upload the package, it gets reviewed by the release team. When we accept it, it arrives in the proposed update suite so it can be tested before it actually enters when we do the point release in stable. Over here you can see how you can add a line in the sources.list to really use the proposed updates. To test the updates, we are planning to include in the point release. Then at the end of the page you can also see that how you should do an update to proposed updates. The guidelines are that the bugs you fix should be filed in the bug tracking system. The reason is simply because if they're filed in the bug tracking system it's easy for us to find them back but also easy for users if we make a description of the change and we include the bug number that they can see the real details about the issue. Then we also like that the patch for the bug fix is tested first within unstable so that there are no surprises in proposed updates nor in the real point release. Then there are some guidelines about what kind of issues we fix in point releases. Definitely security issues and that can of course be through the security team and the security archive because all uploads that end up in the security archive, well on the security mirrors, they are pushed through to proposed updates so they get also included in the point release. Of course not all of the security issues like I said before go through the security team. If in doubt please contact the security team and they will advise you to either upload to the security archive or to contact the release team to have a look if it's okay to upload to proposed updates. Then of course critical bugs and critical bugs is of course release critical bugs but that can also be bugs that actually are not reported as release critical but that are experienced by users or by the maintainer as really critical or you're there again. And then of course uninstallability and failed to build from source bugs these are released critical bugs but they are not always perceived as critical by a user or by a maintainer especially if it's on a particular architecture. And so there is also when something is released and we find out that it's actually also able to build on more architectures is also useful on these architectures. We try to bring the architectures back in sync and make sure that the packages are available on all the usable architectures. And so the way to the process actually for the developer is to first have the bugs filed in the BTS to find the patch for the bugs and so of course the bug filing can happen by a user or someone else. The patch can also be by someone else but we like the maintainer preferably the maintainer to file the patch to the release list preferably the maintainer because we really want the maintainer to feel responsible for the patch and of course for the resulting package in stable. The reason we ask to still send a mail to release is to make sure that we can always trace back when something changed what the change was about. And so even if something goes wrong with with archive server or something like that we are easily able to go back to the state we were before. And of course sometimes we reject things because they're not fit for stable but that doesn't mean that they're not fit for for instance volatile or back ports. But of course this one is about point releases. In two days I'll probably give the the buff about cooperating better between stable volatile and back ports and getting some common policy to to know in what archive fixes should reside and things like that. So now we have the user perspective. They get an announcement that things got changed. They might or might not already have the security updates installed. Then we have the developer's perspective that well you have the fixes ready to be included in in stable or you're trying to reach that goal. And then we have the release point of view. So we have an overview pages of what technically is missing before we can do a point release. First there are some to-do items that are things that well are not really package related. So for instance Decryft. Decryft is making sure that old binaries that are not used anymore in the archive well not not pointed at by packages files and things like that that they can that they are removed. We still have an issue on IA64 that the package has not been built there. It is of course package related but it's not related to an upload already to proposed updates because it's the existing package in stable that's not getting built on that architecture. And then we currently have an issue in stable that the task overrides are wrong because some scripts used still lany-sit instead of squeeze-sit. So it changed the lany one and where it had to change the squeeze one. Then we have the removals so that are packages that maintain your things that are not worthy to be included in the release anymore. And we keep track of them because of course we have to tell the FTP masters to remove them at point release time because there is not a single change in stable between point releases. Every change happens at point release time. A removal, an annotation or an update all happens at point release time. And then we have an overview of the packages. So this page is really useful for the release team but of course also for maintainers to know in what stage of the decision process and of yeah well the decision process the packages is it already in proposed updates or is it still waiting on a review or is it waiting for something else. The resolution pending ones are of course not yet in proposed updates. If you see that it's green strange green there but if it's green it means that we are the release team is okay with the update but for some reason we didn't accept it yet in proposed updates. For Apache 2 that's because Apache 2 MPM ITK has to be rebuilt at the same time so has to be available on all the architectures first and it's still missing the S391. So yellow is clearly that it's not yet ready to enter proposed updates. And you can see that some others are also missing built some some architectures. And then you have Borowski where you see it's still blank. Still blank means that we didn't really review it yet. We didn't make a decision yet if it's okay to be accepted in proposed updates or if we should reject it. So it's still outstanding. There are some others and then you have the pending processing here. The pending processing and currently zero upload that are the ones that we accepted but are still not in proposed updates because the install didn't happen yet. And then we have the processed ones and of course that will be many accepted ones. You can see with some of the accepted ones that are still an issue with the version that the version in proposed updates is bigger than the one in testing. We try to avoid that as much as possible but if it's not fixed before the point release that means that the version that is in proposed updates will propagate to testing and unstable. And if you see a yellow one that means that either it's not a security update and it is accepted and then it only gets auto-built when it's already in proposed updates because when you have a security update it has to be built on all architectures on the security archive before it can be accepted in proposed updates because otherwise you can have that gets built on two different locations and then you get different binaries and that gives problems. But the yellow ones normally are ones that are not built yet on all architectures and then you can see the red ones that are the ones that got rejected. I should also notice that whenever we accept or reject the package we of course send a reply to the mail that was sent to us on the release list and as you can see most of it got accepted. So that's a quick overview of point release from every perspective. The release perspective is of course only highlighting the technical aspect because we also have the social aspects that for a point release to happen we have to make sure that everyone that's involved in having a point release is present and does his part of the job like the press team is available to send the announcements. The archive team is present to make the necessary archive changes. You have for instance the CD builders are ready to build the CDs when the archive is ready. The mirror team is ready to make sure that everything will end up in the mirrors when the archive is ready and things like that. So that's coordinating. It's of course also coordinating with translators to get the announcement translated, coordinating with the maintainer for instance of base files to make sure the file Debian version has changed things like that. But at this point I actually wanted to ask the audience if you think there are things in the process that can be improved. Actually I would really like to see more testing of proposed updates because something also together test results mainly for the stuff that's not a security update because we currently don't know how many people are using proposed updates and it's only moved into stable at point release time and not tested somewhere else and it would really help to get some feedback on that. The change should probably be tested on stable already of course. In reply to that I talked to the FTP master during my time being stable release manager having proposed updates being a full suit instead of a partial suit which would only actually affect the packages file being a little bit larger but that would be all what's needed from the release team point of view to get better testing from proposed updates. That wasn't implemented yet. Yeah the reason being that you don't have to specify stable and proposed updates to test proposed updates. Sorry. Yeah you can also remove packages already through and test CDs and and DI. Okay but sensible improvement already without making proposed updates a full suit would be clearly communicating to users that well perhaps that's been done already but that proposed updates is not now like it used to be in the past that could get random shit like now what entered proposed update proper has been reviewed by a member of the release team and so it should be safer to enable it if it's not a mission critical system because the review process is similar to what happens during the freezing testing so I don't know what could be done or if we want to do it to ask our users or perhaps our brave users to enable proposed updates in their systems. I have no idea if that's something we want to do or not. Well I don't think we want to do that by default because of course some users only want really want to use stable and decide themselves if they want to upgrade but but of course it's already mentioned in the release notes that if they want to test the proposed updates they should add the line in their source.list. The problem with that though is that you still can get problems like this overrides thing breaking and the first possible point that that could have been noticed was when Sledge built the CDs for the stable release update I think by that point you've already basically pushed it out so if you want to be able to catch this kind of breakage in the future you really need to have the ability to build test CD images and things like that. Yeah true on the other hand the CD building scripts and DI stuff could also be extended to have proposed updates as an option. And that still doesn't cover the case of removals and such I think there really is a fairly strong argument for trying to have proposed updates appear as a complete suite. Yes sure. Yeah you know there are ways you could work around various of the other problems that are being discussed but they all have some corner case that wouldn't be caught. Indeed. Making proposed updates full suite would solve a lot of issues through. Are there any other proposals making the process more lightweight or whatever? No one? And then I actually wanted to already have some preview of the and a half releases because and a half releases are actually point releases that normally only happen like once or none at all during a release cycle and where we introduce new packages to make sure the release can be installed on new hardware. So that's mainly means updating the kernel so having a new source package of the kernel that you can install next to or instead of the current kernel in stable. But of course it also means things like the XR drivers and so to make it possible to install your system with new graphic ship sets and things like that. Everyone keeps me asking will we do it again because last time with H&H it was meant to be an experiment. It was the first time we did it. I think the feedback I got from H&H and was mainly positive actually almost everyone was really happy that we did it. One comment I get quite a lot is that while we upgraded the kernel well upgraded we made sure that you could use a newer kernel. We didn't make sure that you could use the new kernel with all kinds of modules so maybe that's something to look at for if we would try it again. And I also wanted to ask the audience if they think H&H was a good idea and if they think we really should do a Lennie and a half? Yes, blatantly. I was bitten as well by random hardware that wouldn't work with the initial release. If anything I'd happily be pushing for a Lennie and a third, a Lennie and two-thirds but I know there's a lot of work involved. The only problem that I really saw with the H&H was the fact that it took so long to get it out and we really desperately need volunteers to help. There was work from quite a few other teams and I saw that unless Dan was pushing prodding people nothing really happened. I mean that might be an unfair view of things but it was definitely the view I had. Well it's definitely true if you do such kinds of and a half release you need more people to really care about doing it because last time around like you said it was kind of only the release team and then Frazier that really looked into making sure it happened. I think definitely the maintainers of packages that support new hardware should come to us in advance so we know which packages we need to look into and where we already can prepare for having them in a point release. What I would like to see for Lennie and a half is the possibility to install Lennie onto a storage area network because the current kernel and I think also the current bootloader doesn't support that. Doesn't support what? Installing Lennie into the whole system into a storage area network. Okay and who will make that happen? So the main issue is the grub. Well if it's just a two line patch I think it could just be added in the point release if it's non-intrusive. Yes indeed. It doesn't need a half release. I looked at the popcorn data for edge and a half. Shortly I hope I didn't make grave mistakes while looking at it and the peak for i386 was at 360 users and for AMD64 200 and you see a dropping of course after the Lennie release so it's mainly used to bridge until Lennie's available but there are still according to popcorn 270 users on the main architectures of course there are still some binaries because it's organized by binary package that use some other architectures but the counters I saw were all pretty small which means 10 or lower and I think the question is what hardware we need to support for Lennie and the half if we really need if it's urgent to support new graphic drivers if we need to support like wireless drivers which are outdated in Lennie and how quick squeeze is coming. True well I'm not going to say anything about squeeze now. Any other comments about a half release? One of the reasons we only did a half release after more than a year after the normal release is security support because of course if we want security support for the normal release and the n-a-half release that means you have more kernels to support in the security team so probably we should make make sure that the n-a-half release if it comes that we have security support and probably that means making sure that there is more people working on security support for that. Actually I think that looking at well we are definitely not going to do half-year release cycles anyways that wouldn't also match our audience and we have already a lot of new shiny things that are not really supported by kernels so yes I would definitely go from plus half release if we could do if we could do means if we have some manpower to do it and yeah well if some hardware options can't be supported as I can remember we discussed about that last time in Edinburgh quite much we said well we can live with it so we don't support any hardware no or so we don't need to support all hardware again the plus half release we just did the document properly things that are still very valid yes true it's not that both kernels have to support all the hardware we really want to have most hardware supported by the new kernel but of course if the old kernel still supports more hardware there is no reason for people in that case to upgrade to the newer kernel does someone still have any comment or question about point releases in general or the n-half release well testing again in general the weekend we do a full release we have people volunteering from all over the place to do cd and dvd testing by the time we get to a point 01 or even worse a point 02 there's nobody around regular volunteers really would be good if anyone wants to help please be around watch the release list and dive in there's always shit for people to do yeah I need to proxy then Frazier from ISC that it's always possible to add or theoretically possible to add hardware support in regular point releases and it doesn't need always and a plus half release it depends of course on the effort to backport a driver but well it's also something to be considered yeah definitely I think it would be a good idea to have more point releases where we introduce a new hardware support if it doesn't affect the current systems any other one comment question no then we have a larger break for coffee or something I guess