 Okay, Steve Langasek and Andy Bart. Thank you very much for that introduction, Natty. My name is Steve Langasek, this is Andy Bart, and we are Debian's release managers. I think we know many of you already. If we don't, come by and say hi and introduce yourself sometime in the remaining days at DebComp. So today we're going to talk a bit about the release process as a whole, how our work as release managers works, what tools we use in order to facilitate that, what your role as developers ideally would be in order to help us get released on time and cover some of what remains to be done between now and being able to release Edge. So the last time I was down in Mexico giving a talk, I asked the audience the question, do you think it's worth the effort to keep Debian ported to all 12 architectures and release with that many architectures and whether that's a good idea? So then a month later I went up to Vancouver and we all know how that ended up. So I'm not going to be asking the audience any questions today but we will have a discussion period in which you will all be welcome to ask us any questions that you might have about the release process. So now I'm going to go ahead and turn it back over to Andy for this first part of the presentation. Thanks, Steve. Now, first we start with some... Well, basically whatever the developer should know but we experience that if that would be the case you would have much less issues so I'm going to talk about it also at this time. Usually if packages are uploaded, they are uploaded to unstable and then they have some rough testing, something like 10 days after that they can go to testing as long as they don't break anything there. And then from time to time what we hope this will happen in December next is the testing is declared stable, moved to stable and the current stable is moved to all stable and it's also experimental for packages that are not meant to go to testing stable so if you make an upload of things where you say no, I won't try to play with it but it's not really meant to go into stable please use experimental. That will help us a much and also you. Yeah, and what's the release management? What are we going to do? Basically it boils down to make sure that the release targets, the release goals are met. We need to ensure the testing is not broken and doesn't get broken. Also, we need to keep the strings together. That means when some developer starts to change anything, we need to make sure that all dependent developers know it and if there's an issue, it gets communicated in time. That seems to be the easiest to write on a slide. In fact, it's much harder to extract information in time of developers. Yeah, and also just to collect information, distribute it again. It's not so much about that we fix packages what we of course also do, but more about we need to communicate, we need to deliver information. It's a task or a role which just needs a lot of communication. So the next slide is who are we? That are of course, Steve and me. Then we have currently two release assistants that is Joy Hess and that's Frank Lichtenheld. But we are currently about to get more assistance. We have currently somehow tests for them running. We called it task as kills for the release assistants where we give them tasks to fix our C-bugs to work on build demon failures and so on. And we hope to see two, three or even more new assistants coming up soon. Yeah, and of course there's a role account. So if you want to get in touch with us, please just write an email to Debian release at least Debian org. This address is public. So if you are more interested into the release, you could even subscribe to it. But that's our generic role account both for the actual release management as well as for the stable release management. And of course there's a website, the least Debian org, where we try to keep the information for you and collect it. Yeah, so we have some tools that we work with without some, it wouldn't be possible to manage the release in the way we do. There's of course for the testing migration, Brittany, that's a script which shows which package are ready to go to testing. Updates a package to send in testing as long as they don't break anything. And there are so-called hints which we could use to make sure or to change the behavior to say, well, this package is for example not old enough but we know it fixes some security issues. It should go in now. And say, well, this package shouldn't go into testing at all currently. That is so-called block hints which we will also use to make the base freeze later on and after the base freeze even for the full freeze. That's all just by changing some text files and not by changing the code anymore. Yeah, then we have another very important tool that has the release critical bugs which apply to the normal packages of course but basically ever the release critical bug means there's something that we as the release manager needs to make sure it's taken care of before we can release. So the release critical bugs are nothing bad about a package maintainer in the first place but just marking to do items that needs to be handled. And so if you get the release critical bug please don't just close it because you think it's an unfair treatment but if there's valid reason perhaps even if it's not in your package so why don't you say, I doubt it's in my package but yeah, but if it needs to be handled it needs to be handled. So then we have the security status and testing with what Shoahez is doing and of course also if a lot of scripts and small things that help us with the release management starting from tools that just show the difference between testing and unstable making it easier to get some hints for it and so on. And how packages go into testing the third way is always via unstable even if we are in the deep freeze and the basic reason for why packages should go via unstable and not via testing with post updates is they get way more testing and unstable and if something breaks we will notice it earlier and also it's just a more natural way that happens that we have unstable and testing and then sync after that and not having something different in unstable and testing. Also we could use testing with post updates but for that we only allow very minimal changes and only if we can be very sure from reading that nothing is broken or gets broken by the change. Yeah, Britain is a testing migration script run shortly after the install usually the install is... Okay, I don't know in this time zone and my time zone is in the late evening my time zone is Europe and after that Britain is run we can see the results directly the results are also published on the web and we could see run Britain during the day if we consider it important enough if there's something to fix or if we just noticed that or if we try to get a bunch of packages in which needs some manual treatment as it happens sometimes. Yeah, and that's all on that slide. So I will now give the microphone to Steve again. Okay, so the mails that have been sent out on DebianDevel.announce if you read that or that have been copied onto Slash. If you only follow Debian news via Slash. Say that we are aiming for a time-based release schedule that will have us releasing Debian in December of 2006. That's this December. We do of course have certain issues that have to be addressed before we can release at all and so we set this timeline early on so that we could get people addressing those issues that we identified shortly after the Sarge release as being the blockers for Edge so that we could get those out of the way early on and not be in the situation as we were for Sarge where we had unresolved issues that prevented us from releasing Sarge that we could not release Sarge without resolving those issues first. We wanted to make sure that that was not the situation we found ourselves in this time. So we identified really only a handful of blocking issues for Edge that were anything other than release critical bugs and you see that list here. The tool chain, we did mark the GCC-4 tool chain transition which included a C++ ABI change that was marked as a release blocker. The Xorg transition, we said we would not release Edge with X for 86 since X for 86 by that point even when we released Sarge I think there had been other distributions that were released with Xorg already and so it was a dead code base for all intents and purposes as far as the free version of that was concerned so we said absolutely we had to get Xorg into Edge The docs in Maine question which is regarding the clarifications of the DFSG and social contract and what that meant for license of documentation that was in Maine and whether we would have to work to relicce some documentation or get that documentation unfortunately pulled out of Maine if that were the case. As well the firmware in Maine issue which is a similar question now we have lots of device drivers these days that require firmware which is not available in source form probably never will be and what we're going to do about making sure our installer is still functional and can install on the hardware that people have to use in the real world how we're going to manage that where the packages are going to sit and how we're going to provide infrastructure for all of that and get that all sorted out. The mirror split was listed as a release blocker simply because it was a precondition for getting AMD64 into the archive and if we wanted to release Edge with AMD64 as one of our supported platforms then we had to do the mirror split first which has now been completed both of those. AMD64 is present now in testing and unstable on the official mirror network although the testing, you can de-bootstrap it but you can't really install any useful software whether you're in testing something about XORG7 blocking things just a little bit but we're working on that and secure app which is requiring authentication PGP signed PGP authenticated releases files that allow you to verify the authenticity of the packages that you are downloading. So as we say there, most of these are resolved the tool chain, XORG, those are completely resolved the mirror split, AMD64, we know those are done so the things we still have on our plate from that list are Docs in Maine firmware in Maine and secure apps which unfortunately look like some of the harder ones on there and that's something that we need help from everybody involved to help us get through those lists of documentation packages so that we know that the documentation we're shipping in Edge complies with the DFSG the same way as any other software that we ship in Edge. But yeah, those are the only blockers that we identified then we did identify a number of what we called PET goals which is transitions that would be nice to have changes that we encouraged developers and the respective teams to work on but which we did not consider essential for the release of Edge. Those included such things as the transition to GCC 4.1 to give a recent example, the Python 2.4 transition he puts it as a PET release goal I think I'm not so keen on releasing with Python 2.3 I'm hoping that we'll make that a non-issue getting Python 2.4 in is the default very soon so we don't have to worry about it. Other examples of PET release goals SE Linux support in the installer was discussed Manoj is very happy about that one we've been talking about that a lot this week and what needs to happen for that and there are a number of PET release goals that we identified which don't occur to me right now some of them have already been pushed through and so they're no longer on our radar others we have maintainers who are working quietly in their areas to get those done and the PET release goals are things that if you have something that is your PET release goal feel free to talk to us about it as the release team we will try to make sure you have the support you need but ultimately it is up to you to make sure that it gets done by the deadline so yeah um boy I don't know what this point was Andy more QA okay the point about the next one is we saw in Sarge or in the end stages of Sarge there are a lot of issues which were identified very short before the actual release which could have been identified way earlier for example how do upgrades work do the packages really co-install together well do the packages leave graphed over and so on and there is happening much more QA that's basically not something we as the release managers or the release team are doing what the QA team is doing but for us really QA and the release team measures very well together and last for example with the PO parts is a great help us there which is one of the good things that happen now in Edge right that's uh yeah how do you pronounce that? PO parts? PO parts anybody who doesn't know this tool it's a great tool for testing whether your package upgrades correctly from one stable release to another it has hooks where you tell it basically what package you want it to install and attempt to upgrade and it does it all for you and then it gives you a list of what changed inappropriately between between installing and uninstalling it or it tells you if the upgrade failed or anything of that nature it could help and I know Lars has been following lots of bugs that he's found using that tool which is really great because it means that the more bugs we find the more bugs we can fix and the stronger a release we can have other things that we've been doing in the QA front Bill Almember has been actively doing a lot of comprehensive upgrade testing I think this time around which we didn't get to until late in the release cycle for Sarge he's identified he's also been working on the circular dependencies issues which affect upgrades and just in general seeing how real world systems cope with being upgraded which has been very helpful because you know it's possible to have a system where you think all the package relationships individually are right and then app crashes and burns when you actually try to do the upgrade and says oh well you want me to satisfy the dependencies of all of these packages that you're upgrading so I'll uninstall X and I'll uninstall KDE and I'll uninstall text and there you go you have a perfectly working etch system with no packages installed anymore so we hope to avoid that and provide as smooth as possible of a transition for etch for as many people as possible another point with etch is that as I said before the whole Vancouver proposal we do have architecture and requirements this time around where if an architecture wants to be a release architecture the porters do need to do some of that work in fact the majority of that work to make sure it's not holding up the release on a number of different points that includes things such as making sure that there are auto builders available that are up and running constantly and are able to keep up with the demand of newly uploaded packages to unstable it includes things like making sure we have a working installer for it so that if the porter machine for that architecture breaks we have a way to reinstall it down the line so that people can continue using it there's a variety of those requirements which I won't go through all of them they're actually on the release.debian.org site but this has actually been a very good help for us in fact there was a lot of controversy about that when it happened when we first proposed it I know there are still some people that are unhappy with it but the reality is that this has made a big difference in how we are able to manage the release because we are no longer spending many hours a day chasing down one architecture or another which is lagging behind that's still occasionally we have to keep an eye on one architecture or another for specific packages but we're no longer spending lots of time worrying about where the builders are for the architecture for the release architecture another change that's come about post Sarge has been the support for version tracking on bugs.debian.org Colin Watson has implemented a patch to DebBugs which lets us keep track of what version of a package opens and fixes a particular bug which makes it a lot easier to track release critical bugs which were present in one version of the package it may have been uploaded but that version hasn't propagated into testing yet so it's not actually fixed in the release that's actually something which was a big problem in Sarge Adrian Bunk went through and did a lot of work for us individually tracking down those RC bugs to see which ones were or were not fixed in Sarge and we're hoping that version tracking is going to greatly reduce the amount of manpower required in order to find those bugs and get them fixed in testing for the release I think this is another point I'm going to have to give back to you because I don't remember what this one is either what we did do is beginning of the Sarge release cycle when it joins the release middle of Sarge release cycle when it joins the release team but only a few things that we could say to Brittany and we got to more and more things that we are able to give Brittany and say so we have more find ground control what we really want to do but just help so that we don't make mistakes because anything that could be checked by the software instead of me manually helps me yeah okay so yeah I remember now some of the hints we added for Brittany in the release cycle can we save that question for the discussion please okay so we for instance now have a hint which allows us to only override the urgency of a package so that we can say yes if this new version of the package it doesn't have any RC bugs that we didn't look at go ahead and push it in before its 10 day waiting period is up and so that's a nice shortcut which means we don't have to track the RC bug count for the package for a 24 hour period in order to be able to hint new version into testing we can just let Brittany take care of the RC bug count like it's supposed to and not override that at all further improvements we have in etch is we do now have a pseudo package for release notes bugs if you know of issues which you think need to be documented in the release notes so that we have information available for users when they are upgrading from sarge to etch please file those bugs that way we have a central place to collect that information so that it can be included in the release notes for etch the question was what's the package name the package name is release-notes we are also making a point to as much as possible use code names when referring to the release rather than using suite names we ran into a number of issues in the sarge release where lots of references were hard coded to testing so we had to change from testing to stable types of places to do the release and as much as possible if we can use the code name we prefer to because then that's one less thing we have to change in order to do the release it just works correctly after we change the sim link and we are going for a shorter and smaller base freeze for etch than we did for sarge of course lots of people have looked at the schedule and thought well actually we are doing the release but the reality is if you remember back to sarge base ended up being frozen for about a year while we were sorting all of the issues out so this is in fact quite a bit shorter and it's what we believe is the minimum required base freeze in order to stabilize everything stage by stage and we are also now thanks to the work of Ryan Murray able to do bin and amuse in a much more automatic fashion across all transitions which means that when a library SO name changes and the package name changes but the dev package hasn't changed at all it's just a small ABI change within the library we as release managers together with the building maintainers can go ahead and just queue bin and amuse or binary non-maintenor uploads to rebuild those packages on all architectures and the maintainer doesn't have to be bothered with it at all basically which is very nice because some maintainers it seems to be a bother for them because they don't you know respond immediately and this streamlines the process allows us to move a lot more quickly on some issues further things that we we have planned to improve these release processes is that we are looking at some enhancements to Brittany would reduce the dependencies between packages between the transitioning of packages for SH libs or SO name changes where we would actually have Brittany keep around previous versions of libraries in testing so that we could have both the old and new library package names in testing which would allow us to do a bit by bit transition with the new SO name into testing rather than what we have today where if a library like say lib free type changes SO name you get a panicked email from a release manager posting to DDA saying okay we have 600 packages that are depending on this library and its SO name is going to change we need some help fortunately that one did not come to pass the SO name change was reverted finally but that's the kind of scenario that we want to avoid where when you have more than about 5 different packages or 5 or 10 different packages using a library in Debian and that libraries SO name changes you end up having to do a lot of coordination work in order to get those packages all into testing at the same time and it becomes far worse when you have more than one library maintainer SO name at the same time affecting packages that depend on both of them and not coordinating their uploads with each other we'll get to another slide about that a little bit later so yeah we are hoping that we will be able to improve Brittany to deal with some of that more automatically so we don't have to worry about it so much and we're also hoping to do a lot more QA work on packages for test for edge improving them as much as possible proactively identifying classes of bugs which can be identified and can be addressed prior to the release another point which point do you... it's a better interaction basically if you look back at Sarge we had for the kernel a lot of different kernel source packages which were uploaded and then you had a kernel which built UDEPs and there are some discussions going on how to improve it or it has already improved a bit because we now have only one kernel package, the rest is done by the auto builders and there's some more discussion I'm not too sure about the details but one general thing how we like things to do is if all of the maintainers that are affected by it somehow are happy with it and so that's the way it's going to be there I'm not too sure about details because for example, France is really keeping track of it and so I don't need to we're also hoping we will be able to take care of in the edge time frame or at least on our list of things to do is get better UDEP handling in Brittany anybody who happens to maintain a package which builds UDEP or microDEPs for use in the Debian installer is probably aware that their packages don't transition into testing automatically the reason for that is that Brittany, the script that handles testing does not handle UDEPs and currently it's not doing so because primarily because of some scalability concerns which we hope we can find a solution to address those so that we will be able to automatically process those packages and get those included in the transition into testing automatically currently what happens is after a packages source makes it into testing an FTP team member has to go through and manually add the UDEPs to testing so in order to make sure we don't end up with a source package getting into testing it's UDEP not making it in with it a new version of the package being uploaded to unstable and the UDEP disappearing before anybody gets a chance to process it which is produced UDEPs currently in a freeze so if you've ever seen any testing output about that, that's why I guess we have a question here Are there plans for publicizing some of the tools I've seen coming up that checked libraries and SHLIB versioning and automatically tell you when you're supposed to be bumping them or not Yeah, the question I'll just go ahead and repeat a bit of that was the question is about there are some tools out there for improving SHLIB handling and allowing you as a library maintainer to know specifically and automatically whether or not you need to bump SHLIB versions those are there are some tools that I've personally been working on as a result of my release team work in order to do that and it's just a matter of getting them as you say publicized and integrated in some way that people can make use of them which is something that I still need to tag a few people here for at DEBCOMF before I leave and see where we can get to on that Checklist with dependencies no out of order processing we did have some problems in the SARGE release which we are hoping not to repeat where we did not necessarily do everything involved in the release process in order to make sure that all the concepts had to be redone because we didn't realize there were other things that needed to be done first so now with that experience under our belt we hope that with etch we'll get it right the first time we're hoping to be able to continue providing more up-to-date information on the release.debian.org website Andy in particular has said he's committed to providing more release information on an ongoing basis on that website we need to make sure that this time around this comes back actually to dependencies we need to make sure that when we built the CDs we are building them from the final archive instead of trying to build them before the archive has settled because well one of the things is okay that means etch is labeled as testing rather than stable and we ran into some problems if somebody remembers what was it 3.0R0A that came out three days afterwards that's why and we're hoping to yet integrate version tracking support into britney today we do have support for querying the BTS to see which versions of package carry a particular release critical bug we do not have the ability for britney to use that information directly in deciding whether a particular package should be included in testing or not whether or not it's an improvement from the team's standpoint viewpoint over the version already in testing and I guess we're going to be doing some final QA for release CDs and DVDs which is another one of your points just we need some way to finally if it's my point before this that slide which is just for the search release cycle we didn't have to do any real QA we didn't test install of CDs, DVDs which worked well but we didn't look for example what exact line is for security in the sources list which turns out to have been the wrong line which even stayed after the search was released and we don't want such a repeat okay we have a question back here wouldn't it be a good idea to make a pre-release of the CD images to avoid this type of mistakes and would it there was an issue too with Spark architecture if I remember correctly okay the question is would it make sense to do a pre-release of images for the release the problem with that is things change between pre and actually releasing you can't very well master a pre-release CD image and have it look the way the final release will look until the archive has changed around so that the version you're looking at is actually labeled as stable in the archive because the Debian CD tools that we use today and the Debian installer I think I'm not sure Joey does Debian installer today does it look exclusively at the code name okay so Debian installer is fixed for that but the Debian CD tools I believe still have some references where their behavior changes based on whether based on which suite it is not just on which code name it is so the problem is you do a pre-release and you're doing a CD build of testing which is not a good predictor of what the build process will look like when you're building a CD that is stable but it also applies to the things that we had before we said we don't want to do out of order processing because basically what we did last time was to build the CDs a bit prior to release to be able to check them a bit but the real issues that came up we didn't expect that there were real issues which appeared because of the basis CDs are built there would of course be a way to do it for example we could say okay at some certain point in time we stop the daily install the daily FTP master scripts running so we don't get any new packages installed unstable anymore we could then change the stable and hide it on certain servers and do the CD builds for those first people but if you look back at the last search release cycle when it became a bit apparent that search is now really going to release people were finding search images and CD images in all obscure places for example syncing the CD images around the world was delayed because some people get noticed where CD images are put online for the other mirror sites to get them from so that was really bad and we probably need to put even on a password protected server for some time to be able to just be able to provide our mirrors with the CDs so it's a bit difficult to do it a lot of developers would be unhappy if we say well for one week no more packages are processed but of course that would be doable would it make sense to do something like using union FS during the freeze like just do it mirror test the thing with union FS create the CDs test them and if that works then do it in the final release well I guess the question is using union FS to test the CDs prior to the release I don't see that union FS provides any distinct advantages there basically one way or another what you're doing is you're doing your test build from a modified mirror at that point which means you're not testing using the real thing you had to make a change locally in order to build it and which may or may not it's possible to introduce bugs there because you're doing the work twice in two different places you may end up with bugs that may end up biting you later as far as pre-release CDs though I want to make sure everyone does know the Debian CD team does publish I believe it's weekly ISO images of testing which include the latest version of DI at the time and includes all of the packages that are available in testing so we are doing a fair amount of testing of the CD builds on an ongoing basis prior to release we also have the DI release candidates and betas which have their own particular ISO builds at the time which are it's kind of a dry run at the build process of the final release the main difference is it's not building from stable because it's not yet stable it's still testing so those are the only real issues that we still have to work out there are identifying areas where the change from testing to stable breaks something moving on this is the part where you all should be taking notes because these are things that you should be doing in your own packaging work to make sure that we stay on schedule this is a collaborative process it's not just the release team releasing etch this is Debian releasing etch and we absolutely need the help of every package maintainer in order to meet this goal and part of what we need you to do as maintainers is just simply keep your packages in good shape this is a simple request not everybody interprets it the same way I suppose you do have to keep on top of your bugs or that you do a level of testing of your packages that you're comfortable with if you feel that you're not able to keep up with bugs on your packages then please by all means ask for help whether that be a request for adoption in the WNPP a RFH request request for help whatever it takes to make sure that we're all working on this together make sure that packages are ready to go for the release and that we don't end up finding issues way too late later than we should you can also help with squashing RC bugs whether that be in your own packages or in other packages this is another example of the collaborative nature of Debian work there's lots of opportunity for people to help with release critical bugs the list of release critical bugs is available online where you can go through them and see which packages you use for instance which packages you might have a particular familiarity with that you feel more confident working on release critical bugs in those packages whatever it is there are release critical bugs for everyone we have about 400 actually it was 380 I think we were down to yeah 20 bug reduction from when we came here so about 380 release critical bugs that we are aware aware of currently which affect the edge release that's more than enough for everyone in this room to get one so please sign up for yours now another thing that developers should help with when you're going to upload packages which make changes which are going to affect other developers whether that's because you maintain a library and the library is changing its interface or whether that's because you're uploading a package which provides programs and they're calling interfaces changing completely or you're uploading a version which deletes programs from the package or moves them to different packages and you know there are other packages which depend on yours please coordinate before you make any changes that you're going to upload that implement these changes you don't necessarily have to talk to the release team always if it's a package which you know only has 5, 10 reverse dependencies which use it and there's no need to actually you know do a massive coordination well you don't necessarily have to talk to us it may help we may be able to ease that transition into testing for you if you do talk to us in advance that you maintain the packages that yours depend on that maintain the packages that depend on yours because if you don't you may find that one of these maintainers is 2 days away from going on vacation for a month and there's his package blocking your package from getting into edge because you didn't coordinate ahead of time he wasn't waiting for it he wasn't ready and as a result you know it has to wait until he gets back and then use are a great tool for releasing for fixing release critical bugs they're still a bit slower than actually getting the maintainers involved and letting the maintainers take care of the packages themselves simply because the maintainer knows his or her package better than anyone else does and is almost always best positioned to do whatever uploads need to be done and fix bugs yeah, read and develop before uploading packages as well we send announcements there we let you know about transitions that are going on which will be broken quite badly if you don't follow our directions in those cases it's really painful when people ignore Debbie and develop announce and as a result we have one week further delay for this package that was uploaded not according to the rules and so on and so forth and when you have 15,000 packages in the archive if people don't pay attention to what's going on around them it becomes very difficult to manage what's going into the release because everything is in a way interconnected and it compounds itself very rapidly maintain your libraries in a good way that includes first of all letting maintainers know when you've changed the interfaces when you're changing the SO name of a library that includes things like making sure that when your library does change it's when it adds new interfaces that you do pump the SH libs and that's something that we hope to provide better tools for you in the future to automate this it includes making sure you look at the no upstream version of the library before you upload it and if the SO name changes change the package name don't upload it under the same name because then all the package is depending on it throw an error saying yeah, no library found under this name well, dependencies are satisfied but the package satisfying the dependencies is completely broken who here has actually seen that error before in an upgrade that recommends writing your package such that it fails to build if that happens so it's possible to make that mistake that's a very good point you should rely as much as possible on automated testing so that if your package isn't building what you're expecting to it should fail to build wildcarding your library names while convenient is not so convenient for the rest of us when a recent example was TASN1-2 was uploaded containing LibTASN1-3 which broke GNU TLS which half the archive depends on today kind of inconvenient when that happens so please if you are a library maintainer keep an eye on those kinds of issues if at all possible your package should fail to build an error like that that is a definite bug if you have questions about how to do that I was going to offer myself to help them with that maybe I should just offer your services there oh he's ignoring me now I was just volunteering you to help them if they have any questions about making their libraries fail to build okay no certainly you can contact Debbie and DeVal if you have questions about that please contact me personally if you feel that you must Debbie and DeVal though does certainly have many people who have the knowledge necessary to help you and most of them are more than happy to pitch in and make sure that our packages are future proofed against changes upstream so that they are not going to break on us down the line so I tried to tell Andy this first one didn't really belong on a how every developer could help slide this was actually an error that we as release managers made during the Sarge freeze process was that we second guessed our freeze guidelines which ended up letting a package in that included some changes which it shouldn't have and it broke a few things and we had to do a bit of scurrying around in order to get that fixed for the release but in a sense this does apply to every developer as well please don't second guess our freeze guidelines we say that libraries need to be coordinated please listen to us because there is a reason we say that it's not just that we like to be control freaks it's that when we're not control freaks then our lives are hell because we no longer have any control over anything like I said it compounds very rapidly when we do not have coordination between library uploads and major changes in libraries due to the way Brittany runs today please don't upload packages to unstable that are not targeted for the upcoming stable release unstable is not a dumping ground for experimental packages we have a suite for experimental packages which happens to be called experimental so if you would please do not upload things to unstable which you're not going to try to get into testing and from testing into stable especially if it's a package which is dependent on by other packages that makes it very difficult for us to do our job of getting updates into edge for the release and again please don't re-upload packages if there's an ongoing transition part of this comes back to library maintainers do need to be coordinating part of this comes down to maintainers of packages some libraries that are transitioning also need to be paying attention to what's going on and we have unfortunately had occasions where maintainers just were not aware that their packages were in transition even though they had every opportunity to be aware of that do you want to take this one? so now is something to get motivated for the releasing edge in time Lars already offered a bet that he will get tattooed with all the release names he worked on if he release in time or even earlier I think we all want to make sure that this really happens but if you have another bet you want to offer we are of course also going to add it so any more bets or do we do this evening while drinking beer? yeah it remains to be seen whether Lars's bet is going to inspire enough people to help working on the release to offset his own stepping back from fixing release critical bugs which he tells me that in order to hedge his own bets as it were he's not going to be fixing release critical bugs for us anymore so come on people step up and help us get there the question was can he stop filing them as well I don't know do the bugs.debian.org admins have an opinion on that can or can't cool just the final question which is quite often asked to us are we there yet? not yet but we could we have still way too many RC bugs with something like 380 RC bugs as of today there needs to be something done on it we have still some transitions being prepared not too invasive but still it will take some time we have broken packages and unstable for example currently xorg which change from a single binary to the modular ones is of course hurting us some packages are not binary NMU safe which doesn't probably mean anything bad to you as maintainers but it means something bad to us as release managers because we can't just schedule and bin NMU of that package if we need to so on the other hand we are some time away from the release of edge so we still can make it we can make it at the proposed time but only if we really want and we doesn't only apply to Steve and me it applies to us all to all developing developers and package maintainers I'd like to insert here as well a comment regarding bin NMU safe packages and what that actually means the reason packages become not bin NMU safe is usually because you have a source package which builds both an architecture all package and an architecture any package and declares a strict version dependency between them well when we do bin NMU of packages we are only rebuilding the architecture any portion of the package we are not rebuilding the architecture all portion which means the architecture any package gets a new version number the architecture all package does not and as a result the dependency is no longer satisfiable so what we've gotten I'm glad to see that the deep package maintainers have gotten a change now into the latest version of deep package in unstable where there are new variables available you can check the names of those I think is that mentioned in the deep package change log I'm not sure but I will make sure that it is set in the next version of release updates there will be a release update that mentions exactly what those new variables are but there are now ways to declare yes I want to depend on the version of this package which matches my source version my source package version as opposed to the binary package version that I'm building which is going to help the relationships between architecture any and architecture all packages so that we're not breaking down okay question what is the issue with simply doing source NMUs in those cases the issue with doing source NMUs in those cases is simply that they take longer to do given that we do have policies where you do not NMU prior to first of all filing a bug report so that it's documented and then file the bug report you're expected to wait and give the maintainer an opportunity to react sure but in this case given that it's not necessary to merge any changes since it's basically a no op upload it seems reasonable to make an exception in order to allow the release process to continue sure I mean that's a thought also a source NMU is more work than just binary NMU because binary NMU basically is more for A in architecture to schedule bin NMU so that's very easy to do and also sometimes we don't need bin NMU in all architectures but just in one or two and well I think that bin NMU is a great thing to do and it's actually most packages they're not broken they're only where few are yeah and that is as you said there is a point there where occasionally we have breakage in one or two architectures where we don't really want to do a source NMU and push it out to auto builders for all 12 architectures because only two or three of them are affected and in those cases it would definitely be helpful if we can do bin NMU for just the affected architectures and save ourselves a little bit of time there are certainly good reasons in that case to do bin NMU but I think it makes sense to have both options available because they have different characteristics for example any developer can do a source NMU in this case as opposed to having to rebuild the admin basically yeah any developer can do a source NMU in my experience it's actually faster for us as the release team the turnaround is much faster if we schedule a bin NMU and just get it signed it seems to actually work better it cuts down on the turnaround than having a source NMU because even to have one of us do a source NMU well we have to go and pull in the build dependencies and build it on our system and so on and so forth and go through those processes regardless and with a bin NMU the auto builders are all set up they all have their mirrors in place and it's just one more thing for them to do in the day which they're already doing alright thank you so I think the next question is here please take a microphone if you okay who has a microphone I have questions from the people in IRC first Martin Zobel asks if there are plans on how to handle the late changes because Debian release list wasn't scaling very well for the charge freeze so if how it will be handled for it tell people not to make any changes no I mean I can't think of anything that really promises a better solution than using the Debian release list because one way or another you basically end up with a queue of issues that have to be dealt with whether it's in someone's mailbox or on a web page or what not I don't think it makes any difference unless we actually try to split those up in some fashion automatically which is also tricky because then you have to you know fiddle with it to get it just right for people's availability how much time they have to work on those issues and so on and so forth so I don't see that we can really improve on that where we simply have people mailing Debian release if that's if they need the release team to do some work I think that's really what we should stick with one thing that we hope that will get a bit better for us this time is for example we have done we have reduced the amount that we have the base freeze reduced the time amount that we have the real freeze because basically as soon as something like that starts we are on the release team we are really I have a lot of mails going on to us for example if I remember correct we had it's a real freeze something like 30 to 40 mails per day requesting some package updates which basically means that every one of us has to work for something like two hours only on package updates per day which is very very much time one thing that should help us as well with that is since we do have version tracking in place people don't need to tell us that they have release critical bugs that they fixed in unstable which are still present in testing because we can look at the release critical bug count and see that for ourselves so they shouldn't need to send us any mails about that and that was a fairly large percentage of the mails during the Sarge freeze was about okay I fixed this bug please let it into testing for me well now we can test that without having to get a second mail about it I think we will have an automatic list of issues fixed in sit but not an edge so that we can just go through this list but that's something we can just implement before the freeze so it's not important as of today but of course very important just in front of the release okay a question from Martin Sobel he says that there are still many arches that have no real building redundancy and if that will change before the base freeze comment there was another comment there that I didn't catch I building redundancy is certainly something we want to have in place for all release architectures it's something that historically has not been in place and so of all the requirements that we discussed in Vancouver that that we've moved forward on it's the one which we are enforcing the least strictly simply because you know we don't want to toss all architectures out of Debian for not having this because the reason this requirement exists is it's a safety feature in case the primary build D goes offline well it doesn't make sense to exclude an architecture from the release to my thinking if it has one build daemon and it's working and it's not failing then why do we cut it out of the release before it fails but what that means is if you don't have any build D redundancy in place we're not going to be too lenient if it does fail and goes down for two weeks and you're not building any packages we're going to say okay well you were supposed to have build redundancy in place you don't have it and this is what happens when you don't have it so we have to cut you from the list of release candidate architectures at that point so that's something for all porters to be about if they don't have build D redundancy at this point is you know what happens if the build D fails and there's nothing to pick up the slack but if you go through so the redundancy actually is on some architectures running so for right architectures so the service build D redundancy are mips it's the mipsons, the power pz and sc90 for example eC86 I don't think it's too hard to find another machine suite as a new build D if we need to and basically also in case it goes wrong well we might kick an architecture out fairly soon because they're basically supposed to and also this number of architectures with redundancy are going up and up so that we have more of them having some redundancy yeah I think the fact that spark is listed there as being only security redundancy I think that may be out of date information now I don't see James or Ryan here in the audience to ask them directly but I'm pretty sure that we do have build Ds now that are building that we have n plus one capacity on the build Ds for unstable at this point on spark yes I think this table is just updated from time to time and the number of build demons was not updated so much, the only thing that updated recently was the architecture status of ARM because they basically failed to match our criterias and now match them again okay well I think we're actually done with all of our slides is that correct yeah so we're fully in the discussion period right now we have a question here with Peter and then I saw Matt had his hand up over here and doko also so if somebody can run microphones around as needed I'm really curious about what you said earlier about making a change to Brittany to allow multiple versions of something in testing I assume that that will not happen in a stable release that you can't have multiple versions of package in a stable release so you also have a plan for a process to then eliminate the duplicate packages when it gets close to release yes that's a very good point and I did in fact mean to mention that the old versions of library packages are only meant to be kept around for a transitional period the idea is that Brittany will automatically remove them as it always does when nothing else depends on them anymore in testing so the solution for getting them out of testing for the stable release is make sure nothing depends on the old version whether that's by removing any packages that have failed to transition correctly or push expediting new packages in that use the new version or what have you got we're not going to be releasing with two source versions of packages of library packages in a stable release but that's certainly something that should be manageable once we freeze that we should be able to knock through all those issues fairly quickly we don't have support for it in Brittany yet anyway so that's somewhat down the line at this point I'll be very quick is there bug squashing party, bug squashing night at the conference and if no is it worth making one well it would be worth thinking one if it would have proper net access you mean every night isn't bug squashing night here what are you people doing bringing strong alcohol I noticed okay another question from IRC Martin Kraft asks if maintainers of UDEP packages are supposed to do something about the testing block that you mentioned earlier or if that was just information okay yeah the question about whether maintainers of UDEP packages are supposed to do something well it would be helpful if they would email Debin release when they think their package is ready for an update in testing we I think periodically scan for those or not so periodically I'm not saying something about how long the periods are but we are doing that so if you have a package that builds a UDEP and you want it in testing email Debin release and we will push it in for you it's just something that we have to know that someone on the FTP team is available to do the other side of it once we've done that a lot of work seems to be done for the release in getting packages from unstable to testing we are the Brittany program currently I see the release team but recently the release team asks people on package maintainers to actually trying to reduce the number of dependencies for packages by splitting source packages into multiple smaller units with less dependencies that enforces some work on the package maintainer for some deficiencies in the transition process from unstable to testing is there any for my thinking that only cures some symptoms about the transition process is there some work being done in simplifying the transition process from well yes we've identified the issue of the changes of libraries around during a transition period that would greatly simplify a number of our transitions but I don't think I agree with your premise that we're asking people to make changes to their packages which only benefit the testing process things like reducing the number of libraries your package depends on when it doesn't use them those are things that benefit not just the testing process that reduces the number of times we have to rebuild the entire archive for stupid reasons because if a package isn't using a library that library changing should not require a rebuild of the package bin NMUs are one thing that make it easier to rebuild the package but it's still wasted work that we shouldn't have to do at all if packages you know depended only on the libraries that they actually needed for operation one example which may support your position is that recently I did ask the avahi maintainers to not ship C sharp bindings from the same source package as the C bindings that all of KDE and NOM depend on so I'm not sure if I would say that's that's uh you could argue that that supports your position that it's a change that only benefits testing on the other hand I think it benefits the release in general because the reason I asked for that change was because the C sharp interpreter the C sharp packages were not in a stable configuration yet they were going through ABI changes and holding things up as a result of that so if we have language bindings for a language which is still in transition we do need to have the option of pulling the language and things that depend on it from a release if things are not releasable at the time we go to edge yes I did just say we should be able to see C sharp out of the release if it's broken then many people ask about the status of Debian volatile I'm sorry I didn't understand what you said Debian volatile and the outlook after edge the status for edge and okay I think that's now a question for me to answer about basically I think Debian volatile has some other benefits of volatile is that we can push new packages in as soon as they appear if their target is stable whereas for a point release we can just do a point release every few months so for example something that really moves as fast as Glamov does we need something like volatile because obviously we don't want to put a new version on security if it doesn't fix security bugs so I think we will continue to use volatile there has been discussions whether to add just the latest version of Glamov to point releases also but that's not decided yet and basically our approach as stable release managers was that we just got two out so that we have a lot of packages now as part of stable and then said okay for S3 we want to work on the kernel on the kernel updates which needs some help from the Debian installer people and all the rest of good ideas is then moved to L4 to the 10 minutes apply to only the talk or to talk the discussion that okay I can see Joy Hess I got one thanks I just wanted to point out that we have redundant building machines for HPPA and I-64 they're in the process of being set up they're not turned on yet but the machines exist and should be soon redundant building machine for alpha so a few more of those boxes should turn green and I think at that point there will only be a couple left that aren't and it should still be a good release requirement well basically what if you look at the fields cause one means our requirements are fulfilled so that one says something broken and the yellow one means well actually it's not like we would like it but we could live with it for the moment next question I think Joy I was just wondering about the firmware issue if we have any idea one that's going to happen etc okay the question was about the firmware issue yes I wonder too basically this is one of the last bad things that we need to work on and we're just really a release blocker because we have decided or my childhood of developers has decided it's starting from edge on we need to make sure that everything in Debian being a documentation firmware also needs to be DFSG applies to it so yes I think it'll hold a month or two for the installer to be updated to support non we get it into after it's kicked out of the kernel so then I agree with you Joy that this is really one of the things that could delay the release so we need to work on ahead some discussions with France here about it yeah but we need to work on that definitely so next one m68k is listed having 15 built demons and well and still only secure redundancy only for security what's the problem with that well basically if an architecture doesn't keep up how can we say it has redundancy it keeps up with security fixer so it has redundancy for security but it has no redundancy for the normal archive because it even can't keep up that particular entry though I believe is about four months old at this point four or five months old I don't think we've reviewed recently how m68k is keeping up the view how much it is keeping up and it's still not keeping up well is that actually that it's not keeping up with packages or that it has a bunch of packages that are in failed because of compiler issues we could check up now but anyways m68k is currently at least with gnu6c4.0 not another release candidate if we move on to gnu6c4.1 it might change that but it has to be seen once we are done with gnu6c4.1 my understanding based on conversations with several of the m68k porters including Adam Conrad and Ingo Jürgensmann is that at the time we were reviewing that they were still in the process of bringing some faster m68k build these online which would allow them to keep up better and reduce the number of build these required I have not reviewed since then how they are actually keeping up with the package queue I know their current and built packages is low but I know there are a number of compiler issues that are blocking them there as well basically gnu6c4.0 is not good for m68k about the m68k is for build packages that do build that let me go ahead and repeat that since that microphone was cutting out on her that was saying m68k is keeping up now with those packages which m68k is able to build it's just a matter of the ones that can't build due to architecture specific tool chain issues next one last one just a small point that the same thing applies to the arm consideration but I don't know if you can see it but you have to decide something has improved on the slides but in any case you'll notice that anything is wrong on this page the easiest way for us to fix this is if you send mail to another lease because then it's definitely in a prominent place in my incoming mailbox and I will process it we seem to have about 5 minutes left here before the end of the discussion period so anybody has anything else Lars apparently does yeah I would like to know what font you want me to do what did you say he asked what font he wants us to use on the tattoo I think that is our decision we need to make in a capitalistic manner do you want a GR about that should we ask the tech maintainers for advice okay I can see another question oh Frank the project secretary wishes me to announce that there is a moratorium on GRs until SE Linux is completed in Debian I see the DPL is not happy about that so maybe there are some in planning is there a point in the release schedule already for choosing the new release name for the next release actually we wanted to do it yesterday evening but unfortunately I was sick it should have been on the last slide would you pardon I have one name in my mind but we didn't like the unit yet I didn't know that character is also in Toy Story yeah for those who don't know the two remaining characters in Toy Story 1 the movie which are toys and are named on screen are Lenny the binoculars and the remote control car whose name I believe is RC Buggy so we really think we're not going to use that one for a release name yeah has everybody seen the new Debian Buggy release okay what do you consider trying to block acceptance of packages that are involved in transitions rather than just relying on maintainers as I told basically I think it would be a bad idea to actually enforce it via the FTP team's tools because actually at least my expectation of developers is that they all all know how to do it and should be able to listen and if you say well, we can't just this maintainer not upload during the transition time or we say well, he says some parts we really should consider if it's not a better option, that is to remove upload slides at all sure, but no one's doing that yet either right, it's also we've discussed this before and it's been my impression AJ just shook his head at me like he was surprised we discussed this before yes we discussed before the idea of doing something like this people have suggested in the past it really seems to me that it's sort of a wash if we have to go through and coordinate with the FTP masters to have people blocked from uploading particular packages only to have that, only to turn around and unblock them at the end of a transition as opposed to just dealing with whatever issues happen to come up which relative to the number of transitions we do is actually fairly small and also we had one or two times with really bad uploads and usually people learned if we use a name and then we'll announce next time I think there was another question at least the microphone is wondering to someone otherwise I think we are there yet at least for our talk now you're going to have some ice cream