 Please we're ready to begin our last talk for today is about Debian Release Management by Andreas Bart Okay, so I Will Doesn't work the microphone. No good So I will just start now for the first slide side for the first part I don't really need the slide urgently because actually everybody of you should know what I'm talking about At least if you manage to go through new maintainer, but well Still it's important that we all have shares the same information in Debian. We have more or less We have we have we have more or less different distributions Usual packages are uploaded directly to unstable from unstable They get after some rough test testing go into testing and from time to time We release this testing to stable the former stable becomes all stable And there's something else also that's called experimental The packages can be uploaded that shouldn't go to testing and shouldn't do harm and unstable So that's so one of the basic things What we do about um testing, okay, you need full screen and you need yes, it's the right slides It's a yeah, yeah, so okay now be just if okay. There's a comment line option for it. I learned on the other thing So Yeah, like this one. Um, so It was xpdf. I think with a full screen Okay, thank you very much for having the side of another laptop Full screen And it's mounted to where? Oh, great Okay, so now there are also slides at least on my laptop. You can't see them But that can be fixed. I think that's one of the very typical things with the lease. We are ready The others are not so but we have learned to cope with that Yeah, well is it something with the laptop? I'm just Function fff8 usually does it F5 oh, yeah Oh, yeah, I should I don't mind if I don't see anything here, but the people should see something Okay, so Just let me take a look at my at my next slides so that I can have some ideas This is just the function so that I can see okay, so okay So what's the target of this continue? What's the target of the lease management? The lease management means we must make sure that actual testing meets the lease targets We must make sure that anything that happens with testing doesn't break testing and that if tasting is broken. We unbreak it We have to keep those things together this testing And one of our most important tasks Actually is to collect and distribute information from from one team to the next team So, okay now it's starting to get to the parent of time. Well, I would like to see you able to see my slides So Okay, so I will tell you something about who is the release team who does the release that are five persons to release managers Steve uh and and Colin and three assistants And there are three release assistants Oh great Thank you very much for it. So now we can continue with this nice slide Three release assistants that the joy has who mainly focus on the deputy installer. You all know him That's Frank and that and that's myself Oh, there's something else. What did happen now? I don't touch it with somebody else I guess it's something with a I see this control mode just changes this array output. Something is happening there. Um, at least we have So the lease team has Okay, we'll just continue never mind. So the leases team has a common role account That's definitely released at least there being orc. Everybody could subscribe to it who is interested in it But please keep in mind. This is not a discussion list And I say I have multiple related places In the basic reason for it is that everybody if in the release team is supposed to read every major center of this list And if there's a discussion that's that's can get quite ugly for us. So please don't do it And there's a website for us released. They've been orc It's not as full blown as we hope it but there's still a lot of good information on it. And I hope that We will improve that website. So if you think there's anything missing This just ping up one of us on irc or send us mail or whatever And we'll add it to the website So what are our tools For testing migration, we have a famous script called britney Which is actually not a script but a combination of shell scripts pyson And some c back ends. I learned a lot of pyson, but okay, and It's an important tool for interacting with britney for us as the release team us is so-called hints With the hints, we can add packages. We can The move packages we can say, okay the package might be to young blood But please bring it in anyway. So that is the gateway to britney as a hint We have released good little bucks for most of you they see you consider them just say block release But actually they're very very very valuable tool for us because we know where we have to take Action and they have a most important management tool and for that reason we have two different websites With telling us how the seabuck status looks a bit different because the different looks have different advantages We have also now a page of security status Which is maintained by joy health as part of securing the testing distribution He will give another talk about it, but that is also one of the very valuable tools for us and beside these Major tools a lot of small script a tool that we use medicine Grip deconsole and a lot of all other small scripts that are very helpful for us to get the release done So how packages go into testing one thing I really want to emphasize for all of you is We prefer if it goes via unstable Even if you're in a deep freeze the basic reason for that is That in unstable the package gets some rough testing if it's got up to testing or post updates, which is also possible Well more or less nobody tests it and for that reason the patches needs to be very very very minimal And so even case we said oh it's a buck, but we won't fix this via testing or post updates Sorry because it's too risky for us The testing script britney is run shortly after the install the install is a script that most and more or less Just takes the packages that are in incoming. They'll be an org installs them into the pool and pushes the mirrors And shortly after set we run britney makes a testing transition, but you can see the results only after the next day We however directly publish the output and excuses of the britney run. So If somebody asks a question this the output says it's going in but it's not in well It is in you will see it on the next day. Don't ask us in the same day about this It's more or less the most frequently asked question I ever heard And what we can done But we don't really like is we can see run britney during the day if there's some urgent issue For times like now we just say oh wait if you run another time next tomorrow But if it's just something like oh half a day before release we do it So Now the question is what worked well, I think that's a very important question for all of us What worked very well is the devian installer It's from the manageability point of view much better than it was before At least as much as I heard because I've never done anything with poolploppers Except installing one potato system upgraded to hoodie But other people who have more experience that is much better We have and that's I can say from my own point of view also very decent management tools That we use during the freeze. We have very granular hints For example in previous times was just possible to say okay. This package is not ready, but this package goes into testing anyways That's of course is nice, but it's not all because You can by accident break some other packages now. I can say okay. This package is ready, but please ignore that It's so young It still checks that it doesn't break other packages that it said it has no sc bucks. So that's much better than it was before And just another small script called D and I can just say D space package name And it tells me what's the difference between unstable in a testing And also if I if I finish reviewing the differences, it just gives me the outer light hint to put in my hints file So that was very nice tool for coping with a few hundred unfreezing the request per day So also we have very decent bucks questioning tools the view of the least good bucks improved very much We have the sergeant nor now edge ignore hints of Text to the bug report which we can use to say, okay This is we don't consider that's the least good right now without having to play funny games with the With the buck status That's also something that really helps us because it avoids all these things avoids that we make mistakes Because we don't have we have to check less things and the rest is done by the computer and computers do only mistakes when You tell them something wrong But also worked very very well is the partial freeze we started with freezing some packages and intended it to more and more and more packages And in the end the full freeze was only a very short period of time and still the release is very stable So that was a great help for all of us and I think also a great help for the package maintenance because it's easier if they don't have to go an extra Around around the management as a release team to say oh, please push the package in And frankly speaking this is a large number that sarge has and that edge really also has probably It's almost impossible to have a longer full freeze than this very very limited time We had what also worked at least in our opinion quite well But in the end that's your call to make not ours There's a regular release updates via depth in development to keep all of you in the loop That's also of course helpful for the release team because we have to make a lot of difficult decisions And if you are in the loop and know why we do it, it's easier to just get them done. So it's I think good for all of us And Yeah, something else that we had to do but it worked out fairly well was this time planning where I said, okay We have some late n in the future where we don't know when it is But depending on the state n we will do some actions and that worked out Very well in the end a better as then we had hoped Also a very nice thing for us was that we had experimental for example the whole good gnome to a transition Was done experimental worked and then done an unstable And I've never ever seen such a smooth transition of such a large number of packages into from unstable to testing Um, I think the transition was complete something like 12 days after the first upload And if you remember that it's 10 days to minimum wait time. Well, there's nothing else can be said about it anymore It's really worked very very well Another thing that we had very long was an nmu policy with zero day nmu's allowed Actually, I think that helped keep uh debian as a whole in a very good shape and to say Oh, well packages are first of course a responsibility of the maintainer, but in the end Packages are the possibility of the whole package because a package is the percentage how the packages were good or bad And I think we should continue something like that And in the end all the management that are at least at large was very very successful Of course, there were a lot of glitches that I don't like and then we come to those glitches very soon Well, one of the things that Is that really could have been a bit better was Survey survey said some Only a very few maintenance work with us, but it's really important to remember Don't second guess how of these guidelines are we publish them and debian development now and so we do it for a very good reason Don't just stop on some on a random. I see trying to say can upload a breaking Upload to unstable right now and break a transition with it, but just happened in the past not only once i'm afraid Also, please don't upload a package So they say oh, yes, that's a new major of observation of the library That's not intended for sarge and then you upload it to unstable. That's not good So if you upload anything that's not meant for the next stable release upload it to experimental That's the place that meant to be not too unstable And please don't upload a package if you are right in the middle of a transition right now With the c++ or we were with tiff and if the least says please don't upload then please really mean it don't break the transition with it And something else Actually, everybody should read the well step in develop now and but if you upload a package for a lot of packages Depend on please make sure they really did it at least Especially if the message is more than one week old and we ask please don't do it And there's a lot of other small points, but I think the summary of all of them is if you're a good maintainer and just Doing what you should do communicate with other maintenance in a good way And just speak with the release team so that everything works well And if there's anything you can always ping any of us on irc on a mailing list, that's always okay So just do it if you're unsure and ask us and we will give you the right answers But of course there were also other things which were more on the management point of view which could have worked a bit better The first points that really hit us very bad versus a large number of Unfluous requests we had in the final stage. It was really impossible impossible to read the release at such a point of time And I just said okay at that time some days I just did that been that been released and nothing else anymore Usually read a large number of mailing lists because it was just too many requests to cope with We need to get better at that point definitely Also the base freeze was a bit long. We started with the base freeze in august last year At that time point of time we expected a little bit of help release about december And it was very sensible to make the base freeze because it was needed to get the installer Shaped up finally so it was not a long decision at that point of time But if we would have known what we know now, we probably would have done a bit later Also, there were perhaps a few packages too much in the base freeze I don't really see any reason why we need to base freeze mud for example Just to name one of the examples and we already discussed it. We will improve at that point also one of the questions is Which development packages should be frozen a bit earlier because it's we don't really consider it as fun if The cdbs maintainer makes an upload five days before the final freeze and breaks such a lot of packages building and such Yeah, that's not really funny. So you also probably feel this package Other packages like that paper were not frozen, but the maintainer was truthful enough to not do it Which made the reason because it's part of the of the release team but well their mind So what we also definitely want to have is a version second back step in org so that we can say, okay Which a cbox are actually open inside not to say, okay, which a cbox were either open any distribution Or are uploaded into the maintainer reopened it Actually the code for that is there and I really hope that show that calling managed to Commit it or to make a productive very soon And that will also help us with the very first point because now we had to say oh if you upload an a cbox fix You must send us mail or we won't get it into start for edge We can say if you upload an a cbox fix never mind We will find out by ourselves and that is very much easier for all involved partners to maintain and so the release team Also, we want perhaps a little a little more padding at the end so that it can happen We release too early because it's not really fun to write a mail and say oh actually want to release Day after tomorrow, but we will need another week But of course it's better than to say nothing so we send that mail but Yeah, we should do better Also, we had a really a madness on monday with the deal downloads where everybody tried to find out Where's a mirror so didn't so didn't prevent downloads of the cd images This half hour before we actually released it and that was really awful because the CDs Didn't really install because the mirrors were not pushed You need to do better than that and we definitely need to do better with a final quay of the release and the CDs And one of the things would be so well this time we just said we have the release more or less ready But didn't change the sim link then we Builds these from the spray release status One idea of that would be to say we make everything including changing the sim link on ftp master Builds the CDs and publish afterwards But that would mean for all of you that there will be no deal install about for one week Or so I think we can cope with it and it would be better than have another cd HLO a release but some people probably won't like it and we will have to discuss about that That's the least point of view. It's good from other points of view. We will see Also, we should use more of the code names when we speak about the release Because we could for example have committed to the installer two months ago that just Any case when it installs it just installs security with such updates So we wouldn't have never had such a problem. So we should have perhaps done it and also at other parts Yeah, but we also need we need a bit more console over how britney works because there are things that still don't work For example, if somebody says please update my script of my package. So well, it's fine But I want to say that it waits 20 days and unstable before it goes in So currently I really can't do it. There's only one way to do it So just write in my calendar do it as a hint on this day when I do it We should have a hint to say, okay, this package is okay, but the waiting time is 20 days now Just as an one example Something else What should definitely work a bit better than this time is the security our chef after the release I think one week is okay one month not and I'm better sure that we all are clear on that And so we really work on that. So that's not only the least possibility But as I said before one of the major tasks of the least team is just to Keep those things together speak with everyone involved and it's relevant for the release. So we have to speak with everyone involved Um, there are more things you want to do better than edge It's a very outstanding issues. We really want to make a time-based release. We want to release in december 2006 That's a goal that that we set for ourselves And we already started and making changes Allow us to make some meets the goal One of the most important changes is we don't we have don't have a large list of things that we said Oh that all are release targets. No, we said release targets are as few as possible Of course, we need a new toolchain xorg Well, doc's a main firmware main I don't want to say what's my opinion on it, but we didn't have a choice because the developers voted There were definitely no opinion on it So we have to trust they have to do it Pot stable and orcs. It is a that is the split of the of the mirror in two parts Is in my opinion very very important or well for our understanding is that the ftp must have support to get amt64 in And for us amt64 is literally a release blocker It was very bad. So the release starts us out and it's not possible to release edges out And something else you want to have a secure app But that's almost well uploaded to unstable and I hope the remaining issues are sorted out At least i'm very optimistic that we will Meet these things in time There's a bit larger list of things that we would like to see That so we called it pack release goals That doesn't mean they're unimportant. It just means We are probably it's a moment not holding our brief for them But if it happens that just one or two packages don't comply with them because they'll say two months before release Okay now say a blocker's because We're not holding this target up for one or two late packages Something else that was really very bad for us We have a lot of dependencies between package or the namings especially libraries Transitions if it comes with a built in it was really very bad for us So we must improve on that definitely And currently all packages have to wait to go to testing did all packages are ready and that's Just can be very ugly especially if some nice things come together like well something like tiff lip tiff kda is then related to this gnome and with some other packages It's really ugly and we decided to just keep the old libraries a bit longer in testing Because in this case we can just let the packages just throw in by themselves We don't need to take care of them at all anymore And ensign is actually started to work on the code for this Also something that needs to happen is just more quality assurance and that's a part that's really everybody can join So if you want to work on quality assurance, you're very invited and this is the most important thing because quality assurance Quality assurance qa So if you do it because that makes true in times that everything works And for example, one of the things you want is that every package is automatically tested if it really installs So that if installs and uninstalls that means as a as a as a scripts, okay That's all that they've been worked on But there are a lot of more things one can do and if one has a good idea Please feel free to work on and we'll definitely support it Also one thing we must make sure is we want to have less pain with some specific art Let me just tell you what i mean with that And for example, what has happened this time we had one architecture That from one day to the next all built email just dropped off the net Which is very very bad And then we started to hint at the machines and then I was informed that one machine was somewhere in europe Near a certain city and then I got on irc of Of the irc channel of that regional dibbian Language channel and said okay any of you living near that city. Yes you okay Please go to your car drive to that place pick upset box and install it somewhere with network access or that ammo can Install a build demon on it And frankly speaking that is not the task of the release team I don't want to say that I want less architectures I'm happy with this number of extractions we have and even with more if you have so yeah 50 minutes is that okay That's my last slide already so But so but it can't be the task of the release team to keep up all the architectures If supporters do a good job I'm very happy with it because I like to have more architectures than we have But if they were not so good job, perhaps we just have to say one day. Okay To two possibilities either the developers managed to get for example a second spark built demon up Or we need to add it to the architectures. We don't really care about anymore That's nothing. It's not meant bad But it's it's also even better than it was now because in previous times There was not really a discussion how it's changed and now we really want to say if it's bad We will tell in time so that it can be fixed Yeah spark is actually one of the architectures really very bad about currently So then I think the general impression which architectures are bad and so on which really hurt us are different If you don't mind too much Well, if it's a a question how a man something please say but otherwise I will throw questions after the talk and so there's just four more points on this slide. So okay So and please raise your hand so I can get up to you with the microphone Yeah, well, I think if there are many folks we just Start at this point and go on the other side. So we don't have too long ways So something else we want is a better interaction between the kernel and the installer actually the action was Much better than it was with bootflop is but there's something that can be improved because currently if we do a security update for the For the kernel We need more than I think currently three to four rounds of rebuilding packages and vaginal Still some are installed and the building again because there's a lot of many dependencies And actually what I really would like to have we have one source package You you change the code upload it and you get Ready packages for all architectures just falling out of the build demons because that's why we have build demons And also we need a bit better u-deb handling in britney because currently all packages that produce u-debs That are the packages for the installer are frozen for a similar for one simple technical reason That if we update such a package one one of us needs to to brought an ftp master say please update this package in the u-deb file also Because well should instead what should work automatically because that's not too hard Also we decided to make a checklist with dependencies to say okay If we go into full full freeze for example We need to make sure that we start now and working on the release nodes Because the release nodes are just necessary to be finished when the full freeze is finished And we don't want to have a full freeze much more than five weeks. And so the release really needs to kick off now Also, you want an absolute package for the release nodes right now Because if any manager maintainer makes a change in one of his packages that should be part of the release nodes You should be able to just file a bucket port right now and forget about them And we will be picked up again when the release nodes are prepared. We learned a lot about that A question of understanding what we can avoid we have one more point So then as a last thing we want to communicate better with all developers with other maintainers with other teams And for that reason we want to improve the information on the the least step in our website a bit For example, we won't have a list of current issues of current policies Where for example now it should list please don't upload any package Which is cheap as plus unless you speak with doko before for the So we don't like the transition that would help I think maintainers a lot But we are not at that point yet But to the general question that we were asked quite often are we there yet? Yes, we are with the average there yet We manage it quite well and we will be with with edge in time Okay, so thank you for your attention and now I expect some questions. So who all wants to ask a question now? Okay, I see a lot of questions. Yeah, I will try we can just Then has has already had the keynote question yesterday my other talk. So give it some someone else And I have another and one thing I really would like you as I guess I expect a lot of questions Please don't make co-statements just ask me your question And if there are more detailed questions, we can just continue to talk about them after the talk So that everyone has a fair chance to ask me a question. Thank you all Okay Well assumed can can you yeah assumed the release team would have a really nice dinner and the The food is contaminated by some salmonella and all the people of release team would be Sent to the hospital for fear four weeks Is there any documentation it would help to replace the release team completely? Well, actually I would say I can remember a car drive where all fdp masters was me sitting in the same car when cover No, you know, you don't know carcass in north america Well, actually there is a lot of documentation of course, but on the other hand Well, if Well, we have a lot of people who are not in the release team who know a lot how it works For example, france really helped us a lot and knows a lot how it works The same is true for sobel and for a lot of other people who are just around and help the release team So the best is the worst thing that can happen is some food contamination here, but thanks god the two release managers are not here Well, but a lot of things are documented on the other hand side a lot of things are You learn a lot of things if you do it and one thing is Nobody with good management skills grows on the trees We need actually we need more people who can do good management in debian And I think the release team is one of the teams that works better But of course there's a lot of things that can be improved And well, I think that's a major major answer on it on the other hand Please reconsider at the time where the last step come for the scheduled I was not even a developer. So that was the reason I was not there So it happens quite fast. You can say you can come in. Yes Next question, please It's a different topic as you probably know i'm peter and also from the school linux project We have been using Are still using woody for the schools in norway and one of the issues we have Been able to have been forced to face is the lack of hardware support in the stable release This is a major pain because Sarge will slowly be harder and harder to install on modern hardware. Do you have any ideas on how to address the Stable release hardware support Well, okay, if you just ask for ideas, that's very easy. I have a lot of ideas. Thank you. Thank god. You didn't ask for solutions So, okay, there are two a lot of different things So the one thing is a stable the decision means it is stable. There's a good reason for it And for example, if you say we are a support a new kernel Then we would have to we we need to do secure support also for the kernel There are good reasons to support a new kernel But my information at least is that this would would mean a rebuild of the installer And you can get a lot of fun with it on the other hand side. I don't think it's too bad to say after let's say Half of the time we expect this Release of edge. We add a new kernel set. I'm even coincided Vaguar is a vaguar is a coinciding to do that perhaps via a volatile But please don't side with that because it's not yeah, it's just I'm coincided if it's too to do and I think there's one thing that can be worse than No new kernels that is if we do it bad So but yes, that's as other ideas to go But for example, if you add new kernels you need to add a new you default for example So you need to place some packages around because the current kernel model is that you don't have a stable kernel But they're just well, I would say developments they have shots that you can take or leave whatever you prefer so Yeah Not too nice, but that's a lot way it is. Okay. So the next question Any more questions? So I think now we go over to overfand I had a couple of questions. The first one was You mentioned a pseudo package for the release notes. I'm wondering why don't we just make them a real package? Um, have them, you know, they wouldn't be base necessarily But the bootstrap could install them that way the users would would always have them But that's less important than just you know, unless you really need to be updating the package like Just answer Okay, the answer is please ask fans pop fans fans pop said to us He can size it better that way and so this team just lost the opinion of other people as always I'll repeat Even after the lease Okay Okay, I will I will say Okay, so the answer is I will repeat now. Oh, I was going to do that. Well, thank you. I'm giving the talk. Thank you very much Sorry So fans answer which which of course was the right one is um said The results are updated very late. So I uploaded Even after the archive was frozen because this time we frozen the archive a Saturday around I would say two o'clock you to see so the lease happened on monday Sometimes and the lease notes really were updated a lot of during the week And I can remember at least three calls or something later So if I remember correct, so you see and it would be bad if we have an installed package that has outdated information So that's one of the main of the basic issues other other packages are more Yeah, more time at developers likes the developers of friends are in there so okay, my other question was um One of the things I'd been kicking around was the idea to make uh didn't put in volatile I know you you haven't approved this was I wanted to make a stripped down version of xorg that just builds the xorg server to Put it in volatile because my ass got so burnt over all the complaining That we weren't moving from x86 4.3 even with patches to the drivers to support more hardware Um, and you know, therefore debion was you know fundamentally inferior to a bun two and we all just suck And I know I had the support of the release managers and staying with x for 86 and none of us expected the freeze to go as long Um, if we don't use volatile for this, what do you recommend because it's something I'd like to do I think it's something our users would appreciate um And if somebody's willing to do the work Can we maybe do it in volatile? Um, because it doesn't change external interfaces the way the kernel does Because the kernel guys screw with stuff and break all the all the modules Well, okay, so that would be of course an easy answer because I'm just currently just having my release team head on And volatile is an official service, but I don't think that would be a fair answer So I will now just switch on my or to take on my volatile team head and say, okay Actually, yes, I think it's possible to put it into volatile. We have a volatile two sections One is the section. We say, okay There are only very easy ones like currently glama for glam of data and the other section have a Okay, we can update we can publish packages there, but they're not installed by default So we make extra hint for up to say, okay, not selecting by default Only if the admin selects it you think for that section would be okay But we should perhaps talk about it not Right during the talk about details on that. Okay. Okay. Thank you So Jesus is the next one. I think uh, and then that's one. Yeah, okay Jesus So, uh, the question is is the job of the release team to drop an architecture I mean if the if the job of the release team is not to make phone calls to Place in Europe to to get one box Move it to 100 kilometers and reinstall Why not drop their architecture make the release not have to wait two months And then do a point release one month later when they're But it's okay now a set of different questions. I will cite our different answers So the first one is in my opinion is the task of the release team to make sure So that the release are as good as possible and of course different people have different markers for as good Uh, so some people with the marker to say, okay, um, we want a very Very good predictable as Peter said in his talk about hp for example That are very important thing also in our opinion because not only for companies unpredictable But it's also for our private life unpredictable So well, I will take this weekend off and this and this and this and this that has passed too much So we all need predictable releases. You want all to have as much good architectures as in as possible You want to have as few bucks as possible as fairly working as possible everything else And so in the end the lease team has to set priorities And to say, okay, we have now conflicting priorities and we decide which one wins a bit from losses And I really I really hope that it won't be as time where when I start texture loses But but but but let me just finish that but if that happens that would say there isn't architecture That can't keep up or that breaks us too much. We might say, okay This architecture will stay in testing, but we won't we won't Yeah, we will just ignore it for any purpose of testing migration So that well if it works it works if it doesn't work it doesn't work And if it stays too long in that state Then it will just from the some states and one day perhaps after two months or whatever We'll say, okay, now it's really broken. Nobody picked the task up to the pad. You really kick it out For example, that has even happened in the past facade, but just nobody noticed Because we drop say the support for 80c 86 processors before single reasons New lip see was uploaded with a patch such a device e 486 instructions That's necessary for speed and for compatibility with other distributions So there was no discussion about whether to handle the peck patch and then there was a kernel patch That worked to emulate these but this kernel patch is unfortunately a root hole So we said, okay, there are two ways. I see somebody fixes it or we drop the support And we said that on Debian develop announce I think for three times and nothing happened and we just dropped the support now It still works, but it's not officially supported anymore and that might happen with other architectures And about adding a package of support release. There's something very bad on that because I will tell you why because a great advantage is that Of Debian is with the different architectures. I can say, okay I have to have some broken of some certain version that works on this laptop very well Besides then I go to my arm box and works the same very same version works well on my arm box And if you say you edit it later that probably means there are some problems with the architecture like the toolchain was not ready or so But then you have different packages. So they behave different and we lost much of that I don't I don't say it's not possible to make a snapshot of another architecture But just believe that it's easy possible to add an To add in full architectures a point release is at least problematic. I would say at least problematic and but in the end that's even not so It's a call of the team that of the release team that I'm part We have actually in the release team two different targets. One is the stable release. That's a Martin Schultz. The other is a testing release team where I'm part of And in the end it's a call of joy and if all of you who know him We know that he will say in a much more definite way that this won't happen So I understand the reasons and I really think it's it's not so good to say Oh, we added it sometimes later, but if there's an issue we need to fix it and we need to fix it soon And for example, how should it work if there if you don't have a really a Porter team who keeps care of an architecture and we notice later on that there's a a root hole is a kernel Specific part of that architecture who should fix it if there's no porter team. So if it doesn't work It's very sad to say it but sometimes one of the release team manager has to say there's things that doesn't work Okay, next question please. Yes I might be a bit slow, but I didn't quite get the Explanation why you couldn't update the packet in the release just before you released it The release notes is a packet. I knew the release team can't update the packet in the release. What's happening? Okay, I can I can tell you how the leisurely worked in time with time slots about on This time was Saturday about I guess The 40 you to see you said, okay, you have two packages open Please push them to fdp master right now because the install will start after you did it I de-uploaded them and in And about 10 seconds after set an extra d install run was started to install the packages that we needed an extra Brittany and hold how the britney run was done and after set britney was put off the ground tab so that no more updates Are testing were done So usually the install was done about half an hour too late and then there were some rough checks And then the wallon said, okay city his teams start building your cities right now and from that moment on So the release team can't do anything except say it's so ugly. We un-solid and do the release one month later and we didn't do that That's all There's one point in history where we just say okay go and if I say we said go we can't do it back anymore Is that as the lease notes were updated after this point with the least three important updates That were called to the newsletter so please update the translation because there were Very stupid errors in it Yeah, that's it works And perhaps it might work a bit better if next time we really start This is pseudo package in time so that we have the updates in time But I really doubt it because a lot of issues just came up in the last second It's for example that a very nice debug on sent mail So that if you upgrade this upgraded sent mail with messages still being in the queue The message in the queue will ignore it from sent mail at that point on So first is a per-script to circumvent it We'd notice that I would say Friday evening that there was no chance to re-upload sent mail with a fix for it So we just say we documented the release notes and we had to fix it because it can't be that we say oh Yeah, you updated sent mail you lost just some well 2000 messages. Bad luck Okay, I think we'll have to end And there at least inside again, you can feel free to continue the conversations outside I just let me say some one final thing about that I will now just go back to the to the dorm and whoever wants to speak with me Of course is invited to do it now or at any time later I'm really happy even to discuss about some more technical or more deep into issues that are not possible to discuss here With so many people and thank you all for coming to this talk. Thank you All right, so we need to clear out pretty quickly If I can just make the announcement one more time There's a bag out there for your plastic bottles and cans only You know recyclable ones and we can use the money we get from that to build up our patent portfolio