 Je suis Lucas Nussbaum et je vais parler de la qualité d'assurance. Je dois vous pardonner. Comme vous l'avez vu, le décompte est grand, car vous pouvez découvrir beaucoup de variants anglais. Nous avons trouvé le jarglish, l'italish, le spaglish. Le spaglish est demain, je pense. Et aujourd'hui, nous avons trouvé le franglish, qui est le meilleur de tous. Bonne chance d'entendre moi. Ok, à la fin du cycle d'assurance, nous avons fait beaucoup de QA, nous avons rébuildé les archives plusieurs fois, et nous avons joué plusieurs parties sur le monde des archives plusieurs fois. Nous avons réalisé environ 200 RC bugs fixés en H. Ceux-ci sont, je pense, une bonne chose, parce qu'ils permettent de donner le même niveau d'attention à l'ensemble du package d'Adebian. Ce n'est pas seulement de relier aux humains pour trouver des bugs, ils permettent aussi de l'évolution. Peut-être que le plus important c'est de garder les maintenance busses, pour qu'ils sachent qu'on va relier les H et qu'ils ont des bugs fixés. Mais, comme le résumé m'a dit, ces tests sont trop tard dans le processus, il y a eu quelques packages pour mettre les H. Beaucoup de packages sont RC bugs contre eux, qui n'étaient pas fixés et qui n'étaient pas fixés. Et beaucoup de choses n'étaient pas testées. Donc, nous devons être plus efficaces dans l'organisation de l'organisation et de l'élection du cycle d'assurance, pour éviter tout cela. En première partie, je vais décrire deux types de tests, pour ceux qui ont attendu mon premier talk, mais ça serait plus intéressant. Il y a plusieurs types de tests que vous pouvez faire. Le premier exemple c'est de rébuilder les packages d'assurance. Le second exemple c'est de faire quelques parts. Il y a aussi d'autres tests, comme l'Intia et l'Inda, mais je ne les connaissais pas très bien, donc je ne vais pas parler de elles. Le premier, de rébuilder les packages. Le problème c'est que quand vous appliquez le package, il n'est jamais construit. Les packages avec l'architecture all dans les contrôles contrôles sont seulement construits sur la machine de développeur. Donc, après le package, c'est apprécié à Debian, le changement de build, il y a de nouvelles versions de compiler. Le build de build, c'est le cas quand le package migrate à la testation et la testation de GCC, c'est le build de l'inventable, ce qui peut arriver. La build dépendance ne peut pas être disponible. Ils ne sont pas utilisés par Britney pour la testation de propagation. Donc, ce n'est pas un problème, parce que tout le monde devrait pouvoir construire votre package. C'est important, pour exemple, pour les updates de sécurité. Il y a plusieurs outils pour rébuilder les packages. Le premier est le build de build. Si vous utilisez le build de build, si vous n'avez pas... OK. Donc, le build de build, c'est l'architecture à l'architecture. C'est très facile à utiliser. Si vous n'avez pas l'architecture, vous n'avez pas l'architecture. Et il y a aussi le talk sur Saturday afternoon, je pense. Ensuite, il y a sBuild. SBuild est le nom de l'architecture de build. Il y a aussi un package d'Odivian, qui est disponible sur ce code base. Il s'agit d'architecture, donc c'est un peu plus difficile à installer, mais c'est beaucoup plus puissant. Donc, un autre test de QA c'est Q-parts. Vous n'aurez pas Q-parts. OK. Q-parts c'est un outil pour tester l'installation de packages. Il installe le package dans un système d'Odivian avec rien. Il y a juste un app et un déposter installé dans le chute quand il installe le package. Il s'agit d'aller vérifier. Il s'agit d'aller vérifier le script de maintenance par exemple. Il s'agit d'aller vérifier tout ce que vous utilisez en post-installation. Il y a aussi un couple de parts qui peuvent tester d'autres choses, comme les upgrades. Ils peuvent tester les processus de l'installation de packages, et d'autres trucs. Il y a aussi un couple de problèmes que j'ai réalisé pendant ces tests. Le premier est que il y a 10 jours si vous utilisez un single computer. Les packages sont très vite à construire par exemple, 90% des packages sont construits en moins de 100 secondes. Mais il y a des packages qui ont pris un très long temps. Il n'y a pas beaucoup de surprise. Le second one génère des fonds et c'est pourquoi c'est trop long. Quand j'ai re-buildé l'archive, je l'ai fait dans un C-grid. J'ai eu un problème. J'aurais pu utiliser 100 notes si j'avais besoin. Mais je ne peux pas, parce que l'archive a pris trop longtemps. C'est plus ou moins d'utiliser plus de 40 notes parce que le temps sera boundé par le temps que j'ai pris pour construire l'archive. Et c'est même le worst de l'archive, parce que la nouvelle version de l'archive n'est pas la même chose. Donc, maintenant, nous avons des multicards. Nous avons des laptops d'archives parce que les billets n'ont qu'une CPU. Donc c'est sympa d'avoir une interface d'interface pour le package, pour les règles de Debian que ça devrait utiliser plusieurs CPUs c'était très longtemps et basicalement si vous readz le blog, tout le monde s'agirait qu'il y ait quelque chose comme ça. Il y a deux proposations et c'est un peu comme le problème de l'archive. Tout le monde s'agirait pour l'un ou l'autre, mais personne ne le décide. Donc, c'est sympa si les policiers le réveillent et décident quelque chose. Donc, nous pouvons juste impliquer qu'il y ait quelque chose comme ça. Et pour exemple, il y a des temps de build de Linux 2.6, nous avons déjà offert une interface d'interface avec plusieurs CPUs. Sur le système quad-core on voit que le speed-up est très bon. Vous allez de 5 heures à 1 heure et demi. C'est vraiment ça. Et ce n'est pas pour tous les packages, mais c'est pour ça, qu'il y ait quelque chose qui va bénéficier de ça. Donc, un autre problème est plus de parts et plus de positives. Donc, quand vous arrêtez plus de parts dans l'archive, vous avez des temps de plus de positives. Most of them are you can't do anything about that because I just there are still some packages that install non interactively that don't allow to be installed non interactively. That is, I prompt the user using read, for example, in bash. So, deepconf is nice because when a package use deepconf you can use a non interactive frontend. But it doesn't serve everything. For example, packages that need access to database, that's what I just said. So, there's a policy bug against starting with 2 that would like to make all packages use deepconf except the packages marked essential because those ones cannot depend on deepconf. And this bug log has been acquired for about 1 or 2 years, I think. So, that would be a nice release goal for landing, maybe. So, a few parts. There's a lot of things to do about these types of pure parts. First, it's supposed to be maintained collaboratively, but there there have not been a lot of activity on that. There's also a donated system called Piatheeddebian.org which could be used to run tests. So, if you are interested in that, just contact those who maintain pure parts and they can give you access and you can help develop pure parts. Because for now, using pure parts, I only file bug about problem during installation and removals. I haven't tested upgrades, I haven't tested files left after a removal. There are lots of things to do with pure parts. So, I have a question for you. Is John maintainer or the web calendar maintainer here? Okay, that's good. Does somebody know what they have in common? No idea? Can I help you? They were both in charge and they are currently unstable. Both are useful. I use both of them. No idea? Well, none of them are in edge, actually. Both of them are at an RC bug that was fixed after the release before the release. But the maintainer forgot to ask for the bug. So, they are left behind, not in edge. Many packages actually missed the release. After just before edge was released, we looked at packages that were unstable at the time of the freeze in December, maybe 10 days before because of the testing propagation delay. And we looked at those that were not included in edge. That is, all those who weren't able to edge. So, that's 433 packages. In many cases, the package is fine. It could be in edge. But, for some reasons, the maintainers forgot to request non-blocks. So, it couldn't get from unstable to testing. So, there are lots of example bugs for that. So, I think that we really need a way to keep maintainers informed about the shape of the packages. So, that's DDPO, of course, but which is really nice and very useful, but only if you use it. And who regularly read his DDPO page ? Who doesn't ? Ok. Hum? Well, one of them, one of you did. So, ideally everybody would use it as start page, but some maintainers don't understand why. I understand. So, the idea I'd like to propose today is to send one monthly email to each maintainer with the most important information about this package. The goal is to avoid spamming the maintainer. My idea is to only send a mail when there's a problem and when the problem is an important one. Like, open RC bugs, packages that are not interesting. And maybe, but even that I'm not so sure about it, important bugs file which attack patch. So, this would be, of course, opt-out, but it can't be opt-in because maintainer wouldn't subscribe to it, so it would be useless if maintainers. Oh, just another poll. Who subscribes to the BTS for this package? Oh, quite a lot. Ok. So, there would be an in-your-mechanism, so you could say, ok, this bug is a dummy bug just to prevent testing propagation. I don't want to hear about that. So, I've already done some work on that. And basically, I imported the BTS metadata to POSGRASDB but I used the bug scan as a base and the bugs I have to fix first not in bug scan in my code. So, I finally used BTS tomzimmer.net as a source for the list of RC bugs. And I computed the testing status for all packages that is, for how long a package have been in have not been testing because the numbers on the BTS are wrong actually. So, I'm ready to send mails but I'm still not sure about that should I do that. Ok, so, QA tasks used to be done by motivated individuals. Some people talk about QA team that's really not a big active QA team like there's a KDE team, for example. And I think that working as a team would help a lot, would bring a lot of fun and also it's not calable, which is really important when you have to maintain when you have to take care of 13,000 packages. So, there's a project on IOS called Collab QA which can be used to share results about QA tests like rebuilds of archive or run of pure parts and to keep the results for history which is important so you can get an idea of all the things. And it makes things much more fun efficient, I think, than before. So, I have a quote from Lunar. We really enjoyed reporting FTBFS. So, inside that project we already worked on packages at missed edge even if that is not finished yet. We worked on several archive rebuilds even after the release of edge until the 14th of June. All build failures except unsatisfiable dependencies one have been failed. So, that's about 400 or 500. Someone is working on file conflicts between packages and there are plans to restart runs of pure parts and also for any ideas that you can have that you find interesting. For example, the double build tests that were done by with an idea from Martin Zobel was not done inside Collapua but we are open to other ideas of this kind. So, don't they say to join the project. So, to conclude let's make QA work for Leni let's make Leni higher than Edge was. Please join the Collapua team and help. So, currently all discussions happen either on ISE on Debian QA or on the Debian QA main list. If you subscribe to the main list you should be well informed. And now I have two questions two questions for you. The first one is what do you think of that GDP by mail ID? Do you think I should send those mails ? I started with quite low my criteria for the first mail are quite low, that is package has to be out of testing for alpha year and must have bugs open I see bugs open for more than 30 days and even with that I get 280 maintenance to mail that is 280 maintenance that has at least one package in this state so that's a lot and I can't reduce that number much and the other question is what about a team or working on packages in a questionable state because that was right during the 15,000 packages both this morning so we could have discussion about that as well so the idea is that we have too many packages we have a lot of packages many of them are not useful anymore or just the archive without they could be removed and we have no way currently to tell ok this package could be removed ok so what do you think So on your first question I'd like to suggest a show of hands so who thinks that this ML is a good idea ok so does anybody think it's a bad idea ok who thinks that the criteria that Lucas was suggesting are very generous to the maintainers Well the problem is that we aren't really representative of the world Debian developers well we'll probably I'll probably get flamed but ok so be there to attack ok so I'll do that and what do you think of the archive state do you think we should work on that or do you think it's ok to maybe on the first hello ok maybe on the first question maybe it's an idea to send a mail to DDA first announcing that you're going to send to send these mails and also provide a place where people actually read what kind of mails are being sent out simply generate them and put them in some directory then you can preempt any questions or just do it so in terms of repository wide qa there is currently a note on the lintian.debian.org page that says don't do math bug filings based on lintian tags because the lintian maintainers will periodically do that well the lintian maintainers are not periodically doing that speaking as one of the active lintian maintainers I don't really have time so if someone would like to volunteer to go explore lintian tags on the lintian.debian.org site see which ones might be worth math bug filings and proceed to go and do that I think that would be great it picks up a lot of pretty obvious problems and it picks up a lot of fairly major problems that I think the maintainer may just not be aware of though it is worth pointing out that not all of the problems that lintian does throughout our serious problems so I think devin lintian is pretty happy to take questions on whether something is actually relevant on a personal comment like it's been great fun to actually feel FTBFS bugs sorry for those like who has received true and false positive yeah there have been some false positive with like Python packages but I think it's really something like this the problem, the main problem that you rebuild the whole archive and you get like 600 build logs to review and that's where like Lucas has a problem and has to ask A and it helped that I've rebuilt all that thing in 8 hours with this huge grid of distributing computing and so it was very easy to jump in I mean like it takes maybe 3 or 4 minutes per build logs sometimes I did a few checks by myself like tried not to get too much false positive but still but it was really something I could do I don't know I had like half an hour to kill ok I'm going to do like review 10 or 20 build logs in that time and it was great fun to do that together and I mean like everyone can jump in yeah I think that's an important point that's reviewing logs is something that you can do when you just feel bored and you have 30 minutes to lose you don't have to dedicate 4 hours to that just do 30 minutes you can still review probably 10 bugs for 10 failures maybe find 10 RC bugs if you are lucky and that helps a lot about the build logs I'm wondering in what's a way in how far can these build logs be analyzed somewhat automatically is there some common problems that you can maybe filter out somebody was doing that for certain common common problems like warnings for unsigned versus signed mismatches then Frazier then Frazier has done some work on that ok oh yeah now a recall having seen that somewhere so maybe it's an idea to centralize this code that analyzing code somewhere on Collab UA for example so that yeah we can see in what way the workload can be reduced by filtering out some of the common problems of course a lot of things are going to be need to be manually checked ok for rebuilding build logs we already have quite a good workflow I think we have quite efficient at that but that's place for improvement but we are quite good at that it doesn't take a lot of time to review them so I'm wondering if let's say I have an idea that I want to test by building the whole archive or digging into the whole archive and I would have no idea where to start for building the whole archive and just assuming that I have the needed hardware so do we have tools which can be used easily by normal maintainers to try a fuller build of the archive or something similar well the problem is that I have strips to do this but that's quite specific to the platform I'm using it could be use less to use them as well but S build is really good and you can just wrap around S build basically my script just do that so another point is that well Luca is available and if you need a rebuild in specific conditions just mainly explain him how to do this build and he will do it for you and then you have lots of build logs that you can analyse and you can create that for the new DPKG SHLIP depth stuff and well you can make sure that your new development is going to work on the whole archive and that you don't break anything so really use it either to file bug or to test new development it's cool yeah but basically if you can provide me with a patched S build that does what you want, that's perfect even if you don't I can see that but please don't come to me with just silly ideas and first test for example if you test on 100 packages it still shows that it's a good idea quick advertisement because we scheduled the buff really late there is a lintien buff Saturday 2pm there's currently only 2 people signed up for it because it just got scheduled today but everyone is certainly invited to come talk about lintien there's another thing that I'd like to mention so I've been doing some stuff for Canonical which involves having functional tests in packages now we don't really have any packages with functional tests but the idea is eventually that we'll be able to install every package and actually check to see whether the installed version works at all after you've done that and there's an interface for that it's called auto package test and I'm hoping to be able to be running that on a regular basis fairly shortly my hardware's a bit stuffed at the moment but I'm expecting that to be fixed in the next month or so and after that you'll be able to get your package automatically, functionally tested so all you have to do is glue a very small amount of tests into the directory according to the spec I've written I have a question for you because about these testing things I think that it's a really good idea but I'm wondering why it's could you move the microphone sorry I'm wondering why it quite failed to get wide acceptance around a lot of packages using it currently why it failed to what to get wide acceptance among the DD basically because at the moment nobody is running these tests I've not been able to go to maintainers and say hey look I've plumbed this test in and get a bit of push adoption because it's difficult to sell it given that at the moment it doesn't really do very much how many packages are using it sorry how many packages are using it currently just about two packages that are used as test cases so it really is nowhere the one of the packages is Mork the other one was something that the Ubuntu security team had some tests for that they already had so I'm afraid I can't remember the name yeah I think it was it was Dovcott well as I say my hardware is a bit stuffed at the moment this is going to be unstuffed very soon it's on the critical bar for lots of other things I'm hoping to be able to be running it on the archive, on the Debian archive probably on testing starting sometime in the next month maybe a bit more on the self testing packages there are some packages that actually do a self test running a test suite when they build and we should include that one problem I've had with some packages that are written for i386 when you introduce the self tests later on it often fails on some architectures Gnash is an example I tried to convince Miriam to enable it by default and make it a fatal error but it's a bit late to do it when it's already built on all the architectures and half of them start to fail and then it stops propagating into testing how can we actually handle these test cases where some architectures were never actually intended to work with the program and they are still in the archive and it's a real pain to get the FTP masters to drop the architectures and get the package into testing I think if a package is not actually working then that's a perfectly legitimate reason for it not to be in testing it's if the devs are stuffed if there are minor problems that don't really affect the core functioning of the package that's one thing but I think the test cases this is just that it's completely broken on that architecture and I think having to cope with either fixing it or getting FTP masters to remove it is okay in that case right right so it is some work I certainly agree with what Colin said also I'm trying to do post hoc testing and that does mean that you don't discover it before the package is built but it also means that something in the package's dependencies causes that package to no longer work you will discover it so Debian has pretty high quality standards anyway and the regression rate is generally quite low so I'm hoping that testing packages after they've been built and put into testing is a useful thing to be doing because the failure rate at that stage will be low enough and that you won't regret very much catching these problems at that late stage but then you get the problem of being able to test enough package so that it makes sense so it's a bit of a chicken egg problem no other comments okay so I think we can all go have dinner