 A gente vai ter o teste-driven de desenvolvimento de Debian com Ian Jackson, Tom Marble e Stefano Zacrioli. Olá, e muito melhor. Muito obrigado por estar aqui. Primeiro, se você quer nos ajudar em tomar notas, por favor, coloquem o Gobi. Tem como fazer isso, e tem um documento que já existe, que é chamado DC11TDDD. E também há algumas perguntas de trabalho. E se você puder ajudar em tomar notas para esse Gobi, isso seria ótimo. Alguém está tomando uma foto da tela, para que eles possam ir para o documento. Muito bom. Então, discreta. Eu posso? Você pegou a cichura? Ok. Então, discreta. Então, o título desse Gobi é uma mistura, porque há um desenvolvimento de metodologia, que é chamado de teste-driven de desenvolvimento, que é sobre, primeiro, criando o teste, e depois, trabalhando um pouco para fazer o teste passar. E o Tom vai falar um pouco mais sobre isso um pouco. Mas isso não é só sobre isso. Então, há dois tópicos importantes. Então, ambos de eles são lançados para testar. O primeiro é simplesmente como podemos ter mais testas em Debian do que o que temos agora. E, com isso, eu digo que ambos são testados para testar. Então, como testar o package individual mais. E o segundo é também para, na verdade, ter mais testes de testa disto, que é algo que nós não temos muito em Debian. Então, esse é o primeiro tópico. Como ter mais testes em Debian. E o segundo é, realmente, como properamente aplicar a metodologia de testes para Debian e se isso é algo que queremos, se isso é algo que as pessoas gostariam de ter. Oh meu. E, como fazer isso se nós realmente queremos. Eu não sei se isso... Oh, é raro. É que, mesmo que o desenvolvimento de testes seja uma metodologia de software fashionável, apenas porque nós usamos o TDDD como nosso acrônimo de juras que não necessariamente significa que todos os speakers nesta panel necessariamente engordam essa metodologia. Absolutamente. Então, essa é a ideia de compartilhar nesses tópicos, como nós ensinamos nesses tópicos no planeta, e ver como as pessoas se sentem, e que nós podemos mover para eles. Obrigado. Então, esse é um breve estado de arte do tipo de testes que hoje nós fazemos em Debian. Então, o primeiro tipo de testes que nós fazemos em Debian é realmente construí-los em packages. Então, todos devemos construir um package antes de aplicá-los, porque, em general, nós precisamos de files para existir quando você faz um upload. Então, isso é o primeiro tipo de testes, as montanhas fazem construí-los em packages de seus próprios. Nós também construímos, construímos um build periódico, em Santa Lucas, e em uma grande classe que construímos um build periódico, todos os packages em archive. Então, isso é um tipo de testes que os packages podem ser construídos de sítios. Isso não pode ser muito, mas isso é realmente um pouco. Então, não todas as garantias de distribuição como nós fazemos, que todos os packages são realmente poderes ser construídos de sítios. Então, isso é um tipo de step zero. Então, nós temos um tipo de testes automáticos de policy compliance. Então, nós temos Lincoln, e nós usamos o Lincoln possivelmente para automaticamente rejeitar os packages. Então, se o Lincoln diz algo particularmente ruim sobre o seu package, o seu package simplesmente não será capaz de entrar no archive, porque o doc vai rolar o Lincoln e vai reforçar para aceitar o seu package. E depois, ainda na frente de policy compliance, nós temos poupards. Periodicamente, nós cobramos poupards nos packages para testar que eles são instalados em todas as possíveis condições estranhas. Então, nós testamos, nós tentamos testar assim, os vários pathos que existem que estão instalados em um package e veem que o package se comporta como uma polícia que eles precisam. Depois, nós temos alguns testes de build-time, meaning that several packages actually at the end of the build run specific upstream generally test suites and those are generally the test suites that can be run at build time, which are not necessarily the same that can be run once the package is installed. But we do have that, several packages do that, even though we don't have, so I think we don't have a specific target in the policy which can be invoked to actually run build time test suites. But this is something we do, but I think we do not really enforce it much, so we can do more in encouraging that. Then we have some sort of integration testing, meaning we check if packages can be installed together, we check that packages actually can be installed at least in some possible way. For instance, we're using the ADOS tool that are available at ADOS DebianNet. So this is part of what we do in QA and for instance we test that we do not miss conflict among packages. So we test combination of packages and we verify if they can be co-installed because if they cannot be, well we should declare a conflict among them. So this is what we do in integration testing and then even if it's not, it's probably more for instance for upstream than for them in itself we're starting to do some automated code analysis. So some month ago Rafael Geissert announced this initiative which is called DACA for Debian Automated Code Analysis which is essentially an infrastructure to be able to run code analysis tool on all the software which is available in Debian. So right now there are some simple tests like CPP check I think, like some Python specific stuff we're trying to integrate other static analysis tool like Coxinal and this is all to run bug in the software we ship in Debian. So it is just started but it's promising as an infrastructure. Just another word about that. There's some other interesting things like OLO statistics that just give you lines of code and percentages for the different kinds of input files that you have and that's actually pretty interesting stuff. Absolutely. So this is what we do today. Maybe we have forgot something if we have forgot something, okay. Microphone. I've been running two archive-wide testing at least previously before squeeze I did complete upgrade testing installing GNOME desktop KDE desktop and I think it was a lot of... So you mean upgrade testing from one suite to another? Yeah, I did a squeeze upgrade testing and discovered a handful of bugs using that. I built basically a learning installation in a change route and upgraded it and checked the results from that. I did both apt and aptitude to see if they resulted in the same packages. Also, I've been running archive-wide checking of the boot sequencing the dynamic boot sequencing to ensure that all packages with in-inscript could actually get predictable ordering. Okay. So that is great. On the Gobi, there are a lot of questions. One is what are we doing in archive-wide testing which we forgot to mention here. So if you can add a line with pointers to those initiatives, that would be great. Okay. So this is what we're doing now. And the question is, do we need more than this? Well, just to... I think we do. We do need more tests. In general, well, it's a bit of a rhetoric argument but if we claim that Debian is about quality well, we need more tests to keep up and guarantee that we're offering packages of high quality. And in general when we add release goals it would be great to have a test to actually test that those goals are fulfilled. So this is in general, yes but I think there are some more specific reasons to do that in Debian. And the first one which I personally care about is that we need to reduce packaging contribution barrier. So we need to empower the maintenance to actually feel more confident when they are touching packages and in particular when they are touching packages particular when they are doing NMU. And I think that having more testing can actually help maintenance feeling more confident when they do changes in code which is not their own code. Also, as... sometimes we need to test some archive-wide changes. Sometimes we propose archive-wide changes which may be on the vela or not that well received. And sometimes we do have a need to say, okay, I think this change is very good even if people in theory would disagree. So I want to show you that the change is good. And I think it would be very nice to have more testing to actually be sure when we do these changes to the archive that we are not breaking stuff. And this could be, I think, better used in conjunction with something IPPA where people can change packages of other and actually show it's running fine and it's not breaking any existing test and so on and so forth. So that'd be sort of an extension of what Malkin was talking about this morning with an experimental archive for a whole a large set of changes que actually merge. Exatamente. So this can be also a kind of not a test bad, but an environment in which we test changes before migrating them from experimental to unstable. Exatamente. So what can we do more than this? So there are a couple of directions we have been mentioning here. The first one is something that has been discussed by Lars on Planet Debian recently is system testing. So we do not really have the sort of testing in which we test that some basic functionality which our users expect to be there actually work. For instance, we can imagine we have tasks in Debian and tasks generally correspond to some profile of our user. For instance, the desktop task is meant to be granted. There exists a desktop base installation which the user can use to do office thing and it would be nice to have for each task a set of tests which test basic functionalities. So example which have been made by Lars are like if you choose a server installation and you want to test this install that you can log in with some basic user and this kind of basic testing functionalities. So the idea is that to have some test suite which is not tied to a specific package but rather which is system wide and maybe associated to Debian tasks. And I think that what would be particularly nice is actually to have a collaborative test suite in which users can provide their own tests and integrate that in a Debian test suite. And we have upgrade report, installation report of this kind of stuff. The users say ok this is not working, it would be nice to be able to add a test to a shared test suite in which we ensure that this kind of stuff will not break in the future. And of course all this should be able to be run automatically before doing RLAs. Do you forget anything on that front? I don't think so. So this is one direction. Another one is TDD, the proper test driven development so maybe you want to take over here. So and just to maybe make this a little bit easier I'll show this diagram, I realize that this is probably impossible to see especially in the video. But the idea of test driven development is basically this waterfall model where you start with writing tests and see if they pass and if if they test pass you need to write more code but really the core idea of TDD is that you write your test before your code. You basically decide a priori what the functionality is by defining that in terms of a test and you write any implementation that you can to pass the test and then once the test is passing now you go and refactor that code to make it perhaps more clean or more performant but you're sure that the functionality is working as defined by the test and so the parallel to what we're talking about here is that in the archive we want to be comfortable to make archive wide changes or perhaps packaging changes along the lines of multi-arch and yet feel comfortable that we haven't broken anything and so that's really the parallel to test driven development and some otherwise TDD is really perhaps more of an upstream concept than it would be for Debian but I think that some of the principles can apply here in terms of code quality and I think the main thing that Zach was talking about is giving us the confidence to make changes even changes that are very tricky and know that whatever we've done will not break anything especially as we move to something like rolling where we want to make sure that we're always operational and actually that could help help us evolve faster because we will feel more comfortable in making more radical changes ok so and then the another direction we would like to discuss with you is actually what is called auto package test which I think Ian can describe way better than I could do it is actually the other side of build time test suite which is actually run time test suite or as installed package testing right so is this on seems to be so basic idea most packages a lot of important packages anyway come with some kind of upstream test suite which is better or worse and also sometimes we could add to that and sometimes you run that in the build sometimes you don't because it takes too long but often what you really wanted to do was test the dot deb that you were producing rather than just the thing that you install because actually when we make mistakes often the mistakes that we make result in the build succeeds and you end up with some thing and the files are in the wrong place and it doesn't ever start up or some key functionality is broken or it doesn't find its plugins or some library you use has broken and now the same binary doesn't work anymore even though you know it worked before and we don't have at the moment any way of detecting that other than when somebody tries to use it on their real system and they go why does this thing not work so I said about knowing what I thought was like the smallest possible thing to make it possible for a package to declare how it should be tested how its dot deb should be tested you want an example I have some here so what you do is you put some notations in the source package which look a bit like that I don't know exactly what this is I'm trying to setting up your example the idea is you put some small test control file in your source package the bottom one is yours so this is the example I did a sort of proof of concept with the package morc which we all know is vitally important and it says there's a test called morc test and well that's the upstream test actually the upstream test is called morc test and it says it needs the build tree to be re-writeable that's like a declaration to the test runner and actually that morc test is a nine line shell script that sort of does a bit of celery and a bit of wrapping on the upstream morc test script and makes it not run the one that it's just built it makes it run the one that's installed and that's all you have to do to the package and there's well what auto package test is there you go that's the morc test script mostly it undoes the thing that morc tests upstream does to make it run the one from the build tree so it gets rid of the path setting and apparently it produced some useless warning version output or something like this is years ago I can't remember exactly what it does but it's not very complicated so auto package test is a program that interprets this debian tests control file and the scripts that will live alongside it and will automatically build the package if necessary install the resulting binaries on some kind of testbed like a churrut or a zenvm or an schurrut or however what you've got and report the answers via logval and the exit status now that thing last worked in 2008 and the machine I was running it the machine I was running it on was rotted and for reasons that don't need to be gone into I didn't feel like picking it up again so just recently Stefano has been encouraging me to pick it up again and I just today managed to get it working again on my laptop interior on my laptop as well I was going to have a demo but I'm told I'm not allowed to have a demo if it doesn't go through the streaming video and my laptop won't drive the streaming video and it doesn't work on Zach's laptop I mean 1, 2 ok while we were reviving things we might want to have a look at Debian test which was a package that I wrote I don't know 15 years ago which had rsync tests because I was also maintaining rsync and I don't think anybody ever picked up on that either so there were some features that are quite nice in that a specific depth in its own installation environment because what I think is very what I found very interesting in how to package a test is that you can for instance have the test bed which is a virtual machine in which you can do the testing of a packages for instance you can test a network server you can set up Postgres and connect to it and do this kind of testing yeah that's an entirely different thing from what Debian test used to do Debian test was a framework for providing tests all these packages provide tests run them all ok in which environment did you use on your machine you can just say Debian test all run the test for rsync one of the pieces of auto package test that I kind of scooted over a bit here is actually it's in two pieces one of them is like a testrunner program and then there's a sort of interchangeable test bed control protocol where underneath you have a program that does things like revert the test bed back to what it was before and ok you could just use a churro in which case well you don't get to revert it and hopefully that's alright but if you have something a bit sophisticated I had a Zen set up where it would save the VM after it had booted and then every time you wanted to run a test it would resume the VM from exactly that point and run the test and then it would throw the VM away and um you could start demons you could wipe the file system you could do any crazy thing you could crash it ok, that would be counted as a test fail Potentially you can test a very dangerous thing like not exactly a bootloader but you can test a kernel you can test a very dangerous thing which you don't feel like testing on your own machine There's a way to declare in these restrictions you can say breaks test bed and if you say that you can run your test unless it's got a sufficiently clever virtualization thing it's confident about that ok, sorry do these tests come with a package yes the tests are in the source package now you don't have to have the tests in the same source package as the binaries you intend to test but that's probably most that's the usual case and it's sort of set up to do that I'm not sure if that's a good idea for example, how do you do bisecting bugs if you only have the test in the latest version of a package Right, so you can run one version of the tests against the different version of the binary package because the tests come out of the source package and which binary package you run those tests against is entirely you know, that's controlled arguments to the test runner to say bring my binary packages from over here to the source package ok, but still I'm not entirely convinced they should be coupled with a package or come with it so tightly so I so I think we should keep in mind that we are doing testing and we are a distribution so I don't think the scenario of debugging upstream application is something which is should be our main scenario and if you think about only testing what is relevant for packaging and you can do bisect in that version control system and see which distribution specific change has actually induced a problem but even more, in theory you can also do actually the bisecting on the upstream code imagine you have a distributed version control system for your package in which you also have an upstream branch well, no problem, you can do bisect there as well so I think it's a very interesting use case but I think it's the main one we should focus on you still can do this exactly what you would want to do you can say I want to run the latest version of the test from the latest source package which is the most sophisticated and best version of the test and I want to run it against this other tree here which is the one I'm bisecting and it will automatically build copy that tree onto the testbed build it, produce the binaries wipe the testbed, install the binaries check that the test runs if the test passes you say bisect good then you go around again you just need to get all the arguments right to the runner ok so I just want to mention what is the status of all this so today is how the test is working again just about but the code has existed for like 4 years, 5 years but as far as we know is not really in use so it was some package in Ubuntu we choosing at some very old stuff I did one proof of concept in Ubuntu then other things intervened and I didn't get around to adding test to a bunch of more packages so as of today in Debian no package is actually using something at that to declare their own test which can be interesting both to have different specific test but also as in the mock example to actually make a bridge between upstream test suite and run the upstream test suite when the package is installed in a Debian like machine so to actually advertise a bit more this possibility so we started a DEP which is DEP8 you can find a DEP URL to actually try to standardize the interface with the testing tool and actually encourage people to use it so if you want to know more about all this you can have a look at DEP8 we are looking for some help to actually make it a proper specification because it's right now in a readme state so it's pretty complete but it might be a bit more for introduction and material to know how it works and so this is detail that you have also discussed so I think we can now open give time go ahead just one more thing I wanted to add is this buff really was coming from Stefano's decision to take some discussions that we had online and bring it into the buff and I want to give credit for where credit is due for the other people who have been part of the discussion as Zach already mentioned Lars I'm sure I'm going to miss up your name Zenius who is also in IRC contributed a lot to this discussion and has several blog posts about it Zach has a blog post about it also involved in the discussion were Michael Henke and Yoroslav Halchenko and also Justin Pop so I want to make sure that those people have credit and with that we sort of thought of about testing in Debian and we put those up in Gabi and perhaps there are some new ones ok so in particular the general question I think for the public is how can we do how can improve both package testing in Debian and distribution wide testing in Debian so this is kind of the main topic and I guess any of us has a specific question so personally for all the thing which regards auto-PKG test I'm curious about what are your experience so do you have packages in which you have build time test suite and if yes are you running them do you have packages which are run time suite which are particularly difficult to run in particular the case which are interesting are those in which you cannot run an upstream test suite because it's dangerous because it needs network which is something we don't have in buildies because those scenarios are those in which we can benefit the most for something like auto-PKG test and other topics we would like to air your opinion on so do you know initiative for package testing in Debian that we did not mention like the one that Parer brought up as well as would it be useful for us in Debian to actually offer Debian as a platform for upstream testing one simple example comes to my mind is that we have way more architecture than the average upstream so we can imagine offering Debian as a platform in which we run upstream test suite in a huge variety of architecture and actually give a service to upstream as well and finally for a specific case of system testing how can we set up some collaborative mechanism in which we collect tests from our users and bring them together and ensure they work release after release so please Hello I think I have kind of a special case of testing because I am considering scientific software and if you have some input data and command line what to do and you have some expected output data and want to compare it and I would like to have a test when package is builded inside the package build process to really verify that the program works it is also planned to do this I mean that's up to you in your package it's very easy all you need to do is a build target run the test I can put it in the rules file sure but we can somehow formalize this in my opinion say DH run testbed or something How would DH know what to do Actually in DH7 I found it out the other day Joey in the room can kill me ok is not So there is a DH auto test common in DH7 which is automatically run and what it does it check if your make file has a test target make check I think it test both check and test but ok there is some heuristic to find out whether the make file have a check thingy so by default if you respect the if you respect some let's say DH7 already does that so that should work out of the box the problem is that one part of the problem is that we are not encouraging people enough to actually use that because for instance there is to the best of my knowledge no LinkedIn warning which for instance check if there is some non-standard test target and say ok beware you are not running that test target and also I don't think we have a policy target which is sort of mandatory to actually test the package so these are the two direction in which I think we can improve but by default DH should already do the right thing which is pretty good so excuse me Nils you got the note about a new LinkedIn warning ok you will yeah I think the word ok Neil Williams then ok ok so do you want to start ok so in two packages we have upstream test shoots one is Libia BI driver the other one is Nova Compute for OpenStack and I run both them in my build target but I found very very important to be able to say ok now I'm just modifying my package and I don't want to build it to run the test shoots because it takes too much time so it's it's good to have it run by default but it's also very good to be able to say this time I don't want to run it ok ok one at a time I don't know I think there was Neil first and then we go around we need an answer to this point and I don't have an answer but there should be a standard way for you to ask for example that DH there should be a way a standard way for you to tell DH not to run the test this is what I was going to say there is a standard way of doing that with the dev build options you've got to make sure that if you are going to run tests you use the dev build option for no check or no tests and you enable that in your dev build rules file and as long as you are then you can then specify the dev build option to not run the test for that particular operation particularly important when we're talking about we've just had multi arch so then we're cross compiling you've got to make sure if you're doing tests during the build you've got to have a way of allowing that to be disabled because when you're all kinds of situations this should be in the policy is that what you're saying? no check it is the no check option it is in the policy I have no idea how much it is encouraged it would be worth looking at policy to check that policy says what we think it should which is probably you should probably run the upstream test suite unless there's some really good reason not to and you should respect the no check that build option the other thing I was thinking of one of the very hard things to test is things like grub grub and boot loaders because come release time we get so many arcy bugs against boot loaders and problems like that do test, how do we deal with that kind of situation what is the framework for dealing with big issues like that where the system is so disabled if you actually get it wrong the sensible way to deal with that is to test a boot you can only test boot loaders properly in a VM so you need machinery that will install your system in the VM and install the new boot loader and then check whether the VM still boots that's in principle possible I can say it's a simple matter of coding but nobody's written it yet it's probably not that simple I have a few questions relaying from IOC the first one is do you happen to know if there are any packages in the archive que implementa the dev8 pack no, Stefano é o que o Stefano estava dizendo então há mais de 5 anos há 5 anos mais um package em Ubuntu mas se você voltasse a um Ubuntu 5 anos seria em Ubuntu mas acho que eles mudaram isso parte disso é tentar popularizar você tem outra? sim eu vou lhe dar a outra a próxima pergunta é em relação ao testamento de build o que seria o momento em que tempo por exemplo, em um laptop convencional é isso ok para testar durante o build eu não tenho ideia eu diria que uma lei de thumb você deveria comparar quanto tempo o seu package faz para construir e se o testamento pode fazer mais tempo do que o restante do build então você vai adicionar um grande custo e você deveria pensar se isso é sensível algum build maintainer se... ok então é interessante saber o que eles consideram aceitável e o que eles não consideram aceitável é uma figura interessante pode usar o dev option no check se é muito longa excepto que você pode colocar isso nas build e o... não há maneira para o build para dizer que esse package particular vai ter 3 dias de running um testamento no rml então o que você quer é algum jeito para o package para saber que o testamento vai dar agência para esse host isso tudo vai ficar muito complicado o melhor é se isso vai acontecer com o seu package não faça o tempo ok, você falou sobre o que deveria acontecer quando o testamento faça o build, o package deveria faça se não é um build test se é um testamento de um computador ok, então isso é interessante eu tenho a resposta para isso eu acho que essa informação deveria ser na página QA de seu package o resultado da última testamento deveria ser na página QA de seu com uma pequena color para dizer se é certo ou errado e no momento não há um código que vai dar o output do testamento de auto-book e vai apresentar em algum jeito o que faz no momento é que e-mails a pessoa com um parcelado de fail particularmente e-mails a você com lojas de fail e é não é uma coisa muito útil para fazer mas eu não tenho tempo para escrever como um web front-end então parte da ideia para re-reactar tudo isso, depois de ter o código e depois de ter um speck e depois de ter sabido que tem alguma infraestrutura que periodicamente é o teste de todo o arquivo potencialmente em diferentes arquitetos mas essa começa a ser uma bastante informations consumindo e do que nós modificamos para a collapse de QA então o investimento periódico e publicando o data e integrando por exemplo em a PTS e depois, depois de um review humano periodicamente você pode ir na frente e e editar a deonie isso é exatamente o que acontece com os built Então, eu acho que pode ser útil para poder especificar algo assim, como esse teste é crítico, se esse fail, então é um bug ou algo crítico, porque todos os testes não são criados igual. Se algum teste de GCC small test fail, isso provavelmente não é crítico, mas se um bug derrotar todos os seus files, então é ruim. Certo, então... Boa sorte, Peter. O teste de testes é muito fácil, como o Arabic 802, é muito fácil, mas dado que no momento não há uma forma de reportar, tem também um problema com a atualização de bug. Se você tem muitos testes, você vai dizer que se esse teste fails, o 시간이 é sem chance, e você deve ficar automaticamente embalado de bug, Então, se a Lipsy derrubar, de repente, cada mantena de qualquer package vai ter um bug report dizendo que o seu package não funciona mais. Porque o testor vai instalar o sistema, tentar correr o teste, descobrir que o teste não desaparece e decide automaticamente tirar um bug e, no momento em que não temos uma boa resposta para esse problema. Eu acho, apenas um segundo, que o PEOPART as pessoas possam acertar esse problema por fazer um teste seguindo a trinta de dependência. Então, eles primeiro testam o package que está na rota da trinta de dependência e, se isso não desaparece, o teste não foi abaixo. Então, Olga não está aqui, mas eu acho que ela está fazendo algo assim, que pode mitigar o problema. Isso seria uma coisa útil para fazer. Phil? Uma forma de escolher o que é testar, é testar uma coisa que se chama de bug. E isso diz diremente se é um teste crítico ou não. Porque se o bug for um teste crítico que você está testando, então você faz isso para fazer um teste crítico. Eu não tenho certeza de que isso é isso. Então, cada vez que alguém reporta um bug contra o seu package, não é uma boa ideia se tínhamos uma forma de escolher um teste que desaparece antes de fixar, e se acesse depois de fixar. E se isso é um bug crítico que você estava só fixando, então é um teste crítico. Isso é bastante definido. E também se esses testes são coisas que você pode empurrar as pessoas para correr os seus sistemas, aparte das que trazem. Então, seria bom ter um programa que se chamaria de testes seguros. É, a ADT Run vai fazer isso se você for ver as opções. E isso pode ser integrado em um bug de report, também. Eu imagino que, idealmente, você iria fazer isso durante o upload. Se alguém reportar um bug contra o seu package... Eu acho que o pbuilder, ou o debuilder, deveria ser mudado para ter uma opção de dizer, oh, e correr os testes. Você está usando um True Root, de qualquer forma, correr os testes, correr os testes seguros no True Root e ver se... Absolutamente, mas se você é um usuário comitado, você pode decidir fazer um teste agora e de novo com todos os package que têm testes. E se você está reportando um bug, se você for ver todos os testes disponíveis para esse package e um loader que faça, talvez você faça um bug no seu sistema ou um bug no package. É isso, sim. É uma pergunta aí. Então, não que você tenha mencionado a pure parts e, talvez, o lindian. back to some of his question about, do the test need to be in the source package. For example, if I want to test all Emacs packages, and I want to test that they should do something when you install and Emacs is not installed, and something else, if you install Emacs, would I need to then put these tests to some Emacs and common package and declare dependency on every single package in the archive, Emacs package in the archive or how would you do that? Or do I just put them to some git repository and make some manual hack to... So, what you could do is... So, auto package doesn't require that the packages you're testing come from the same source package as the one containing the tests. That's just the easy default. Yeah. If what you want to do is run a test on some particular package both before and after you install... Well, the problem is that... Lindian, it doesn't... You cannot use Lindian to check the maintainer scripts because they are not run because it's build time or static. And with pure parts that you cannot do much logic there. It's just checks that you can install and remove it. But you cannot say that... Yeah. I don't think auto package test could do this at the moment, but it wouldn't be very hard to add a feature that would make it possible to do that. You'd end up writing a source package whose sole purpose was to contain the tests. Or maybe you would put it in an Emacs common package if it was convenient. You certainly wouldn't want to make a thing that said, well, we depend on everything in the archive. Em packages that you are testing. But there is a question there about how you would decide which packages you want to run the test on. Well, every time somebody uploads a new version of an Emacs add-on then the test should be run. You know, and I know what an Emacs add-on is, but in order to do that automatically there will have to be some code that automatically decides which packages in the archive are an Emacs add-on and run this test only on then. A polícia precisa dependê-la de um e-max e um packet. Então, veja se isso é dependente. Então, o teste é aplicável. Então, a melhor forma de fazer isso com o teste de auto-projeto seria para ter algum tipo de... o script do driver. Seria para selecionar um teste de package. Eu não sei... Talvez eu coloquei o teste, coloquei ele para a repositoria e coloquei ele contra alguns packages e enviou-o para você e você pode pensar sobre isso. Ok, podemos mover? Tem uma pergunta para o Paul no fundo e no fundo. Então, em um Quilt 3.0, você pode ter múltiplas torbulas. Você pode colocar os testes em uma torbula de torbulas. E eu já faço algo muito similar para outro package onde eu tenho o primeiro package, que eu tenho um número de quantos, digamos, matizadas de um material de beta em diferentes packages de torbulas. E eu posso, selectivamente, mudar quantos outros packages de torbulas eu inclui. Mas o base de torbulas está idêntico, o base de torbulas. Então, sim. Você pode colocar testes em um torbulo de torbulas e eu acho que é geralmente muito bom para um teste que vem do Labyn. Ou talvez eu não tenha sua pergunta. É só dizer que é possível standardizar o porquê uma forma de nome para testar o tarbol, então, em vez de testar ou algo, depender em testar. Então, sim, isso deveria ser possível, mas eu não acho que isso seria desejável, porque impulsando para ter um tarbol separado, se você quer testar, é tipo... Eu acho que é... Sim, se você olhar o Ork, o Ork, ele teria um tarbol contando um fio de 2 e um fio de 9, e isso é um pouco sério. Então, realmente, o que você queria foi simplesmente para empurrar o tarbol com tudo o outro. Se você tiver um package de upstream, que tem um enorme testa de testes, com grandes fios de data, que vem distribuído separadamente no upstream, então você quer o tarbol separado. Eu acho que é importante considerar uma convenção de nome para testas que compram múltiplas implementações. Nós falamos sobre isso um pouco. No caso do Ork, tem Mark, Nark, Gark e etc. No caso do Java, temos múltiplas implementações de Java, mas o testa-suita é mais ou menos o mesmo. Então, há alguns refactores para fazer. Então, eu acho que temos mais de 1 minuto. Algum outro comentário? Justo um comentário da ISE. Liv said he has a few tools. Ah, sim. Na verdade, prototypes... Eu esqueci de mencionar, mas obrigado. Para testar coisas como bootloaders e x-applicações, se você olhar o blog, tem links para isso. Então, sim. Lá tem um tipo de library que se chama systest, que pode ser usado para testar o sistema level. E eu acho que seria um tool muito interessante para usar como base para criar um teste colaborativo para testar tasks e instalações básicas. Então sim, isso seria muito interessante. Ok, então, para concluir, se você é interessante em tudo isso, em particular em L2PK g-test e revivê-lo, sinta-se para contactar com mim. Há muitas tarefas para ser trabalhadas, então você pode ser de ajuda, se você é interessante em esses tópicos. Então, eu acho que talvez falhamos um pouco sobre o follow-up. Devemos fazer, talvez, uma página no wiki? Ou devemos enviar e-mail para o projecte? Então, eu diria que esses minutos são para discutir e eu acho que é um projeto também. E sim, por que não? Nós podemos anunciar uma página quando temos um, com o ponto de testar. Ok, então vemos que há muitas pessoas em Gobi, e eu acho que há um monte de IRC. E todos aqui, muito obrigado. Muito obrigado.