 Okay, good morning everyone, maybe a bit of a strange subject to have in a QA meeting, but I did try to make a talk at least QA related, so I'm going to talk about quality assurance in the DI development process. And the main thing we have to do is to manage breakage. Basically the install has been broken in testing and unstable for the last two months ever since the big transition started. We have had occasional days that were good days and we couldn't actually test it much, but most days have been basically hell. We have not been able to move anything to testing from unstable because there just wasn't a point to do it. We would have liked to release better release for Edge by now, about a month back was the original planning, but we had not been able to do so just because the whole environment was continually so much changing and we were not able to stay in life things to get a release out. So this is about how do we try to handle that? It's in three parts and let's start with a little introduction. Well first of all what types of breakage do we see? Breakage can be either general, that means everything breaks all insulation methods for all architectures or it can be specific to a certain insulation method or to a certain architecture. That's mostly the case when there's something wrong with boot loaders or with the entire configuration, something like that. You're going to have build failures. That's not that bad to be honest because if we have build failure then we just don't have a new image for a day but the old images will still be there. Anyway, the breakage in building can be either while building the DI images themselves or when building CD images that include the DI images. The DI images are basically the kernel that's used in the image in the CD and the entire D. That's used. You can have basic installation problems. The installable boot, you will have problems with selecting keyboards the keyboard won't work, whatever. You can have problems during the basic installation phase that is when the bootstrap is broken, which it was quite a while after the migration after Sarge was released because basically Edge was not supported yet in the bootstrap. So we had to wait around for that. You can have problems in basic configuration that's after the reboot, after the first reboot so that's problems with things like task cell. You can have problems installing tasks because dependencies are broken and that's basically been the problem for the last two months with the major transitions going on and things depending on each other and not wanting to be installed and a totally separate type of problem is internationalization problems and those can have them in either the first stage of installation or the second stage. The environment for languages is fairly different between first stage and second stage so often we see that something goes right in first stage but it fails in second stage and especially when you have languages like Arabic where there's really different issues than normal Latin based languages, well you can have very funny things happening. DI is really a complex environment and should be seen as a miniature Linux distribution that has to support all 12 architectures on it and that's a challenge in itself. It really is a full distribution because it has its own kernel, it has its own environment because you have the microdabs, the U-dabs instead of normal dabs so we have the problems associated with that. We use library reduction to make sure everything fits in a small space as possible so we can do floppy based installations and no-memory installations so there are different challenges in creating a DI distribution if you would like to call it that from doing a normal Damian distribution. Having all architectures means you also have a great variety in stuff that you have to support like different disk labels, different file systems, different bootloaders, stuff like that. There are some components we control ourselves, there's quite a bit of programming involved in DI mostly scripting but also some scene, some other stuff. A lot of configuration, of course, but others we do not. Well, if something breaks there we often are dependent on the maintainer of that part of their being to provide solution and very often you have U-dabs, the stuff we use in the installment that are really produced as a byproduct of normal pages and that means that very often the U-dabs and what happens there is not really in the front of the mind of the developer. At some point we will have asked him, OK, could you do this and this for us? But it's not really something developer think about when they are preparing changes or releases and especially when there is major upstream changes. We are almost never informed, OK, this is going to happen in that version, please prepare for it. We have a complex build environment and that includes again both building the DI images themselves and then later in a second building the CDs. And I don't know if anybody has ever looked at them in CD, but that's very interesting as well to configure. And there's a lot of ways you can go wrong. And also DI builds for different environments. And then the basic daily image that we produce are using in-a-tardis and U-dabs that are built in unstable but installs testing as soon as we have a mix of environments there. When we go towards DI release in testing we will switch that to using U-dabs and in-a-tardis from testing and still installing testing. So there's a lot of configuration going on there as well. The reason for that is that normally when you are doing daily tests you want to be as up to date as possible so you want to use stuff from unstable that has been developed for DI itself. But you really want to install a stable environment that users can actually use and for that testing it's better, especially now the security support is starting to happen. But when you start to prepare for batteries, well batteries are supposed to be based on testing because that's the more stable environment and we can still continue doing some development in unstable while we're preparing the data release. And DI is also a fairly complex operational environment. For instance we start with an in-a-tardis that is based on a bunch of U-dabs but then we have to add other U-dabs to that either from CD or load them from the network or whatever to create a full operational DI environment. We have to configure the environment for either framework operation or for done terminal, all sorts of things like that. Often you will make a change in DI or it will be a change in an upstream U-dab that will work for one environment but that will break something in other environments. If there are questions feel free to answer them. So why does DI break lots of relations? Really too many to enumerate. We can of course break DI ourselves which is the easiest case because if we break it ourselves we almost certainly know why we broke it, how we broke it and hopefully how to fix it. There are new upstream releases that can cause breakage. To give a random example there was a change in D-package very recently which was the result of a witness buck report submitted by someone whose name I won't mention but who's a ridiculous DJ here. He requested that custom headers that you can use in Ups, in control files. We use Debbie's folder menu item to determine the order in which things are shown in the Debbie's folder menu. He requested that those be capitalized, the first letters. DI used lowercase and the software built in the menu only checked for lowercase. So a new version of D-package grown complete manufacturer of DI. It took some tracing and debugging to find what cost it. But now of course you have library changes. There was one I think dated for yesterday that of course Joey has some headaches because well he said there's a new version of LIP wireless, LIP IW. So he decided let's try if I rebuild and if that solves the problem. And then he forgot to copy the Udips that he created that way into his build environment. So he tested with the old software anyway and well he spent a lot of time debugging the problem even when a simple rebuild actually did solve it. But stuff like that happens. As I said earlier, transitions can hit the I fairly hard. Especially in major months of life we've had both sides. We've basically been stopped from doing any serious work of the last period. Because there were so many changes waiting for things to get into testing. For example we have switched for I386 and Spark now to the I864 as well I think. But those terms are still not in testing. We will use the 2612 kernels in the installer. But as we install testing for the installed system, they still, the data build still install 268 for the final system. So if you have a system with a SATA controller, which support which was broken in 268 or at least it's very incomplete, you will have perfect installation. But when you reboot you get 268 kernel. And, well nothing happens basically. You get a pivot route out. So that's really blocking. Another very nice one is kernel configuration changes. So when a kernel maintainer decides, okay this module was compiled in, let's modularize it. And we won't have that module in Europe yet. So basically it's not available during installation. So that can break networking for some adapters. Often we have to wait for installation reports to tell us about it instead of... But that would be something that's very difficult to solve. Can you be able to make this change? So you can say okay we set that unit and then it should go. Yeah. Tell us that would be very nice. But we've seen in the past that... Should make it possible to track the config file? Yeah sure it should be. And maybe we should do that. But we have to check anyway whether there's changes for formative kernel releases. Like when we change from 268 to 2612. But that's really funny thing going on sometimes. There's also, for instance, if highman support is enabled by a kernel maintainer. That won't have direct impact on the UDEP situation. But it may break the eye in other interesting ways. So it's often very subtle what happens. Another thing that we are looking into together with FTP buses and release people is that currently we have to migrate UDEPs manually from unstable to testing. Which means that often the package that the UDEP is derived from is already in testing while the UDEP is still in unstable. And so we have an old version in testing. And in some cases that can cause problems. But is the moment packages produced in UDEPs are blocked? Yeah, but there are, if at the moment there is a request to the release managers to move into testing it will almost certainly just happen. But with the data. Come to the email services and say who's the UDEP. The first request will be from the maintainer of the normal package to let the packaging develop automatically and at the same time let the UDEP go. What I do is if I get a request. I ask them if they are good as they are. I have a request and if they don't get a normal and they follow the 24 hours so I just do it. But the next thing is for example if I am not a package it might still be blocked by other reasons like unassolability. So all in all I can do is successfully get to testing. I can ask an FTP master to all of my greats UDEPs. So at least one day it will be different. In some cases it will just have to be more than one day because it will be a business the next day or whatever. It's something really should fix because it puts a bit more and the most is more than the least of the FTP masters. And we will have a nice one going into the play. And what also is an issue there is that UDEPs just like normal packages have version dependencies. And version dependencies in UDEPs is a little bit less strongly defined and controlled than it is in normal packages. So we get all kinds of minor temporary issues from moving things from unstable to testing and I think that we have to be alert for it and we'll have to solve some parts. And that's a very nice one and that's the build dependencies that TI has. This gives you a nice overview. That's the first page. That's also the largest bit, but I left out all the version dependencies but you can see how it covers all architectures and what the differences are. And well, with many dependencies in the build process it also makes it clear that something can easily go wrong if a new version comes in or not helps. Packages. The second page is actually a bit special because those are dependencies that are only relevant for building the manual and not really for daily builds. So they are only used for official builds though where also the DIG or the installation guide is built. There's a very nice one here that often causes a lot of laughter because that's actually part of KB which means that the build environment in a stable was unusable at least if you were not a little bit creative with one live from a stable to a built machine because KB was in transition and we prepared for the G++ transition. Do you think they're migrating to PO4A? Well, I have looked at both POXML and PO4A when I was trying to decide what to use for PO translation and there is one major difference between them is that PO4A will break strings that means have a footnote in them if you have a paragraph with a footnote in the middle of the paragraph PO4A will take the first part will take the part in the footnote and will take the second part of the original sentence as separate strings. And I don't like it. You can specify some text to ignore. Okay, well, I've looked into it. We would be happy to help you on that topic. Anyway, it works perfectly of the original, but sometimes... Why does the ID bent on KDE? I'll walk you through another program from the console that the Kenton KDE list. It's only one lib? It's only one lib. We do. In fact, it is. Because that package is maintained by a great partner. So we can use one in a temporary fashion to develop a package for us. It was probably the one lib from Stable and everything was fine. It was just that lib from KDE was only removed from Stable. If you were just keeping track of Stable, it was really no problem because the lib would stay around. But if you were creating a new build environment, it would just say, okay, how can build this? Yeah, right. The lib would be missing, but it was such a minor problem. There was only really one portal that made an issue of what the app says, okay, let's install that old lib and not worry about it. That one guy actually deleted the building that was seeking temporary. Just do it. But it's just an example of how things can break. So how can we manage it? Well, we try to revent breakers as much as possible, but it really is something we cannot do. So the alternative we have is to detect it when it happens. How do you get this? We have daily builds for all architectures of the I. Using the latest UNEPs that are in unstable. The page that is linked here, I'm going to switch to that browser. From the other desktop. All right, so I do... Sorry. There was a panel at the top, so just to let you cover. Move the mouse to the top. There is... No, that's not what I expected. No. Otherwise, it's too loud. Just the app for P2. That's right. The app for P2. Shout for... Shout for P2. Go outside, Peter. Come here. You just have to accept that. Yeah, I'm scared. Franz, just go on as soon as Peter is here. Yeah. Anyway, I showed you the page in a moment. The main thing is that Joey has set up a test lab at his home. You can see he has currently six architectures there. And another two are being worked on. He would like to ask about how we see it. It used to work with keyboards, until you decided to... No, it doesn't work anymore. OK. No, decided to find a keyboard. Are we starting now? Smaller. Oh, this is nice, actually. Because yesterday, all the successes were failures. Bigger is bigger. Control plus... Yeah. So yesterday, all the successes you see here were failures. Because of that library problem I told you about earlier. So you can see that just fixing it, or just looking at this list, you can very quickly see if there is a problem. And if I open, let's say, Spring 19, if I open the link here, then you get the build log. And you can see this simple case of a built-in latency, not satisfied. So this really helps us to quickly see what's wrong in the daily builds. Is there a need for a minutes build? Or minutes... Because... On your page, you have something about that. On your PF slide, there were only... Oh, no, that's for the test lab. The builds are covered. Ah, okay. But you need to store ribs for the test lab? We could look into that. That's true. But it still has to be working as a hardware factor for the better devices. He works on ARM. Ah, okay. Today, this was prepared to more... architectual attention. Sorry? What was prepared to more attention to this lab? Yes, it is... Well, he has common running, the hurtless emulator as well, but so that kills off S390 test lab. But I can do that. I can regularly do that on my laptop. And S390 isn't that important anyway, so if it breaks... There's not many installations happening all the time. And obviously he doesn't have it in his test lab. I think he was going to get the machine from a simulator, but I'm not sure if that has happened yet. But that will be... You know how it was in 1664? Sorry? I'm trying to go back to... Iron's brand. Sorry? Iron's brand? Yes. Iron's brand, right? Yes, America's brand. So, the second thing is the test lab, and that also has its own logs, so let's do the same thing again. And that's this page. And you can see there's quite a few failures. That's just the names of his machines. Oh, okay. He's the destination names. Chicken and what? Donkey. He was using animals. He was using animals as well. And you can see there's quite a few failures here. Just look at it. Six failures. Success. I think you can also look at the success of that word. No, that's not the case. He works in... What? Well, he works in Conqueror. He's a bad model. No. Let's forget it. One thing about that thing is that they are a lot harder to read than they'll want to be, as case sequences for the graphical environment end up in so much. This GNOME environment is... So another way we find out about breakage is through installation reports. So we really welcome those. And we are sometimes told about things going wrong on DRC. Because of, especially the daily build structure and the test lab, we will often notice things breaking before we are noticed as noticed in the general gaming environment. Of course, we won't know the breakage in stuff like office packages or graphical editors and all that. But if there's something wrong in boot loaders or stuff like that, we will often be the first to see it. And that also often means that you will have to track it down. So how do we do it to fix breakage? Debugging and tracing, of course. If we find out what's wrong, report it. If it is something that's under our own control, well, we can fix my load relump. If not, well, then it will often take a lot longer to get something fixed file back. And how to maintain it? Keep after them to help us out. But there's quite a few packages that are not really maintained. And in those cases, we often have to M and U. For instance, the bootstrap. And we don't have mainlips, syspeed in it. All kinds of basic stuff. Well, we are very lucky to have Joey as the team because he knows Demian so well that he can often judge what is possible and what is what's safe, what's not safe. Well, of course, we test things locally, which means manual builds having the right environment to test in. So you really need the architecture if it's an architect specific problem and the test lab comes in. And then you have to wait four weeks to make it to the daily builds. For some times, start of problems, it's really outside our own knowledge and we have to get help from quarters or specialists. Internationalization problems are often like that, especially when it's to do with character combining and weird stuff like that. There, there, we have Christian's variability coordinates that part of the eye very heavily and does a very good job of that. So, this is my last slide. What do we do to keep everybody informed that this is something we have actually added quite recently because it was breaking so much that we thought, well, we have to do something about it. Isn't it a wiki.wm.arch in the meantime? No, no. We are waiting for the transition. Why, everyone? You mean the homework? No. The D-I-T. Yeah. I'll go there. So, this is the list of problems, issues, basically. Well, the top one we can remove now, that's been fixed. We haven't had CD builds for the last week or something like that because we need to update something and it's a monthly list that the day-build coordinator is really busy with university work, of course, just at the moment. So, he does have time for updates. So, Joey has decided to update himself, which was perfectly all right. But the problem is that the scripts that Monty uses to start builds, check, and defy some of the environment and, of course, that fails. So, now we're going to have builds, which make testing a little hard as well. But I think that it's flexible, we'll fix it. So, any questions left after this? No questions. I have seen this. Yeah. Don't continue. We're going to switch to CD build stuff. I think we already used it, but I'm not 100% sure. So, D build stuff is more than something that we might remove Yeah, I would guess so. Maybe even French, but that depends if everything that uses it, including D build stuff like that, D build is already switched. Or it's already switched, yeah. It doesn't work either, so... Yes, I'm really excited. We've worked on this like four times. Sorry, everybody. This is from your point of view. CD build stuff better? Well, D build stuff needs a list of packages to install pair architecture. And CD build stuff will analyze what's required for standard information or for imports, something like that. But I don't know the details. That's what I've understood from hearing things. Yeah. That's the basic difference. I'm wondering how the... who test that works, how do you automatically test these or how do you do the second stage? Joey has set up a power control center using some home device that's available and that can be controlled through serial interface. So he will switch on the power for machine that way. And everything he does is network installations. So he will just make sure that the correct image is somewhere, make sure that he has scripted it well. For the details, I can give you a link where he explains the bit. It's a number I know. Certainly nice setup. I can imagine. Okay, other questions? Right, thanks very much for your attention.