 Hello everyone and welcome for the last talk of this morning's session. Cyril Brutbois will present us news from the Debianny store, so please welcome him. To divide this talk in many two parts, the first one will be some kind of retrospective of what happened during the stretch of this cycle, and then we'll focus on what is currently happening for the Buster cycle and what is planned, what we are trying to smooth out for the next release. So let's begin with stretch. That's not the first time I'm giving this kind of talk, and usually I compare the release cycle to the previous one. For Jesse, we had one alpha, two beta releases and three release candidates before the actual Jesse release. It was already much better than what we had for Wheezy, which had fewer releases. The timing wasn't that good at the time, but we felt there was some kind of room for earlier releases and more releases. I mean, once the cycle has started, it's not too good to wait a few months until you release the first alpha because there are so many changes piling up and when something breaks, you cannot test anything because everything changes from the past release. So for stretch, that was really much better because we managed to squeeze eight alpha releases between July 2015 and November 2016. So the distinction between alpha and beta was really artificial. That was just a name. So we simplified these bits and only called those alphas. And then release candidates, when we are reaching something that shouldn't move too much, especially when the free starts, we are calling the DI releases release candidates because we are not supposed to do that many changes at this point. Of course, that's what should happen in an ideal world, but there are so many great changes and last-minute changes that we need to squeeze in every time. So that's why we had five release candidates between January and June. And yes, we did some release candidates just a few days before the actual stretch release because there were still some moving parts, even if we don't like that, but that's like. Let's concentrate on its selection images. One of the big changes is that the full sets are gone. You must ask why, but then I've got numbers. We had up to 80 series like for MD64. On the last bit, in 2015, we had 854 images which piled up to several hundred gigabytes for each DI release, so times six and now 13. So that's a big storage space that wasn't really useful because we don't have any stats for downloads, but you can imagine that if you need that on CDs, maybe DVDs are better bet. And it seems that we are moving towards more network mirrors and you only usually need a few packages which are on the first CD or the first DVD and we don't need that anymore. We had some really scary stats as far as building goals. We already had 12 to 24 hours delays between release and the publication of the installation images, which breaks the momentum of it because hey, look, it's released, but you cannot install it because we don't have the installation images yet. So right now, the only CD option is a network-based install which is not a net boot. Net boot is for booting for PC, so that's a different thing. And we kept a single extra CD which is called desktop-available, which is XFC. For all the others, you can select the desktop you want to install directly from the installer since JC, I guess. Previously, you had to type some boot options to obtain this or that desktop. So now that it's easy, we only focus on that single CD. On the port side, mid-64 EL was added and poor PC was retired because nobody was maintaining it, so it bit was a bit. For unofficial ports, we are now having some more machinery to allow for building against the Debian ports archive which makes the testing and installing easier for people working on ports. So that doesn't change anything for the official arch, but that should add them a bit. We had some change on the multi-arch image, so MD64 and Intel in 32-bits mode. We are now defaulting to 64-bits, which is, yeah, except it's 2017, so it's a bad time. We managed to do that with some SysGlynex trick. We have a small module which we can use to select MD64 when it's available on your CPU, otherwise we default to 32-bits. So we did a two-part update with that change. Another default change, you might be aware of one of Manuel's escapades, and there's a famous strip for people familiar with the Debian Boots RAC channel, which is this one, when globally you would be asked to configure your system and select some Bosnium, which is apparently a bit more upsetting than choosing your daughter's first name. Basically, we are familiar with that blue screen and grey and red, and that's about it. But now we switch the default and we are installing with a graphical installer by default. I mean, it's been many years, like 12. It was first released in 2005 with Edge, but it was only a technology preview. But it remained this way until this release, because we never flipped the switch. We can call that resistance to change, or just lack of opportunity, or we didn't think about it until it was too late. That's what happened to Jesse. So finally, at the very beginning of the Twitch release cycle, we flipped the switch and we are now getting the graphical installer by default. What is it about? Basically, AI is driving depth comp and prompting you for answers and presenting you with choices. Basically, the text installer and the graphical installer are working really the same, except we can do more things with the graphical installer. Especially with translations. We have many languages which are either not available in text mode because of missing fonts or it's not possible to render them or not render them properly. And we can have a few interesting things, like with Hebrew, which we are going to see on some screenshots right after that. So that's Hebrew with a text-mode installer. So that's a right-to-left language, but it's displayed on the other way around. The progress bar is going the wrong way. So that's with the text installer. And with the graphical installer, everything gets back in the right direction. I mean, for them, because we can have some better rendering. Like for example here, a version. I'm really not into languages. I mean, I speak some bit of German and that's about it. But you might think that maybe we can understand which glyphs and which characters are displayed here, but they are only individual characters. There's no ligature or liaison or whatever that's called. It's only character next to each other. And now we are getting proper rendering with GTK. Everything is smooth and you get letters combining with others and so on. If you know more about those languages, feel free to educate me so that I can give a better look next time. We've got Punjabi. Punjabi is not available at all for the text installer. It's not about better rendering. It's not possible to display this language in the text installer. It's a bit of a shame because there are like 100 and 100 million people talking this language. It's a bit of a shame not to display it. So with the graphical installer, we can finally reach more people. And that's the same with many languages in India. Small languages have millions of people. So it's not really French and Canadian French and so on. It's a bit on the larger scale. There's also Sinhala, which is another language from Asia, I guess. So how does this work? We're using GTK for the rendering part. Progressive that was working on Direct FB, but that technology was retired and deprecated and with many bugs. So a long, long time ago, it was proposed to switch it from Direct FB to XOR. And somebody called for help because the graphical installer was never released, officially. And it was already on the way out, even with all those maybe not marvelous but interesting features. So I knew nothing about installer. I knew nothing about the Direct FB or XOR. So I joined Debian Groups at this time. And basically, in one week, I had some kind of a working proof of concept. If you're interested in details, everything is on my blog. There are four blog posts, so it's quite easy and quick to read. And that's basically the reason why I then maintained X with Debian for a few years. Basically, the one to claim is Julien, who said, oh, you're a grown-up. So please come in your pathways and please join the team and tag your it. I'm going to do release work. So thanks, Julien, that was really interesting to get that kind of a chance to join the team. And a few years after that, we had nobody left working on the release management part of the Debian installer. So hey, you know something about Debian installer, don't you? Yeah, so please go maintain it. So that's why I've been having a step in the release team and in the DI team for a few years now. The other person I would like to thank again is just now, who is the person who called for help in 2010. So everything started because of him and I'm really happy that it turned out this way. I'm going to talk a bit more about funds. We had a big change in the CGK world, CGK for Chinese, Japanese and Korean. That's a group of languages with similar problematic forms combining and so on. So before we had an interesting package, so TTF for two type funds, CGK, easy and compact, because we were extracting many funds, one for Chinese, one for Japanese, one for Korean. And then we were extracting from those funds which are a bit huge, only the place we needed for the Debian installer. So we needed to have the list of all the lists, so all characters we needed to render and then extract every rendering, every images from the fund and then combine it into a single package which would be smaller than what we had before. And of course, translations evolve. You've got characters disappearing, orders appearing. And in the end, you've got query images and everything is working. So thankfully, I guess Fonts on the Red is a Google font, a primary really targeting font and so on. But it's also available in Debian and we used its UDED, so it's micro Debian package to get a better coverage of all characters we needed. And we're no longer playing the game, I'm going to extract a few bits and then everything breaks every few days or weeks. So fragility is gone, no missing lists, so everything is good, right? If you look at the screen, you might notice what is happening because that's the selected entry. We have Korean with squirrels, so that shouldn't happen because we've got the whole package with everything inside. So what is happening? Maybe a few characters are missing. Now, if you pick Korean, everything is working again because hey, that's a package. It has updates and sometimes the updates are working at home. So we need to keep an eye on what is happening with the rendering, especially for non-lighting fonts, but also for lighting fonts as well. And once the package was updated to use a different font file, we've got proper rendering in Korean. I mean proper, it looks better, I'm not sure it's correct, but it's definitely better. On the artwork side, we had Juliette get selected again for a new artwork, which is software use. She was already selected with the lines artwork for Jessie. On the integration side of things, we've got desktop base, which is a command package for all desktop environments with lock screens, wallpapers and everything. And for the Phoenix Solar, that's WordScale GTK, which is of course used for the GTK part. We've got the banner at the top of the screen, shown during the world installation process, but also bootloader parts with greetings and so on. I'm getting back to that a bit later. So if you didn't install Stretch or upgrade it, well, that's the default wallpaper, lock screen, and I'm going to talk about networking. We got a proposal quite early in the release cycle to switch to another kind of way to name interfaces. So some people disagree because it was pushed by system-based people or whatever. I don't really care. All I care about is that the installer system was still using the old naming, so 80H0, while the install system was using ENF3, which is a bit annoying if you configure the install system with the old name. So essentially that was called during testing, I guess, for Alpha 1. And Martin Pete and Michael or Michel Biber were kind enough to provide with fixes and that was a near miss on the releasing side. But hey, that's on Alpha. That's what it's for. Then I'm going to talk a bit about priority. Basically, when you install Libyan, you get task cells. So you select the tasks that you want to install, which are basically meta packages to install desktop or everything related to French or to English like dictionaries and so on. But we've got a set of basic packages, which are fixed based on their priority, so you can have required packages or optional packages, etc. So Ansgar proposed to remove quite a bunch of old packages from the important or required set so that the images would be smaller. That's a case for installation of wear metal in shoots, in containers, in VMs, and so on. So maybe you are going to lack this specific demand that you've been expecting and using for years, maybe dozen of years. But basically we are reducing a bit the minimal footprint of Libyan installation. At that point, it seemed that maybe we could have some fun with the init system, what is system D going to be installed anyway, and so on. And everything was prepared with the init system address, I guess, package. And we had nothing to change on the Libyan installable side. Everything was handled by the FTP masters who are the final reviewers or the actual priority that we put into packages. Then we've got HTTPS support added. We had to get a way to actually talk HTTPS that was done by creating a new user for the WGIT package. Before that, we were using the Vezvex WGIT. I'm going to get back to that in the second part. And we had to trust HTTPS, so we have the CA mafia and so on. But we have to trust a bit the other side, so we have an extra user for CA certificates. So Marga and Philipp pushed for this, I think, to remember because Google was interested in getting more HTTPS and less HTTP. But we had to adjust some bits, because even if you force HTTPS installation, depending on the installation image you would use, you would be missing act, transport HTTPS or CA certificates in the installed system. So we had to fix some images to make that actually working. But a question remains, how do we trust HTTPS mirrors? How are we getting our mirror system to work with private keys and so on? So that's not a huge issue, but it's not entirely transparent. You can just flip the switch from HTTPS to HTTPS on any mirrors yet. Some work needs to happen on this side. So that's it for stretch, that's a really short presentation because there were so many more changes, there are so many people to thank about it. Sometimes I identify like the guy guy, now there are dozens of people pushing many patches, so that's only a very... So let's look at Bustina to do work which is carried over from release to release, especially secure roads. Now it's not possible to install Vivian when secure roads is enabled. And that's a shame, it's been the case for way too many releases. And we need a user to actually disable it, which means setting some kind of root password for the buyers, no UEFI, and then disable it and then maybe unset the password because you're going to forget it. So it's a bit of a heavy process. The solution is known for quite a while. I guess the plan was already outlined in 2014 in Cambridge Media Development. So we already have signing keys for secure roads. We need to update the infrastructure to actually use them and that's the heavy part because we've got FTP master and some packages need signing and we don't want the key to be stored just as a fly, so we need some kind of crypto tokens and there are so many moving parts and so many teams involved that it's not there yet. I'm not going to throw rocks at anyone, it's just not happening yet, which is sad, but hopefully at some point that's going to be a... yeah, mic. And once everything has done, maybe we'll need to adjust the AI a bit, but probably not, it's just about getting the actual packages signed and everything should just fall into place. So maybe we'll be able to backport part of it for stretch just by picking a few packages and adapting a few things. So maybe through some kind of point release we could have the official visual images and some testing once with secure road enabled and see if that works well. But what's up? The GTK side, we'll be lacking a bit because we're still using GTK 2 while GTK 3 has been first released a few years ago already. I actually had packages for it as soon as May 2013, but we needed to introduce new UDEPs. So the GTK in the 3 variants, a needed KRO check and then we discovered that we were missing the 80 SPI grid and then LibEpoxy, what is LibEpoxy? It's basically a wrapper for method functions. So then we were... with everything in place, every packages having UDEPs at home, we were failing at runtime because we were missing MESA, which is a huge library and we definitely don't need any 3D inside DI yet. I mean, we are not going to play any games here at this point. I'm glad I found so many new contributors. And fortunately this issue with LGS being mandatory just went away. A bit before Stretch was released, it was expected to look into it during the Stretch release cycle, so it's going to wait until buss up. So we need to clean up the batteries that shouldn't be handled, but we need to port how many we've worked out. The artwork is integrated, like how to highlight in this or that color, how to set the background colors and so on. So maybe write some JavaScript, and I don't want to go yet. Another thing also for the graphical installer, we are using default input modules, so FDEV for everything like keyboard, mouse and so on, and separate components, so keyboard and mouse for 3DSN and HURT. And we should be able to use, at some point, libInput, which is kind of a merge or a new way to handle keyboard, mouse, touch pads, which have been quite a pain in the DI context. They were not working during the installation, but they were working once the system was installed. So hopefully everything should fall into place with this new way of configuring inputs in the DI. We've got some booting improvements suggested by Steel. At this point, we are using grub or maybe syslinux or irolinux, you can call it that way. And the plan was to get rid of everything and just focus on grub, which would reduce the surface of X and maybe make it maintenance easier. So, Steve will be working on this, right? Talking a bit with others who didn't want me to prepare this slide, I came up with a few interesting suggestions, like speeding up the installation process. We have many, many, many fight system options. You can disable some safety stuff, like disabled barriers or maybe set some white-back stuff and so on. But I'm not a kernel developer. I'm not a fight system expert. I don't want to become one, because life is too short. So, the big issue is that these options can speed things up, but also alter data safety. Maybe you are going to write stuff in a wrong order or not save it the proper way and so on. The first tool is to use those fight system options which are really making stuff easier, but only during the installation process within the installer context. And you just forget about these options when you boot the actual system. So, that shouldn't be too hard from a safety point of view, while still allowing for speed ups. So, here's an example which I'm told is working fine with electricity suppliers somewhere in Europe. So, data white-back and some asynchronous stuff and no barrier. This came from Rafael. So, you can draw any comparison from it. I'm leaning towards pushing for LGM a bit more because there were many objections. So, there's no need to address that to questions at the end of the talk. We've got a thread going on on the devian boot mailing list. So, if you want to voice in or out of it, that should happen there. From my point of view, LGM makes it easier for people to actually change partition sites afterwards because once you've got the DOS or GPT setup, it's a bit trickier to reduce some fight system or that other one. Like usually, the root fight system would be too small and when that happens, it's a bit a mess to actually make it larger. Maybe you've got to move some directories and then bind them. It's really tricky. So, that's just an idea. Feel free to follow. Another one is speeding up the debut sub-phase. Basically, on Intel hardware, it's going fast enough for casual use, but once you've got some ARM device to set up, it's a bit lengthier like you take one coffee and then ten and then it's still not finished. So, we could maybe stop looking into the available packages and then computing with dependencies and then fetching packages one by one and maybe failing and then everything breaks and you've got to start over and maybe just pack everything together with maybe some kind of tower and then download that and then install from it. It might not be perfect, but I guess it could be an option that some people could use to speed things up. So, we could benefit from it in the installer for cloud images because people building cloud images are also running debut sub-phase and when you build custom images, I don't know, for some customers you may want to build a device based on Debian and not actually run DI on it, but just copy wood partition over and we could benefit from it in various areas. So, this was inspired by Aurélien. And I said that I was going to get back to Vizibox. What is Vizibox? It's a tennis set of tools for units. You see here everything that we use from Vizibox in the DI context, so many basic commands are available there. Unfortunately, we are lacking a maintainer within Debian. It's been the case for at least one running cycle which is really way too much and apparently, Optimum is active and friendly and responds positively to bug reports and enhancements to questions. So, it would be nice if the collective view would follow up on the request for help that I opened a few months or maybe years ago. We have a component which is called OS Prover which is trying to pro for operating system looking into devices, disk partitions and so on so as to detect what which other system might be installed so that it can fill the list of extra entries to the boot loader. So, you can boot your Microsoft system or Fedora or whatever. The problem is that probing should be like I'm looking into it, is there this or that? Okay, I'm doing this and that. But the current or at least the previous code for probing was having some kind of side effects like corrupting systems which is not to go. If you had some kind of virtual machine you could corrupt stuff on the host which is really bad because you can then pollute all of the apps. So, there were many really critical bugs on OS Prover. Colin is looking into 2.0 version of it and he wants to make it testable because every time we wanted to fix a bug we would break another thing or it would slightly change another output and so on. So, testing is important. On an implementation level he wanted to make it decorative so instead of having way too many shell scripts and fragments calling each other and then taking sometimes up to minutes on a real modern hardware that's really a pain at the moment. So, make it declarative so that it's easy to write a detection code but also quicker by not using so many fragmented shell script parts and just having some core configured with these declarative statements. So, if you want to learn more about it he posted some detailed plan so that sells OS Prover 2.0 again on the Debian Group meaningless. Speaking of testing we are currently relying on actual people which is great but a bit of work and some metrics of everything that many people tested on release day or night or was it day? I don't remember because it was lengthy. It was both. It was both, yes. Several times around, I guess. So, it's really, really great that we have so many people testing actual installation images and finding bugs so that we know a user can know that this image cannot install this and that combination and that order and so on but we are already past the point we already released a stretch we copied all the stuff on FTP Master and then we built installation images and if there's anything to fix it's already too late we can just document and what doesn't work so the idea is to actually test stuff in advance and basically I'm usually when uploading dependent installers the source package I'm doing a build and then testing in QMU or LeadBuild a few things like basic installation next, next, next, next, next, next, next and then, oh, but you've got to type password and user and it's really boring I mean, really, really so I wanted to script that I knew that Phil had been working on some kind of automatic testing stuff and so on I honestly didn't see any summary and so on and I was like, I know exactly what I want to do because it's exactly what I'm doing right now for a few years so I wrote a tiny tool called nickname Kdit because no imagination and so on and basically it's driving the installer to X11 events so you start QMU or LeadBuild and you just select a playbook which says I want to make sure that this screenshot is shown and then I press this key and then I want to make sure that this screenshot is shown or this or that one, or that one and then I do this or that and that makes it way easier to actually test images because you've got automation even if I'm really aware and only focusing on some installation process it would take up to 10 or maybe 15 minutes to go through everything with this tool I got in solution under 3 minutes so you can test a lot of stuff and in some kind of automated fashion at this point it's not integrated into check-in it's not run automatically but I really would like to make that a reality to actually make it easier to spot regressions because right now I'm finding them when I'm about to upload a new release so we've got some minutes left so I'm going to try some demo because hey so I spoke about Japanese so maybe we can do some short demo with Japanese so that's a network image with GTK and I want a Japanese version so it loaded a playbook which is a textual description of every step we need to undertake we've got a few screenshots which are expected at each step and once the screenshot has been obtained we then respond by typing some keys and so on so it's going a bit quick because it's polling it's taking screenshots quite regularly and once the wanted image is shown it just type and confirms the screen and it's going on and on and on I'm not going to do the whole installation process because it's a network image so everything is going to be fetched from the network and it's going to take time so I'm going to just kill this one and start a different one like Alvinia you saw the selected entry go down and up just to make sure that it's not totally broken because we had SysLinux fun quite a few times already and then this time it's a whole CD so every component should be already available except for packages which are only stored on security and as you see there's no need to actually interact with it HTTP is going to work directly I'm typing autotest which is nice because it tests whether other key and quality are working fine which is not always the case and then we reach the part-man menu and then we are switching from the top entry to the middle entry as well as to enable LEM so the installation is going on I'm going to show you what the playbook looks like on the left we have a basic install with several screenshots which are possible so ISO Linux, higher menu, higher menu ISO 386 and some unofficial stuff because depending on the image you're using whether it's official or not official and with some late changes as to how the menu is shown we can have either of the four screenshots here and you see that in which spawns we are sending the down and up and return that we saw on the first screen then we are working on the language screen I set an extra key to make it possible to accept some kind of diff between the wanted image and the one we got because the scroll bar on the right of the screen is not exactly the same every time like when you fix Korean it was broken with squares and then it's shown properly but a few pixels off so it breaks so you've got to tolerate some differences for example when you use EFI and not traditional QAVU you've got some slight rendering differences like a few pixels with a few reds or blue changes so we've got to accept some differences but not too much because you don't want to accept any screen anyway so we've got some 20-25 steps with some input like the name, the password and so on you want to set and it's a bit boring to obtain this so I wrote a tennis court to record what I was doing so that it can be replayed afterwards and once you've got the basics you can then inherit from the basic install that's LVM on the right and it's the whole file for LVM and I'm just changing part menu to insert to change return which is accepting the default to I want the next entry and then return of course we have a few differences in the next screenshots so there are some other stuff to change and we've got to confirm LVM is set up as we wanted so there are some changes removed or added steps compared to the basic installation so in a little while the screen wasn't rendered so the screenshot was failing but then it resumed when I switched back the focus on it and then everything is working because I shouldn't have switched from one desktop to another but hopefully with XFIRE, XVLB and some other technologies to decouple the rendering and your actual work it should be possible to see what is happening but then not disrupt everything so LiveDevo and XFIRE Playbook were shown we need more Playbook so I need to document and then show a bit more what we can do with it it needs to be integrated into Jenkins or whatever system to actually get test run and report it because for example we have shoot installation and upgrade tests for tasks like desktop, XFIRE, GNOME and so on but they've been working in May and June right before the release, we didn't notice thankfully there was some kind of BSP and one guy said oh, VLC is working and everything falls down but we need to be more proactive and the running test is good but checking the results is better so we need to do something and this kind of tool need to be integrated with that kind of thing in mind so I'm going just to show you the style and not really talk about it so that you have some tips and tricks if your package ships a UDEP it's going to be used usually in the deviant server so if you change your packaging and that affects a UDEP we really would appreciate if you talk to us so deviant boot, before doing your change you can contact me directly because I don't follow everything going on on the mailing list and that's basically about it if you found affecting the deviant server in some other package please put deviant boot in copy of your report so that we know about your issues and that's about it we are out of time for real life questions there was one question on IOC about this automated testing stuff do you have any plans for doing testing on actual hardware and contrary to that remotely? not at this point I guess we would need some kind of robot to actually input stuff in some keyboard or doing stuff over a serial console and then you're not actually testing the interaction through the keyboard and so on you're testing a solution over a serial console but maybe Civo would be... yeah exactly so in Lenovo and Lava we have been doing some testing or installation or whatever and in an on-wheel hardware that is exactly the problem actually picking up on a wheel installer not just the one on the serial console it's much much harder to do well thank you thank you I just remembered the things that I couldn't remember when you asked me yesterday we were going to do regular back port builds we spoke about it last summer and I told you about all between every one out of time to make it happen I would love to get us having a regular back ports build every couple of months because we just need to get around to actually do it yeah help would be wonderful people thank you