 Okay, so our next talk is by Holger and yellow on reproducible builds. Let's welcome them So this is yellow. I'm Holger. We talk about reproducible builds aiming for bullseye Bullseye is the next stabby and release after Buster because Buster is frozen and boring Reproducible builds have the purpose to enable anyone anyone to reproducibly reproduce identical binary package from a given source That's what it's all about and the rest I save you because I guess most of you have heard it Yeah, we have some project goals We want to change the meaning of free software also that it's only free software if it's reproducible because then you can then you Can only be sure that the binary is really coming from the source We gave many talks at many conferences, so there's also available with more background information And longer ones and since this is a Debbie and mini-deb corn we compare Last time we compared from Debcombe 17 in Montreal now that the changes since last year in Taiwan We finally have a logo which was a year ago. It was still planned We had the force reproducible builds summit in Paris will have another summit this year as well The location is still to be determined Um, we are now a software software freedom conservancy project Since last year in August or something So that's we have some funding from some Company some initiatives and some people of the project are paid from these resources not everybody and Besides Debbie and Arch Linux is about 80 percent Open Zuzer 93 We've now included the results from open Zuzer and Arch Linux into our database which where we have the Debian results Then we will generate web pages similar To the Debian ones in future Since yesterday Alpine is being tested KPC YRD has added that he's not here yet Net BSD and free BSD the base install is 100 percent reproducible the port system we are not testing yet and the tails ISO image have been made reproducible and Most OpenWRT images are reproducible as well and We do a lot of collaboration and it's really great because when I came here yellow was here and gave me this nice sign Which is there and it's lighting. It's green because the Jenkins maintenance shop is successful It will turn red if it's not successful Okay Okay, thank you, so I'm here for I'm happy to be here by the way Hager invited me so I could join the reproducible build sprint this Week and we got actually got some work done to further Get us nicer in the reproducible the belts website and into the testing infrastructure So at the moment In the test structure framework, which just builds two packages and then compares the results We are about 80 percent reproducible And our thing is a bit different from Debbie and we have three repositories so actually four and core is our main is like that the most important repos repository it's 85 percent reproducible and We are aiming to first get that hundred percent reproducible and I'll work our way around to the other reproducible to the other repositories because they are larger But The test framework is nice, but in real life things are different. So when we actually started testing if Package we have in a repository can be reproduced Using the build info file which we have in our package. We found out that files we saved file size of a package in the package and on Different file systems you will get a different file size report if you just use du so That was an interesting issue We also had to do some infrastructure changes so we also had to have a build info file which Contains all the packages or we have installed on building and For our techniques you have some flags and options you can give with a build. So we also also have to record those we also already kept an archive of all the older packages and We needed to Make tooling to be sure that we don't remove packages from the archive which we need to rebuild packages otherwise we have an Unreproducible package because the old dependency is not there Yeah, and we also created the tool which reproduces which takes a repos repository package and then try builds builds it again and then compares it and We are still having to fix a lot of packages because which are for example are already fixed in package in Debian or because in Debian if you if something records the the timestamp or Put sedate into the binary you can usually Give it as a build option. You like use that the the timestamp of the change log in our things, we don't have a change log. So we have to Fix those packages in a different way and we have some packages which Debian doesn't have so we have still have more to fix and Something we actually got done this week was getting Jason output from for our cynics. Debian already has it because and We will soon integrate this on our web page So if you view a package then you can see in a test framework, which is reproducible. This will hopefully then give the incentive for people To say why isn't this reproducible? Let's fix it So that was the short update on the art status for me It's interesting to see the things there are some things they implement the same way and sometimes they find different solutions And I find this interesting in this couple of collaboration to see how different Solutions then evolve and which becomes better. It's really nice aspect Buster we have this in policy. That's also that is new or that was already maybe last year We now have Debian installer images Where in the last three months lots of progress was made and Then there was the patch which thought to be reproducible and then was something found on the buildings again So there's another Fixes which are still missing which might not get into the Buster release But into the Buster one if in the first point release But it looks like we can have reproducible di images with the release of Buster and Once we have them we can look at the the big CDs or the full CDs I've also looked at this graph again. This is the patches we've sent since 2014 Yeah, since 2014 and it's There's 2014 on the left and this is two thousand five hundred bucks with reproducible builds patches filed and Except for three hundred eighty all of them were applied So that if somebody wants to do a big and then you campaign you have three hundred eighty bucks to take But not six or eight hundred which are why I was thinking so that is quite some time ago that we had more What else is missing? We have this problem with GCC that it embed embeds the build pass in the object So you have the user home directory in this case in the build pass We have this patch, but it's not been accepted by upstream for Buster and maybe it will not get it for bullseye because we just use the workaround to rebuild in the same pass and We need GCC to be fixed and Adding this patch might not be the best thing. So we just say for the moment Rebuilds have to be done in the past where it was originally built Hmm This needs somebody to drive that Was the GCC didn't like the way the patch was implemented So they want something else. They don't dislike the idea, but the implementation LLVM has something implemented and We basically lack somebody who can drive this like I'm I'm not good enough in seed to talk No So that is something but it's the workaround is so simple. So it's okay. Let's do something else But if you want to tackle a hard problem, there's one Yeah, and there being is wrong everybody misunderstands the graph we are not at 93 percent We are Because that is in theory we just do the CI test and we don't compare against the Debian archive and There's some blocking box blocking bugs, which I'll present now which cross this somehow so This is this difference between theory and practice 93% is a lie on March 5th I sent a mail to Debian devil Where we only had 54% of the packages distributed from FTP Debian org where we managed to do a rebuild Now we are down to 31% And you will ask me why and I have to say I don't really know why yet There's 24% for 6,800 source packages have not been uploaded since December 2016 when the change was in deep package to sort the Archive elements and reproducible deterministic way and Everything uploaded before and not been in you've been in a mute since then Is unreproducible and I'm not sure if I or we should do 6,000 or whatever uploads of packages. We don't use and just We upload with us We need to discuss this with the release team. I think I would need to drink beer with evil later You find that This problem is mostly or is biting the security team Because people it's a detail. I don't have that much time Been in a muse I'm generally also problematic For backups because m times and our sink So maybe we should change the way been in a muse are done though We are now able to reproduce been a muse. So it's not they are not unreproducible per se, but they cross other issues That's what we learned the last year and this mastery build is this bug. I already mentioned Then we have the problem that the build info files are not exported outside of FTP master They're exported to Corsair Debbie and org but that's only again Debbie and developer accessible machines. We have workarounds for this bug now Then we don't we also want them to be in the archive that doesn't happen yet and for the security builds the build info files are only synced with the point release so that they are also only available after months and We should fix this somehow. It also affects LTS and Then we have built info files We made this hack that they are available on built info Debbie and net and then I created built info stabby and net Built info Debbie and that allows submissions of built info files from everyone So and then there has a downside you go there and ask for the built info file for whatever this package this version And it gives you 20 files or something or more and you which is the one for the FTP archive So I created FTP a built-in for stabby and net which is just the FTP view like it's on cokea. There's a Build it's build date base. So I created a link farm to have a pool structure So you can just go there and get whatever the build info file for this package this version But there's a stabby and net and there should really be a Debbie and org machine providing these built-in profiles in the long run It's not that many files. It's a million files in total for the archive Which is 12 gigabyte file size and 4 gigabyte of links the same amount of links obviously Yeah, of course we Then upshot maybe worn That would be good for bullseye if a packages unreproducible upstores the warning but really the goal is not to install or run unreproducible software and Then there's also the question. What does that mean and which rebuild I do I trust so in total what Lucas does present it is Way better than this up warning But it's also more complicated to design We have now for test reproducible builds we have the results saved in a common database We from that we generate Jason for Debbie and Arch Linux open WT and alpine open Susie creates their own Jason's and we just imported We still want to do shared node and cross distro links This is that we something I talked too many years about and We want also want to do two kinds of tests at the moment We have this CI test where we build software twice and compare it against each other But what we really want to do we want to compare we want to try to match the what's provided by the Projects FTP or whatever server because it's nice to do these CI builds But we really want to see whether we can reproduce the package people are using This is from last year The last thing I would not write anymore today because Buster is now coming and maybe we get better than this But a bullseye is coming. It's really far away. We can still do anything. We just need to do it and This is bullseye and That was the logo Hans Christoph Steiner choose for his Android talk at foster 16 I liked it very much and I still like it. Well, the new logo is great as well. That's probably better representation So thank you for taking these updates for your interest and for your contributions Thanks, Hogan yellow arm. We have time for some quick questions. Please line up at the microphone How many toolchain patches are there left in the Jenkins CI? Artificial setup zero That's cool. We used to modify Debbie and packages, but the mostly since deep packages and we had Ocamel have had some tests there in our infrastructure, but nothing serious and let's remove now the patches been taken So there's nothing left. We used to have a patched DCC, but we stopped doing that as it was news already Yeah, no, but that's no news compared to last year any further questions will be here for more days Arch Linux has some packages that are not in devian. How is that possible? Ah Because so we we are less strict we we don't have an arch Linux policy Which is comparable to the devian policy regarding free software and So we also package steam for example Which is I'm not and we're not sure even how we're gonna handle those things Yeah There's also arch has this diff what is the arch Linux repository and the arch user repository and that's not arch Officially as I understand that's true. Yeah, and the arch user repository has way more packages than there be in But those are not built so you yeah They are not tested by us because the same with contrip and non free for us. There's might be some dragons in there But I do believe we have packages which are not in devian, but I don't have a list but soon as we may mom if we get the the Package comparison pages which compare the reproducibility between Debbie and arts. It will be E and open Susie It will be easier to find out But but naming is an issue there because another two reality yellow also use this weekend to package Pac-man the arch package manager so we can also now enjoy pac-man on Debbie in the real Okay, if there's no further questions Then let's thank the speakers again