 Okay, I think I should start. Hello. My name is Holger. I will talk about a story about reproducible builds and Yeah, I'll start with slowly. So as I said, my name is Holger and I don't speak Portuguese. I'm sorry. So you need to listen in English Go by the nick Holger as well. It's just written differently So that's me on IRC and I'm involved or I'm interested in computer since a long time 25 years or something. I'm a Debian user since 23 years and Contributing also a long time and I'm a freelancer Which I've started by the way after I've been to Fizzle in 2004 the forum international software Libre Which was one of the reasons I became a freelancer and Some bits more about Debian and me. So I started 2001 contributing and become a Debian developer in 2007. I Was involved in organizing Debcon. So if you have questions about Debcon, I'm happy to talk about this as well I started the video team Stefan was now the better person to talk about video I Do work on Debian Edo still Debian Edo is Debian for schools and universities I One of the two people running pu parts Debian org which tests package installation upgrade and removal and checks that the system Is still the same after you removed the package That's basically the quality assurance and this led me to set up Jenkins Debian net Jenkins is the basis of many things of I would discuss on this talk, but Jenkins has 1,300 jobs, I think and only 200 of them are related to reproducible builds the others tests some other stuff in Debian and I'm now a cubes OS user which so I don't run Debian as my primary operating system anymore So if you want to talk about learn more about cubes, I'm happy to talk about this as well And I use core boot so I don't that's for the BIOS So the BIOS is normally a non-free blob, and I run a free software there. I'm also happy to talk about this If you want to know this So ask me anything either now during the talk or the next days. I'm happy Yeah, and also in this talk feel free to interrupt if you have questions. That's fine or later But this talk is about reproducible builds and That was started in Debbie in five years ago by Luna and Has bit more history and I've been involved in it since 2014 and These slides are taken from Chris lump who's also involved since 2014 and reproducible builds We work a lot together and his slides just I was bored of giving the talk with my slides So it should choose to switch to his slides This is now the second time I give the talk with these slides Chris also wrote a sudoku solver in PostScript So if you think PostScript or some files are not code that is just wrong PDFs can also execute whatever they have database connectors and So it's not always just text The same is true for office documents or other stuff So but to tell the story about the three developers so First we have Alice Alice is a software developer. She writes code Whatever maybe a Bitcoin valid or some other cryptocurrency thing and she um Releases the source code and Some access on that Debian package or some rpm and it's cool. It's works and there's the source code You can review the source and Then suddenly sometimes somebody knocks on her door and says hey You don't have a dirty secret But maybe her sister has or she has a house or whatever she's blackmailed into Modifying the binaries so they asked you still release the source code as it is but So the source code stayed the same the sources put as always Just the binary is not the same as from the source code anymore and People check the source code and think the source code is fine and The downloaded binaries they are different than they don't come from the source code people review And that's bad Um The other scenario is Bob Bob is a Hacker or something he has a strange keyboard and he maintains a lot of machines so he maintains the servers and Stuff is being built. There are lots of softwares built and It's cool. Bob knows a lot about security and thinks the system are cool But the systems are on the internet 24 7 and sometime they get compromised and They build these binaries for people who download them and then they get they get compromised and that's bad and This is Carol Carol is a nice person and Carol is not the problem. The problem is if If puts while that is the evil major text for Carol Carol leaves a laptop alone either at home or in the hotel or whatever and then if comes plugs in some USB thing and Backdose her computer and then when Carol gives softwares to somebody else the softwares compromised and The four freedoms from the from the free software foundation Define that the one purpose is that you can redistribute copies to have your neighbor But if you give your neighbor Trojan software, you don't help your neighbor So that is a problem So the general problem what I've described here is that we can review the source for malicious flaws or problems but Users install pre pre-compiled packages. So whatever you review It's not really what they get you really you get something else than you review and The question is can we trust this compilation progress? process and Well, this has happened This has happened this has happened this This this this This this everybody gets hacked So this Bob thinks he's good, but he isn't and it goes on and on and on This is also this even-made thing also happened that I take to this to get the encryption The disk encryption compromised Yeah, there are many examples you've probably seen many of them either self So what is the solution for that? so We we start with the same source and the same built environment and And Then we ensure that builds always have identical results and when I say identical I mean bit by bit identical and every bit is the same and then we compare the results and So if we do this in the There's David has some hash and Aaron has some hash. That's the hash of the binary and There's freight and freight has a different hash So what does that mean? It's either means that freight has been compromised or David or Aaron But one of them is wrong or the software is not reproducible but if the software normally builds the same bit then if you get these results then something is wrong and So if Ellis will be blackmailed this will be discovered because you rebuild it get the different results Everybody gets the same result except Ellis, then you know Ellis has been compromised The same with Bob everybody else gets a different checks and we know Bob is compromised and Also if the laptop will be turned but we will notice that so and it also reduces the Intensive to attack in the first place because if you know your attack can be detected you probably don't do that You would do something else Maybe you would compromise the source code then but you would not attack the compilation process anymore and So reproducible builds really allow verification that no flaws have been introduced during the compilation process Not in the source. Just this reproducible builds are only make me only make sure that the compilation process is secure So It's not about reliable builds. That is that you can repeat it It's also not built with the same dependencies. So that is we can reproduce the build and just do it again and again it's about identical build results and It's not only the build with the binaries, but the software has but the whole package Because we just want to in the case of the Debian package Debian package contains of binaries and documentation and Scripts and images and we don't want to say we exclude some of the Debian package We just want to look at everything because if you want need to inspect it doesn't scale anymore You need to look into it and ignore this and write complicated code to ignore some part And then the code might be buggy. So the thing is the whole Debian package should be identical and I would as I first heard about it. I was I thought that would be the case That software would be identical But there are many reasons why software is not identical when you rebuild it So there's hash ordering database ordering Dictionary ordering which can be different in each build and then the build result is different or parallel builds if some software builds parallel and the Some parts are executed in different order than the result can be different Timestamps the few deaths does a timestamp in the build and you rebuild at a different time and the timestamp is in there Then you have different results again the build pass can also leak into the binary and Also, the file system is not always deterministic if you do LS LS will sort for the output for you But if you do a find on the file system, then X4 will not have the same results Some file system have all a deterministic ordering, but not all of them So that can be a problem. There's there's more reasons, but these are the main courses and And of course user group the environment variable all this stuff can be in the bit so your username or my username or whatever can be in there and While we did this and looked at lots of software and built it several times we found that reproducer builds have other advantages Because if you have no changes normally between two builds and then you do a change for example There's an security update Then you have a very minimal diff and you can really look is the diff what it's supposed to be That is pretty useful The other is also if you care can have the same result you have better caches Then you save time and money and also co2 so you save the environment with reproducible builds and This is no joke Google does reproducible builds because they save compilation time so they save developer time So they save money with it And you can also detect corrupt build environments So somebody doesn't upload for stable and builds it an unstable you would see it or whatever you've broken dependencies installed And you can also use it to remove build dependencies just to see is this build dependency used at all So if you get the same result without a build it dependency You can just remove it and we found lots of bugs which I will have few examples of that So one bug we found was a predictable predictable open ID secret so This was in the code which then Resided in this Secret and every installation has the same secret and so it's not really a secret And we found this by looking that this happened every every build has a different secret ID but That's not a secret also sometimes Men pages if you build them with different locates or different It's mostly locates for men pages then you get different men pages and that is a bug It's just not the reproducibility issue, but the men page should look the same. There should not be random characters there I leave this off This was also a lovely bug so you have two functions was a test case and It generates two strings Which are 16 characters long and their strings should each string should contain a B and C and Then there's an assert which should make sure that the resulting string includes a B and C but of course these strings can also just include a and B or C and B and not a and so Then you get an assertion error because the string does not contain one of the letters it should contain and Somebody did the mass and so it fails zero point four six percent of the cases It's really rare, but these things happen and then you stand there and why does it fail? And of course this is just a byproduct of our work, but it's useful So what did we do to find this? We started in Debian and In 2014 I set up this what it has become a torture test Where we vary a lot of things so we We do two builds and then we Compare the result whether it's the same and these two builds vary by the time and the date So we have one machine or one test is running in the future it's running one year and a month ahead and The time is also different. We changed the host name and the domain name of the machine We use different file systems or we also wrote disorder FS This order FS is a fuse file system, which will give either a random order or You can also say that it should one Forward or reverse sorting that we will see if the code does not do the ordering itself We have the builds are in a different time zones and different locale, so we built We only built with Latin locates at the moment We don't do Arabic or this stuff that would probably find more bugs, but we find enough bugs with this already We built with different user ID and different group ID. We vary the kernel and the CPU type and then then we look at the results and And When we did this in 2013 for the first time 24% of packages in Debbie and we're reproducible while now we are at 93% Which is quite good, but 93% of 25,000 packages still means there's 1,800 un reproducible packages So it's still a lot of things to do and as we looked at these packages for four years These are also the harder cases So it will take still take some more time and This is on this is the graph for unstable and so the green packages are the reproducible ones the orange ones are the un reproducible ones the red fail to build and the black ones are some others and You can see or cannot see probably but the graph started in October in 2014 and it's going to March 2018 and This graph actually doesn't have 93% but only 88% I think because this is unstable and There's one problem. We haven't fixed yet, which is the build pass the build pass is leaked into the binary of objects of many of Compilation objects, but also man pages. So when we test Debbie in testing so Buster where we have 93% We don't vary the build pass Because there's too many problems with the build pass. We have not yet fixed What an unstable when we test unstable we do vary the build pass. So there's 5% less reproducible packages So there's probably I don't know the number 2800 or something and the fix is really easy just built in a deterministic build pass for example Slash build slash package name slash source code as version and then you get this 93% and it's important because This is similar with the In deviant for our deviant has to be very the timestamp at the the locale and the time zone Because we want developers to be able to build in their environment if you have your system set to Portuguese We want the results to be the same as if somebody builds in English But it would be easy to just always say always use UTC as a time zone Always use locale C and always use this build pass, but we really want to fix it properly because it's in the end it's It is compute. It's called computer science and if you have some deterministic input and a deterministic Compilation process the result should be Deterministic as well and not just random. So that's not science This is the URL you can easily go it will just say yes or no I Will also get the percentage, but we'll have this domain for some more years, I guess So and beyond deviant we started testing this for Debian and Then we thought okay, we can also build other stuff So we now build core boot or this doesn't there are more projects involved But on this Jenkins Debian at there we built core boot Open WRT net BSD free BSD arch Linux and F droid are all built on Debian resources And that's really nice because we have now the reproducible reproducible builds team is now a Cross distro project where people from many distros Collaborate together and share the fixers because we all use the same sources. We have some modifications But the base source is the same and then there's other projects which joined and tails as an example Where it goes beyond reproducible builds, but to reproducible installation Which is a different problem problem in the case because then you Install packages and then the post-ins is executed and post-ins again does unreproducible stuff It creates a directory which has a timestamp it creates user which where the user ID might be different in which Due to the order opt installs the packages So if you do opt install foo and then 20 packages are installed the order in which these 20 packages are installed It's not deterministic and tails has fixed that for tails and we look to work on getting these fixes back into Debian Yeah, and That's So that's these are two problems We have in this reproducible builds area which one is reproducible builds and the other is reproducible installation and then this reproducible builds Problem is also against blitz into two problems Because what we've done so far is just showing in theory that Debian can be 93% reproducible But to actually prove that and do this we also need infrastructure that people can Test and prove this and all this infrastructure and the other tools are mostly still needing work so for Debian Or in general what we came what we came up with So to reproduce you need to have the same environment which was originally used to build the package And so we came up with dot build info files which record the environment in the Debian case It's a list of packages where it were installed So if you want to reproduce some soft as something is uploaded to unstable today and say you want to reproduce it in a month Then unstable will be different So or buster will be different. So you need to recreate the exact same environment and for that we have built info files and then it should be possible But this is for me this practical problem of reproducible builds, which we have not really taken much so far So far we are only still working on this theory and Yeah, I explained this already We also to work together we had reproducible builds summits, which are like mini could have comfort with conferences Where we for three days sit together and we don't heck instead we sit in a circle or in several circles and just discuss brainstorm or Do these things and we take notes and Have published them on our website and there were Around 40 people at each summit with from 20 projects. So all these projects have been at the summit and more You can go to reproducible builds Orc slash who to see which projects are involved So when you look if you want to look at differences between two packages the one the tool Everybody knows is stiff and if will show you the differences between two text files nicely and you know that But if you do this onto Debian packages you get this which is not helpful and So we should build a better depth to a diff tool and Luna started this and This has become differ scope in the beginning. It was called depth in diff the Debian binary diff But we renamed it to differ scope because differ scope can now Deal with many more packages. I'll show them some so the way differ scope works It goes reek you give it to objects And then it goes recursively through these objects We give it to Debian packages and Debian packages are archives Then in the archive there's two tar archives and in the tar archive There is whatever PDF and the PDF includes a PNG and in the PNG the timestamp is different and differ scope will show that nicely and It will show it either in plain text or an HTML version or also Jason and This is the list of formats differ scope understands now So it does every so does CP CP IO archives RPMs You can give it to directories. You can give it to ISOs. You can give it to file systems To anything to open office documents and if there's a file format which differ scope doesn't support please file a bug and Give us the test cases and we will support it. It's also does Android archives and TCP dumps and many many things and there's Honest There's differ scope org is the web page and there's also try dot differ scope org Well, which is an online service where you can upload two files You don't need to install it because if you install differ scope on a plain system It will install about 1.5 gigabytes of packages to be able to deal with all these formats and This is how differ scope text mode looks so you see there these control and data Tar archives are different and then it shows how they are different and you see here These are time stand difference. I have some more examples later Yeah, and try differ scope is really Go there give it a look upload two things. It's really nice and Differ scope is also it doesn't only work. It's on that on Linux. It's been ported So it's been put available in Arch Linux and Fedora. It's been ported to BSD It's available for net and free BSD. It runs on Mac OS. It does not yet run on Windows But it's pysons so should be possible also and It can also show differences in security uploads because if the if the packages are reproducible before then the security upload will only have the changes you want and you can see it with differ scope and Differ scope is not the definition or the tool to find out if something is reproducible The tool to find out if something is reproducible is just diff or CMP or chart whatever some So it should have the same hash Differ scope is the tool to investigate why something is not reproducible and you can also use it to Inspect files. You don't know for example router images or whatever binary software you get you can look into it with differ scope So what's left to do? The source code There's still programming errors in the source code that can be back doors or obfuscated code in the source code weak algorithm MD5 is still insecure even if it's reproducible and Code was testing mode. There's so the source code. We don't care. We we don't look at the source code So the source code still somebody needs to look at the source code Reproducibility is just really to see whether the compilation process made something with the source code You don't want to do and then the other thing What do we want to do with this? so this is a Up then it says there the following packages are not reproducible install them anyway and We don't think this is a good user interface because what should you do? You want the software so you will not say no So what we think to have any good user interface is just to have all packages Reproducible and just prevent installation of non reproducible software But this is still some time to go what else needs to be done We still need to fix two chain issues GCC is the one example because it has it's the biggest problem with the build pass But the build pass is also leaked leaked by lots of documentation systems Which put them somewhere in the generated PDF or whatever thing? Spinks was just discussed. There's many Toolchain issues and toolchain is just not where in Debian has 40 compilers or something there's cobalt air lung or camel Whatever there's java script and these is this one part of the toolchain and the other toolchain our documentation system And there's many documentation system in Debian and they most of them something into the object We want to improve our tools further, which is different scope on the one hand, but also this all disorder fs and We want Mendating that Debian packages must be reproducible last year in August during that come 17 Debian policy was changed so it now says packages should be reproducible, but it's just should so if it's not reproducible It's a normal bug and we wanted to be a serious bug so packages must be reproducible So that unreproducible packages don't go into testing and Then we get closer to hundred percent, but I we will still not be there with this must because if we are Faced with the decision whether we kick out we cannot release without the Linux kernel We cannot release without many many tools if they are unreproducible the release team will just ignore this bug and release anyway So policy must does not change it at all or changes a bit but together we really need to fix the software and not just have policy and We can with reproducible builds we can also defeat this trusting trust problem if you don't know that trusting trust is a problem identified I think in the 70s by the CDC Creators so you have a C compiler and this is usually already a binary blob and this builds the next binary block and that's bootstrapping is the problem and Our idea is that you have two tiny C compilers Which are able to build each other and then you build each other with the other one again and Then the if both results then are the same then you know that this compiler is fine But this is also just theory at the moment. We are working towards it, but have not reached it yet So please get involved Visit our web page. That's also documentation with common problems like Whatever gzip by default does not sort the stuff So you need to use gzip minus n and there's some there's documentation what problems there are We have also a package which is called the unreproducible package where there are several problems are explained and their fixes We we are on Twitter And we also do a weekly block since three years every week we block so we are now at issue 155 I think so next year next week. It's our three-year anniversary And we are an IRC This is reproducible builds is our general channel. There's also Debbie and reproducible which is just for Debbie in There's Arch Linux reproduce reproducible Arch Linux Arch Linux reproducible And we generally don't care. We are happy to talk about non Debbie and stuff on the Debbie and channel We are generally helpful and wanted to fix the problems Yeah, and there's also one eight or eight hundred Patches we have submitted which were not merged. We've we've submitted 2,000 patches which were merged So one third not which is two-thirds were merged is a good good ratio, but still you can do lots of enemy use If you're a Debbie and developer, please help us and I'm you this stuff because the bugs are laying there since half a year or longer That was that's for me So hi, I don't know if I understood everything really well, but my question is is the Linux kernel reproducible If not, why not? The Linux kernel that the Debbie and Linux kernel package is not reproducible I think because there's some problems in the documentation the kernel itself is reproducible and There's one if I want if I start packaging if I want to test if my package is reproducible So I'd like to test all those cases about different dates different Locales is there a tool I can use There's there's there's two ways to test this for one We have one tool which is called repro test it's in Debbie and so up to install repo test and Repo test will do the variations for you and repot test has also a mode where it Varies the variation so it builds with all variations and then it sees it's unreproducible And then it removes the local variation and if it's then reproducible then you know Aha, it's something with the local and if it's not reproducible then then it removes the next variation Until you know from which variation this is caused The other Way to find out is just to upload your package to Debian because then we will run. It's on your package Automatically because we test all the packages anyway So everything which gets uploaded is automatically tested with all the variations and this is Then also shown on tracker Debian org So this is tracker for the Debian policy and you can see there does not build reproducible So if your package has this then this is the problem or then you know it You don't know why yet, but And if I have another question Is there also another tool to check if my system got compromised it If my the package I installed is exactly the same if it is reproducible You cannot really you cannot use this to detect whether your system is compromised But you can what the only thing you can do is build it on your system And if you know it's reproducible and then build it elsewhere then you might see okay One of the two systems is compromised because the second has some doesn't match and it should match But it's only the reverse proof Thank you if you Either you can either go to tracker dot Debian dot org and through your package there Or you can also go to reproducible dot Debian dot net slash your source package And this will give you this this is again for Debian policy and if we have investigated it We also leave notes there. So we have some Different issues issue user and groups and talk about tab all timestamp and tab all timestamp in Zisa piters Different due to umas different timestamp The time stamp in documentation generated by e-max org mode Time stamp in ps generated by DV ips So these are all issues the Debian policy used to have because these are the notes from version 397 and Policy is at 414 at the moment so this these notes are outdated we need to work on those this is the Stretch differences so it's policy 398 over there, which is the version in stretch and I want justice as an example of difference group output and So it says here that the This step is different and then you see inside the depth There's this differences in the control and in the data um file and Then you scroll further down and You see here some file sizes are different. So that's not helpful not helpful but here you see 2018 and here you see 2019. So you see this is this leaked the build date into the binary and This was the old policy version You now look at the new policy version. This is all gone But what's still there besides this which nobody don't understand I don't understand here You see the year and the months is the same, but the build date is different And this is because the time zone is there because we built with GMT minus 12 and GMT plus 14 So it's more than a day different. So if you have big dates, which are just different by one day It's due to the time zone variation So you need to manually look at why it's More questions So thank you. Oh, I'm also involved in a few initiatives that I Expand the whole Debian archive and there's always this issue of people not picking up the patches and it's hard to push things forward because People don't care Do you think you have insights from reproducible builds on this type of initiative and what we can do better in Debian? To make sure that the people pushing these initiatives Don't get blocked forever Your question is what we can do to that our work is not lost What we can do as Debian to make sure that initiatives that spend the whole archive like reproducible builds or auto package tests or Stuff like that doesn't get blocked forever with unresponsive maintainers and Patches sitting in the BTS for I'm not sure what can be done I think when I when these patches we submitted Half of them are acted very quickly like in a day or two those are superb and then there's the other half or maybe it's just one third where there's not read there's either no reaction or reaction or something and Which I Think that the problem then Debian is which is one of the strengths of Debian is the strong maintainership model but also what Debian has since many years now has Changed the way we do NM use non-mainer uploads They maintain uploads they used to be very frowned upon and now they are not that much anymore They are what more widely accepted and I think we need to be more actively doing more NM use So we tried this Chris Mattia and me try to do NM youing NMU campaign for reproducible builds we announced it announced it on Debbie and devil people were happy with it And I think we did maybe 100 or 200 or maybe it's three or 400 now, but we are still I Could do 500 MM use, but I don't do them so you could blame me and So it's not blaming me, but it's blaming It's not only the maintainer who don't accept the patches but it's also it needs people who actively do that and Lumby does it love Lumby does one NMU per day or something But even that will take two or three years till he's doesn't got any more thing to do and he does file a new Patches, so it will take a long time So I think the way out is to do be more friendly with NM use and One way what I what I like about Fedora and Fedora you can clone every Fedora source package with Lifts on git Fedora project org slash git slash source package name Well in Debian it can be in git it can be an SVN. It can be on salsa. It can be on github. It can be somewhere So that's that's a complete mess and this mess is good on the one hand because it allows us to experiment with new stuff And the mess is really harmful because it's a mess So that's the same but this is the same with and Packaging tools nowadays most most packages use step helper 9 or 10 maybe even now But they are still CDBS. They used to be Yada There are some packages which don't use any helper tool and this is both a strange and the downside because Strangest we can develop new tools downside is you don't know what tool has been used. I Would like to ask about the tracker tracker Debian.org that Usually says that it's not reproducible Then you go there and see that hates saying about GCC captures built bath that I should ignore that It's about the two-chain that But then if there's ever a regression and if I start ignoring Hints that it's not reproducible. I won't maybe notice that Do you think there's a way to improve like? Hey, you you have these Bugs and I'll have these more bugs or Kind of regressed. I don't think it makes sense to to differentiate between Has three unreproducible issues and two Which might it makes sense to see the package was reproducible and it's not reproducible anymore But whether you introduce another Variation or not I think is meaningless So and we are not really there that we can say these regression can track these regression very well But I think once we there we will do that because we've done this with pure parts results with the testing migration And that is sensible regarding tracker is also on tracker. We show the results for testing for Debian testing and not for unstable So if you go Of the packages unreproducible only an unstable but not in testing tracker will show that it's reproducible Because that is just the build pass variation and the workaround is so simple just built in the same build pass. I suppose most of these patches are upstream patches and A few of them are actually Packaging patches, but we only Apply them because when the fix to arrive before upstream gets a new release. Is that right? I don't know I'm not sure whether most of the patches a bit bugs are really upstream bugs there's many or there's many fixes which we can be done in the Debian packaging Which are then nice for Debian, but don't help this free software world in general So we try to fix it upstream But that is also something we have not done that much. We Debian people There's Bernhard Wiedemann from open zoosa who has been way more active than us and pushing stuff upstream Which is partly a manpower issue also Yeah, I also I mostly heard about reproducible builds from Debian Are there many people from the other distributions helping this initiative? Many yes, this is what I had here. There's there's and so the the open zoosa people are really actively Doing lots of upstream work and they have they also test open zoosa or it's only Bernhard mostly doing that The basal is also interesting basal is a build tool by Google which is designed to do reproducible builds and That it's it's also used it's used by Google, but it's also usable By other projects you could build your Debian package with it with it The Arch Linux people are now quite active FDROID is interested or is working on FDROID as an application store for Android The NixOS and Geeks people are also actively doing that. There was just a scientific paper by the Geeks people for bioinformatics software, I think and they had 98% of their software reproducible There's many people working on this now This is What I what I didn't tell that Debian started in 2013 on it The there were two projects that worked on this earlier which were tour who made the tour browser reproducible in 2011 and the Bitcoin people because they were afraid if there was Bitcoin was worth four billion dollar in total and They were afraid somebody would Release a Trojan Bitcoin Client and then steal all the Bitcoin and they would be blamed so they did it they were the first to really prove possible and then we started and It's become many projects now so so this Baselok is a Debian based distro Electro BSD also was the first BSD to show it Electro BSD is a BSD It's a free BSD fork and they had the base system reproducible before free BSD and net BSD Fedora is working a bit on it leaders open double So there's more and they all work in various Things It's the tails tails the tails that the tails 3.5 release was reproducible It was the first tails release which was reproducible Then tails 3.6.0 was not reproducible and then tails 3.6.1 is reproducible again And they also had it in their PR material So it's getting there and this is the other thing. This is reproducible only work with free software But we have now safe driving cars. We have pacemakers some people have pacemakers or other medical devices in their body There's nuclear power plants. There's whatever weapons. They all run with software and Nobody can really know what's inside if you don't have reproducible builds You cannot really be sure that the source code you review is really the source code which is running So this this is really an advantage of free software Which people start to understand now the other thing is What I just said that tour and Bitcoin were the first in 2011 This is not true as I just learned last year and I've been working on this for four years because last year We learned that the GNU auto tools from psych Cygnus Where did commercial GNU support in the 90s they in 1992 released the GNU auto tools bit by bit reproducible for nine architectures But every the code bit rotted and the world for God even the GNU developers for God So that was And we hope we now made enough Fast that people will not forget that this is both possible and desirable Do you have a BTS user tag? There's a unreproducible user tag, but that does not mean what we mean That means I cannot reproduce this bug And we have We have we have you we have user text there's a tech, but we have user text Do you have one yeah, we have 11 or But we have for each issue like leaks environment or there's It's a really a complex topic So it's also I skipped a lot of things in this talk because there's too much information. I think I was there's too much information already and there's So the BTS user tag is No, that is that is the Jenkins things. No, it's in the wiki Started we started in the Debbie and wiki and then moved everything to reproducible Bills because we didn't want to be in the Debbie and wiki because we wanted to be cross-distro project if you go to what is that you're You go to wiki Debbie and org slash reproducible builds then bug reports There you find the user tech and so you attack is reproducible me a minus builds alias I will take it as a reminder to make these bugs easier to find again Thank you so I'm not to understand why some Binaries are reproducible for us in a specific category, but not for another Why? What was the question? I mean like you can I have seen many of Reproducible status status there like this is reproducible for x86, but this is not reproducible for arm for example What is the reason for that? that depends Can be that We have bit the variations we do are a bit different on the different architectures we test AMD 64 i3 86 arm HF and arm 64 and Some problems only occur on some architectures for whatever reasons So it's some problems are only on there's there's some bugs which are only on 32-bit architectures So I'm a 64-bit architectures reproducible on 32-bit not or vice-versa That can be The problem in the source code and the other could can be that the variations we do are different And so it's that's not visible in our test framework, but it's there are some things which are not visible on all architectures but for the other thing For example, we on i3 86 we vary We're once bit with a 32-bit kernel and once with a 64-bit kernel because you can do this on the 32-bit architectures Well on the 64-bit architectures you always need to use a 64-bit kernel. So if the kernel What is it the kernels? The architecture is leaked into the binary and you will get different results for that package only on arm HF and i3 86 There's lots and lots of lots and lots of details It's great details are good and you have mentioned that the reproduced reproducibility helps with with caching and build Compile time how so if you If you have a big source source code and you only change a small part of it And then if you want to rebuild this big source code, but you know That the results will be the same for the stuff you can use the cached result You don't need to compile it because you know the result will be the same because it didn't change that area of the source code so you can get the build environment after after the the build and And based it in another machine and I quite don't get it No, you need to do it on the same machine You need to be able to reuse them and I'm not sure how basal this achieves internally But it is possible to do that and there's tools to do that Okay, more questions. So thank you hogar Thank you, too, and I'm happy to talk about this the next two or three or four days