 I think we should start. I guess everyone can hear me okay? Yep, brilliant. Welcome to the Reproduce World Builds team. Here we've got Benjamin Hoff, myself, Chris Lam, Horga Levson, Stephen Chamberlain and Begur Kaskadian. So thanks again for coming to Reproduce World Builds Talk. Who here has heard of Reproduce World Builds? Great, yeah. I think for the case of the video, we'll just do a brief introduction of what they are, what's the goal. Just so we're all on the same page and for anyone following at home. So quick whistle-stop tour through that. Yeah, feel free to rest your legs. So I like to outline and perhaps sort of frame this as a bit of a story. So I'd like to give you a tale about three developers and getting some feedback there. I'd like to tell a story about three developers. The first developer is Alice. So Alice, we're going to go through Alice, Bob and Carol. Alice releases free software, she's a free software developer, and she releases, say, a Bitcoin wallet or something. And on her home page, you can download the source of it or just be convenient because most people use pre-compiled software. You can also download the binaries. And she compiles these herself and uploads them to, you know, wherever, Sourceforge or whatever. One day she gets a knock on the door and someone says, you know, you've got a nice house, you've got a nice family, wouldn't want anything to happen to them, if you know what I mean. So yeah, basically says, you know, every time you release that source code, don't change what you upload to, you know, get hosting provider or anything like that. But when you do so, just make a change to the source code that I give you and upload those binaries, not the ones that actually correspond to the source code. And so she's been essentially blackmailed into introducing flaws into the code, into the binary code that gets uploaded and essentially what most of the people use. And this is hidden from most users because the actual source code on GitHub, Salsa or whatever, it looks absolutely fine. There's no exploit in there. But the actual code that people are running does contain the exploit, and no one is aware of this at all. So as I point out, the source code doesn't correspond to the binaries. Let's talk about Bob. Bob is a Sasanin for a well-known distribution, as you can tell, and he's extremely elite because he has a fancy keyboard, which is all the rage, and pretty cool headphones, actually. Yeah, they're pretty neat. So as I say, it's a Sasanin for a big distribution. But unbeknownst to him, his personal computer, the one we just saw, was slightly compromised, and therefore they managed to compromise the build farms for this distribution in this scenario, which means that all the compilers, when they go to compile the software, they are introducing backdoors into the software being produced. So a maintainer uploads some extremely secure software to the build demons, for example. But then the build demons themselves, the compile farms, are spitting out compromised binaries. And those binaries end up on people's machines. So again, anyone doing an audit of the source code, doing an apt-get-source, yum-get-source, whatever the equivalent is. There's nothing wrong here, but their actual servers themselves and users and machines and laptops all the way down the software tool chain are compromised, which isn't great. And so when they install, you know, do an apt-get install blah, the actual code and their machine is now compromised. They aren't running what is corresponding to the source. This is Carol in her stock photo glory. But this is actually Eve. So the real person here is basically Eve, which is the typical evil maid attack. So you leave your laptop unattended as you pop down to your breakfast at the hotel. The maid comes in and installs some extras, shall we say, on your laptop. This is a problem because Carol likes to give software to her friends and distributes them and things like that. But now the software can't be trusted. It contains backdoors, Trojans, leaks of privacy, steals of bitcoins, all this kind of thing like that. And this sort of somewhat goes against the four freedoms now. Naturally, the Free Software Foundation's four freedoms have a strict definition, of course. But I mean, if you sort of squint and interrupt with them a little more prosaically, are you really helping your neighbor if you give people Trojan software? I don't think so. So yeah, that's one way of looking at it. So what's the general problem here? The general problem is we can view source code from malicious flaws. We can send it to auditing teams. We can look through it ourselves, et cetera, et cetera. Most users are basically installing pre-compiled software. They're downloading ISOs and running live distributions. They are doing app to get install blah, et cetera. Perhaps even just downloading EXCs and things on other platforms and things like that. And can we trust this compilation process? Can we actually trust what is going on between the source code and these pre-compiled binaries that we are all pretty much using? Our solution is as follows. We all start in reproducible builds. Our solution is we all start from the same source. So we agree that this source is absolutely pristine, contains no vulnerabilities whatsoever. Then we ensure that any build of this always has identical results. So I build it on my machine, I get one result, you build it on your machine, you get the identical result. And then we compare, for example, the checksums, or we basically literally compare the files that we've got. The ones, for example, that I built and the ones that you built. And this means that if there's any deviation between these two, we can start to investigate why. And if there are, perhaps this is because one of our machines was compromised in some way, or we might also compare the binaries that have been generated from a build farm and things like that. So just to run through an example here, David, you don't necessarily need to see that text, that's a char one sum. What happens is that David builds a piece of software and gets a particular hash of 7a48. Okay, great. And he tells everyone, hey, I'm David, I built this piece of software, I get this particular hash. And then Aaron builds the same software. I too get 7a48. Okay, this is looking pretty good. We all seem to agree on if we start with the same source and compile, we get the same binary. And then Fred builds it. And Fred says, I get a different hash. I'm getting something different. Now, this should immediately say, hmm, why is David getting a different result here? Is it because his machine has got something else in? Is it secretly his GCC has got some backdoor in it or some sort of clever trusting trust attack going on? But whatever it is, it's certainly an avenue for discussion, for investigation and to work out what is going on with his particular build. So how does this help? So in Alice's case, the blackmail will be discovered. So Alice would still publish, as I say, the pristine perfect source. But the exes and the devs that she produces and things like that. If someone else builds them, they'll get a different result. And they should always get the same result that Alice gets because it's a reproducible build. So the blackmail would be uncovered, or at least that change, the uncorrespondingness, if you see what I mean, between the original source and the binaries would be uncovered and would be questioned. I get a different result. Why is that interesting? In Bob's case, the compromise of the build file would be discovered because someone would build every Apache or whatever it is on their local machine and say, hmm, but the build farms are churning out different binaries. Why is this? Investigation should find out the reason why. And Carol's case, her Tampa laptop should be uncovered in the same way. And this basically reduces the incentive to attack in the first place. Because if the goal is to compromise secretly or surreptitiously, or blackmail someone, it's going to be of no use if you get rumbled quickly or, you know, oh, we've compromised it, but you'd just be found out, and maybe it can be automatically detected. So it just, what's the point of blackmailing Alice if it's just not going to have any effect? You're just pointless. Or maybe you might better compromise the build farm, but none of those binaries actually end up in anyone's machines. So it just cuts out that avenue of attack anyway. And by reproducible or build with, you get the identical result after a build. I don't mean you build with the same dependencies. That's often used in, for example, the JavaScript world. They might use the word reproducible in the sense of you always use the same version numbers so you aren't going to get, sort of, literally feature differences between two builds. I mean, you don't, I don't mean reliable builds in the sense of, oh, you always, if the internet is down or I'm pulling from this random git tree somewhere on the internet, I mean identical build results. So when you compare with char1, char256, cmp, you literally do a bit for bit. Check out the files. They are literally identical. No tools required just comparing the bytes. Easy peasy. But wait, isn't software reproducible already? Why isn't it reproducible right now? So one big reason is dictionary and hash ordering. So for example, in Perl, if you iterate over the keys of a Perl hash, they're in a non-deterministic order. So if you build once, and the output and the build process of that program relies on that ordering, then if you build it again, it may end up with a different ordering and so they end up with a different build. Paralism in build, in build, for example, if the output of the binary at the very end result is dependent on whether something gets built first and then gets appended or something else, that can also affect the build. Time stamps are perhaps our biggest problem where you end up with, say, people love to include the current date in the build, for example. You know, just this was built on this date and time. Okay, great. But it means that if you build it twice or I build it and you build it, we're going to get a different result because we're going to build it at different times, basically. Build paths, you might build it in home whoever, and I'll build it in home lambi. Again, if this gets encoded into the binaries, then you can end up with an unreproducible binary and therefore we won't be able to play that comparison game with the checksums. Not deterministic file ordering, oh, sorry about that, is also another issue on the, for example, on Unix when you do a reader system call, the order in which the files are returned are not defined to be in alphabetical order or indeed any order. It just so happens that on X based file systems, they tend to come out in the same order that you added them and that kind of is usually in alphabetical order. So most of the time we think that file systems are returning things in alphabetical order, but this is a bit of an illusion based on LS sorting them and things like that. Once if a build system does just do a raw reader system call, they can be returned in any order the file system wishes and this is more prevalent on BTRFS and other file systems like that, but you just can't rely on it even on X. So if I build it on my machine and then you build it on yours, you can end up with a different result if it's just, for example, naively looping over directory and appending a bunch of files or doing that kind of thing. Again, if the build process encodes the current user or the group or your UMask, if we have a different setting on our machines, or different binaries and we won't be able to play the game that we played before with the hashes. So are there any other advantages reproduced with builds apart from this security and et cetera? So it also means that if you reduce all these differences between my build and your build and or just builds on your same machine after a different time period, it reduces the diff between any two particular builds and so if you make no changes whatsoever you should end up with no changes in the binary, which is great. So when you do make deliberate changes, so you add remove a feature, you should only see those changes in the resulting binary. This is really helpful for doing quality assurance and things like that. Or if you're adding a feature on top or a security fix. So you apply a patch and then you build the software, you apply a security patch and then you rebuild it again. The source function should just be the patch but with a reproducible build you end up with the binary differences between the patched and unpatched binaries should just be what you've patched, if you see what I mean. And if you have all sorts of other changes then you've probably screwed up and you probably want to be quite conservative when pushing out security fixes pretty obviously. If a reproducible build always produces the same result you tend to get higher cash hit ratios saving time, money, CO2 even. A very large search engine said they were saving quite a lot of time and essentially energy in our environment simply because they made things reproducible and said they were just doing less compilation. Things like that. You can also use it to remove build dependencies if you literally get the same result depending on whether you've included this particular feature of this unfeature you've turned something off. Then you probably don't need it because the reproducible build process pretty much guarantees that as it makes no difference to the end binary you don't need to include it in the source to begin with because it's just not being used because it makes no difference whether you turn it on or off. So you can just get rid of it that way. You can also find a whole bunch of bugs. I don't have time to go into them today but it can surface bugs of all descriptions from the rather curious and the obscure all the way up to quite serious security vulnerabilities that we've found as part of our reproducible testing. Great, and now Holger's going to talk about Debian and reproducible builds. Hi. So what we've done since 2014 is kind of a torture test then where we do two builds and vary the environment as much as we can. We build with different time and date. We have machines running in the future. We change the host name, domain name. We change the file system between those two builds. The time zone and the locale, the user ID and the kernel and CPU type and we do this to find bugs because we know then if other people try to rebuild they will build in a different environment and we try to emulate this. We don't find everything because we cannot test everything but this is our aim and this is the stuff you probably know. Got another idea here at Debtcon from Adam Conrad who suggested to just build twice in the same environment to find other kind of bugs and to see if packages are easily reproducible if just the time changes. So there's more tests we can do but this is what we have been doing so far and as you know we found several thousand bugs with this test. We still don't fully know if this package might be unreproducible under other circumstances but with this we find lots of variation already and try to get there where packages are reproducible. We first did this in 2013 that was still a manual test. Only 24 packages were reproducible. Nowadays it's 93% without the build pass variation because we only vary the build pass when testing unstable but not testing. So this is the progress we've been having since 2014. This is Debian Unstable on AMD64 and this big bump in the middle you see is when we introduce build pass variations in unstable. When you look at the graph for testing you don't see this bump and you see the 93% here it only reaches 88% and a bit cut out on the very right. There's again a little bump which is because of the switch to GCC8 which we'll explain shortly what consequences this has. So at the moment this is probably 88% or 86% only. And we have this web page is Debian reproducible yet? Which always says no or not always, in the few years it hopefully says yes. If you want to find out about your packages you can either go to testreproduciblebuilds.org slash testdebianblah or you can still go to reproducible.debian.net slash source package name and there you can see whether your package is reproducible or not. Number three. This one. Okay, so many of you are probably familiar with the diff tool. It's a pretty basic tool and it can show the differences between two files. But it mostly only works with plain text files in a meaningful way. So several years ago ah, so here we have the output from binary diff comparing to .deb files. As you can see there's a lot of gibberish information in here. Like I said, diff is mostly only useful for plain text files. But inside of those .debs you have plain text somewhere or we can come up with ways to get it. So numerous years ago folks started building an improved diffing tool which can actually unpack archives and continually unpack them until it can get some sort of information out of the archive that it that can be represented in a format understandable to humans. And so once upon a time it was called debbindiff but since it was useful well outside the scope of debbian packages it was renamed the diffoscope. So basically the diffoscope as I was saying it can unpack archives so you take tarball which maybe has an ISO image in it which maybe has a compressed file in it which maybe has plain text in there and it'll go through and unpack each of these objects and then try and show you a diff of the resulting thing. If it doesn't actually get down to something reasonably diffable it might use a tool that will output information about the file that is somewhat human readable. So... Alright. So here I don't know how well you can see this but hopefully if you can we'll make those slides available for you to read directly but basically it's got a series of a series of different file layers that it exposes and then we'll show the differences between the various files at the different layers. So it's kind of like a layered diff it can produce output in HTML which makes it readily readable when producing a web page or something to output differences between files. It supports a few file formats this is largely due to the fact that it has a really nice plug-in infrastructure so if you are using Difascope and it produces some binary output and you have some clue of how to turn that binary output into something more usable you could add to this list to the point where I doubt that we can realistically fit this on even probably two slides and as you know a wall full of text doesn't make a great slide but it shows you just how many file formats that it already supports and anything you find that it doesn't support is relatively easy to add that's one of my few contributions to Difascope as I added support for the device tree blob format and also because Difascope supports so many different file formats it has a lot of recommended packages to install and some of those are very large so fortunately there's a service available where you can just upload two files and it'll do all of that output for you I think there are some limitations on if you're trying to compare multiple files like it doesn't do much with a changes file or something but if you upload two devs it will produce your typical output in a web consumable format it can be useful to show the minimal differences in a security upload to make sure that if you made a small patch it doesn't result in a bunch of unintended unrelated changes it can be useful to determine why a given package isn't reproducible when you're testing it but just because Difascope doesn't produce any differences that's not to say that your package is reproducible there's a much more involved process to figure that out but it's helpful for revealing that sort of information and it can, with router images like OpenWrt it can be used on these things but you may often end up with just hex dumps of raw binary data there's probably a lot of work left to be done on that and I think I'll hand it over to Steven Okay, so since Stepcon 17 when we presented a similar talk quite a lot has changed in the Debian infrastructure for example everything had to be moved from Alioth to Salsa for the reproducible build project that meant some of the... not only the repositories but some applications had to be moved for example the scripts that produced the weekly blog that we posted and at Debian everything had to be moved to the new infrastructure and that's already complete we had a sprint in Berlin we should definitely try to arrange another one of these sprints this year so that was the end of... toward the end of last year we met in Berlin with some guidance in organization and with scheduling we had workshops and many of us split into teams for knowledge exchange and discussion of ideas which people continue to develop the rest of the year something else relatively new is we have this patch that landed in GCC this fixes a lot of build path issues that we were having I will give an example later also we made a decision on the project logo because reproducible build is not limited to Debian it exists with the participation of many other open source projects so we have a separate web domain and now a separate logo we're still most of us Debian developers but of course we have people from other projects participating so the build path issue that I mentioned when we started to consider normalization of the build path as essential to our definition of reproducible builds we realized that GCC often embeds the build path for example when we compile this relatively simple test case like so the directory in which we run the compiler gets embedded as a string somewhere inside of the executable which could be considered a privacy violation in a way if the developer's name gets embedded in the binary that they distribute also in an organization maybe it leaks some internal information about the server infrastructure so we ideally do not want build path to be embedded in executables but so long as they are we do have this patch in GCC now that Shemin worked on who was paid to work on this project this adds a new parameter to GCC so when we compile the same test case with this extra parameter then all occurrences of our local build path will be replaced with something normalized and this makes reproducible builds much easier that said there are many other ways that a build path can end up inside of a binary for example it's not only GCC but compilers for or tools for generating PDF Scython I've seen OCaml, Erlang, Golang so tools or compilers for many other languages so our binaries may also embed a build path so to get the last 7% of packages fully reproducible we need to consider some other work around just to give a hello an overview about other projects is this on other projects which do also reproducible builds ArchLinos is at 80% mostly because the way they install Python packages they put the Python packages in the package while Debian installs them at installation time so that the Python path ends up there OpenZusa has reached 93% like Debian NetBSD and FreeBSD have reached 100% for their base systems not for the whole ports collection but the base system can be reproducible they don't do it yet but they have shown that it's possible that they lack the source code is ready but their infrastructure not which is the part Benjamin will explain for the Debian stuff we need to hurry up a bit I guess by the way Tails creates 100% reproducible ISO images the ISOs are reproducible they are made reproducible out of unreproducible packages OpenW.t and Coreboot also reached 100% for some targets and there's also work on Yocto, Angular and other stuff Alright, thank you so now that a lot of work has flown into achieving reproducibility the question is how can we bring the property to our users in a way that they can have a high level of confidence that all the binaries they run are actually built reproducibly and the core idea that we are going to use is that we will try very very hard to make sure that every installation the exact same metadata and packages from the archive and this should hold true even if components of the central infrastructure critical components even such as the sign-in key are compromised so that's a property we currently don't have that we can have and that perhaps we should have and this gives us the nice property that all verifications that we do on the packages are directly transferable to every installation and there can be no doubt that it's applicable to a particular installation now in order to do this we need some additional infrastructure so we would add a log server like in certificate transparency approximately there needs to be an augmentation of the up client with some verification parts and we also have verification components that verify that the log server and the archive behave appropriately so to say and in particular they can verify the property of reproducible builds on the archive and now we're in the nice situation that any monitor and anybody can run such a monitor any monitor can detect the violation of the reproducible builds property and this is directly applicable to everyone who runs the Debian software on their machines because everybody has the same view on the archive so any monitor can detect reproducibility problems and we can benefit from this detection mechanism and the absence of any alarm can have us assured that reproducibility is indeed something that we can take as a given there is some more information on this in my debconf talk that happened previously you find the slides here there's code for a prototype you find the repository here the code will move out of this as it's converted to patches for the actual Debian components if you have any ideas or concerns or feedback please do feel free to email me that's my email address here and of course there are also other interesting projects looking in the area for example the in-todo project we are a bit in a hurry we prepared the talk a bit last minute so I'd like to correct Ben please don't just send a mail to Ben send it to our mailing list so we can just open because it's really it's all this is material for Buster plus one probably and as said this is in-todo stuff and we don't really know what to do for Debian Buster you know we have this in policy it's reproducible that's in fact an achievement during debconf 17 last year but there's more problems what I'm missing for Buster now we don't have Debian installer images yet so this is the bug in case you want to read more about this so the DI images are not reproducible we are not comparing packages against the archive yet we just compare with our builds and funding yeah this is another topic actually about funding I'll be funded to work on the last thing there in this list from December to September starting but besides that our funding as a group we had four people funded working on this and that has run out since the end of last year which has led to some we have the new progress has been slowing down we have some things go backwards like this GCC patch we only had it for GCC 7 but we don't have anything for GCC 8 and we also have impact on collaboration and this is all nothing you can do anything about it we just want to inform you why we have slowed a bit down the progress we kept up our weekly blog there's still been lots of progress presented though I think it was a bit less in recent times so this is something we are working on we've talked with quite some foundation it's all in progress but there's nothing concrete yet except the funding for my side for this work on this archive comparison which is only for me because it was a fund which was only available for people living in Germany and yeah the hardware funding on the other hand is still fine we still have support from Profitbricks who still give us half virtual machines with half a terabyte of RAM in total and 100 something CPUs CodeSync still runs our ARM64 Machines and Vagrant runs his ARM-HF form with also some Debian funds so the hardware situation is still fine but we lack funding for the actual people working on it and another problem as I said we need to be very quick now Debian is not wrong everybody is wrong most people are wrong because this 93% is a lie or it's a misconception the 93% is about the sources but we still need infrastructure, processes and policies and testing of the stuff if packages are really reproducible and we need to continuously do it so it looks like some people think we are at 93% but we are really not and then we have a list of bugs which are of course assigned to individual teams but these are bugs the individual teams cannot serve alone it's a fault by Debian and I think this is a bit stalled because people think we work on it we will fix it and we will not fix it we cannot fix it we need the help from teams in Debian Debian and everybody so the point of these bugs is to point out the problems and so the major blockers we have is that d-package and then s-build d-package producer if you do a source upload it will create an AMD64 build info file this gets uploaded and then when the build debills it's also created an AMD64 build info file and then this upload cannot be accepted then we have a problem with binning them use where the build time is the same and this causes problem the bug actually has a very good approach plan for this problem just somebody needs to implement it and change the way binning them use work which is always probably not a good time now anymore in the release process and this build info files the build info files are still only accessible to Debian developers so we need to get them out to some public machine build info debian net is only one machine then we would like to include the build info files in the archive so they get automatically mirrored and for security builds the build info files are also not outputted because there is embargoed build and the stuff so there is another problem there which we need to tackle so my fear is that stretch we released was a deep package which could do reproducible builds but we didn't rebuild the packages so stretch was not reproducible as it could have been Buster lacks the infrastructure and maybe even Bulls always still have not sorted these things out so we need to continuously work on it and make sure this will not happen and the good thing is Buster is still not released is still not frozen so we can still change some things for Buster so hopefully we do that and we not we reproducible builds but we debian so if you want to get involved visit our web page join our IRC channel or also we plan to have another summit at the end of this year if you are interested in this contact us, join the summit get involved so thank you for listening I think we have three minutes or so for questions so while you are thinking of your questions there were two more slides that somehow must have got merged out of the presentation because they were only written about 20 minutes ago but I wanted to mention that built-in profiles there were two use cases one use case is for a particular package, version and architecture I want to see all the people who have been able to reproduce this package and the GPG signature saying that they downloaded the source and got the exact same package that I am about to install how we can have third-party verification of the software we are about to install in order for those third parties to reproduce the actual depth in that archive there really needs to be some bulk download of what I would call the canonical build-in-force not related to Ubuntu but I mean the canonical build the official build that resulted in the depth of the package mirrors what build path was used for example with that information with taking that build-in-force and the source a rebuilder, a third-party should be able to get exactly the same result as the depth that we ship in the archive and it would be awesome if we could get one build-in-force per architecture to the archive and out on the depth in mirrors and even better if we can get signature in the release file allowing people to properly authenticate that so that's something we need to talk with FTP master on the exact details and I really hope we'll have that in place before the next step so any questions now we have like 30 seconds no questions but maybe suggestions because I've been reading your blog entries at Planet Debian but recently I was just skimming new packages were going and so on so maybe instead of just putting this wall of text put some call-off for help or call-off what is the problem what should be done because initially I was reading through them but then some packages got reproducible, cool, everything is progressing so cool I guess this is a problem that we've gone to the part where the software patches are getting kind of boring after three and a half years of producing the same results but at the same time we are a bit stuck because we work less on it and the funding is gone and also because we get into areas where we can develop ideas like what Benjamin does but there's an implementation from other parties as well and we can prototype things and this is stuck at the moment but thanks for the suggestion okay I think we're out of time thank you everyone