 So welcome back everybody after lunch Just as a short announcement dinner will also be in the Cantina tonight if so everybody's on the same page and First talk in our afternoon session is by Lucas on building reproducible builds into app with in total Is the microphone on yeah, it is Welcome back from the lunch break. Thanks for having me here. I always feel very Welcome at these Debian events, and I'm glad to to come My name is Lucas. I am a developer at a research lab security research lab at New York University and I will show you how we built reproducible builds verification into Debian app Using a supply chain verification Framework that we developed and which is called in total. I have one slide about reproducible builds I suppose this is not news to you Let's still walk through it a build is reproducible if given the same source code build environment and build instructions Any party can recreate bit by bit identical copies of all specified artifacts. So this is a Feature that's pretty desirable for a build because it allows us to compare builds or build results and that in turn allows us to Establish a consensus on what the correct result of a build is and by reverse conclusion It allows us to detect incorrect builds under quotes So why should we care about incorrect builds? This is a fine selection of Attacks and compromises on the software supply chain. So somewhere between writing the code building packaging in pet packaging it releasing it attackers infiltrated the software supply chain and introduced backdoors malicious code for attackers, this is very attractive because with a single successful compromise they can Impact hundreds thousands millions of users So we should care about checking whether the bill is correct So the question is how we how we how we check that how we should should we all Build every software that we want to use ourselves and then then compare it to someone else so we we need some tooling for that and we developed a A setup for rebuilders Actually, it was together with a few people from the reproducible builds community One of them is here this week. He's not here today KPC YRD and some folks in Norway and At our University too they created the setup for rebuilders which are servers that periodically scan the Debian archive infrastructure for new reproducible builds and Take the bill info file from there and then just blindly rebuild them and Make them available to To a client to to apt so that when So actually they make the results available to the client so that when you install your package You don't have to rebuild yourself. You just consult a threshold of trust rebuilders and Ask them if they agree on the on the hash and And you can decide whether it's okay or not and in total provides the very case verification protocol for that in total is I will it's not so much in total talk, but more Like a reproducible builds apt in total talk so I want to do the whole in total spiel but Explain more with the example of reproducible builds, but in general in total allows you to specify all the steps in the supply chain that That you want to cover like tagging a release running tests Creating a build Putting in it some into some package you can define all of this in a So-called layout these steps in our case. It's only one step. It's called rebuild and Then you say who's supposed to carry out that step We call those actors functionaries You just list the key in your project definition your layout and say this person is allowed to provide rebuild results for me And you can also use thresholds what we do with the rebuilders We say okay We want K out of N of our functionaries to provide us build results in our case we have we have currently run we have a rebuilder run at in our New York University infrastructure There is one running at the University of Bergen and I think it would be very good if Debian could run a rebuilder but basically anyone can run it a rebuilder anyone who has a mildly powerful machine can just run Rebuilders and the more there are the easier or the less we have to trust individual rebuilders So we can define these thresholds in our in total layout and We can also define For for a step of the supply chain, we can say what should go into a step and what should come out of a step So the materials and the products for a rebuild step We would define okay the sources go into the rebuild step and the binary comes out of the recent rebuild step and probably a Building for file or some other metadata and The layout this project definition is the root of trust or not the layout itself but it's signed by a root of trust key and And serves also as pki it ships out the Functionary keys so for instance the keys of the rebuilders are embedded in the layout So you don't have to worry about those keys You just need one key or if you sign the layout with multiple kills keys you only have to care about the worry about those keys and And as I said before each rebuilder just runs its regular command like as rebuilt and robs it in in our in total tooling in order to Generate evidence for the steps or this one step of the supply chain And this evidence lists all the files that went into the step by its ashes And all the stuff the files that came out of the step by its ashes and puts a signature with its Functionary key on it and when we want to verify it on the client we just grab all the The evidence files the link files from the rebuilders we take the layout It's as I will show you it best best way to handle this the layout is already on the client it could be installed with apt and it takes the package from some mirror and Runs in total verification team And I have prepared a demo for this On my demo machine This is a little They've been client that Has trend that the apt are apt plug-in for in total and reproducible builds already installed and pre-configured there are Two or three configuration file that we can take a look at there is one and In here It lists our rebuilders where we can get the link evidence that the in total link files from that says I rebuilt this I came up with this result This doesn't establish any trust relationship as I said earlier the keys for the rebuilders are in our assigned layout that layout Is here at this path I configured it with with my app client that the layout is already there And I will tell the app client to load it from there. And this is the root key. That's used to sign the layout so we can take Look at our GPG key chain. Oh Yeah, it's there. It's just a demo key. It's in the in the demo repo That's used to to sign the the layout and the layout is at total root layout and It has this section with steps And as I said, it's only one step that's called rebuild and Here are the The keys from the rebuilders. It's only the key ideas. The keys are themselves are at a different Different section in the layout And one very important thing that I want to show you is a Rule that is executed during the verification routine on the client And that says okay Match all the depth files that I currently am downloading. So it's a it's a generic rule It would work with any package match all Match any depth files that I'm currently downloading and trying to install with the products that came out of the rebuild step And then there's another one. It's like it's a little bit. It works a little bit like a firewall There is another rule that says and Just allow all other depth files that I have here Okay, so we took a look at the config file. We took a look at the layout now We have to do one more configuration step We have to tell Apped to not just download the package from the archive as it would do usually but use our in-toto plugin. It's called a transport To perform that verification And then we say get update So you can see here. It's already doing something unusual. It's trying to run in-toto on the files that are Fetched while while updating but we only care for depth files at the moment. So it's kept Let's try to install a package. It's in our mirror Get install demo package Oh, I made this bold and the most important thing blue so that you can distinguish it from the other Parts and the app log That's just for the demo here. So What it did here is it requested the metadata for the demo package from the two rebuilders The in-toto link metadata and it got it and Then it loaded the layout it verified the signature of the layout with the with the root key that we configured Then it ran the in-toto verification with the with the rules for the artifacts and the Hashes matched so everything's good Now I will show what happens if our mirror is malicious that I will first remove the package To download it again later Oh package and Then we update the sources list To not use our good mirror But our bad mirror Which will serve as a malicious That package for the for the demo program we have here and I do it get update To get the files from the bad mirror And then I do an up get install demo package and it blew up which is good It did the same thing as before it asked the same rebuilders as before for the same package to Provide the evidence for the rebuild Then it loads the layout it verifies the signature and it does the in-toto verification Which as I said earlier when we looked at the layout tries to to match the Binary that I'm trying to install with the binary that was created on the rebuilders and Yeah, we can we can also we can take a quick look we still have time Yeah, we can take a quick look at this link metadata the evidence that the rebuilder service It's it's really simple, it's not a big file It's only because we only we only record we don't do anything with the the materials Which are the sources so we don't record them right now, but later we should record them So we at the moment we only care for What got spit out of a rebuilder and that was the Debian package and we say we stored a hash and we give it a signature and Then total verification routine can use that together with the layout to see if everything is good And I have one more slide so you might think okay And it looks cool, but it's still kind of complicated. I did it in total talk at Debcon 2017 and Back then I showed how you use the command line tools and you have thousands of Command line arguments and people told me back then that it looks really cool But it's way too complicated and I think it got a lot better Now you have three or two config files that are maybe already in place when you install it and you just run up to install and You will need something to To to compare the hashes locally with the hashes from rebuilders, so that's tooling and the cool thing about in total is that you can It's it's really generic and you can arbitrarily Extended up the supply chain, so you don't in this example We only check for evidence from rebuilders, but the supply chain is longer There is a there is a downstream repo and we want to be sure that only what came out of the downstream repo went into the Rebuilders and even before there are there is an upstream repo and in total allows for so Total is only concerned about steps and all of these are just steps for in total and you can say I have these steps and and then we have this little rule language where Where you say what came out of this step must go in this step and some other rules too And you can have testing steps and other building steps and so on That's it for today Thank you very much. I'm open for questions. Please reach out to us I was able with help from some of you guys to package in total and to transport This week. It's not yet uploaded, but it builds with with as built So, yeah, I hope holger and I can sit together later today or Sometime and upload them so that it's really as easy as app get install in total and then up get install anything and It's verified against rebuilders and also the rebuilders themselves. We need help with that day For this demo, I just mocked them There are running rebuilders that actually rebuild but we could do do some more work on those If you have extra time and like the project reach out to me or my colleagues Some of these friendly faces here Yeah, thanks Thanks for the talk. We have time for a couple of questions. If there's any please step up to the microphone Was clear just app get install this it's clear. Yeah, it's easy. It's super easy. Just wait for any Oh, there's one. Yes Hi, so I'm not sure if I totally understand so you replace Normal apt Repository with which is signed by Debian keys, but by those repositories that are signed by other keys Basically because I've seen many keys and signatures and so on and currently in Debian. We have this Debian key ring which Which has all the keys which which are used to verify signatures of packages. How do you? Yeah, manage keys for rebuilders That's a good question So it's not not invasive. It doesn't change anything with the Debian mirrors Unless you want it to unless you want them to to order the Debian archives to Provide link metadata as well that can be matched with the what the rebuilders provide But those are different entities. So you have Let me see this was my attempt It's not really good. There should be another component here That's where you fetch the depth file from and the rebuilders. They don't give you the depth file They don't give you the binary. They just tell you I built the binary this binary that you're trying to install and I came up It yielded this hash. So You just have something in addition that provides you meta data You can check against what you're trying to download from from Debian and And as I tried to say in the layout you also ship out the keys of of these rebuilders So it has its own peak in pki kind of you don't rely on the under Debian developer keychain So you just change the transport not the rest of the uri. Exactly. Yeah, so actually what I did is it's a man in the middle apt and HTTP so I still use the HTTP transport But I just rely everything that comes from apt asks me or tells me I want to download this thing from this HTTP URL and I Tell it HTTP transport to do it and then when HTTP comes back to me Telling me I downloaded this I wait to tell up to install it and do my verification Only if it verifies properly will let tell app to go and install it Okay, so I have a question. What if a package is not reproducible? Will you just refuse to install it? Oh, in that case it just installs fine. Oh, there is a Configure, it's also a good question. There is a because on our current rebuilders. Oh, I removed that a bit bigger Yeah, and also I have to show you a different configuration file from the tests Sorry well that That does show it No, whatever switching between yeah, and sublime is weird. I Have this no fail option That just that just Let's just still if even if you use the transport and do the verification still lets you install it even if it fails Right, but what if I wanted to fail if a package is reproducible, but There's it's it's the wrong hash, but if the package is not reproducible I just want to install it without in total interfering it. I mean you see when there's a third. Yeah, and that's good Okay, that's currently that that's possible Currently it will also fail if there is no If there is no metadata or it doesn't it doesn't this it distinguish between packages that are reproducible or not But it could be it could be changed. I mean ideally at some point everything is reproducible until then It's probably a good idea to Yeah, exactly, but if people see either it fails on the first package They want to install which is not reproducible and then they just switch it off and then They switch it off. You know you see what I mean. So that does make sense That's why that's that's why I have this option where it still it shows you the log I think the log could be improved It could and it could even Be maybe Interactive that it tells you it failed either because it's not reproducible or because it is reproducible, but There was another problem Do you still want to install it? You might have said it and sorry if I missed it, but what is actually taking into account when you compare? Packages, is it just like it's probably not just a hash of the devian of the depth file But what is actually it is just a hash of the depth on okay, so what wouldn't the Regenerated depth file would it can contain like Times M's change locks up with something that might not if it's reproducible So that's the goal of reproducible for reproducible builds that when you build it under the same Circumstances same build environment. There was the first slide I had Okay, I thought I would just apply to the contents of the the package, but it's also applies to the package itself I Think Not sure Holger Can you can you repeat the question? What is actually being looked at when you try to compare or when you try to to detect the Reproducible do you actually look at the contents of the package or is all just a package as a whole So is the content the package itself also reproducible? I think it's really that's the goal. Yeah, that's the goal So in my demo as I said, I just mocked the rebuilders But we have rebuilders and I looked at some packages where both rebuilders Rebuild them and they have the same hash So I always figured that the depth file the goal of reproducible builds is that the depth file is Can be compared I think great what you're doing, but I always worry about end users. Yeah, how are we gonna get the configuration? There and is it all guessable or discoverable and if it is discoverable, how do you keep that secure? I'm not sure I understand so it shouldn't be secure Secured by a secure. Hmm. You got these rebuilders. Yeah Yeah, and you've got to find them and put them into your You've got to get these Rebuilders exactly and you've got to find them and put them into your config. Yeah, that doesn't sound like a very user-friendly Process at the moment at the moment. So How I created the package now in the last days I copy a config with the rebuilders that currently exist and if you My idea is that if you update this app transport in total app transport You update it with with new rebuilders that exist and that are endorsed by the Debian community and the the URLs themselves are not directly the Trust the they don't establish the trust connection, but it's the keys So the this route layout and there we have a discussion who who is allowed to sign a route layout Is it the one who? manages the the app transport package or It could be a quorum of people. So on the on the github page This give github page. There is an issue where we discuss this It's a it's not an easy question, but I think most of the configuration can be shipped out when installing the transport some of the configuration should The user should be able to to put it there him or herself I'm always for security faults But if you if it then ends up that you can't use it as Michael said then then it's not worth it so it's better to maybe have a An easy adoption or easy rollout configuration Any more questions? If not, let's thank Lucas again