 Welcome and good morning This is the reproducible builds team talking about stretching out towards trustworthy computing So yeah, we're far on stage But actually this is a team effort all these people listed here have contributed to the project at one point We're only like the four of us. That's Luna me death dole Chris Lam lambi and Holger But actually this is dev conf and so a lot more of us have been or are currently here And so if you want to thank anybody to what let this working on this you need to actually thank all these folks because yay The people in blue are here Let's get started so Quick quick up on what we're talking about So we have software It's made for source sources reliable bad humans or at least a good amount of humans In this room. It's good binary reading it readable by computer and some tiny fraction of humanity and Going from source to binary. It's called build or like building or compiling And we are doing free software and free software is awesome because well we can actually run these binaries Like we want and we can actually study the software the software how it is being made by studying the source and By studying the source we can assess that it does what it's supposed to do and not something else that has not like have malware or trillions or like security bugs And so we have the binary that can be used fine. We have the source that can be verified Problem is that right now? The only way we know that a binary that we get has been we have to trust like a website Or like a Debian repository that says well this binary has been made with this source, but there's no way we can actually prove that and Well, this is actually a problem that been well explained by McFerry and session at the 31c3 in in Hamburg last December and for example session he made a proof of concept exploit for the Linux kernel that when GCC was called The kernel will without modifying anything anything on the disk when when when the kernel detects that GCC is going to Read us a C file. It will insert some extra lines of codes and These line of codes can actually be very bad thing in the case of the 31c3 so he was just recalling but So actually you can have you can even have like developers who are in very good faith Who like I've totally secured the machine or they thought they have who have like reviewed all their source codes for any bugs and well We would still get totally at home as soon as their computer gets compromised or one of the build demons from Debian gets compromised for example and well, this is not like Hypothetical threat threats here. We're discussing a couple of months after a second and Mike's talk at 31c3 The intercept revealed from the Snowden leaks that at a CAA conference in 2012 there was a one of the talk that happened was about a project called straw horse and straw horse is about modifying Apple Xcode which is the Development environment for for my question and IOS applications and while they were modifying Xcode so it will produce without the developer knowing Bineries with Trojan's malware Modern Mac binaries lots of bad things So Solution enable anyone to reproduce analytical binary packages from a given source because if Using a source using about the same environment Multiple people on different computers on different networks at different times can all get the same thing from the same source all the same binary bite for bite Then there's a good chance that Well, everybody could get it could be on but let's let's be more like like joyful and say that probably if everybody gets the same result they will actually there was no problem and Everybody's safe, and we call that solution reproducible built So actually it's not only about security For DBM, we have if you're doing multi arc same packages Well, they all need to have the same bytes if they are built in is from for different architectures the files in the in the package Debugged packages you can create them at a later time like if you forgot to you have the debug packages in the first place and Well, you can pass the no strip option later And you will get because the package is reproducible you will get the debug symbols that work for software that has been cheap already We do only detection of fail to build from source that way because if we try pretty quickly to reproduce a build then it has to work It's useful for build profiles We can get smaller dot-deb deltas Because from one Russian to the next we might have the same content We do you can do validation of cross builds helmet grown is can't come to you with that and also Nils tire was told me that he was very interested into reproducible builds because It will enable him to test developer better Because you can if the package builds reproducibly then when he makes a change to developer You can rebuild a package and you were like the same version of a package with a newer developer and see what has changed And these change can be isolated to only what he has worked on developer for anybody else and Oh my the wall walls is watching us since since Quite like two years or year and a half Everybody I meet in security conference in hacker conference in three stuff track conference. He's like are you working on that? That's awesome And I mean I've been the one doing quite a lot of talks and So he really comes to me, and I'm like whoa. This is way more way bigger, but we're actually leading the field here and Yeah, yeah So we are not the only ones leading the field Bitcoin and tour made their software reproducible before us Corbwood also succeeded a few bit Corbwood without any payloads is 100% reproducible Free VSD has so page on the wiki since 2013 saying there's five bucks in the basis a five reproducibility issues in the base system and We're in the moment trying to confirm this So on this Jenkins debut net I've also set up now tests for free VSD net VSD Corbwood and open WRT So if you go to reproducible it's a debut net slash these projects you get that tested And there's more in the pipeline. So there's Other projects are interested as well that net VSD also has a variable m car repro Which you can set and then it builds reproducibly though They think I'm keeping some timestamps. It's fine and then filtering them out later. We disagree So this is how Debian looks like Debian sit, but this is a lie. This is not the truth This is just our test setup Sit is not like this for sit. It's all orange. There's zero reproducibility in sit today And we are talk what we'll talk now and in the following round table is to actually create make sit reproducible so the current status is We are working on this in Debian since two years We have weekly reports about our projects now since May and We've given several talks And especially in the last year and all these talks presentation also other stuff is linked in the wiki There's a about page with information about Debian these psd's other Linux's upstream Softwares all on this wiki So since Debian 14, which is merely a year ago we've made quite some changes We have introduced stripped non-determinism which will Take is called by the by DH after after the at the end of the build of the package and will normalize some things Which will do a little you will Chris will explain later We have we decided on a fixed build pass because the build past is leaked in the binaries and then several things We didn't find a way yet to make the build pass arbitrary We just designed a way to record the build environment because to rebuilding You need to recreate the build environment We set up this Jenkins setup We have wrote We wrote differs cope which used to be called debindeth which shows differences between two packages or two directories or two file systems by now There's source date epoch, which is a way That the tools expose the last day last modification of the source Because the build date people want to include the build date often because they think this is a meaningful indication when a build was done or which software is used But if the build is all always create the same result the build date becomes meaningless and the really interesting thing is the last modification of the source and We've written patches for the tools So stripped under the rhythm is Andrew higher in the audience. Yay. So he did it It's it's Simple pearl. It's really in pearl because we didn't want to have a new build dependency on all the bin packages But basically it takes anything and try to normalize it as much as it can so replacing Time stamp or like file permissions or removing some Issues it's it's it's working very well on many formats. It's made to be extensible so we can add actually more things And it's actually run by gh at the end of the process as a whole good said So the build info It's currently like a proposal. We have not yet Contrary agreed, but we're generating them as part of the tests we have and basically to file It's a new control file that will tie the in the same so in the same File the sources degenerate binary the package that we're all used to build this binary and their version and the idea is that we can use this file to Reinstall all these specific versions from snapshot so we'll create the same build environment and Then we can just start to build from that source that was mentioned and See if the binary that being generated matches Match and so that what it looks like for now. So you see there is a source binary the build path because Currently we don't have any good Processing tool for build path in elf and dwarf binaries So we just decided to use the same to specify the build path if so when we do for for later We build we can use that path and be safe So the source it's the DSC that binaries is dot deb and a list of packages with version We currently use the base files Version to know which DBN like stable release is to be used as a basis Yeah So what the general procedure for testing is we built the source We save the results we build it we modify the environment and we build it again and compare the result and Let's start it with a shell script in last year, which I put on Jenkins and then it exploded a bit and Now we have 67 Jenkins jobs running on seven host Since the last week we have four Armhf small boards so we can will be able to test armhf, but very slowly We have two new AMD 64 build nodes The code is now split into Python and bash scripts There's for this boy all the other distro testing There's a lot of bash code now Which is mostly boiler play played and it's actually five lines or something to build free BSD and Five lines to build net BSD, but there's hundred lines boiler code around so it's really not that much code We do test testing unstable and experimental For arm. We will only start with unstable because we do lack hardware. So if you Have hardware to donate to us that would be great. We need SSH in and root basically We are as I said, we are testing Corbwood open WT and the BSD's soon. I will also set up Fedora test I don't want to test all the 20,000 Fedora packages But just 200 or something the base system of Fedora to examine how RPM works to get really the whole free software world reproducible and This is all run on profit with hardware since 2012. So thanks to profit bricks for that and This is the variations we do for Debian So it's the hope the host name the user name the time zone the locale First we'll explain how this what modification these causes Variance is we are not testing at the moment differences in date So the date is always the same the time is a bit difference Well, almost because we cheat with the time zone because we use one time zone that is like GMT minus 14 And then GMT plus 12 and so it's more than 24 hours apart And that's on the first of the months. We sometimes find new bugs where there's packages was record the months And we don't have variations of the CPU type at the moment, but this both Time and CPU type variations will happen in about one or two weeks. The nodes are being prepared at the moment and then we will test More all the meaningful variations we could think of there will be probably some packages Which will be different according to the number of CD drives attached or whatever things but Those will be fine by you And so we're doing all these tests because we want when you will build a package on your machine That if any of these is different from the build demons of Debian you get the same results And we want you to use we use this to detect these problems early Before you actually make a false predictive that we have to investigate when someone rebuilds a package on their machine To understand actually the difference what we found from from one build to the other we so it started like also as a 10 lines sharp script and Then it felt like a cage and so Python and Now it's a lot of code and it's actually grew way beyond just even package and we changed the name It was called Debian diff, but it's absolutely not tied to Debian anymore at all. So it's called defascope Thanks to the slant for the name and basically what it does it tries to get to the bottom of what is different between two archives or directories Because it's not useful to compare bytes That are compressed by jzip or xe that will not leave you to understand what what is different You need to uncompressed and look at the uncompressed data and if the thing that actually compressed is a table Well, you might actually want to compare the files inside. That's our ball and if there are like a PDF inside this archive Well, you actually don't want to compare the bytes of the PDF You want to compare the text of the PDF? So this is basically what? Defascope does is it tries to transform anything that is a container and compare things in that in that comes in these containers And if they are like can be transformed into human readable form it will try to do that and compare these human readable forms and If it doesn't find any difference, but that's the difference from the byte. It will fall back to binary comparison Try it Extended is Python. It's smaller. It's great. It already supports like squash fs and ISO and rpm and Get text and all files and so many different things you can have HTML output Like that. So this is what is displayed on on many examples we've shown so far and also to make it easier for copy paste and Pass processing we have the text output You can also use it to review packages before uploading them to Debian It does fuzzy matching So even if the directory is different in an archive, it will like fine like a little bit like it does And so it's it's way it's green way more beyond just like build reproducibly useful tool. Yeah So in order to solve Time-stamp issues we are proposing the source date epoch variable This is because most of the times having the build date embedded in a package is not useful for the user because you could take a Really old package build it today and that they will not be useful So we are standardizing replacement for build dates so that tools can use it and When this value is set the tool instead of Embedding the current date it will embed the date taken from source date epoch, which will contain a Unix epoch time-stamp This is a general solution. We're trying to Standardize so that not only Debian uses but uses it but other free software projects and distributions And in the case of Debian we set these variable to the latest Debian change lock and three time-stamp and We have already been Sending patches to different packages mostly its documentation generations So here's a list of bugs. We have opened which are which have been closed and Merged so it's helped to mine a pedagogos script taxi to HTML and Sphinx We are both sending these patches to Debian and upstream so that all the distributions can make and use them and And we have also been sending patches to other Packages which are still open. So we encourage you to take a look at these packages if you're the maintainer and merge the patch so thanks, thanks Daniel Kangemo and Shimino for pushing this proposal forward and also like a lot of these patches I've been written by Akira and all as part of their Google some off code and it worked really great The GCC yeah, well he would Yeah, the GCC patches GC uses to macro macros which are data and time which embed the timestamp and I wrote a patch so that If source date epoch is set instead of adding the current time It takes the time from that variable. I send this patch to the to GCC It's still there forgotten with many other patches, but hopefully at some point They will realize that this interesting and they will merge it Hey So like to run you through some very quickly run you through some very simple Ways of fixing packages the details don't necessarily matter It's just to give you a good idea of like what needs to be changed and basically to point out that it's not rocket science So you can just come in and jump in so for example Gzip It's a very old peak tool and they decided to add timestamps when you generate it But it's an easy fix. You just add dash n flag Some other things quite easy to change And some Python stuff had to tag date true, which if you can see it But as a timestamp to eggs, you just change it to false or get rid of it static libraries you They are just a our archives. So the same format is dot-deb and you can just use bin utils or our strip Terminus and tool Ping has timestamps for some reason you can get rid of them. That's image magic. It's a bit ugly But also strip non-determinism Tables are quite interesting. They will by default capture user group You just pass on a route blah blah blah Ordering this interesting as well. So it'll usually just use file system ordering Which is just completely non-deterministic. So you need to sort LC all Oh, wait go back Think about the local because sorting other very from local to the next quite. Yeah, that's fun They also takes timestamps again, you can set M time or you can muck around with find excise And load of other files have timestamps Erlang files for no reason even upstream don't know why they added a timestamp Yeah So we source now a patch for a source state epoch, which I think landed a couple days ago. It's great Here's interesting one not necessarily the current build timestamp. So So this is a time zone dependent date which Ruby loads and then saves incorrectly as your local time when it just gets commingled So that's patching. So I'm going I'm kind of going from Changing individual packages to more to tool trainee type things as you can see Upstream configure scripts you can maybe see at the top that it just uses hostname for their own reason Sometimes you can override in dubbing rules just by exporting something or passing a variable to you know DH auto build or whatever. So that's just a little bit more involved You have to look at them a bit more carefully Pearl hash order. Yeah pearl lot of pearl uses data dumper to just output a bunch of stuff, which is just not the term in a stick So often just setting sort keys But sometimes it's completely different solution Head of files. So you can maybe see that they're using the timestamp essentially as a unique identifier You probably have to start rewriting these using something say no or Because it's just wrong. She's the timestamp. Anyway, just sucks Again, we'll make files. They mean the the deeper the Timestamp in the upstream package the more you have to start patching. So these kind of start sucking a little Yeah, so we've made a lot of tool tone changes. Some already mentioned some of them already merged You can see more of this link again details don't matter. Just check it out. It isn't crazy. It's just Working out what's different In terms of the work done. We've sent these many patches to hatches a day, which is not too bad I can't clap because I've sent three or something like that so three per day Poggers three per day And this doesn't count other Just bugs we found in the process of building packages like fail to build the stuff on that This is blue presumed. Well, you can see in the key blue is one that were opened and Once we've done so you can see that someone went a bit crazy in February filing bugs and eventually Eventually they're all being fixed slowly And actually the we fired more bugs cause these bugs the fails to build from source bugs are excluded I think we fired 300 fails to build from source bugs in the last two or three months So yeah, those include fail to build because of reproducibility things as well, but we haven't spit them up Anyway, great next What's what's left to be done because I said the whole guy said The cake is no the graph is a lie And so while they are issues the main the main thing that is blocking A lot of work is is is a GPKG Right now it the output of the big edge is both be non-deterministic a hundred percent of the time because of time stamps and At least the file ordering We also have a patch that creates these Building for file that we've shown that works We need to it's not submitted yet to the BGS because we need to agree on the format At least we've FTP master or maybe the big edge. Well, we've a lot of people and that's why we're going to do the next hour developer also has a few changes the Mac M times might also not be Developer might not also be the best place. Maybe we want that in the BKG I've been trying to put patches in tar so we can make it easier It's it's complicated to see where is the best place but so far we've been doing this test our test with This frame and it worked this this in our repository We have these bug packages with these bugs fixed so when you want to test reproducibility issues on your own machine You need to use the repository which has these patches applied at the moment So pure set you cannot create reproducible packages So also I heard that the source that epoch Patch is in git already. So it's going to happen Ah CDBS also needed to you export that source that epoch And we are starting to do more also infrastructure work Josh mainly on and I clear out on this build Which are because we want to have this s rebuild Script where you take you give it a build info and we'll do the rebuild and it needs changes in Build demand for the build path and and also like couple of changes and I'll build in as built itself And the script is not ready yet this finishing means we do this our repository at the moment And we won't need to change it to only use it and snapshot So there is the the build the issue that we need to discuss And we also need to see how we could include or not or somewhere Give these building for control file to the world so they can rebuild the packages So it's not yet clear where is the best place to you store them Because while adding twenty two thousand files some some people got cranky about it It's more than twenty two thousand files. It's twenty two thousand source packages multiplied by ten architectures But there's a lot of art art spits or probably that's hundred thousand build info files Multiplied with stretch and sit that's two hundred thousand files on more on the file service and on the mirrors We would like to have it that's the same amount of files which are currently there the mirror operators are not happy They will not take it so our current idea is now to just concat all these files into one file That's hundred forty megabyte uncompressed 40 megabyte compressed that's easier to handle and then probably have a service built in for Debbie and orc Where you can download individual built-in for files if you need them And so when we build on we've all we will be done with all that We can maybe add a final patch. We would be to Debian policy Meningling that Debian packages be replaceable Yeah, why not My so I can I can say that dream again my dream again of mine is that we would stop uploading the ebb When we upload a package, but instead just upload the ash of the binary Have to build a build again this package and only if these two matches they can enter the archive So we are sure that we are at least sure that to machine the developer machine and the building and agrees that they've built the same thing But Of course, I'd share this dream, but I think having this in policy as a must requirement is certainly something only for stretch plus one but I'm curious what What you if we had fixed the deep package and depth help on now What do you think we should upgrade all these wishlist bugs too important now? so makes What? Okay We'll talk about this later soon, so but before that we actually have work to do so So in order to fix your package The first thing you can do is go to reproducible Debian net slash the name of your package and you can see The web interface when you can where you can see nodes on the package We have tax to identify different issues that make package not reproducible with links to the wiki about how to solve them um When you see this you want to click on this step bindi fling it's still called that bindi if not different scope This will show the differences if there is a note if there's not in if the package is unreproducible And there's no note it would automatically display the debindeth and if your package is fine. There's here a son You can also see an entry in the tracker stating if your package is reproducible or not and you can also find information in the DPO and Dmd you can find tips on the wiki. It's the reproducible builds wiki We are working on a how-to to have a detailed Tell steps on different issues and how to solve them Lunar gave a talk and in CC come where there's a many issues really really well explained and the solutions for them and You can also come to our IRC channel, which is that you unreproducible and ask for help or go to the mailing list So in order to test locally if your package is reproducible Right now we are using a script that uses to be builder in a custom configuration You need to set up Our reproducible repository there's in the how-to in the wiki There's the steps on how to set up the CH route and everything that it's documented in the wiki and Default scope is in stable and today it's going in stretch We plan to add These scripts to rebuild packages in different settings in depth scripts once the PKG is good and We Welcome you tomorrow to the hacking session at from two to seven in the stock stock home room And so the that's for fixing your package. Please do that if you want to have even more fun than just your own package Jonas The this is the past year of my life has been awesome because the team has been so great It's been fairly atmosphere lots of new understanding so many things you didn't want to learn about that you had to learn about and Basically, it's like it felt very good to be part of this actual changing the world thing Okay, it's just software, but it's all like, you know, it has some profound effect I've been told that the work we've been doing is like being tossed around in like Cisco and Google and Facebook all these big dot-com companies blah blah blah. They'll actually want to do that as well Even they're not doing free software, which I find weird, but whatever So what do we do like we review packages like we have these notes when we actually try to identify so when maintain a calm They don't have to think too much about the problem and just fix it We tried to identify common trends So when many packages have the same problem that we make an entry and explain and maybe think about fixes that could apply to the whole archive We work on these reproducible that debian.net Jenkins setup the scripts We hack on the default scope tool. We make tripped on the system and that's an even better We propose change for the tool chains when there are needs some it off matches Almost all the bug we have submitted are been on individual patches Well, no, sorry some most of the bug we have reported on individual patches have bug patches Whatever Yes, and also we are actually writing some more general documentation from the understanding and experience we we have been having we are preparing Reproducible builds how to to explain to the world free software world how they can do it. So it's about Some of what Chris explained but also margin our consideration on what if you're not debian And you want your thing reproducible when you distribute as an independent vendor and so we want you have actually Walk on reference documentation. So the world can actually do that And we do a lot of talks as you've seen and it's been fun and we all these Presentation we've made so far. It's all in git and everybody's free to take one of these slides decks and run with it somewhere translated do that, you know And yeah questions we have to run with the microphone because there's no mic anymore Wait Okay So I just wanted to make two quick comments. So first of all Difascope is really awesome Not only for reproducibility, but also so for example if you change your debian rules in some way and want to see if the package is The same afterwards because you just clean up a bit. That's really awesome for that. So thank you and Also, I think this is something the work you're doing now is something that in 20 years time We're going to look back towards it and think well, of course build should be reproducible So, thank you very much for your work Reproducibility becomes Or will become part of the debium policy Will be a lintian me minus minus report usable I don't think Lindsay and can detect that because Lindsay and works on the source packets and you need to build the package for this I mean things that could be detected at by lintian is in from a static analysis point of view I'm sure like looking for gzip without n for example, but it would never be conclusive One thing that I really wanted to add to Difascope at some point that the code is made the way that is possible is to had hints So when it actually looks up differences between two packages, then you have It can like have an idea to also like suggest you hey you need to remove that time stamps or or you should sort these keys It's not done yet. But if anybody wants to do patches, it's totally doable. I don't everyone Thank you for the work and have you thought about reproducible images? It's on the to-do list Before images you need reproducible package installation and then you need to reproducible images like squad of s has some things Which are not reproducible, but the package installation is not reproducible at the moment because up installs packages with an arbitrary order And then the post ins create for example users which get user IDs in the order the packages are installed So for that to fix either up needs to up needs to have a way to install in a Deterministic order, but it's on the list. It's in the to-do file Perhaps started a wiki page a couple of months ago that's called like reproducible install This is very important if we want tools like tales To actually be reproducible. So we will I mean some people will work on that I would do want to work on that from it's quite a deep problem For example, DI will install different stuff depending on your hardware. So that's immediately not reproducible. So It'd be great So I've been working on a couple of my packages to get them reproducible bill, but I was often wondering If I should fix it in my package or actually that it should be fixed in a package higher up And I guess I've been adding some fixes to my package which may in the future not even be Needed anymore, and then it's just a code and necessary code as well So how do you see where things should be fixed and how should we as package maintainers go about with this? There's many things which which there's the easy fix to whatever set the time zone in Depth helper or better in deep package to MUTC, but that will not fix the upstream bugs So actually it's not better not to fix set the time zone or other things deterministic in these tools But rather them have them fixed upstream. That's what we want It might be desired what some things we will need to fix in that deep package to get a meaningful result But basically we want rather there's distributions which just built from source Which don't have WN rules and they just built with upstream makefile. So we want the fixes to lend there Well, what one way we've been experimenting for two years and it sees a lot of trial and error You know trying something see how it feels Oh, maybe we can do better than that and changing and I know this can be frustrating at some point because you do changes and all they All become I need it But in the end this is how we we make stuff that matters It's because we are not trying and we move forward It's not because we're trying to make the big peter at once and I know in demand Sometimes we try to do that and so we experiment and and learn from it Yeah, an example that I'm now looking into is actually that the documentation is built For this package by looking in all the files and and generating but for instance the index file is sorted But I guess upstream would say well if you set some Ordering in your LC parameters you want this page to be ordered as you want Instead of forcing it in the source So I'm really wondering should I now upstream this or should I just fix it in my rules because that's the logical place Both No, there's no good answer. I mean the so I'm I'm I'm a quite a strong proponent on the idea that if you use a computer You should be able to talk to and have the computer talk to you in your language language that you choose So if people wants to have GCC error messages in in in German they should have it So I'm but local sorting. This is the kind of LTO That can be very local and that you can do for just one tool. It's fine to do that Do you have ideas on making sources for a crucible like up screams calling make disk or this infamous auto gen.sh files? I don't know. I don't think that anybody in the team has looked into that yet source Sources is source files are easy to Analyze way more than binary packages. So it's it would still be great to have easier ways to have source Tables be bite-for-bite analytical, but it's not as An issue as it is for binaries But if people want to look in that they should yeah So, do you know a way to make it archive build something reproducible? Well, Pristina. Yes, but without it There's one tool you want to use a new one and write it Why not use that tool, which does the job? Pristina does it Well, we I mean this is this is again This is like for source and so that's another issue that what we're actually currently working on We are welcome to join the team and extend our scope to sources Took how many questions to To two more questions to offer you. Yeah, so if there is a couple of other environment Variables that could be be set in the environment to increase reproducibility where to put them in the rules file or in In the generic build environment of all packages or where should these things be placed? well It'd be nice if upstream fixed it. So if we just changed it in Debian rules, that's just only helping us so Often talking to upstream will be like the ideal solution. Are you referring to something else or well? For example many hash maps have randomized data in the hash functions So if you have some code that relies on hash order At least some implementations of hash functions are letting them be seeded from the outside rather than using Something Random for a building but you want the randomness in your hash functions for normal uses because that is your Hash maps Get filled with or is open to attacks Correct. Yeah, I mean We're in that in these cases. We add we send patches adding sort everywhere for the keys and it's folded For very few cases we for pearl for example, you can set an environment viable and some maintenance prefer to do that But usually we try to push these changes to upstream because that's simple enough and I like it Because it makes it's actually makes testing easier than There was one in the back there last question, that's the last question a Follow-up question to what we had here before You showed an open bug report against GCC to support source epoch date to cover the M date and M time timestamps So I have patches to patch them out in my pack packages Should I remove those patches and if so when? Have you seen any more emails from the GCC maintainers? So the mail is forgotten, I guess we should ping it again and See if they reply Because what I read from the GCC Website is that only the Replies from contributors are the one that matters and I think I mean the maintainers and no maintainer replied to the message so should ping again and See that was just an example My question was more general at which time shall I remove my patches to fix things which were fixed Higher up in the toolchain or shall I just leave them in there once they are in suit Okay, thanks Okay, I guess we're out of time. Thank you for listening Fix your packages