 Yeah, the many arm monster of reproducibility, so Hi, I'm Begrin Cascadian and I may have created a monster So Just a really very brief introduction to the idea of reproducible builds Basically, the idea is Ideally if you build a package from a given set of source code You build it with a particular tool chain the binaries in a really ideal world They should come out the same identical exactly the same They usually don't We're working on fixing that and this is a project that got me into the big mess of trying to demonstrate that it works on Basically trying to find more variations by testing on an entirely different architecture You can find a little bit more about reproducible builds at reproducible builds.org But I'm basically assuming you get what reproducible builds is about at least on some level So what led me to do this crazy thing I'm the U-boot maintainer, which was kind of a crazy thing in its own right and I noticed that U-boot was marked as reproducible And I knew that was absolutely ridiculous because every time I booted U-boot on all of these random systems I've been testing you get a date string with the seconds in it I mean how many times am I in and it the U-boot builds like countless targets They're not all building in under one second Usually takes few minutes at least So I knew something was wrong here and obviously Some packages build differently on different architectures. Some packages don't even get built on every architecture And so I thought well, I've got this handful of arm boards that people have donated to me Just to largely to test U-boot or just because they thought oh here vagrant have a board And I was a sucker to take them But now finally I had something productive to do with them I figured well, I could like set up this whole infrastructure to reproduce these builds myself But why not share? So I decided to contact the reproducible build team and say hey, why don't I set up a few of these boards for you? so We started out pretty small in August we got You know to dual core and to quad core machines running That's an example of one of them. That's a Humming board I to EX One gig a ram dual core pretty cool and Then it went live in September. It was building about 200 source packages a day With over 20,000 packages and really it's probably more like over 24,000 It was looking like it's going to take over a hundred days to build the entire archive just for unstable What a mess that's not you know, it's a pretty big development cycle to test the entire archive so That was just wasn't going to do So I bought a couple more boards and I got some donations I kind of solicited some donations got one from Beagle board and then Holger was like vagrant. Are you buying all these boards yourself? Yeah Well So Holger helped by soliciting to get a donation from the DPL to and we wrote up a proposal to Double or triple the capacity of our build network picture. There's an example of a Beagle board x15 That was one of the ones donated But the majority of our network got built out from funds from Debian This is basically Roughly what it looks like when I started building this out. They're just cables everywhere Cable management is not my strength, but even if it was it's a lot of cables So what's that? Yeah, there are a few So yeah, we can see a handful of systems there But that's basically what it looks like these pictures were taken a few days ago The basic gist of the current setup we've got a few dual core a few quad core You know anywhere from one to four gigs of RAM The till down the four gigs of RAM is well only one of our boards actually recognizes all four gigs We've got a few Q box I four by fours Which recognize only three point eight gigs of RAM, but that's still pretty good So how many what sort of thingies we got, you know, I did a little math because you know, we're pretty good at math right at least with computers 86 cores and 43 gigabytes of RAM But you know, we're comparing apples to oranges Some of the cores are a lot faster than others and some of the RAM is faster or slower So that's those are kind of stupid numbers really So a useful number since we're building packages is we're up to building about 1,400 packages a day and Actually the last few weeks. It's looking more like 1600. You see some of those red spikes And I think one of our best days we got to around 2000 Builds all of unstable testing and experimental in about a month and a half Which that's more reasonable. I also kind of feel like maybe we shouldn't bother testing testing and all that and then we Could get a real development cycle going but So anyone want to take a guess where we started getting a lot of donations It's actually kind of funny Basically, it's basically we started all the new machines went live right in this deep dip right here That's the way life goes But there are some technical glitches and then a few days later We started bumping up to more around like a thousand packages a day And you know getting into some more reasonable numbers The whole build network runs in under 180 watts there's a picture of the Power remote power controller so I can manually reset them They don't always stay up 24-7 Some of them crash we rarely go a day without at least one crash But that switch has some features to auto ping them and reset and reset them when needed and so you know Yeah, and right now. I think one of the boards is actually down at the moment And it's probably not getting back till I get home But and by comparison if you don't know what on what one watt is or 180 watts is The space heaters use that dubcom 16 used between 400 and 800 watts So we could easily run two to you know four of these build networks in the Energy it takes to run just a single one of the space heaters. We've been using Ah This uses a fair amount of bandwidth, and I didn't really think about this when I when I when I started this project, but The vast majority You know we've download about a hundred some Hundred some gigabytes of you know source packages every day, but the real surprise is how many we upload We upload about This is in June we upload like over half a terabyte of You know binary packages because we build them twice, and then we upload somewhere where it does all the comparisons The proxy server, however, I did an apt caching proxy server that delivered You know almost five terabytes of data in June, so I guess the proxy server is working pretty well But yeah, I was a little surprised we started hitting near the bandwidth limits of my ISP at one point and They never complained which I'm glad So we've got a number of different types of boards one of the things one of my goals was well if we're going to try and Create variations. Let's throw in some variety. Let's get as many different types of boards as we can get and Most of them are free-scale IMX six base boards, but we've got a handful of other ones And and there are some multiples on a number of these And Yeah, I really I doubt if we've actually found anything that builds reproducibly on one set of boards and not on another but But in theory it's possible something might actually check the CPU type running and build differently We'll find out And then so as a result of all this I actually ended up enabling a number of platforms and Debian that work out of the box Or almost out of the box got you or Linux support for the Odroid firefly the Odroid zoo for Beagle board X 15 Enabled you boot which I'm the maintainer of for a number of other boards and Enabled Debian installer support for some boards some of the boards actually already had some of this support So I didn't have to do as much so that's why these are not all like perfectly lined up here, but And then there are platforms It's kind of sad So some of them lack mainline kernel and you boot and I got a couple of those I picked up the Odroid C1 myself and we got a couple cubie boards for the build But I just don't go anywhere the vendor kernels suck the vendor you boot it like it might boot But even you know rebooting once a day. That's maybe tolerable, but rebooting every hour or two Not really viable Some of them also lack mainline kernels support the QB truck plus and the leemaker high-key Haven't really been able to use them because the vendor kernels not so great and and Well the mainline support maybe it exists, but like Martin Mikkelmeyer said in his last talk Yeah, you know serial console is nice and all and you can see the RAM and it might even recognize all the CPU cores but not particularly useful if you don't have any USB support or Or you know The micro SD or anything like that and a number of them We're using but they require some non-free binary blobs, you know Raspberry Pi the Odroid family are all disastrous in that regard and I'm not sure about Firefly That's a rock chip based one it might have some binary thing I had trouble resurrecting one when it had a terrible fail and Then I've worked. I'm running some patched u-boots for The Firefly 4 gigabyte variant and the keybox i4 by 4 Need to work on some sort of auto detection to figure out So that you can run the same u-boot on multiple different boards right now the If you try to run the 4 gigabyte variant of u-boot on the to get to a gigabyte board model It'll just crash or something. So doesn't work out so great So we're looking at some other boards in the future We've got a few pine 64 boards waiting at home and there's been really great Progress on mainline support for both u-boot and the kernel I don't know how Odroid C2 is coming, but I'm I'll spend some time on it And then we might get access to some limaker cello or even I've heard rumors of getting access to some HP Moonshot stuff so that would be pretty exciting and One thing I'd really like to try is building the architecture independent packages on two different architectures So we build arch all on one, you know on the RMHF architecture and then build it again on AMD64 Do they come out the same? How will we ever find out? I'm guessing I'm moving a little too fast here, which will eat plenty of time for questions So In the distant future, I think the autonomous reproducible build network will reproduce the entire earth This is kind of the monster we're heading towards Many thanks to all sorts folks Holger has put in an absurd amount of time getting this infrastructure set up We've gotten hardware donations mostly from Debian. They contributed probably about two-thirds of the build network Limaker has donated a handful of boards directly or indirectly not so bad and Beagle board donated the Beagle board X 15, which I don't think it's generally available yet So Debian supports a board that's not even really on the market yet surprise and solid run also donated one of the boards The reproducible builds team is awesome. They've really put up with me in a lot of ways I often pester them about hey, I've got this idea how we could do this or that They've been really supportive even if they say no And my partner did most of the photos in this and So we've got I've got you know, I'll introduce you to the staff We've got a humming board one that basically it's got Two 13 port 13 port usb serial consoles and that's basically just doing some serial console monitoring on all the boards We've got an rpi. We have two rpi Raspberry Pi two boards Look it's a board Most all of the boards are running on 120 gigabyte SSDs some of them with SATA some of them on usb 2 some of them usb 3 um This one is the cubie board 3 aka the cubie truck Uh another our raspberry pi This is one of the cubox i boards. It's one of the most uh, it's literally a black box Yeah, but more open than your typical black box, I guess uh The odride zoo four Those are one of the few ones that actually has a fan which kind of annoys me I guess they just recently made some very large heat sinks so you can run them fanless I would love to get those because they make little worrying noises that drive me crazy But this is one of our best performing boards despite the annoying fan It's uh Perhaps because of Yes, um But it's running. I think a cortex a15 and um, it's got oh, no, this is the octa core board And even though it's only got two gigs of ram. It seems to consistently be one of the higher performing boards in our network This is a one board quad and this was one of the first like fairly capable arm boards. I started working with And uh That one that one is one of the boards that rarely ever reboots Which so if you want a nice solid one, uh the one board quad It's not the most powerful thing on the market, but it's reasonable and pretty stable Really well supported We've got the firefly Uh the orange pie plus two You can really get lost trying to figure out all the orange pie variants Another firefly board another orange pie plus two And I think that's it So, uh, I know we're probably almost out of time which works for me, but How many anybody got any questions Do they uh crash because of kernel bugs or because of heat or what do do you have any idea? Not entirely sure To be perfectly honest, um, some of the they'll definitely I definitely see some kernel panics Sometimes the usb kind of dies when when your root fs is on usb that can be bad Uh, there are all sorts of random things. We don't really have enough time to really troubleshoot all of that Um, but uh, yeah, mostly, you know with 21 boards running We get one maybe two reboots a day that are due to who knows what Um, but it's kind of one of those, you know, you Large rate array One of those this is going to fail. Um, kind of same kind of problem Yeah It looks like we've got a handover in the back corner here. We have also a question Ah, okay, I received so the question is rich boards have been gave the best performance for this implementation Um, I mentioned the the odroid xu4 Was definitely doing pretty well, but it's also one of the ones that reboots the most So it's kind of it's kind of this like well, it still gets the best numbers, but it just crashes randomly. So um, and then the wand board is a little bit less good numbers, but but much more stable so Yeah, uh They're all pretty I mean they range I think they range just under 100 builds a day up to around 200 builds a day per board. Um, I don't yeah Well, first of all Thank you for Doing all this setup is quite impressive And the question is if you grow your network Uh, would it be possible to share resources for qa? Like for example hooking some of the boards to to lava Which is a framework Leonardo dash for doing our validation and stuff like that Yeah, I think it's possible. Um, we could maybe just do some scheduled downtime for the the reproducible builds network It's not absolutely essential. We're running it, you know 24 7 um, and uh We briefly chatted about uh, actually how to do some u-boot testing over lava and uh But yeah, I'm open to the idea. Most of it most of the network is done with deviant funds too. So Yeah, I would love to share Yeah So, okay, thank you for a talk. We are out of time now. All right And so um, I think that the discussions can always start Yeah at the tech company and thank you for your presentation. All right. Thank you