 Felly mae'n bwysig i gael'i gweithio i'r Gweithredu'r ddebyn ar gyfer arnau sydd. Ddod. Ddod. Mae gennym ni'n cael ei gwaith, ac rwyf y gallwn i'n gweithio i drydech ar gyfer ei ddebyn. Felly rwyf i'n gweithio i'r gweithio i gyd, ond dweud y dyma'r ddebyn ar gyfer ar gyfer ar gyfer ar 64 ar gyfer, Dyna'r gweithio i'r gweithio i'r ddebyn yn cael ei gweithio i'r gweithio i'r ddebyn, ... 살 but this is just something we did. As within ARM and I think it is quite an interesting example of companies using our stuff and finding it good. I quite like some feedback from anyone else who has been doing mass rebuilds about how you do it. The reason why is there was something wrong with an ARM core, there was a particular instruction sequence... ...which could go wrong under obscure circumstances. Felly, mae'n bod yn gweithio yn edrych yn ei eich cyfnod o'r meddwl. A'r yn cyfein i'r hyn arno'r meddwl yn ei eich cyfnod. Felly, mae'n golygu gweithio yng nghymru, y cyfnod o'r meddwl newydd, amserullad methu, felly, mae'n gweithio'r meddwl yn ei eich cyfnod o'r meddwl, mae hynny'n cael ei gwybod i'r meddwl mewn meddwl i'r meddwl i'r meddwl. Mae'n meddwl i ddim yn cael ei gwybod i ddim yn ei meddwl, They'd fixed binutils in this case to not admit the sequence and rebuilding all of Deby and Ant, and Android and some other things. They'd be reasonably confident that this problem wouldn't affect people. They happened to know that I was the Deby difficulties build guy. Hopefully hours now and you've had to do this, I was asked to help. This is an internal project, companies are sensitive, chip not working erhouses. felly wnaeth dwi'n gweithio'n gweithio fel y byddwyr hynny. Felly wnaeth wnaeth hynny'n gweithio'n risau. A oes, mae'n tuerunio'n gweithio, mae'n gweithio'n gweithio'n gweithio'n gweithio fel y byddwyr hynny. Rho o'ch meddwl, rhesyn gweithio'n gweithio'n gweithio'n gweithio'r drwy'r ddebyl a'r rhan o'r hanes, felly rhaid i'w flynedd i'ch gweithio'n gweithio'n gweithio. Edmund Grimley Evans rydyn ni gweithio bod yw'r hyn o'r gweithio gweithio'r gweithio a'r ddysgu'r gwaith. Thomas Pridom, sy'n ddebyg yng nghylch, rwyf yn ddebyg y tŵlchain, ac mae'r dda i'r ddaeth. Yn ffwrdd, rwyf wedyn eisiau yn gweithio'n gweithio, ac oeddwn i'n gweithio'n gweithio'r Lennaro, fel Nghymryd Lennaro yn ymgyrchol, i'w dweud y byddwch i'w ddyn nhw'n gweithio'r cyflogwyr i'w ddechrau'r cyflogwyr yn ei gwasgwyr yn ymdweud. Yn ymweld, dyna gweithio'r cyflogwyr yn ymddangos i'w ddechrau. Mae'n gweithio'r cyflogwyr yn ymdweud a'i gweithio'r cyflogwyr yn ymdweud. Mae'r cyflogwyr yn debyg, mae'r cyflogwyr yn ymdweud. Felly, mae'r cyflogwyr yn ymdweud. Mae'r cyflogwyr yn javascript. Mae'n meddwl y gallwch yn gweithio'n gweithio'n gweithio ar y cyfnodau. Yn y gallu'n gweithio, nid yw'n gwir yw, mae'n cael gweithio'n gweithio. Mae'n gweithio, rydyn ni'n credu'r gwithio, mae'n gweithio'n gweithio'n gweithio. Mae'n gweithio'n gweithio'n gweithio, 14 mwntadau ar gyfer HP ar y ddechrau Centry. Mae'n cyfrifig ar y ddweud. Mae'n gweithio'n gweithio ar ddiwaith fel y dogfwydiad. is done an x86 everything they shipped to customers is for x86 they don't do anything on their own bloody architecture it's terrible so some of us have been going on about this for a long time so you ought to provide software to people so they could use your architecture because you're kind of expecting them to maybe anyway stop ranting um so anyway they had some nice new machines for us to use they were running a bunter and they didn't really want to change that but actually it turns out building all of devian in a charoute on a bunter works just fine and we had route access so it's easy to set up oh look fucking slides of stuff this could get a bit tedious i want to go faster than that normally i turned javascript off to make it stop doing this but of course these slides will javascript so that won't work there we go so the obvious thing will be to reuse what people have done before for cloud building i know lucris had done lots of rebuilds um but it turns out his stuff is all for amazon web services so we couldn't use those because that's external and also it turns out they were in ruby and all three of us had no clue about ruby so that kind of put us off and it looks at all the other tools that might do this rebuild and debil and maybe want to build um but all of them had issues so you couldn't actually install debil uh earlier this year apparently you can now uh as far as i know rebuild d has no idea of multiple build nodes it just has like the one um and want to build is designed for the ongoing process of building stuff and that's exactly what you don't want you want to make a list of all the packages now build them all uh and not have new ones turning up uh and so you'd never finish then so you want to build a static list uh and also setting up want to build is a pain in the ass and we'd have to have email working with all the machines and uh so i was looking at pi bit because that's actually installable and it can do multiple machines and it's quite sensible but in the meantime um edmund wrote a 40 line shell script that did the job and so we went okay well we'll do like that then i guess i he's off he's building things already uh so it turns out that actually building all of debian is surprisingly simple the hard part is setting up all the machines you need to build it on that's a bit fiddly um so they asked us to be in a hurry and within about three days we've actually got a first test build running so it wasn't really very hard uh so the basic design um i recommended was trying to use the standard tools as much as possible we've got lots of cool stuff in our ecosystem uh s build we'll just do builds and you don't have to do anything complicated use app casher so you're not downloading dependencies over and over and over again use rep repro to make a repository that's it really ssh um shell and pearl that was the sum total of our toolkit oh and gpg um to avoid the problem of new packages to actually we did we did all of this in jesse but also to avoid any issues of updates you use a snapshot um and then you've got a fixed set of packages that isn't changing whilst you're trying to rebuild it um put test tools i think that's a typo i ignore that um uh yeah so the um the updated toolchain which you're testing in this case although you could be testing an updated anything it's not specific to toolchains that was just this use case goes in an overlay repository which is always used you just have to make sure it's got a newer version number or some pinning to make sure those packages actually get used in the build uh and one of the neat things if you think about this for a few minutes is you have to build the longest jobs first if you're trying to build all of debian as fast as possible it's no good if the last job you do takes 14 hours because the other 13 machines have finished and in 13 and a half hours the last one will finish so you do all the big long open office and webkit and crazy shit at the beginning uh or one each machine and then you fill them all in with the tiny jobs at the end so they all end together so that maximizes your uh throughput over time now actually it turns out that's a little bit more complicated than that because at the end you're building hundreds of tiny jobs which all take like 40 seconds uh and the master that's controlling things gets a bit overloaded so actually you probably want to alphabetize the end of the list rather than sort it all into the tiniest jobs but to a first approximation slow jobs first uh we use a package tool chain because that made it easy it means s build can use the infrastructure as soon as you're using binary tool chains your life gets more complicated and the purpose of this exercise was to have the binaries left over at the end so we could run a scanner on them and check that this uh bad sequence never occurred um and if you're doing this strictly because we're not doing it in dependency order at all we're just building things we're building them against all the stuff that might be polluted so potentially your new binaries are still polluted by the old binaries if they've just included things so you really ought to rebuild it all twice to be sure now in practice we found that wasn't necessary in this case but obviously designing a system that can do that is a good idea so making the tool chain is pretty simple um so long as there isn't too much skew between the version you're working so they're working on latest upstream binutils today with this fixes in um and we want a debium package tool chain out and actually the easiest way to do that is just take the upstream table stick the debium packages on top bump the version so it's always newer and we'll get used in the build and build it uh and that's working fine now you might have a problem that if you're trying to test a very old release um binutils is a long way ahead then the debium patch diff won't actually apply without some more work but we just had a little stupid script that updated that so every time I knew because we ended up trying quite a lot of versions of this fix um so there were several rebuilds of the tool chain and then rerun the the set oh and the slides have stopped again it's just marvelous um if I could remember what was on the next slide I could tell you here we go yes so we had to make a package list to build now this is the ugliest part of the whole thing this is embarrassing this is the crazy pearl script that uh Edmund came up with to produce a set of all the source packages um for you to build which have an architecture dependent part um and you go there has to be a better way Edmund really sure that so I'm pretty sure you can do it with grep decontrol but I spent a long time trying to get grep decontrol to produce all the source packages in debium uh and I could only ever get it to produce all the source packages with an arm binary in debium so if someone can tell me the right rune for that I would be appreciated because I'm sure it can be done in like a line really um so anyway you end up with just a great long list of um packages ordered the same way they are on build ds with an initial number an initial letter and the actual package name and all the versions which you can give straight to s build sorted by build time so that the build d just starts at the top and works his way through so we designate one machine to control it all um and we run a building script for each build node it turned out we could run two of those in each build node and they didn't get any slower so maybe we could run three um and all the script does is ssh to the build node and run s build next line in the package list that's it it's not very complicated and then if if it doesn't work you do it try it again a second time and a third time uh I would go into more detail on this but I don't have time if anyone cares uh I will be sticking all this on a wiki page so you can do it so that is all you need to run uh you know rebuild everything in debium 14 servers uh two things per server create a package list and start them all off oh and the thing's stuck again so um I will show you the script just because uh I'm quite pleased with how simple and mindless it is it's not perfect by any means um but that's it so basically for each uh line in the package list run ssh to run s build and then if it finished it runs the post build command of touch a file so that you get a list of which ones worked and which ones didn't build it in jesse so the interesting question of whether to build with dash a capital A or not so that's building architecture all packages so in principle the only only architecture specific packages vary uh on your architecture and could possibly have this code in but it's also actually interesting to build all the architecture all packages on an architecture that's not amd64 because you'll discover a whole load of things that don't work because no one has ever built any of those that arch all packages on other architectures because there's no need for it uh so it turns out that some of them are broken so here's the certain amount of stuff about how to set up the machines again I don't want to go into too much detail but we have an overlay repo install the standard tools set up archive keys uh make that available over htdp is the standard way of doing things the charutes we just made with s build create charute which is nice because it sets up the s charute config as well it's just one file you don't have to edit there's a few little wrinkles we're using apt casha so you want to tell each charute to use apt casha and you have to have this check valid until false thing if you're using a snapshot otherwise a week after you start all your packages will stop installing because the key things expired and that's incredibly annoying now so that's just a little wrinkle you discover about four days into the process and you'd knock off signing key for uploading through repository on each build node all you have to have is ssh and s build and some keys so that you can do passwordless login to run those ssh s build commands it's best to have a user for doing this so that you can throw away all the config on those machines easily we actually did it as the s build user the s build install but that's the wrong answer and you should use a specific user for this uh and you just copy over the charute you made uh is it really so some results so of 9889 source packages which should build an arm 64 binary turns out four of them in jesse have missing build dependencies that's probably a bug um 15 just broke for various reasons seven sometimes broke which is interesting um a qa results uh and 99.7 of them worked so that's pretty good we also actually built all the archial package no sorry the even though we asked it not to build all the archial packages it still tried to build some of them and i don't know why that is i haven't investigated that properly that seems a bit odd um i don't know if anyone knows why that is build these so maintainers don't test them properly maintainers are bad people yeah so there's there's there's this is a useful qa exercise in that you will find out some of this stuff that we ought to just fix and we have in fact filed a fair number of bugs from things we discovered so the bottom line here is it only takes 48 hours with 40 nodes to build all of debian on arm 64 and everyone knows arms are incredibly slow and shit so i thought that was quite good actually i was expecting it to take about a week at least um so yeah that was good and even with the archial stuff then you about 5% of stuff that's broken and my understanding of this is it should be much faster if i built in tempifes instead of real file systems but it didn't actually seem to be i don't know if that's because the kernel is really clever about its memory management these days my experience these days is if you use um eat my data you get all the benefits that you used to get with tempifes without the pain okay that is it that's a default in some esbult setups might be in yours right possibly and apricacia was really helpful that does make a big difference even with a fast network so that's pretty much it um oh yeah we get quite a bit interesting bugs if you have two users called root there's one package that won't build turns out all the arm machines in the cluster have zero and one being root okay that's a bit weird but apparently that's perfectly legal and it should work uh and equally uh audit tests users uid 890 um and expect it not to be a user but we've got quite a lot of users so that's just wrong um and some things just fail sometimes uh and those are clearly bugs um and you don't find them unless you keep rebuilding stuff you know in the build these keep trying things until they've worked and they go okay it works right but actually there's something wrong uh we did discover that dash j4 is not the same as build options equals 4 because it parallelizes the make as well as that so it parallelizes the debi and make file as well as the rest of your build right and it turns out that quite a lot of our debi and make files are not parallelizable and there's a way of declaring it but they don't right so we really ought to put the uh there's a there's a make rune for do not try and parallelize this make file it'll explode uh and quite a lot of the files that should have that don't um the key gen thing in sbuild is really annoying you don't want to run it on your server because it takes like half an hour to get enough entropy so you generate those once and then copy them in uh and there was a bug in sbuild we discovered which confused matters so there's the summary of this is actually the rebuilding part of everything is trivial right it's a tiny shell script um you know it doesn't cover all possible failure cases it's a bit stupid if the machine goes down uh it thinks everything doesn't build but one of the retries should catch that um but actually the complicated part of this is setting creating itchy routes and setting up your builds with all the right keys and all that stuff which is what the amazon scripts do but those scripts are no use at all if you're not on an amazon cloud or at least they're not much use in their current form so we don't have a generic way of doing this and i think quite a lot more people have enough machine resource to do this than used to be the case right you know it was a big deal rebuilding all the debi in a few years ago and now uh you know if you can get 14 boxes together you can do in a couple of those i think probably quite a lot of people have access to that sort of resource so people at arm were terribly impressed that we could rebuild the world in like it took us a week from a standing start um and they got the answer yes your fix is fixed uh so you know we got some kudos for that uh they they thought the tools were nice and it wasn't an impossible thing so really the next question is what uh does anybody else do this um what should I do with these scripts i can put them on a wiki page um should we perhaps do more work on this collab qa scripts at the moment it supports aws we could we could jam this incredibly primitive version in as well and say here's the dim version for people like shell and pearl and don't understand complicated systems um you know there's quite a lot more you could do to make this easy um i wonder how much interest there is in that thank you very much wiki are there any questions from the audience you had explicitly stated that uh solving the dependency chain down was not a target for your work in this case yes um for the general purpose of making sure um a system is really bootstrapable that would be a requirement that have been works on that finding the proper order for bootstrapping which appears to be quite complicated with the approach just rebuild with existing binaries if you even if you build them twice or three times uh i currently don't see a way to make sure that there is no more pollution from previous binaries in the room just missing some something uh no i think you're right uh you can think of really obscure if something just copies something in uh and it keeps doing that down the chain then yeah no number of simple rebuilds will necessarily get rid of that copied in thing um so yeah you're right it's not perfect um but it's so it very much depends what you're testing and i have no idea how much would actually get you know raw inclusion of code in other packages you know statically linked library pieces i suppose is probably the biggest risk um so yeah you're right uh it's not perfect and yeah there's a much harder problem doing it in actual rigorous bootstrap order uh as you know but we're also working on that any more any more are we done time to stop hi so um in the pearl team uh we're rebuilding most of the things that depend on pearl particularly the the excess packages against newer versions of pearl and we do have a simple dependency order resolution script for that use case which so and i'm aware that other people have probably done the same thing as well but there's not a particularly good way to find these so i think a wiki page which just listed these simple techniques would be really good right and we can start to generalize things a bit because there are quite a lot of little wrinkles having done quite a lot of this building you know there's a load of stuff like oh yeah you need to set this obscure um apt variable otherwise it'll break and just how to get your keys in the right places and if you haven't done this before it will take you weeks to get all that stuff right whereas if somebody explains it on a wiki page then even if you haven't got the nice script to do it in all the general cases it's fairly painless uh i also wondered if anyone knows whether there's a way to have us build this uh create your route automatically set up lvm snapshots because that seems like a really nice way to do things rather than yeah i mean it's got the dash dash make me a table but it hasn't got a dash dash make me an lvm yes the muck charoot yes muck s build there is no reason why we have s build great charoot and muck s build right they just have a different set of features and we read someone someone should really just make them both do the good things or one of them do all the things okay we have reached the allot at the time thanks again wuki for this interesting talk and uh wuki will be around later for a discussion tomorrow