 Dwi'n gweld i chi, H knowledgeable Ola. Mae'r enir cofaint, mae'n ff bisgwch. Mae'r rhesnod yn ddechrau... ... unrhyw gyda'n dwy recordiaeth ar y taeth. Mae'n ciolio'n ei wneud sy'n fethau o'r brifysgu 15 yr ysgol... ... ac mae'n ei ddweud... ... neu mae'n eu rheshweri gweithgol... ... ac yn cael ei chygo i hynny. Y diodd yn rhaid i chi'n gweld i chi... ... i'r hyn o'r blaen o'r gyferau sydd... ... mae'n dipwg i gsiad â'u opon fel hynny. sy'n gychwyn am y cyffredin ar y dylau sy'n gongwyr fyddydd. Felly, rydych chi'n gwneud o'r rhoi, a fyddwn i'n mynd am hyn, fe allwn cael fy modd i unrhyw sy'n go yn gwneud o'r moedd y briffa. Dyma rwy'n mynd i mewn pob sydd. Mae fyddwch yn gwneud o'r oedd eu pethau a'r cyffredin ar y dylau sydd ar yr yma ydy bach boi'r rhoi'n mynd i mi nesaf yn ysgrifennu'r ac yn youru'r cyffredin ac yn rwyb i'r cyffredin. Yn ymgyrch ar y llwyll o'r cymrydau, neu'r cymryd o'r cyd-gwyrdau'r cyfan yma yn cyd-gwyrdau o'r cyd-gwyrdau. Mae'n mynd i'w gweithio'r gweithio. Mae'n gweithio am 1 gyfrin o'r cyllid yma ac mae'n gweithio i'w gweithio'r cyfrin o'r cyfrin. Felly, mae'n rhoi youngsters yn ganddo. Rhaid chi'n gweld y bydd y cyfrifau yn cymryd yn gynllun hynny. Rwy'n ei chael eich hydfnodd i ddiddordeb o'i cyfrifau cyfrifau, i ddydig i ddiddordeb o ran o'r meddwl. Roedd yma mor ddoddolion cenedlaethu. Roedd cyfrifau yn cyfryd yn cyfrifau cenedlaethu. Mae'r ddoddolion gyma'i trechu, a'i cyfrifau a gydig i'r ddoddolion gyfrifau. The conf.sure.something is a common pattern that I try to use. The dot something on the end will deter, it is something that you supply on the command line later to tell it I want this specific build. It means that for example doing DIL for releases, beta releases and whatever becomes quite easy. I'll show you in a moment what's in the config files. Then the top level script ftp.cron is a cron job that runs every five minutes on Casulana. This is the job that starts the daily and weekly builds. Yes, it would be much cleaner if it was triggered via SSH, but at that point we then end up with cross security domain traffic and whatever. It's actually easier. This will check every five minutes and see if the timestamp on the archive on this machine has changed. If it has, it will do a build. Ignore the horrible shell. The actual logic is down here. We work out for a daily what the next build number should be. For a weekly down the bottom we check to see if it's a Monday and if it's the first time the archive has changed. The weekly build day is always Monday. It just is. Of course I can run these manually, but that's how it triggers. Finally at the end, the most recent addition, once this is finished, if we've actually done a build, we will sync across the log files from what we've done over to a static CD builder logs we host elsewhere. Various people in the past have shown interest in the log file output of WNCD. I don't know if they even do any more, but we still sync the logs in case. The two cron jobs, the top level cron job, depending on the day, will call cron job dot daily or cron job dot weekly. Cron job dot daily basically sets up a whole load of variables at the top. This is more config than script. We then work through a list for arch in arches, do stuff, and there's a whole load of conditionals depending on what things we want to build, but fundamentally the core thing that matters is there are multiple standards calling testing CDs with a whole load of configuration. Some of it, the only thing testing CDs cares about as an actual command line. Again this is a horrible style of script, but this is the way it's grown. I've been rubbish about updating this and documenting it and everything. The only thing that testing CDs takes as an argument is the architectures that you want for this build. Not the set of architectures for your entire build, but the individual architectures you're doing in this individual CD image set. So, for example, this particular one, if you notice, does AMD64 and i386 because we're doing arch multi arch. It sets up the rest of the parameters. Conf.sure sets up a base and then we overwide them with a whole load of extra settings here. These are all configurations to the underlying WNCD code. I hope everyone's following. None of this is hard. I'm just more and more embarrassed the further we get through this, but it is messy. I have no qualms about admitting that. So, the specific set of options that we've won here, hopefully are mostly self-explanatory. You know, for example, on the net instance, we don't include release notes. We don't include installation manual because we want a small image. The code inside WNCD that knows if we don't have the installation manual on the image instead modify the weed me in the root of the disk to link to the one online, that kind of thing. It all falls through. No recommends, no suggests. Again, if that isn't obvious, please shout. Complete equals zero means we are not looking at every package in the archive. We are just going to take what is specified in the task package. The task package determines what set of packages go on the media. Debian install at plus kernel is what actually makes a net instance to net instance. It has DI plus the kernel. Who would have thought? If you just want what we used to do a business card image, in fact, we didn't have all of DI on the image either and it would have to go and grab bits. We didn't have the base system. It would have to go and grab bits. Debian install at plus kernel here is a task. I will come to tasks later. Max ISOs, max jigdoos determines what our output format is going to be. We can build ISO images. We can build jigdo images. Do people aware what jigdo is? For net instance, we always do both for certain image formats and I'll come to that in a moment. We don't. We also specify here what else we do in terms of which version DI we're using and all that. This script has support for building both testing and SID versions of things. We do a mix. We don't always do the full set because it just hasn't been useful in a long time. So be aware there's a lot of script. A lot of this is actually not one. It's actually conditioned out. That is the multi-arch ones. We then have a list of the individual architectures and again, it's exactly the same thing. It's just we're passing in an individual architecture. We then have our special Mac only builds which need a slightly different set of config. Again, these are all basically inheriting the same base config, what we're specifying specifically boot method equals BIOS only. There was so much conditional stuff available in WNCD to support lots of different variations on images. Then there's the stuff I've just been working on literally in the last week with Holger to add Debbie and Edu variants of a net inst. Again, you can see it's just a lot of conditionals so you can turn these builds off if you don't want them and then a whole set of here is config to describe how things should be run. Now, the thing I haven't explained has anybody been noticing that each of these scripts is run with an ampersand on the end. That is one of the reasons why this is a nested set of scripts is so that we can make the most of our humongous big 88 core machine with all of the IO in the world and all the memory in the world. We actually don't want to be running one at a time. Did I just wander off, Mike? I saw you had it earlier, didn't you? Sure. As part of each of these builds, there is a macro. There's a shell function called build started and we specify a build name for each of these. This is my own grotty, again, not particularly well documented because it works for me way of starting a whole load of builds in parallel. These days, when we build the daily set for AMD64, i386, we have about eight variants. They all build in parallel. By building in parallel, we actually benefit hugely because of the file caching and directory caching. Most of the input data is identical from one image to the next. We can build eight images in the time that it would take to maybe do four if we were doing them serially. On Cajolana, it's actually much faster than that. It's probably we can do eight in the time of two because, of course, we're not seeking any more. It's all SSD. At the bottom of this of that loop is the magic catch parallel builds. I'm not going to show you the gory details. It will sit there in a loop every few seconds checking. Have all of the builds that were started or any of them still running if they are, wait. Once they're all finished, it then checks for the error that came back, then it deals with actually building for each architecture because we build an architecture at a time in parallel. It then goes and deals with catching the results. Finalize Archer. This is where we actually grab all of the different outputs from all of those parallel calls and we stick them into one directory tree. We make sure all the gdos are together, all the isos are together. We generate the checksums for them, all of that. We parallelize as much as possible during the individual build process and then we stick it all together at the end. You will see there are various extra script helpers. If you want to see each of the script helpers, I'll go through it. Most of them are fairly obvious. There are comments in line. This has grown organically and so things have been refactored as I've found it makes help make sense. So that is cronjob.daily. Once we have done the, we've collected all the bits together, we make sure we remove any of the older ones. We only keep a few dailies around at the time. We then put them into place in our output directory. There's a lot of moving files around and whatever because we then finally sync things over to Peterson, which is where we publish. So this is where all of the data goes from Casulona sat in Iraq in the UK. We sync across the actual output data for publishing goes to Peterson in Sweden. Peterson used to be the build machine as I'm sure most people know. For now until we end up replacing it, it's still the target machine for pushing to and then copying on to the huge big publishing machinery that the University of Remeyer host for us. Well, in fact, not just host, but provide for us. I'm astonished at how good they are to us. So there is an individual bits and pieces that we are sync deliberately. I exclude ISOs when I'm pushing to Peterson because we have jigsaws for all of the builds. Peterson on the other end, I can tell it with the publish on Peterson script. It knows how to then convert back from jigsaws to ISOs using its copy of the mirror. So instead of copying potentially terabytes of ISO images, we can copy, we can actually send over less than 1% of that data and get a huge, huge big win on performance. Although the machines both have really good net connections, still there's no need to copy a terabyte for no reason. So that's the daily bit script. It takes currently about 20 minutes end to end. As I said, it runs each architecture in parallel. I count, I think 11 architectures, including our definition of multiarch, which is the AMD64, i386 together on the same image. That's the daily script, which runs twice a day, of course. It used to be daily, but then we changed the archive sync to be more than once, archive pusher to be more than once per day. We actually then went to four per day. It is pointless for us to build CD images four times daily because not enough is changing. So we deliberately, so in fact at the top of, no, not here in ftp.cron, there is logic to work out which daily bit, on which build we should be doing a daily out of the pushes that we've seen. I hope that makes sense. So cronjob.weekly, guess how often that runs? Once a week on a Monday. I said, on the first sit, the first mirror push that casual analyses on the Monday morning, it will do a weekly build. It will then do the daily build straight after it. But the weekly one is, to be honest, is the more, more investing build. And so I did the daily one first because there's less of it. The weekly one is similar in style, but there's a lot more to follow. So again, there's a lot of variable set up and whatever at the top. So you notice earlier, I mentioned about conf.sure.whatever. If I specify release build equals foo, then various of these things will trigger to say instead of using a generic conf.sure, we will use the specified conf.sure.foo. And also because it's a release build, we will not automatically sign with a testing CD's key. It will instead generate everything and put it into a temporary holding directory on output for me to then go and say, now we grab them, we test them, and then I sign them with the official WCD release key. So there were various conditionals through here where behavior is almost identical. Those are just a couple of changes in the flow. We show exactly related to how we describe the images and what happens to them at the end. So again, lots and lots of variable stuff. Each time we pull this, we run these, it will CD into the WNCD directory, which is working coders, and it will pull it. We run directly from Git because although we package WNCD and we encourage people to use it that way, for our purposes, there is always changes going on, particularly in the testing code, which is what we're looking at today. For the release branches, once we get the, say, for Jesse and for Stretch, you'll see, I don't know if you noticed, we had a build.jesse, a build.stretch. The code in there is also Git pulled, but that is on a specific WNCD branch, which only ever sees obviously very well curated changes. They're just fixes that we find after we've done a release. So there is a whole set of images here. Again, for Arch in Arches, we have, you know, is the top loop. There is a whole load of setup. So, we have a full DVD set. Again, this is all configuration to describe that set. For i386 and AMD64, we make three ISO images. We make Jigdo images for the whole set. Specifically, for DVD1, we say the size of it is four gigabytes, so it'll fit on a USB stick. I think that's a cute thing to do. It means that we don't not fit. For source, we explicitly build a full set of ISOs, because we want it to be easy for people to download our complete source. This was a special config that we had a discussion about with people a few years ago, and you know what? It's sold. That means that we will build at most three ISO images. We will build more images available, but we'll only actually make .isos of the first three. That is part of WNCD. I'll be coming to it in a more generically later. Basically, it sorted the things, the tasks that you specify for a full set go on first, then the rest of it is sorted using popcorn. The stuff that goes in the first image, to be honest, we're struggling to make even what we want to go in fit in that first DVD image. You might need the first two or three, but you will get all of your desktop bits. You'll get all of the core things that frankly make it a Debian system. If you want to get obscure things from Debian science or from JavaScript development or whatever, you're going to be somewhere down on DVD 11. You're not going to be able to download an ISO image from us because we don't care. There are not enough people wanting it to be worth us mirroring the full set of ISOs for that. That's the decision we've taken and most people understand. Specifically for PowerPC and ARM64, again, we can put the same image on a USB key as you would write to a DVD, so we make the first DVD fit in 4GIG. You will notice that, apart from the x86 architectures, we only do maxISOs equals 1. So few people are ever going to be caring about grabbing the later DVD ISO images that, again, we don't see any point in making them. These are all changeable. You can see it's all config. It's quite easy to fix up in here. So, specifically, specifically, we specify the task for the DVD is Debian All. I will go through tasks later. Debian All is all of the desktops followed by the popcorn data. We also have a single DVD with firmware image. At that point, we start another build with ForceFirmware turned on. A different name for the image, but everything else is the same. That's the DVDs. We have a full Blu-ray image set, where we specify, I should explain, maxPackageSize is the maximum size of a binary package that we will attempt to fit in the images. It didn't used to be an issue until people started dumping a gigabyte and a half of game data into a single package. The first time Debian CD saw that, and we were still making CD images, it ran for about a day and a half, and I hadn't spotted because I was doing other things, and I was wondering, why hasn't the weekly build turned up this week? It had gone up to CD image 2345 attempting to fit this into that size image and incrementing hoping it would fit the next one. It wasn't great. We set a maximum package size for the various different images. You didn't see this before on the dailies because we never installed any really large packages in the dailies. For the full sets, we set a maximum package size to make sure that things fit. The fact that you can see the same thing is for the DPD. I just glossed over it earlier. On the Blu-ray, again, we allow a really large package size. If anybody does come up with a 50 gigabyte package, they'll break every tool in Debian, but they'll also cause issues in Debian CD. Explicitly, we only generate Blu-ray images for those architectures. Debian CD treats source mainly as just another architecture because it's easier that way. This is not how it used to work, but ignore history. We don't generate any ISO images because I don't think anyone should be downloading a 25 gigabyte file over HTTP unless it's on their local network. The chances of it getting down to the far end in one piece without retwize is essentially zero. We do create jigdoos of all of it. We do specify that all of the desktops will be on there, so this means, again, this comes down to how tasks work later, and then we generate, we start yet another backgrounded task to build the Blu-ray set. We also have full dual-late Blu-ray set, but we did actually manage to have a single disk which would hold all of our m54. That was about three releases back, and it's gone away now. We also build the individual desktop CD for XFCE only. We used to do a whole set, one for every desktop. Most of the desktops don't fit on a single CD anymore. It's just frustrating. I actually killed this altogether until a lot of people said, oh please, can we have a single CD? Fine, here, I've picked one. You're never going to get no one on a CD anymore. As part of the weekly build, we also build a net-inst image, or, in fact, a full set of the net-inst. We didn't use to, but it actually makes life easier. The weekly build includes the small set, the small images too. Is the microphone just dying too much? Okay. Okay, sorry, just picking the microphone. I have a lot of sympathy with that point of view, but there are a lot of people out there who still need offline installation media. It's reducing, absolutely. But especially considering in the past, before I killed them, a full set of CDs, we were up to 85. No one was ever going to write more than the first two or three. So that's why we killed that. For DVDs, it's similar. But I personally have sold sets of DVDs to people who actually want them, sometimes for their own archiving purposes. It's meh. If you're at the stage where you're trying to feed a dozen or more bits of installation media into your computer, I don't think that's a great experience. So we also build all the net-insts as well as part of the weekly build. This is mainly for my convenience. It used to be at the end of a release build, we would have a set of DVDs in one output directory, a set of CDs in another output directory, and potentially others. And then on release day, I'm then juggling to merge those. That is, was and still is, to a certain extent, a manual step where mistakes happen. So I don't want it anymore. So that's why this is here. Blah, blah, blah. So again, you can see all the same code in cronjob.daily is basically here again. We've got the multi-arch builds. And again, at the end of this, you see, look, catch parallel builds. For the weekly images, even more so than the daily set, now we've got multiple Debian edws, literally just added in the last week. I think we're up to 11, maybe 12 parallel builds going on for the x86 architectures. Casualano is a monster machine and doesn't even notice. It's running at a load of 12 or so. And you know what? Who cares? It's fine. The key thing is, you know, I don't have to live in the machine to hear what used to be pettison. I spoke to Mazawani and he said yes, he could tell when it was a build day, because all the lights on the desk on the front would go crazy and you could hear the heads moving apparently. You know, it really was thrashing the bollocks off it. On Casualano, it's probably, of course, it's silent apart from CPU fan. But still, we are thrashing it. This Debian CD on this type of build is about the most IO and CPU intensive thing I'm aware of, which is why I spent a lot of time actually tuning this and benchmarking what a good setup was back in Heidelberg. So, finally, end of this, we generate a set of checksums files for the outputs. If it's not a release build, we sign using the automatic testing key. Things get, we check for errors in the build, blah, blah, blah. We generate our firmware images. The firmware images here are not the CDs with firmware included. These are the extra tabls and whatever that you can download separately to go and stick on to your separate USB key. I'm very tempted to kill these because nobody can ever make these work. The number of times that we get people complaining, I don't understand how this works and I can't make it happen. It's a nightmare for user experience. So then, at the bottom, we sync to Peterson. We also sync CD sources. I'll come back to that in a moment. We also push to Peterson to have it make snapshots for us. Again, I'll come back to it. Now, I did skip over at the top. There was something at the bottom here for catch live builds. Right up at the top of the script, there is, it also triggers, in parallel, a separate cron job.weekly live. That's where the live images and the open stack images happen. It is not going to be that way for much longer because we desperately, desperately need to disentangle some of these. The reason for merging all of this and having it all controlled by one big script of doom at the top was to make it manageable on release day. I've had enough of trying to manage three different types of image from the same script. It's going to go away. I did mention, at the bottom here, we publish CD sources. One of the things that came out back in the Sarge days was WNCD as part of its build needs to extract various things from some binary packages to go on the CDs. Not the packages, but it needs bits of syslinux to make them bootable, for example, or it needs bits of grub. We didn't actually do this well for a very long time, WNCD. We actually had our own copies of syslinux binaries in the WNCD package, which was frankly broken. We realised that after a while because it caused us problems, it also meant that we'd lost track of where the source was. We were not good people. We fixed it. What this does is part of the WNCD build, it now knows I am extracting a package from this binary and it keeps some metadata to list the binary and source packages where that came from. It copies those files, the binary packages and the source packages into a CD sources directory as part of its build. We at least have an archive of all of the sources and all of the binary packages we have ever borrowed bits from, also are kept for posterity. It's slightly wasteful of space. Of course, from one run to the next, we're copying the same files over every time because of the same package versions. It's not huge. We also generate the snapshots. This is less of an issue now that weasel's awesome snapshot.debin.org thing works, but of course years and years ago before that happened, for our jigdo images to work and for them to continue to work after the archive has moved on, we need a fallback mirror where jigdo can go and find the packages it needs. We have a snapshot on Peterson, which is exposed publicly. We also have a snapshot on the UK Debian Mirror, which is exposed publicly through an alias in the DNS. I'll come again, like in many things, I keep on saying today. I'll come back to that in a bit. Basically, we generate the snapshots. Finally, at the end, we date our base files. This is what triggers various mirroring things to happen, and we're done. Where are we? Cronjob Weekly Live will be, the actual code here won't be familiar, but what it does will be familiar to people in the cloud team. Basically, it sets up various things, and then it calls into one open stack or one live. Each of those runs a VM and causes things to happen inside it, just the same way that we do things for building cloud images, just not in as good or as controlled a manner. I will be moving this VM setup over to this stuff that Lucas come up with, so we have disposable VMs rather than long running stuff. Okay, is this going? It's not complaining about battery low or anything, of course. Okay, does that work? Yep. Okay, right. So, Debian CD. Again, this is in the archive. We don't use the copy in the archive, but it is updated fairly frequently. Top level script is build.script. You see, it wants a confdoxure. If you don't pass one, it will go and find one in its directory. It's just a set of configuration in shell. So, it just sets up a whole load of variables. So, you can specify, do you, you know, the code name of your image, do you want back ports? You can tell it where to go and get DI if you want. You can say whether the images you're building are official or not. Only people running this on Debian machines are allowed to call these things official. We have made a point of saying that. It sets up a whole load of working variables. You tell it where the mirror is, where your temporary directory for building is going to be. This is boring stuff. There's very little external documentation, but every variable that matters is commented in here, including things like we have a whole range of different possible sizes we can support. So, the more useful stuff is in the make file underneath here. We have a whole load of things that it sets up. The key thing is we set up tasks. Debian CD itself comes with a set of core unchanging tasks, which are basically a task towards, it's just a set of, it's a meta package, not necessarily the same as the tasks that we know of in the archive and DI users, but a task for me, and I'll show you one, I mentioned Debian all earlier, it is literally a container. This is all expanded at one time using CPP, so we can have the Debian installer plus kernel task always goes first on every image because an image set that doesn't have the installer on, it's not an installer image, obviously. We have specific things that we recommend should be on the first image called 4CD1. I'll show you that in a minute if you're interested. We have what we consider to be the essential tasks. Then we go to the full tasks, and these are generated. I'll come to it in a moment. We then have a set of other interesting packages that we think might be near the beginning of the set. This is almost an all set these days. Finally, we include the popularity contest. Now that, at one time, we go and talk to popcon.debian.org and download current popcon data, and we then use that to feed in to our to a sorted list. Anything that's not already included in one of the earlier tasks will then be pulled in through popcon. Again, deliberately, if you have something that only two people have ever installed and one of them isn't using it anymore, it will still make the full DVD or blue way image set, but it'll be right down the bottom. At build time, it's passed through CPP, so deliberately, if it looks like it's CPP, it is. We can then have recursive includes. I'm going to show you one of the other tasks. For example, Debian Edge U Full. You can see it doesn't have all of the stuff that I just showed you in the Debian All, and the tasks bust a popularity contest. This is auto-updated, but we include the most recent copy directly in the Debian CD source, so you can do a Debian CD build without being online. You can see, Libby Z2 is apparently quite common. Any guesses for what might be at the bottom of the list? The sorting that we then do with these, we have the input list. We then pass it through our own sort dependencies script. You don't have to list every package. All of the dependencies for your given package will be pulled in if they're needed. That way, we can just list the task packages that matter. We can list, for example, the task in here. We can list, say, for the Xfce image. That just lists task essential Xfce. That will get expanded at one time later to pull in everything that is depended on by task essential Xfce. This is how we do it. In this particular case, you'll see this will pull in everything, because it will pull in a popularity contest. At one time, when we build, we build the Xfce image and say maxisos equals one, maxigdos equals one. The moment we get to the end of building the first image, we stop. No, we only build the first CD. We then stop doing it. We then don't build any more. The maxisos and maxigdos thing interact. We will go when we're actually laying out the CD images. We will go to the highest number that you have mentioned in either, and all is magic. It just means go to the end. Ah, where are we? In here, there's a whole load of data which is organised per release. This is just daft little lists of the other specific U-debs we need to include or exclude for various architectures. I'm not going to go through the data details of those because mostly, in fact, they're obsolete. A lot of these are very similar. They're tiny. Don't worry about details. The other important parts of WNCD are the tools boot directory, and this again varies by release. There is a boot script here per architecture. This is where actually a shitload of magic happens, and I wish it wasn't. All of the knowledge that I have picked up about how to do combined booting of AMD64 and i386 for both BIOS and UEFI on CD and on USB key is encoded in these scripts. So this is the magic place at the beginning of each image in the set. WNCD will call tools boot suite boot dash architecture with a CD number. For most of them, we don't care if your CD number is higher than one we just exit. For a lot of our architectures, we cannot build a bootable CD because they don't boot off CD, for example, or they don't boot off USB. That's a pain in the ass. For x86, for PPC, for ARM64, we do know how to make things bootable so there is real stuff in here. And these know where the things live inside the inside DI to go and grab them, and then have a look at the details later if you want. I'm not going to cut to work all the way through this, but we do have lots of things down the bottom here where we work out, where we add, for example, on the x86 ones, we add the Win32 loader bits. It's all integrated here. We also add all of the syslinux bits, and by God is this a pain in the ass to do, because it changed from one week to the next at some points in the past. And you'll see there is a common thing here, add macizofs. So all the while, while we're going through all of these boot scripts, we are doing work, copying files into place, and also appending things that are going to be run as part of the final macizofs or now Zorizo command line in a specific order. So, for example, you'll see here that we're adding, we're saying where the iso Linux boot block is going to be, where the iso Linux boot catalogue is going to be, and various other bits and pieces. Actually building a CD image that works with all of the clever magic, you end up with a thousand characters in the command line. Generating that, doing it by hand is easy. If you want to do it with lots and lots of options, and it takes some work, and I'm just skimming through. We also have support in here for using a different splash image, so this is where the Debian edu different graphic gets dropped in. We have to tweak the init ramfs for it, blah blah blah. We add grub. We create a fat partition if we're doing it. If I booting, I'm going on and on. There's a lot of stuff here, but you don't need to know it most of the time because it should just be working. We generate a list of all the packages that we're going to do. We then start laying out. We sort those packages. We then start laying out the temporary image trees. On Casulana and on Petterson, the best way to do this is rather than copy all of those gigabytes of files, we create trees of hardlinks into the archive. This is a bit that causes me to be forever chasing DSA every time we reboot the machine and say, please will you let me create hardlinks. It does save a vast amount of IO if we don't have to then make for 12 different types of image, 12 different copies of large chunks of the archive. It saves on disk space. It saves on IO. It is a really big win. We work through. I have the script. Make disk trees. Guess what it does? It makes disk trees. It knows how big the size is that you're meant to be creating. It works through the list of packages. It will guess how big the image tree is that you're currently working on. The point when it goes gets close to the size that you've told it is the maximum for the image size you're working on. It will actually then call Zorizo and say, how big is this really? We guess because it's faster. Once we get close, we actually validate with a tool. We want to get as close to the maximum size as possible, but obviously we can't go over. The closer we get, obviously, we might end up with one fewer DVD image. That is absolutely worth doing. If people are going to download a DVD image, we want to get as much Debian on it as we can. I hope that's fairly obvious. The script goes through and optimises. It does not change the order of the packages because ordering matters. We always make sure that the dependencies of Package A are inserted in a list before Package A. If we then went and changed the ordering here, you'd have things on the CD that you wouldn't be able to install. This is a big pull script that just iterates. It causes, again, huge amounts of IO making trees of hardlinks and then calling MacArthurFS to say, are you at the right size? It keeps going until actually it's gone one package too far and then it's gone over the size that's allowed. At that point, it knows I can't do that one and move back. It's easier to do that than to guess and so on. This stuff also supports the multi-arch images. As I said, we have had in the past, we don't any more, multi-arch images including source and a constraint on that would have been even more so than do your package dependencies fit on the media. The source for any given package would go on the media first too. That way, we could guarantee that if you gave out one of these multi-arch with source images, you were guaranteed to have the source for all the binaries. That took a lot of effort and it's a shame that I have to turn it off. Unfortunately, a DVD just isn't big enough to have two architectures and source on it and actually get anything useful. That is the overview of the make disk trees. At the end of each time make disk trees comes to and it knows it's finished an image, it will go and start the next one. Once we've done that, we simply iterate through all the temporary trees and say, make me an image, make me an image, make me an image, make me an image. Or make me a jigdo and an iso, make me a jigdo and an iso, make me a jigdo, make me a jigdo, don't do all the isos. Then we get to the end of the run and that's where we collect things together as we saw earlier. Following me? Good. This is obviously totally ad hoc. I haven't prepped this in any way so I know I'm not doing a good job of describing ordering and how this plugs together. Once we get to a release day, the first major release is easy. I do cronjob.weekly, it gives me all the images. I go and grab those images from where they're in the temporary output directory on Peterson. We can go and grab them, we can go and test them. The only time I'm ever prepared to put the official release signature on images is when they have had some manual Q&A as well. I've been wanting to go back and plug in some automated Q&A into this process for as long as I've been doing it and I'll admit I've been crap and there hasn't been enough time, there's always been more things needed and the Q&A hasn't happened. I will get to it, I promise. Once we have done the Q&A and I'm happy, then at that point on Peterson, again, not Cajolana because that's where we're publishing, there will then be a set of scripts to basically pull together all of the checks from files, I make a tab all of those, grab them down to my local machine for signing and then push them back again, obvious stuff. The key is not on one of the public machines. On a non-major release or on a point release, there is the added complication of the update images. Are people aware of those? No, I didn't think so. So it's probably not useful anymore because people don't have full CD sets. I'm tempted to actually start dropping these. If you have a full set of DVDs of Debian 9.0, the last thing you want to do is when 9.1 or 9.2 come out, go and download and burn, or even worse, buy a full set of 12 DVDs each time. Apped is awesome, it makes a lot of this easy. We can instead, however, create an extra image which just contains the things that are new since the last release. That's exactly what the update images are. So if you feed your whole set of DVDs into app CD1, of course it will then tell you put in disc 2, put in disc 3. It can also quite happily deal with here are a set of newer packages which are on update DVD1 or update DVD2 and it can happily use those two. So each time we do a point release, we build a set of images that just contain the changes between the .0 and today's point release. So you can then just have, so if you're at 9.5 like we are today, in fact I couldn't even tell you exactly how many. I think we're up to three, but let me check. No, we're probably up to two DVD updates. So you can then just go and grab those afterwards. So there is an extra process that runs alongside the normal CD build which is I first of all have to generate a list of all of the .debs, .DSCs, of all of the contents of the archive that would normally go on to a CD or DVD. I generate a list of all of those and I maintain that where I actually have a separate repo which contains all that metadata. So I can then track what we had in 9.5.0, 9.5.1, 9.5.2 and then I can generate a diff and using that diff, I've got an extra script in WNCD called update CD which basically you feed that diff and it will give you a set of parallel images to go alongside. On a point release day, sorry, oh god yes, on a point release day I do then need to merge those images and their checksums with the DVD build, the rest of the CD DVD builds. It's again something that I should merge better because it can be error prone and the Simkins normally is round at my place on release day and was horrified to see how much typing I was doing just doing that bit at the end of the process once we've actually tested everything. I've been rubbish because I can be because nobody else is doing this. I really desperately would like other people to dive in. A, to tell me how crap some of the code is, I know how it is, but also to really identify the points where actually somebody else would struggle to follow what's going on. So if we have currently passed factor one, so if we lose you, we have big problems really generating images quickly. Generating a complete set at the moment is hard for anybody but me. I know that and that's one of the reasons why this session is happening. Going back to Casulana, as I said, readme.release and even those people not in the WNCD group, if you're on Casulana you can read this, it's publicly readable, it lists what happens. It says it has a script to follow which is basically disable the automatic daily and weekly builds so you don't end up with Casulana trying to trigger a normal weekly build just at the point when you're trying to do something else on the machine, obvious stuff. It's like we disable Chrome on the Debian release when we're doing it, that kind of thing. There's details here about snapshots. The primary snapshot that we point to is now snapshot.debian.org. As I said it's awesome. I don't have to worry about it. Weasel does and he's better than I am at managing some of this. That's why he's in DSA because he's awesome. I know this isn't being video of his benefit but genuinely I am in awe of how good Weasel is for some of his stuff. The thing that nobody else could do apart from me and Phil Hans at the moment is do the secondary snapshot that is on the UK Debian Mirror. It's not critical anymore but because the jigdoos need that snapshot I've been in long, for a very long time I've been in the habit of maintaining three separate snapshots for redundancy. There is one on the Debian Mirror in the UK, there is one on Peterson as I mentioned earlier which is not actually referred to by any of the jigdo images but it's there as a backup. I also have a third copy on my own home mirror because that way then I can know if a disk is dying and I need to put one in. Between the three of those machines I have periodically pulled and pushed things. We do occasionally lose things due to software bugs or whatever or for example we did have a disk failure in the UK Mirror not that long ago and it did cause a few things to have turned up and lost and found but this is not great so that's why I keep redundancy. What else? Here is data about making the update images and how to release things. The other thing that nobody else could currently do right now is actually sign release images. I have the CD release key under my control. Adam Barrett has a copy of it but at the moment he wouldn't know what to do with it. We do have another copy. The flip side of that is the actual CD release key is not as highly trusted as the main archive key. Things do not check this automatically at boot time or anything. We haven't gone that far so if we do need to bootstrap in the UK and I've done it a couple of times it's literally just a case of get a few local DDs who trust me to sign it and of course being one of the many cabals in the UK is really really good for getting DPLs and release team and whatever who are around to sign a key to give it some trust. But still I would like it to be easier for somebody else to do this. So Jonathan and I were talking about this already last week. I would be over the moon if the next time we do a point release somebody else or somebody else's was also on Casulona and Pettisyn with me. We can share a screen session so I can show people exactly what's going on which basically corresponds to the instructions you're looking at on screen but of course this was written by me by hand the chances are I've missed a step or something's changed it you know how that works. What I'd like to go through it at least once or twice with somebody false shadowing me so somebody else gets an idea of what's going on. Most of this on release day it's a I edit a couple of files to change config to say here's a version number go and then the hardest part is actually testing the images coming out. Most of this is really really trivial the problems come out of course if you actually need to start going and fixing bugs in Debian DD because sometimes happens on release day we find a certain scenario that we've never come across before due to some weird combination of packages or for example when we did a dedication and we've done dedications before but new dedication file names were not exactly the ones we expected and so on release day it's a oh crap start again tweak the code to actually cope with you know to cope with a new reality. It's how it happens it's not great but unfortunately the thing I have found over a number of years is as Debian CD in particular is one of the is another completely independent implementation of tooling to handle the mirror to handle the Debian archive I find things that other people haven't because of course they've changed implementation and changed their tools it's it's happened a few times that I find bugs in behavior they're often not a problem you know I can tweak around it on the day it's only ever happened a couple of times were sorry this breaks the build we can't do it and obviously at the point when we've just hit go when pushed out to our mirrors to say Debian 9.6 is out that's not a good time to say guys you did well can we go again yeah so questions. Thank you. If we're going to begin using Casalana to build things for cloud and they're going to be triggered off of git commits and things like that we're going to need to refactor your monster script that does a lot of parallelism yes absolutely totally into jobs that can be run by a scheduler yes okay absolutely I'm well aware of that this is state of the art at the moment second item on my to do list for Debian stuff is we factor all of this into something that is more easily callable externally okay okay how much dependency is there on the big collector at the end like do you want all of the various arches built before you begin the copy over to person all so so I didn't say was the way it works at the moment is each of the architectures is built as a lump we then sync architecture A across to peterson while architecture B is building on Casalana you know to parallelize deliberately right but you but it's important to you that the entire lump of the architecture goes as one block yes I prefer it that way so of course for most of the images we do a daily where of course you only have one or two small images per architecture it's much easier that way for the weeklies for me it's a feature that the set goes together because of course the checksums go together that kind of thing it's easier to build it in one lump I'm not saying it can't be changed at all but I think it's a feature okay so when we when we carve that monster script up into pieces we'll need to think it's going to be a per architecture thing okay okay yeah and definitely oh I record the type script and whatever typically on a release date so I can share those afterwards as well easily yeah and apparently we are out of time