 So welcome to our afternoon talk building in Debian images So, please leave give a warm applause to Riku Voipio Thanks. So my name is Riku Voipio and I work at Linaro in a team called Builds and Baselines Which in this case means that a lot of stuff I do is I make distribution images If you are not familiar what the disk image is it's a It looks like an hard drive. It has everything that the hard drive would have inside its contents There would be the boot sector the partitions the file systems and their contents Bootable disk images can boot them after you put them back on a hard drive so there's some Overlap because sometimes people when they hear a disk image they think about the picture of a hard drive and It might confuse people but if you have been using disk images already, this is nothing new for you and At Linaro we have created our own tool like it seems almost everyone has It's called Linaro media create or Linaro image tools is the package which it comes from in Debian The idea is that we have a generic distribution root file system such like Debian Ubuntu Fedora and then we have a hardware specific hardware pack for We support many different kinds of arm developer boards and each of them would have their own hardware pack Which includes the kernel bootloader and possibly some user space components specific for this very board and Using this tool we merge these things together The idea was not to distribute the images because that would be a Matrix of downloads if we have three boards and three distributions It would be nine if we add more boards. It only gets bigger and bigger But alas it turns out Most of our users were just asking I don't want to run this command. I would like to have a pre-made image Can you provide them? I? Would just have like to have something I can use the Unix DD command to put on the SD card and boot on it So it turned out that We now have a lot of images on our Linaro servers available for downloads fast forward a few years later and Linaro media create is not as good for us as it used to be So people who designed it are no longer working for us and the use case is a bit outdated At Linaro, we would really like to have one kernel that works on all boards not some specific hardware pack per each device and On the other hand, we haven't really reached this goal because we get new hardware Which is always needing a specific kernel before the main lining progress has happened So there's new platforms like the 96 boards or platforms Which are not supported by Linaro media create We would need to do some changes to the source code to make it happen but Nobody has really had the big motivation to do that and instead we have Kind of hacked our way around it So we started looking into alternatives Because it surely must be a problem that a lot of people have bumped into So some of the findings There are actually really a lot of people who do install Debian from images instead of using the Debian installer For example cloud servers You don't run Debian installer on Amazon web services. You have a pre-built image Same for other places like Microsoft Azure or the scaleway system, which is an arm-based hosting provider Embedded boards. This is Close to what we do Pretty much every single embedded board you find that says they have a Debian image for you Raspberry Pi has their Raspbian based fork. It has a disk image BeagleBone provides Debian disk images, Q-box, everyone has their own and Finally, there's the class of OEM pre-installed images and workplace Where there's some centralized admin provisions disk images using clonezilla or something similar They are not necessarily built using a tool, but Manually install one machine and then save the disk contents and continue And if there's still one case there live CD users They actually never install anything, but they always use the pre-built image So how many of you here have used one of these cases to? Everyone Few few people I didn't but almost every hand went up. So why do people do this? One is speed if you're having a factory that is producing hundreds of boards every single minute you do Setting on the factory floor with the board is expensive So you really want to just put something on the device as soon as possible Other one is convenience. I Mean you just Get a file put it on a SD card boot it. That's it instead of Please configure this USB stick and put it on and wait for 15 minutes And then you will have an Debian system after you've answered to quite a few questions and Then for all the platforms that are not supported in Debian Adding the support for Debian installer for these platforms is often a big work and It might seem like a nice shortcut to just make an image and provide it to users So what do tools that make these images do? First there's a loop back file created a file that is big enough to fit the real hard drive contents partitions are put on it The partitions are formatted with some file system like X4 or that file system in case the bootloader needs it a Debootstrap is run to fill a basic Debian system there At least everyone can of the tools I've built uses Debootstrap and no one has reinvented that one I Usually There are some extra packages that everyone wants to install when they're making an image So there's at least a parameter or option provided for this A default user is added or a password is added SSH keys things that let the user to use it for the first time Then we have some customization scripts Some are hard-coded into the tool there somewhere hidden there that they do some little bit mingling of that or they are just defaults that you can override and Then there's some Customizations user can do themselves usually in form of some kind of shell script or shell commands There's a kernel installed so that it can actually boot and The bootloader if it's x86, it's usually grub on our platforms. There's a big variety of these and One big motivation for image tools is that in many cases the bootloader is proprietary and Shipping it together with the BN is a challenge Event might not be even legal if the bootloader is not actually redistributable now There's some if you have Wondered what these tools use under there's usually the same set of tools for everyone There's the loop setup tool you make a pile look like a hard drive using a low setup There's parted it's an partition editing tool It's also has an alternative called f-disk which was previously used a lot, but these days almost everyone uses GNU parted Debootstrap as mentioned earlier Creates the basic Debian install with the minimum of packages you would expect to have like DPKG and APT K-part X is probably a tool that not many people have heard about It makes it possible to easily mount all partitions of a loopback loopback partition loopback image Else you would have to set the offsets yourself and calculate how to mount them QMU is used for the tools that are from Creating cross-platform images if you are running on x86 and you want an arm image You would use the QMU's cross arc emulation and then there's the convenient QMU-IMG command Which is used to convert Loopback files to different formats expected by virtualization tools Then there's cloud in it, which is could be described as a pre-seed file for first boot You're booting a system and you want to put your own SSH key there or set up passwords from a generic image cloud in it is Seems to have taken a de facto standard on it You can for example Set DHCP server to provide a file that has all the possible Configurations you want to do on the first boot or you can do stuff on the kernel command line Because there's no official image building tool in Debian there's essentially Dozen tools in Debian for to make these I make a brief tour of what I found The usually people say that everyone should be using Debian installer. Why are you putting this image? and Well, you can't create images with the Debian installer. It is possible The Debian installer has a cool feature called preceding You create a configuration file that has all the possible answers that you would ever want for setting the Partitions what packages you want their user name password. It's all there done Debian installer But There are problems It needs to boot the Debian installer to run it. You can't run it on a CH route and if you Have a desktop machine. You probably don't want to run Debian installer on it to overwrite the stuff you have on it So you have to use a virtualization tool or like KVM Xen something else QM virtual box So the pre-seed file allows you to customize it and as Long as the options you want to configure are in the pre-seed file or some debconf option it's pretty handy and The vert manager project provides a package called vert-inst Which will do these automated installations really simple. You can do things like Inject the pre-seed file in the init.rd file automatically, which is not By command hand on command line, it's not very easy. There's quite a bit of steps you would need to make so Is this dark background visible? This would be an example of how you would use Debian installer to make an image with the vert-inst tool Do you install the packages Vert-inst and the recommendation lib-vert bin in case you have no install recommends this just there You start the virtualization networking that the tool needs It's based on a lib-vert. So if you haven't been using lib-vert, it might seem a slightly alien There's that system The options We are calling it Debian 8, but using variant of Weezy Debian Weezy it works just find the tool has just outdated names File name is set we have a pre-seed file we put on that configuration and Finally we give it the Debian installer init.rd and kernel Locations where to download them form some Run this command and some 14 minutes later. You have a nice x86 image for virtualization tools so Apart from people not knowing about this option. There are actually some real problems. Why Debian installer is not very useful for making pre-built images As if you have a really weird platform You would need to add support to Debian installer which again is a bit down thing task for some people and If you need to change something that the pre-seeding doesn't allow there's no option for that You need to build a custom DI image Which again is a big job to learn about Probably a bigger job than to write your own image writing tool It doesn't anonymize images because it expects the image to be the one that it will be actually used So things like the SH host key would be shared by every single Board that uses the same image There's others like machine ID and things like this which could make things look very confusing one you look at the network and As mentioned it can be a bit slow So there exists another tool that is officially used in Debian. It's called live build It's builds the Debian official live CDs it's skips the Debian installer booting part by using a CH route where everything happens it can do cross architecture installations with QMA and It can be customized really easily with hook scripts put in different locations The more hook scripts you put the less like Debian. It looks like but it's and It's not necessarily clear which hook you need to run at what point so it can easily create a rather confusing setup It really targets read-only images. So it builds a squash FS setup and so on So if you are using live build to make a read-write image, you might get some surprises And since it's actually designed to be a live CD to it takes care about anonymizing the image by the Hooks that come with it by default It's a quick live build example We will be giving it the Parameter of boot device hard drive instead of CD ROM so it doesn't create the ISO image but something that could be booted They're giving it a custom kernel package For this to actually work you would have to also add a repository which has the custom kernel and Finally, there's the sources list where to pull everything from Okay, after we run the LB config command There's now a tree of files under a config directory where you can happen things like packages you want to install and All that configuration of hooks and so on and Finally, you would use the super user rights to actually build the image Here's a slightly alien called tool bootstrap dash dash beads it is clearly not Something that you would find with when you're looking for a disk building image It is a as a package in Debian But the one that people actually use is ahead of that and it's available on github There I checked just there's a bug report where people are saying that they are actually working on updating the Debian one as well This tool Seems to be the one that is used to build the official images for Amazon and Google Cloud and some others so One thing that sets it apart from the previous tools that it's written in Python it has some plugins and It even has tests so it's been put a little bit more thought than many of the other tools I found in This example command line I I've added here I'm just installing bootstrap beads it from Debian to pull all the Dependencies that this tool has and then we take the git clone of the actual version that is being used and Then you would simply run the command with the definition of the Virtual machine image you want to create Here's one example definition It's for the Google Cloud engine GCE means that in this case You have Basic simple parameters as in YAML format what is the release you want to use architecture disk image type partition sizes and Finally, there's the list of the plugins There's the Google plug-in that does the Google customization and our NTP plug-in to set up the NTP stuff So it's a bit more clean than many of the other Up tools where everything is set up on command line Then we have VM debut strap, which seems to be now under quite active development it's called itself a debut strap wrapper for disk image building and It has explicit cross architecture support. It's been one of the use cases to build arm images with Using QMO as well here there's no configuration file just a command line you worry everything is done and When you run out of the options that the VM debut strap supports You can run a customization script to do the rest or make a wrapper script around it Here's the VM debut strap example provided in the examples Where VM debut strap is around to create an image for Beagle blown back We set up for an architecture arm HF and we tell that we need this QMO binary to run on it Network configuration options It's the option of setting the root password extra packages listed and some customization commands with the final Beagle blown back customized script including the actual setup of the bootloader and Kernel which are skipped for earlier using the no kernel and no X Linux options more tools Open stack tool team has two tools in Debian Open stack dash Debian dash images which builds pure Debian images using tools in Debian and then from upstream There's a disk image builder Which can build images not only for Debian, but for Ubuntu red hat and so on and Quite a bit bigger and Give somewhat alien tool for Debian more Xen tools has a Image creation tool which is not necessarily designed to distribute images, but to build them on demand it's quite Still possible to use those images outside Xen although they have been quite tailored Root strap is a tool that is still in Debian. It was designed to use for UML and I believe used by Ubuntu earlier more extensively These days, it's probably a bit forgotten and out of date Linaro media create is the tool I mentioned earlier. We have uploaded it into Debian takes the Kernel and bootloader included in a hardware pack and a root file system created earlier and creates a booted bootable image out of them GRML debut strap GRML has a tool for this one as well. It's It's basically similar to live CD or live build, but tailored for GRML So so far I've counted 10 tools. Did I miss any? Okay. Hi. Ah now it works. Okay. Just two slight Corrections Grimel debut strap is actually very nice to just the bootstrap Debian But from within Grimel or Debian so it's not only tailored for Grimel live CD and Xen tools actually Doesn't do disk images, but partition images So slightly different, but yeah But you can still make a disk image out of that partition At least I haven't tested that for ages. Okay Any question? Well as for tools, I've also noticed multi strap. I haven't used it yet. I have because I'm currently in the process of Developing how to best do it for my use cases But I've noticed that also Okay, so I think that multi strap is actually like debut strap But designed to be used with several different sources lists and not really a disk image building tool It's a little unconventional, but LTSP builds an image that typically ships over NBD But we've experimented with using it on disk. Okay So 11 I See someone from fire rising their hand Currently we do not have an disk image build tool, but there are plans to do such a thing So I think most of the things we have already only the car politics and Yeah, this this thing is missing but all other things are creating a DD image. We can do this. So Maybe we call it cloud image create cloud image. That's better than this image so another option that I've actually used is to for for architectures that you can build a VM for is Build the VM and install the things that you want in it and then convert the VM image into it into a disk image right, so you can basically use anything like QM or VM where or and It's of course then a manual work to run through the installer and This was only inside the VN If by numbers you count the best thing it's of course spindle that is used to build the Raspberry Pi images Supposedly I haven't really checked the chain because following down the track from the image that goes into the Raspberry Pi to the actual Scripts that are used this trivial. It's hidden in repository after repository Omap image builder and one that I've used previously. It's builds images for beagle blown black and other omap based systems It's quite nice actually Yeah, I'm being builds images for cubics and human board some others as well Camelion isn't it calls itself an appliance creator. That's another Google word to look for image-building tools and They create 5,000 people seem to be using this one VM builder is what was previously used to build Ubuntu VM images for cloud services It seems that it has been depreciated With their own hand-built scripts now, of course in these slides the links are not clickable because You're watching the slides. I'll give the URL to the slides as soon as the talk is over and More more more, but I can't just list them all here. I Think I just remembered one more in Debbie and there's this Superman appliance builder thingy previously called fee boots web Superman. Yeah, I think that also builds well kind of Images but stripped on to Whatever could be possible So a quick look under who what they are Programmed with the BN installer is written in CN shell. So and Live build is shell as well Bootstrap V8 set and VM do bootstrap use mostly Python Send tools is written in Python apparel yes, sorry for that and Rootstrap was written in shell as well as Many of the other ones. There's really seems to be two favorites shell and Python and Many cases the Python code is just calling a lot of shell commands So the subjective conclusion Yes, exactly. There's the saying that in IT industry the biggest things that people argue about are the smallest details like an color of the bike shed when you're designing a nuclear power plant So in many cases, it seems that the person who is has done the tool is also the main user of the tool They have written it for their own taste and they like it and at least they know it completely how it works And they are happy with it and they've uploaded it to Debian in the hopes that other people will find it useful as well and Since people have written these tools making them to switch to another is quite challenge Any tool that wants to replace these needs to be really nice to use and really able to Convince them that they're going to be saving time by using someone else's code How many people do we actually have here who have written one of these tools mentioned here Yep so We've looked into what the tools have been doing well and there's quite a bit of problems when people only think about their own use case one of the really common things is that they allow you to build an image with a default password and It's very convenient when you're doing it for yourself But easily it will become an issue when it's been Distributed for a wide audience and suddenly there's millions of hackable devices which have Listening to the world with the SSH and have the same password There's usually a problem for the tools that are designing embedded Targets the ones that are doing cloud images get this right Probably after a few mistakes Another thing that makes these tools challenge to use is that they have hard-coded customizations that are designed for their specific use case So in case you want to use it for something else it might have removed some important file that you would like to use They might expect that ETH 0 is the network interface being configured and if you have more network cards It's not easy to customize that quite a many tools have Rather simplistic view of how to partition things like just have one partition for bootloader and rest for the root file system Or just one for root file system Make a squash FS and have X Linux load that directly and if your use case doesn't fit this then you're in trouble Missing ability to change things on first boot like the password or host names and so on Unupgradable kernel bootloader so the image sets the bootloader kernel in a very specific way and You can't know once the system is running You can't do an APT get upgrade to get a new kernel on it The hooks that set the kernel for booting were in the scripts that were used to build the image And the only way to update it would be by manual commands or by Using the image building tool to make a new image Another thing that happens often is that the CH route used to build the image doesn't actually take Everything from the repository it's downloading but copy stuff over from the root file system that at the host file system So you have to have some specific versions of packages installed on your host system And if they are not available or are different versions the images produced are different or don't work And of course Think that make it hard to jump to another tool is that there might be something really nice in your tool And you haven't told anything anyone else about it One thing that would be usually nice to have because these disk images are a specific size Is would be to resize the image to fit the actual media on it on the first boot so if you Make a distributable image you want to have it rather small so the downloads are small you make a four gigabyte image for It's would be quite the download even compressed So people make it two gigabyte image and then you put it on a 60 gigabyte SD card You put it and you still have only two gigabytes available And so some tools support features like these and others don't and They use your case nobody potters documenting because it's their own tool and they know how it works is There a way to fix this? Here's one option Just joking So now I've run through my slides and I have a little bit time for discussion. I believe so Here's a couple of suggestions. I've done there was I think someone wanted to talk Yes Hi, just a comment about the live bit live build tool Before the jesse phrase it was in the process of being Rewritten in Python. I don't know what the status is but Yeah, so maybe it well it was quite hard to to see what was happening during the the process and no Hopefully it will become easier to to hike okay So having listed what 10 11 tools I thought you were about to introduce your 12th one which we Didn't have time to write it yet What I'm missing in most of the tools that they only have One post in script So the the main part is done by the tool and then if you want to customize such an image You can write your own shell script and I think that's not a lot of help if you have different Different types of images and I think that would be helpful to have More general structure to like plugins. I think grimoire Grimoire for example is using Phi and they could create automatically every night like eight or twelve different Life images because they use the Phi classes. So there's a structure and not only this You can use one post in script for customizing. I Think that would be very helpful if the tool could provide more than Yeah, please write your own post in script indeed so what has Okay, let's have it please first. Well, I just like to respond to that idea because I'm the current maintainer of Xen tools I took it over like three or four years ago from Steve Kemp. So I'm not the original author I feel a little bit singled out being the only tool written in pearl So I thought about talking to needle with because of VM bootstrap But when I noticed it's written in a different language, there's not much to share But one thing we could share would be all those hook scripts which modify the image So because that's the part which is written in shell script for Xen tools to and we also have some Kind of roles we can Deploy to the generated virtual machines, right? So this is one of the suggestions I had here that these kind of customization hooks could often be packages and if they're packages you can install them with all of the tools and You can do that regular updates to those if you find out that one of the customization was actually bad You can provide an updated version and distribute it to all the people who have installed the image earlier Instead of the current way where if hook needs to be updated then people need a new image Yeah, and one one nice thing Talking as the person who looked for all these things it would be to have backlinks from wherever the image is downloaded That what was the actual tool that made this image? That would be critical for many people who want to change some small bit from the image and it there's not easy easy way for that currently and I think these tools are all nice and probably Fit some use case, but what I think if we want something for officially for images labeled as official Debian and There should be quite a focus on having these images Be as much as it makes sense the same as was Debian installer would produce at the same time and so I think We should really first explore if there's not a possibility to Have a Debian installer component that does all these de-analyzation that is needed to have a generic image Maybe I think it's even possible to Have Debian installer variants that could be run in a change route, but that's probably a bit challenging I'm not Someone would have to try it, but do you have any ideas about how to do that? That's a good question that more you customize the less like less like Debian it is So when you're making official images you want to do as few changes as possible But on the other hand that people make images for many custom purposes and in that cases we shouldn't restrict what is possible So they should certainly certainly not restrict But what I see is that many of these images have kind of things in them that are Very opinionated by their maintainer like the official open stack image uses not the default boot loader It adds few but some additional packages which to maintain things make sense Which are not in a default Debian install and some things like that So one of the things I picked up from this is that that second point about providing a link to the tool and the command line and config VM to bootstrap does actually put its own config as a config file I'll need a document that you can have a config file with with this because obviously it is now It is something I've missed in the documentation But it's trivial with a lot of these tools are very small So you can easily add an option and I'm going to put this into a new version of VM to bootstrap That it not only provides the config used to build the image But it might as well provide VM to bootstrap itself and just stick it in somewhere Maybe install it on the image as an option Maybe just put the package somewhere with these dependencies make it an option and actually then give you Basically the script that ran the build is put inside the build. It's easy enough to do right Okay, so I just like to reply to one thing because I'm a maintainer of the bootstrap vz Which one bootstrap vz. It's slightly outdated. It's my fault. Sorry but Amazon Amis and Google Cloud images are built with this tool and if you are checking Amazon marketplace because there's basically no other place to check from where the army is coming from there is a reference link to our wiki which is explicitly saying how how the image have been built and with what So if you didn't see it, I suppose it's probably because you were building images Well, you were checking Amazon images with which are From marketplace and not checking the actually dead young account on marketplace and Unfortunately, there is not much we can do with it. We can just provide all the information. We've got in this one specific place So okay, so I'm at some might be the one of those places where I actually saw a backlink. So I Didn't it's just I didn't mean that all tools miss them I just meant that usually when you go see an image somewhere. It doesn't explain how it was created So one thing we've talked about is before and one of the things that's come up is it will be really useful if install scripts properly separated install time things from image creation time things all the first boot sort of Aspects and we need we should put that in policy and then a lot of stuff would just work I think we should probably you know, this comes up every few years and we go we should do this and nobody's done this I think that's probably a good idea. I assume you agree. Yes So one thing that might exist somewhere in Debian, but I didn't find that it was some standard way of making a first boot prompt Where first time you boot the image. It says, hello, welcome to Debian. What username would you like to create? Please give it the password And I say there's one other thing while I got Mike. You talked about Kernel images not being upgradable because they're just rammed into a partition And of course the reason for that is because there isn't support for this particular device in Kernel yes, and you know, that's just people do that because there isn't support in flash kernels So we can stop people doing that, but we'd have to have support for everything in flash kernel Which is generally a good thing, but I'm not sure we'll ever be ahead of that curve. That's always difficult Okay, last tool, please Um As someone who's Done a lot of work around producing the if the official live images and the official open stack images And I've got a queue of other people lining up asking me for other official images to be distributed One of the things that is a real pain is the fact that a lot of these tools of course need route to run Typically just so they can run mcnod to make device nodes that go into slash dev There are some tools around that we move that That need If you're prepared to use them a lot of people don't think about it until they've already written these things a little later Yeah, polystrap according to wookie I've heard of a few others I can't remember any of the names off the top of my head and Moving on the second thing it's We have plenty of scope if people want to provide Officially sanctioned devian images from CD image, which is a really really bad name for the images Please talk to me. Please talk to the devian CD team We have plenty of scope to be able to build all kinds of things if it's useful, right? Thanks. It seems that Mike people here are rather nervous. So I think we run out of time Yeah, thank you very much for the very interesting talk