 Hello everyone. Great to see a bunch of people here interested in five fully automatic installation. And we'll have now a talk of, yeah, a better look like Rico Voipio from Finland who will talk about live images for ARM on SBC. SBC stands for Single Board Computers. And, yeah, a round of applause and welcome here for Rico. Thank you everyone for coming this early in the morning. Anyways, most of you were here for the talk for Thomas Lang, which was here before about fully automated installer. If you weren't, there are papers around here in the ends of the tables that give you a basic idea what that fully automated installer is. So the first question usually we get from people who are getting some kind of single board computer like Raspberry Pi or a 96 ports high keys. If they are interested in Debian, they want to have a Debian image for it. They've been experienced that when they install Android or some embedded distribution that they have a live image available for it. Now in Debian, we don't usually do that. We provide an installer, but people don't expect to have an installer for these boards if they come from an embedded background. Usually we only get requests for installers from people who are already Debian developers or deeply used to using Debian. Eventually, a user will find a Debian image from somewhere that someone has made and because users have requested it and someone has done it. Usually someone who doesn't work for Debian for their daily job and does the way of making the image is somewhere hidden. Nobody knows how it was done. Nobody knows where the kernel came from or how to rebuild the kernel or upgrade it. There are some vendor kernel because usually the mainline kernel is not good enough for the board and there are some random changes to Debian behavior which make it feel less like Debian. If the user uses this image for a very long time, they start thinking it's Debian. It's not real Debian anymore. It's something that can install Debian packages, but it doesn't have the feel of using Debian. Now, I work for Linaro and we provide images for some 96 boards platforms and for our engineers who work on kernel and other stuff and are not really interested in user space-side except for something to boot into and to have their tools available on it. This has been an experience for us as well, that the images we make are not exactly very great. We have used a tool called LiveBuild and a custom tool called LinaroMediaCreate which originated originally from Ubuntu. Neither of these alone is usually enough to make an image, but we had a CI script that adds a lot of magic into these images. We had one custom image for every board and the kernel was somehow shoehorned in and you couldn't upgrade it easily except by rebuilding the entire image. We started thinking that there must be a better way of doing things. Surely this isn't the only way we can support the single board computer users around. At about this time, people started talking about fully automated installer and fully automated installer's disk image tool for building cloud images and in general, even though fully automated installer has been available for a very long time, using it more in Debian has been a relatively new thing. So we were investigating into all these various tools that are available for building disk images in Debian and I think we found like 20 different tools and I was not able to decide what is technically the best one, so we decided to go with the one that has already gained traction in Debian which was fully automated installer at this point and to find out how feasible it is to make images for our developer boards with it. So a few months later and indeed it turned out to be a major improvement for us. Our CI scripts got a lot shorter, the images are a lot more cleaner and they are all contained in the file configuration we have rather than scattered around in different places and eventually we were able to make images that work on multiple platforms. This is with some catches, you can't do that on every platform but when we can do that, we have a kernel that has been installed the regular Debian way with just using the fully automated installer's APT commands so it's a package there. We have a repository where you can update the kernel and APT get upgraded and the configuration gets updated and with one reboot you already have a new kernel there. This is something that people who have x86 are used to it and it's like normal life but for some ARM people it's like, whoa, amazing. So let's have a tour of why this kind of image building is different and say more complicated than making cloud images or disk images for x86 PCs. The usual cases that makes things hard is that the firmware needs to be on the same image as the root file system and all that configuration and because the ARM SOCs are built so that they load the bootloader from some say flash which is either an EMMC or an SD card or sometimes an SPF flash it has very little space for the ROM in the SOC to load so it has a very tight hard-coded way of where it needs to find the initial bootloader which then loads the firmware which is either Uboot or FE or some Android L key so that in order to support multi-platform images in ARM we have to have some kind of hardware description included in the firmware it's either a device tree or some server systems we have ACPI and this also needs to be included in the same image as we have all the other stuff and if this is true then the image has to be hard-coded for this specific platform sometimes you can have one image that supports several boards of the same SOC SOC means System on Chip so it's the core part of an ARM single board computer but in the more idealistic case we would have the firmware not included in the image that we're about to build with the fully automated installer if life would be great each board would have a separate SPI flash a bit like how on x86 we have a BIOS on a separate chip but the SBCs compete very heavily on the price point and adding a single chip adds to the cost and when it's more expensive than the competitors people will just skip buying it as a workaround I found out that sometimes you could just use the EMMC as a firmware partition and install Debian on the SD card or USB or SATA if the EMMC is not particularly big like 4GB it's not a big loss if you just install a few megabytes of the bootloader and stuff there and leave the rest unused it's not like a small EMMC is going to be very useful for Debian purposes if you have an embedded distribution like open embedded or yokto or build root based then it's just fine but in those cases you're not really interested in it and then we have the issue especially if you want to use this fully automated installer with Debian kernel is that we use the mainline kernel quite religiously we don't have the resources to support every single vendor kernel that is available but the end users they already have their boards and there is no mainline code yet available usually you have to wait for months or even years before the SOC and the board features are all mainline if ever so we need to find a way to provide end users possibility to install their own kernel so I have created an example image that will be touring a little bit it should be able to boot any ARM64 board which implements the FE API in the firmware usually it's done by Tiana Core but these days Uboot also supports FE and thus we can boot from it let's have a look a little bit under on the last talk we already talked about a little bit how the FE is configured and what kind of commands you use with it we have some kind of configuration space which you can also pass directly on the command line you don't have to necessarily clone it but I find it usually more convenient to clone the repository and then you can look at the files and explore it before running the actual commands you don't usually need to add a lot of packages to the system to start running the FI build in case you're on a AMD64 machine you need to install the QMU user static package to support the cross-architecture building of the images if you're on ARM64 already you don't need it there's one catch here the cross-architecture building does not work if you have already configured your X86 PC to support cross-compiling because then you have the ARM64 architecture in the foreign architectures then we have the usual list of classes that describe the image in case you are not on the previous talk these are the kind of Lego bricks that describe what is going to be included in this image by adding or removing these variables in the command line you can have one configuration space that supports several different kind of images so in this simplified configuration space we have a handful of directories and then the list of various files that these variables will refer to we see in the class directory we have files that describe variables and this class usually every single time there's this caps lock file name it refers to the class variable we gave in the command line in the previous slide I will go through these quite quickly since it was already passed through in the previous talk we set variables in classes the DevConf directory describes things that packages that use DevConf will use in this case we pretty much only set the email the disk config will describe how the system is partitioned the setup storage man page has excellent examples and descriptions how to do this and in the package config directories we describe what packages each of these classes will install on the targeted image finally in the scripts we do things that need to be hacked into that place that can't be it's easily a place where you get tempted to do a lot of things that you wouldn't need to do in packaging but usually if you just can do things you should put them in packages and install them packages to the system rather than try to edit the root file system from the scripts so a little bit about the future one really annoying thing about using any kind of images whether they are for ARM or x86 or even in this case in Windows is that usually every image has a hard coded password and this is kind of terrible for the entire security of internet even if we think that the users of our images would be smart and not put them into the public internet they might do it anyways how do we get rid of the passwords does anyone have good ideas put SSH keys was suggested well that works if people are only going to SSH but sometimes people will want to have an image where they log in from the serial console or even from having a graphical user interface then we still have the same password distributed all around so something we've been talking about for quite a while but no one has yet done to my knowledge is actually sorting out a first boot set of things so you can generate yourself an image that doesn't have a SSH key doesn't have a password and whatever and on first boot we'll go through and ask these kinds of questions so you can make what I've heard called an OEM image or a pre-installed image that can do the right thing it shouldn't need a lot of work but I'm not aware of anybody doing it yet indeed but this is something that we need to definitely do too not only make Debian better but internet better by having less bots available and then we have the issue that even if I make these images they're still going to be Rikus images and not still Debian images so I believe that our end users deserve already built ARM images and we should start making them so we should start building a CI loop that builds them out the same way that we built Debian installer having a generic image and perhaps some custom builds for few platforms which are popular enough to make it justified and then get someone's official stamp for it, probably Steve's Thanks, any other questions? You've mentioned before that the ARM bots do not only need the raw format but some different, can you explain this again? So in some cases people know that the board does not have a bootable removable SD card media or USB media but the board will only boot from its internal flash and you have to update this internal flash somehow usually the standard way of doing this is with Android fastboot where you have a USB connected from your PC to the single board computer and the Android fastboot has the protocol for uploading images that will be flashed to specific partitions the format is called Android sparse images which kind of try to make the images smaller by slicing off the pieces which are empty which is usually the entire ending of the image So what we have been doing is that we take the rootFS partition from the file image and then run it through the tool called sim2emg No sound? Absolutely, yes we probably should be doing some of these images How big are the ones you're making at the moment? About 500 megs Okay, how long do they take to build? Depends if the packages are on cache or not It should be fairly fast Less than a minute So as you saw the images created by Thomas it's the same speed Sure And how many different boards would you want to support? That's a big question Sure So absolutely, I'd love to do some more of these and we should talk afterwards We can happily have these building on the official image build machine we've already got going It then just comes down to logistics and working out the right set of things to do The thing I don't want to do is to have a full set of every desktop for every type of board out there and we end up with 400,000 things and no one can ever test them or find the right image But if we can come up with a good set, yeah, job done, let's just do it Thanks I think we're running out of the time Thank you everyone for joining