 Good morning, everyone. It's fantastic to see nice crowd of people here On the next day of the Debian conference In Taiwan here in Taipei. We had the beautiful days already here at the conference We had a nice excursion and we are now here with full power again and Thomas Lange is here in front of us Shot already some good spirit when he came in here. So, yeah, thanks everyone for joining us Also, thanks for joining us here online and Thomas. You are from Germany in Cologne And we had an interview many years ago. Yeah, which was big fun And I'm looking forward to hear more about FAI fully automatic installation and the next steps that you're Doing here with your project. So thanks for joining us and big round of applause for Thomas Thank you very much for the introduction Today I want to read I want to talk about FIME which is a built server for creating installation and cloud images so I'm a CIS admin for more than two and a half decades at the University of Cologne I became a Debian developer long time ago and In 1999 I had to install a cluster of some very old computers or in those days that were the very recent computers and I was thinking about doing something automatic because before I was using SunOS and there was a an automatic installer called Jumpstart and some ideas I Put into FAI because there was no preceding at this time in Debian So we invented the fully automatic installation for Debian I'm giving a lot of talks They are all on FAI and I also do some fight training for companies So what was my motivation? A neighbor came to me and she said oh my Windows 10 PC is broken Can you fix it and I tried to fix it, but it didn't work But she was very open to say oh you can also install Linux on it And then I said oh what what will I do? How would I install Linux on her computer and I was thinking yeah Oh, I did this great tool FAI with this work for her and I saw oh no The the target audience of FAI was or is still mainly the advanced system administrators And also I also looked at the Debian installer and I thought this is also not very easy for beginners so and this was the motivation to look again on FAI and to see if I can make this much more easier for beginners So their ideas let's create an installer which Should cover most installation the Debian installer is very very good because he has a lot of very Very good features you can do mostly everything with it But we could create an installer that is much easier if we only want to cover like 90 or 95 percent of our users of our Beginner users and advanced users still will use their tools and the configuration management so just ignore the special cases and I wanted to ask all the question that the Debian is installer is asking At the beginning, but also during the installation. I want to ask all questions in the very beginning and Then if we have all the information What the users want to install we could create an customized installation image Then we could boot this installation image the installation would run Fully unattended and then the computer is ready and there are three things that reminds me Creating some customized configuration Would the media go get a coffee come back and the computer is ready. This is what's five was made for So as I said, it's for experienced system admins. You have to adjust some Config files which are text files, but a normal user would never edit config files So what I invented was now the FIMI service and the FIMI service is a web service And I'm not the web programmer. I hate web programming so Even I had the idea Sometime before I Started this when I was sick and laying in the bed for two weeks and I had good Wi-Fi in my bed So that's that's the time when I say, okay, let's do some web programming So this is the web page It's not on the FAI dot me domain not yet. So it's on the It's a sub directory or an URL of the five minus project dot org web page and As you see we have if this big enough or should I increase is this good? Okay It describes you can create your your own customized installation media all data will be overwritten So currently this you cannot create an image, which will preserve your windows partition So first we ask for a username and the user password if it's not set We will generate one and show it to you, but I will just enter FAI then you can select your language and During the dep count I just added the Chinese Hindi and Japanese. This was very easy. This was like Five lines of configuration code. So it's very easy to extend this So you can select the distribution currently we have stretch and Buster and You can also enable the backpods for stretch So it's very easy to say oh I want to install a machine with the backpods kernel just one click and you will get this You can select a desktop. I will select the XFC desktop here We have some collections of packages which you can Also enable and you can also add your own package names. So I will just type This one and give so MC the midnight commander and maybe moon buggy So there's an optional email address if you put in the email address You will receive an email with the configuration but also you will be Get an information when your customized image is Is was made so it's ready. You do not need to use it So and then you just say create installation image. What I'm doing is that I always Um Configure the system with a 105 key PC keyboards because that's very common The the local time is set to UTC. So all this very Things there are a lot of things you can do with the keyboard configuration language You can have very different combinations, but if somebody selects a German desktop I I say oh most people would like to have the German keyboard and Also the time zone should be set to German, but With UTC it's fine for all and I also have here the commands. This is not for a beginner, but If you want to change the time zone and so on so then you just say create my image and Now it's working. So what's working behind this? There's no job So this is the first job and then a computer reads the configuration This is a summary of the configuration and this web page is reloaded every 45 seconds or something like this and when the creation of the installation image is finished the web page changes It will take take about one minute and should be finished. So let's reload Not finished yet yet now. Let's see still not finished Okay, you see it has been finished in 65 seconds There's a URL where you can download the ISO image and you can also download a Log file so when the image is created we have to create a partial package mirror and There's a command called fire mirror and you can download the log file and The most important information. I will tell a little bit more later. It's the five config space You can also download it. So you have every information the the confi configuration space and and at the very bottom you see the two commands that were used Creating this customized image. So you can rebuild this customized image on your own I do not download this now Because I prepared one image which I already have here and now I have a wrapper so I can Call a KVM machine with some network settings and I say please boot it says Create a KVM machine with some network settings boot from CD and the ISO image is this one So let's see if this works full-screen mode Need some time. So now this ISO image is booting up and Currently the local disk, which is just an empty disk is already partitioned you see there are some part at commands and mkfs x4 commands which are executed and Now you see that locale and a lot of packages are Installed and the passes media slash mirror. So the packages are inside the ISO image So everything you need is on the ISO image. You could even do the installation without networking But since you want to have network later When the desktop runs I do the installation also with network So and this is now the fully unattended installation with the configuration That we clicked on the web page What I will show you now is That I go back so we added a Simple toggle so in the beginning the first version of this web page had a lot of more Questions and this is the bed This is a toggle between some basic settings and somehow advanced and in the advanced mode You also will be asked for a route password if you not supply this Zudo will be configured You can upload your SSH public key for accessing the route account You can enter your github account and then we just do a W get from the github Also to receive the public key We have different disk partitioning schemes You can say oh just and this is like in the Debian installer where you have just One partition or one partition using LVM or separate home partition or a home partition using LVM And this is Yeah, these are the The three or four additional information you can enter if you toggle to the advanced settings So let's see any questions so far, okay, so you see now the Linux image 4.17 is installed and this is the back box ports kernel So we have a stable Debian stable, but with the back ports kernel that's currently installed So after all the packages are installed there will be some customization scripts Executed and I will point you to this later These are shell scripts that add a user set the password do some customization on I guess What do we have ETC message of today and so everything that you want to customize Can be done with shell scripts or if you have Some different configuration management it would be possible to add so now you see executing Debian 40 Debian 30 interface Executing demo 10 miss these are the customization scripts and then the installation is finished We call the the f command to see which Partitions are mounted you see we have a slash target slash target boot and slash target home directory Slash target is the the mount point where we change route into it to install all the packages And we see the installation took 215 seconds and then the installation stops so this is not completely unattended, but this is just for showing it It's very easy to set a flag Which then will not stop at the very end, but just reboots the computer. So I just type return We we already are we we again boot from ISO, but no the the the boot order is look at the disk and then at the ISO so Debian was the user FAI password I Switch to full screen mode How does this look over there? Yeah, ah Yeah, I'm not sure if it's really the same Can say it's the same Okay So you see this is a Xfce desktop And as you see The configuration was the configuration using LVM and to separate home partition We have the 4.70 kernel The Debian version oh I created a buster thing so This midnight commander tool was added The moon buggy package is there so Yeah, and also gimp was added So very easy to just add your packages and to create a new machine So we'll just stop throw it away Going back to my slides. So what did we see it's very easy to create an installation media Which also works on USB stick you just have have to do some clicks on a web page You can select certain things and Like distribution the desktops We have different disk partitioning schemes You can add a public repository And Now the same for cloud images now you you saw an installation image What we also can do is and we go on this link creating a disk image which can be Used in a cloud environment in a VM where virtual box KVM environment there you can select How big should the local disk be we could say a five gigabyte we can select different formats There You can set the hostname The root password I said it to will set this to five Oops, we don't need this user password Also set to five US English we will just use the stable distribution with the text console and maybe just MC and moon buggy and I will now add my email address and Then they create image so This will not be the ISO image which runs the installation, but I will get the Raw dot z standard, so this is new type of compression which it which is very very nice very fast so it's very good for big files and So this will last a little bit longer because we have not just to Collect all the packages and put the packages and the live system and the fire Software and onto an e ISO, but we have to run the installation into a disk image So while this is running We have some more ideas what we could do with the FIME service First make it more dynamic that was done during the mini-deb conf in Hamburg Thanks to Yuri who did this so this needs some JavaScript Thing in the web page and as I said, I'm not very good in doing web programming We could do create images for non AMD 64 Architectures, this is very easy because five can build cross architecture images So it's mostly extending the Web page and maybe a few things in the tools that process the jobs these web jobs if I can also Install other distributions, so it would also be possible to create an Ubuntu Installation image or as sent to us cloud image And in the Debian cloud team, we already have some five configurations for the major cloud providers, so currently I'm using the Default five configuration that are in the Five package, but if we would use the configuration from the Debian cloud team We could create ready to go images for the cloud providers Another idea is to create a generic image So you do not need a customized image But you could use a generic image that you boot and in the very first beginning It's ask the job ID that you've created on the web page So I tested it so it there's a proof-of-concept you boot this generic image It asks for the job ID. So just these letters and Chars and then the live image will receive the configuration and will download the packages just from a Debian mirror This works or could also work very easily I Could also create live images so Another ideas to to extend this that somebody says or I do not need an installation image, but a live image so and if you want to do more customization then This is the next step that you set up a fire server by yourself and do everything in your on your own fire server But fire server does not mean this web service But fire is normally using a pixie boot network set up Where you have a fire server and the client install installs a wire Network card and that's what I mean set up your own fire server so let's see the as you see the The disk or cloud image has been finished in 80 seconds It's only 250 megabytes big Although I said the disk image of the machine that I want to boot is 5 gigabyte. This is because it's a spare file and most Bits are zero. I copy the link location And now I try if the download is fast enough. Oh, yeah Okay, this is the downloads So again here we see the lock file. We could have a look at it This is what normally is on the console The five version you see the five version then in fire we have some classes defined These are the like the Lego bricks that will be put together to have the whole configuration and Then some variables are defined. This is the part Then the disk will be partitioned You see the part at and MKFS commands that are executed and then again a lot of packages are installed and set up configured and the end the customization scripts are executed and The log files will be written and so on so you have the log files of the run of your customized cloud image and Here's the command that was executed for creating it So the downloads finished now. I have to uncompress compress this image and You will see it's very fast to extend the 250 megabytes to a five gigabyte file and I would start it And now The last call when I started the ISO image with the 5 kvm wrapper I said 5 kvm CD now I say 5 kvm disk and So I have to distinguish between booting from CD and from disk So boot the image and as you see there's no booting up a kernel with the fire Thing but that's it the machine is running very very fast. We can look in as Debian the password is FAI and You see just a plane. Oh I guess it's very hard this white on black. I'm sorry for this But the midnight commander you see the mid cut midnight commander was installed F10 and Moon buggy is Also on this machine. So this was very easy to create some customized VM disk image So How does Fimey work in the background so first what you see is the web server with the web page But the web server is not the build server So I have another machine a virtual machine and Then the web page has some pearl CGI in the background that validates the input So you cannot say oh, please create a disk image of 20 terabytes. No, this will not work Hopefully I Do some good programming and then You said you see every job gets an identifier and this is in the background a directory So every job has a directory with the name of the job ID and there we write a config and a meta file and Also a status file. So for example if a lot of people are using this Service you will see your job is waiting for processing because another job is currently running And if the status changes, you will see all your job process is now being processed or it's done Or there was an error And We have some This is a one big shell script in the background that reads the directories and looks if there are new jobs And if it has to deal with new jobs and we have some additional scripts for parsing the locks error We create for example if you want to create a buster image or installation image or a cloud image Every night a new buster NFS route is generated so you really get the newest version of buster and We have some monitoring and we clean up the old images if you create one You have like one to two days to download your image. Otherwise, it will be removed And on on this Server that is creating the images. We have three different configuration directories for FAI With with which are using the name of the distribution because we need different NFS routes This is something technical inside FAI So for each distribution another NFS route the five configuration space is the same for all distributions Even if we would install Ubuntu or provide Ubuntu images, we could use the same five configuration space We can read if there's a new job the directories between the web server and the build server our NFS mounted We make a copy of the configuration space or of the default configuration space Then we do some customization that everything that you click on the web page will be added to the default config space and then We we we have two different types If you want to do the want to have an installation ISO We first create the partition package mirror and then create the Installation ISO or we just create the disk or cloud image there. We do not need the partition package mirror Then we update the status write the log files and if the user provides an email we sent the email Oh, your job is ready. Here's the configuration and the ISO image will copy it back to the web server and this is very fast because I love RAM and If I'm creating disk image installation image I always do this in slash TMP, which is a TMP FS So I just throw them away very nice. So if you do a lot of things put some RAM into your machine So fine needs an NFS route a configuration space and then the important things the fire classes as I said It's like the Lego brick bricks and every class It's just a name of a file or a directory and they define for example the class home underscore LVM Defines that the partitioning should be an LVM partition with a separate slash home partition The GNOME class defines. Oh, please install the GNOME desktop a list of packages And these are the the commands that are used for creating it This is how the architecture looks if you do the network if you have your own five installed server and do the network Installation and on the left side again You see the the main parts of FAI the NFS route, which is the live this system during the installation You have the configuration space, which are just some simple text files And you need access to the Debian mirror and these three main parts will be put on to the ISO image for the installation so So normally when you set up and it's installed server and have this network installation running It's very easy to convert this installed server with the two commands into a bootable ISO image This is an example how the disk partitioning is look like when we started with five We thought oh, it should be very easy for a for a sysadmin who knows how to edit files to create configuration files and that was the reason why we would never never ever used XML and If you look at this you will see oh this looks a little bit like an etcfs tab file. Yes, that was our intention So we extended this a little bit and when you see like two or three examples I'm pretty sure you could create your own configuration if you are a sysadmin and At the bottom you see which types of file systems we support. We also support software rate configurations and crypt setups For the software installation, we have a different configuration file format We always use this a different Sub-directory this is a sub-directory package config and there's a file called debian Which is used if you use the file class debian and then you have this lines packages install no rag Which means use apt-get Minos no re-commands And then you just list the name of the the debian packages. That's very easy so some references in The past I was very very bad in creating webpages. I think like maybe six or seven years ago I I Know maybe it's already ten years ago. I redesigned my file project web page which has some graphical elements before I only had text a lot of text with a lot of links like the debian web page currently and I Collected and I have a fire questionnaire and every time somebody says oh, I'm using FAI I say oh, please fill out the fire questionnaire and I started very early in asking. Oh, could I use your logo? but It was only half a year ago when I started doing some web programming web page Again, where I put all the logos that I collected over the years and because I think it's it's a very good Advertisement to say these are companies that are using all that used FAI in the past Before it looks like this. I always showed a list of Companies but this is much easier. Just this is just a nice eye catcher So At the very bottom you see the grimoire project, which is a live CD for cis admins is also using FAI since several years for automatic building daily iso images so questions Yeah Max and do you support the setup of different Sub-volumes in one Okay Currently When we have a good discussion about sub-volumes in butterf s I didn't implement the butterf s support And I know there were two or three people that said oh, you're currently I'm not sure what we support and what we do not support, but there's something with butterf s sub volumes That we currently not support And if you are interested in Please contact me. I can point you to some mails I think it's not that hard, but the only thing that we That we need and I'm not sure how to decide it how to support it or Do we want to be very very flexible or would it be really easy to say this sort of sub volumes? We support and that's fine for most of the people So then there's an ongoing discussion and I'm I think currently We support some sub volumes, but not what a lot of people like to have Thomas yes, so There's a few remarks that I made to you about yeah image Like I should maybe mention it for the others. Yeah like your web page shows How to customize? The SSH key root password and user These are all things that are normally Configured by cloud in it. Yeah Do you intend to remove that later or what what are you going to do is it? I would guess I would like to to Distinguish between the cloud image for the cloud providers But still have the the option that you can say I want to use this disk image just in the local VM Bear virtual box machine, so maybe the also with the host name And then there's not a cloud image. Yeah, so so so we we just have to distinguish there's an installation image a disk image for a VM and a cloud image and You're completely right that cloud image All people will use cloud in it and then some options that we have on the web page has to be removed Me me me maybe we can have a toggle use cloud in it or do not use cloud in it And then you get different things. That was my other question like You never mentioned cloud in it and I And I By seeing the page here, I see disk image format. I don't see cloud provider type Meaning that there is no way to configure for example the data source list of cloud in it Probably that's desirable to have that and also to have the different agents In the image as well The different agent agents could be added here with just with the packages, but you're completely right This is the part where I use the word cloud for some Advertising because it's very catchy, but you're right. These are not really ready to go cloud image for the main cloud providers and like do you also add things like PCI hotplug Configuration block device hotplug and such things or it's not yet there yet I would guess the kernel would care about no, you do need to configure it if you know You need to load the module so it is to go in ETC mode probe D or something Okay, and I didn't know that that there are some special configuration needed for PCI hotplug it's like Otherwise you just plug Block device with whatever cloud command and then the VM doesn't see it Okay, so I'm pretty sure that for for having a ready to go cloud image. We there's still some work to do And then and I hope I could just copy this from the Debian cloud team configuration Hi So two questions for you. Yeah I'll go so you mentioned CentOS. Is that something that you are doing right now or that's an aspiration? No I'm I can do this. You can do. Yeah, I did it on my own several times. Okay, great The second question is about a user scenario. I wonder if you're thinking about so for many people who build custom images For updates, they also want to be able to control the set of packages that get installed onto the machine I think you mentioned that there when creating the installation image you create a Package repository. Yeah, okay, so I wonder if that's So the the scenario I imagine is if that package repository gets saved and then someone can Repeat the build of the installation over time Then the updates will get into that repository and then be able to get onto the user system I don't know if that's a scenario. So you've considered so for the installation image as you see you can add one Public repository here. So and when you add the URL of the repository and create your installation image I will download this package from this local repository and put it on the ISO on the installation ISO I hope I did it right that I also configure sources.list correctly that this URL will also go into the Installed system and then you can do normal upgrades Okay, is this yeah, I think that's what yeah what we're looking so that a scenario that's very common for So I'm from Microsoft and we see often. Yeah for Azure for customers When they are deploying updates to their machines, it's very painful for them They don't want to do unattended updates because some patch might get on that breaks their application Yeah, so they want to have control over which packages So if if the packages can get put into a repository that they can test before making that available for their And machines that would be very so so if I will put the package onto the ISO image But after the installation there will be only the entry in the sources list remain because that the package was installed And it will be not be copied onto the machine. I guess there's no need for it Yeah, after the installation and and how you do the upgrades if you use unattended upgrades or have a cron job or whatever This this can easily be configured and there's nothing that fine says you have to do it in this way Right, okay, so I'm thinking of a slightly in a scenario that extends beyond that, but I'll talk with you later. Yeah, okay Is it also possible from the web interface what the manual configuration to other packages for example if I add Unattended upgrades. Can I also add a configuration file? No currently, it's not possible to add configuration files to this web interface I guess you should set up your own file server and then you can do everything in in the file config space so then the web interface is really or Mostly for the beginners that don't want to add their configurations But just want to say oh Unattended installation with some customization is very nice. It's much easier than Writing a di preceding file. So I just click and have an installation media But this is very easy if you set up your own file server You can put all the configuration files and they can we have a command called f copy So which knows about the five classes and if you have different templates of a configuration That should be used for different five classes. For example, you have a five class Department a department B or location C and location D Then it's very easy to put all the templates into a directory and F copy will know which class to use and copies the correct Configuration file into the target directory So we still happen yeah Okay, so Yeah, I also have a question about that so you can set up your own five server But on your GitHub repository, there are no license Files, so I don't know Yeah, I think you forgot to add them So we don't actually know what is the license of the project on your GitHub repositories No, since the the five packages are in Debian You know, that's an open source license and There's there's a read me file And the read me file also says that just this is gpl 2 plus Yeah, so the whole fight website itself The website itself and the background script is not yet open source So if anyone is interested in it Come to me and we will see how to So at false Asia we have it like a project that is related where like we have a lot of events and we want like this like Some configurations like no screen server no desktop Notifications things like that so we also wrote a small script like to set up Like event computers like that So I think it would be an overlap But like so I wasn't sure like how how could that align maybe we can talk about this in detail more and Yeah, like basically would be great to have more options to for additional configurations for these kind of things Yeah, and yeah, and also like headless clients So I think like with the cloud is very Interesting, but like as we are moving also to like headless client like let's say there's Google home There's Alexa echo, but we also see a number of like free software projects that work on Like these kind of boxes that don't have a user interface Maybe we can also put an options like that into five Again which options to like, you know like the Google home or Alexa echo Devices for the embedded devices sure it would be very nice to and Rico will talk after my talk about if I using on on embedded on our devices We could create Or extend this service to say create an image for a raspy and pie With some very easy customizations Yeah, so we can do a lot of things using FAI and More question should I create another image or I already Using the service and maybe I didn't get it. That's my services currently overloaded So if a if there are no questions. Oh another question Since you mentioned in creating embedded images. Have you had a look at how? FAI compares to There's another group that's working on something called Debbie which uses Debbie in packages with Yachto So for embedded image creation, how would How would FAI compare to what Yachto? So the most important difference to bit bake Yachto and all the other tools in the embedded field is that they're Often compile or build the packages from scratch Because they want to have some certain compile options or remove things Debian starts at the package level. So I do not care about Building the kernel from scratch or whatever Debian is just doing the normal debut strap stuff of embedded devices first create a DD image loop back device And then adding packages to it and customizing it and in the embedded world a lot of tools Recompile everything and collect the licenses. So that's the main difference. I I start from the level of packages And so the image the image types that could be supported and that are the different board support Yeah, sure, then if there's an official package repository which I can use and just Change route into a my my empty disk image and call the the package installation Step that can be done also cross architecture. That's very nice Oh, and you will support cross architecture. It's still it's working since I know maybe a year or half a year For form and not only for arm 64. Okay. Rico says works. Okay, it works like this Okay, thank you So how much time do we have? two minutes any more question Okay, if not, thank you very much and stay here to her And listen at Rico's talk about FAI and architecture