 Hi everyone. Let's talk about ArchEyzer. Jump right in. I've prepared some slides and we'll guide you through this now. Looking at the content, so first do like a short outline, talk a bit about myself, the past of the project, where Elsa is being used in the wild, what we have changed so far and what the future will bring. So my name is David. I'm a trusted user since 2017 and have been a developer since 2019. I mainly package Pro Audio things, Python tools, web applications and stuff like that. Whatever I do, I try to basically improve also on the documentation in our wiki, not only for packaging and tooling purposes, but also for our general workflow as developers or trusted users. In 2020, I took over maintenance for ArchEyzer. To explain what ArchEyzer is, it's probably best to kind of segment it away and explain those parts separately. So ArchEyzer is basically a collection of make init CPO scripts, which define the behavior of the image being created. ArchEyzer is also a script to create an installation in a live medium. This is called make ArchEyzer. This is actively in use to create your monthly installation medium. Recently, we've added the script run ArchEyzer, which can be used for testing an image in a virtualized environment to boot either into BIOS or UEFI based systems. ArchEyzer is also a set of profiles that define what is being installed in the image in use. ArchEyzer is quite the old project and has been with Arch Linux for some time. In 2006, Aaron created the SVN repository and committed all the initial stuff basically. A lot of work has been done since then and roughly in 2009, Gerardo took over and maintained the project until 2019. When he stepped down in 2019, we could not initially find a new maintainer for the project. I looked at the codebase and saw a lot of things to be able to be improved, which is why I took over the maintenance in 2020. The project has been moved from SVN to Git and has mainly been developed on the ArchRelang mailing list. When looking at all the source code, you probably wonder, what is all this and what does it do? At the core of the project, there's makeArchEyzer, which does all the heavy lifting. There are two profiles that ship with it called baseline and Relang. Relang is the one that is being used for the monthly installation media. Each come with a built as age script, which still call the makeArchISO script to generate the image. We'll come to this in a little bit. Each profile also has an overlay directory, which is used for files and directories that are being put into the root file system of the image after packages had been installed to it. Additionally, there's customize AI root FS, which basically is a script to alter and do stuff after you've installed the packages and copied files from the overlay directory. For the distribution specific project, it's always interesting to see who else uses this actually. I've scraped the internet, found a bunch of projects and also derivative distributions that actually make use of ArchISO. There are a few special purpose projects that basically just build a specific image, such as the system rescue system or talking arch, which provides accessibility features for blind and visually impaired. There are a few spin-offs of Arch Linux that basically have chosen their own path, most notably probably Artix with their AR tools, or Chakra and Manjaro using Calamaris or Manjaro using the architect framework. ArchISO has changed quite considerably in 2020, and I'll try to give an overview of the most notable changes. For most, we have moved the entire project to GitLab. This is pretty great because it allows us now to use continuous integration and build the entire thing in a containered way. I can show you around a little bit. While the project is now in Arch Linux, GitLab instance, we can of course use a lot of interesting features such as doing continuous integration on merge request and automatically enforcing specific security schemes as in only allowing to merge something with signed commits, etc. It's pretty nice. It really allows us to at least work towards a way where we do not have to do so much manual stuff anymore for testing ArchISO, which is where everyone wants to basically end up, I guess. Unfortunately, the downside of all of this is currently that the GitLab instance is not yet open for public. I mean, you can browse the code, you can clone the code, etc., but you cannot create an account there yet unless you ask Sven nicely for an account. But I think this will soon change and then we will have proper social logins as options, opening the gates for basically everyone to contribute, which is pretty good and very much needed for this project. When taking on the maintenance of ArchISO, I also looked into licensing the entire project under the GPL3. This was due to the fact that up until then, the entire project never had a license. This is pretty bad considering that all derivative works that are based on ArchISO up until that point were basically illegal in most countries of the world. So when you want to re-license or license a project, you basically have to ask every previous contributor of the project for their permission to license their work under a specific license. This is what I did in May and it took roughly two months or two plus months basically to really track down everyone. So roughly in July, we had all the permissions together. This is a great achievement because now people can actually unproblematically fork and download and modify ArchISO without having to fear legal consequences. Quite early on, I replaced the included UEFI shell that was basically a download of pre-built binary from GitHub with a properly built version that can be packaged. This unfortunately took quite some time because packaging the entire EDK2 stuff is not very pretty and doesn't really have a proper build system. It's full of really bizarre surprises, bizarre build system-ish type of scripts that build the entire thing but are really, really hard to grasp and not very well documented unfortunately. But I'm nonetheless glad that we now have basically the EDK2 shell as a binary that we can build ourselves reproducibly instead of downloading a random binary blob from the internet. One of the first actions to be taken out on the ArchISO codebase was to apply linting and code sanitation using shellcheck. As we have a large codebase with bash scripts or ash scripts in the case of the initCPIO scripts, we had to get rid of many code smells and general insecure handling of variables, for instance, non-quoted variables while deleting things. I can give some examples. This used to be in MakeArchISO and there are plenty of other ones in the buildsh scripts of the profiles where you have plenty of unquoted variables that could lead to really undefined behavior under certain circumstances. All of this has been cleaned up basically and is by now in much, much better shape. A script was added and I noted this before. RunArchISO is used for testing the images in a QEMO environment. In a long-lasting effort, we have started to consolidate the profile scripts that buildsh scripts with MakeArchISO itself. This means that in the future buildsh is obsolete. To show you what that means, I probably best show you a version before the change of buildsh scripts for the relaying profile. If we look at the buildsh script, it mainly consists of a lot of hard coding and then many, many, many boilerplate functions that basically all do the same to some extent than the one for baseline, for instance. We look at the baseline, buildsh, it is pretty similar. It has the same setups, it has the same functions basically with a few omissions and that is just a lot of boilerplate basically that we removed and merged into MakeArchISO. This is now based on the profiles that I mentioned probably earlier and it is using this here to basically override whatever settings are the default for MakeArchISO. The profile looks basically like this for arch relaying, for instance, where we define the standard things that used to be hard coded in the build.sh script such as the publisher, the label, the name, the version, the install directory, etc. and the boot modes that we support. The build.sh script has now been cut down to basically only do this. Print out a warning about being obsolete soon and executing a specific functionality in MakeArchISO that will be removed in the next release. Let's have a look at how all of this works. I'm currently running the latest version of ArchISO which is 48.1 at this point in time. As mentioned earlier, ArchISO comes with two profiles, baseline and relaying. These are being installed to the system and basically consist of a the a rootfs directory which is the overlay of files that are being put into the image after installing all the required packages. Here you can see that, for instance, it is used to enable services such as IWD or systemdnetworkd and reflector. It also features the customized AI rootfs script which is being run after installing the packages and copying all the files to the file system. We have a set of custom scripts here that, for instance, let you see the installation guide or choose a mirror through a kernel command line option. Currently it also still features the build.sh script but this is the last version where this is going to be available. Additionally, outside of all of this, there are configurations for the various bootloaders. For UEFI we currently use systemdboot and for BIOS systems we use syslinux. An interesting new aspect is the profile def.sh that I talked about earlier. This file basically declares all the required information for the image to be built. Here you declare the various boot modes that you would like to have on the system such as systemdboot in various options and or BIOS systems using syslinux. All of this used to be hidden away in a build.sh script or in MakeArch ISO directly. So we removed a lot of hard coding and put it into a profile configuration file. Building an image is now as easy as pointing MakeArch ISO to a configuration. I'm providing the verbose flag so we can see a bit more what's going on under the hood. The first thing that is being done is using packstrap in a work directory to basically install all of the packages that we define in the profile. This is very similar to basically installing your system using this image. The step currently taking the longest is creating the inner drum of s image. This is due to the fact that XZ compression does not allow for multicore and being reproducible at the same time. Therefore I will fast forward this step a little bit. After finishing the creation of the inner drum of s image all the bootloader configuration is being put in place and the squashfs image is being created. Overall we're left with a build time around six minutes. Running and testing the image is now also being made quite easy using runArch ISO. Here I'm now preparing to boot the image using UEFI in a QEMO environment. We can do the same just booting into BIOS by removing the minus U flag. So that's basically it. With all the recent simplifications and changes there's still a lot left to do for the future. At a date in the not too distant future I would like to move all flyspray tickets to GitLab. This can mean just kind of referencing them in the GitLab project for Arch ISO or outright closing them because they're super old. We still have many many old tickets that have been opened several years ago and some of them are maybe not even valid because they pose a specific use case that we might not even want to support actually and some of them just need to be worked on and properly figured out actually. That being said we already have quite a lot of very specific use case issues basically in the GitLab tracker. Here it makes it also easier of course to reference them in the pull request or merge request rather and see whether they have someone working on it already or not. Flyspray kind of lacks all of this and it's also a very old system that we eventually want to get rid of of course. We have a lot of old documentation lingering that we need to convert to restructured text and make available more easily to the user either browsing the repository or using Arch ISO locally. Some of these have been around for a long long time and basically described the standard workflows that are being established when using the init CPIO scripts for instance and what can be used to modify them and their behavior. But these are not the only pieces of documentation that are lacking currently. We also have still gaps in the new workflows that we have designed and worked on considering the profiles although they roughly already describe what and how they work and the tooling is also more or less well documented I would say. However I think workflows can always be described in a better way. While the per profile build.sh script is for sure going away in the next version of Arch ISO this might not be the case for the customized AI root FS script that is being used to do post installation and post overlay file system modifications to the image. To know why we probably should have a look at how this is being handled in the scripts and what they actually do currently still in these profiles. While this is already a solved problem whatever the customized AI root FS is doing is not yet. This is due to the fact that here we are modifying and generating the locale settings. This is something that is actually required or potentially required already during boot and we need to test how this behaves when we for instance stuff this into a system de-unit that is being run at very early in boot. Meanwhile the mirror list is rather easy to fix and to get around it because the current situation is that we already have the entire thing being generated by a reflector. While declarative profiles are already a thing more or less they will see some improvements in the future. While we currently have an overlay file system or an overlay directory that lists all the units for instance that we want to enable in the in the image most of these are of course based on sim links to the current root file system which is not very pretty considering that we package all of this basically. We have a lot of sim links for instance in the multi-user target that all do not really relate to any files in the in the repository. We have an idea how to get around all of this by listing the files that we want to have and listing the sim links that we want to create. This will hopefully make things a little more clear. A for some probably rather controversial idea is to maybe add Arch install as a simple terminal based user interface for installation. This would be an optional package basically used in the image to allow templated installations of Arch Linux. To show you why this would be a good idea and how this all works I can show you around the codebase for a little bit and the video. So in the codebase we can basically find a few profiles that allow for certain use case installations. This would of course not be for everyone or would not be something that might be used by everyone or should be used by everyone but it would be an easy and fast way of installing things without going through the hassle of typing a lot of things. It's probably best to show a video. As you see this offers an easy way of installing Arch Linux because having rather declarative profiles that do the same thing all over again that we ship is a lot more controllable than someone's random bash script that does a very specific installation but doesn't really help the greater community basically. This has the upside for being an easily scriptable thing that is optional and can be used or not by users wanting to install Arch Linux. Earlier I mentioned the TalkingArch project as a project providing accessibility features for the Arch ISO image in a specific way for visually impaired people and blind people. The current maintainer Alexander Eponechnikov just opened a merge request so I better get on reviewing soon which is pretty awesome given that we've been basically dangling around with these fixes being up in the air for longer time and I really hope we can merge this soon. Reproducibility is not only important for the packages that we build and provide with Arch Linux but also of course for the installation media. Patches for this were already provided in the past to at least better the situation but the codebase has changed quite drastically since then so they were not included yet. Also since we now move to GitLab we want to further the automation and image release process of the entire thing which means that maybe in the future you'll have a daily release. At the moment releases are still being done manually by our release bot release manager Pierre Schmitz. You might have seen the tooling he uses for all of this on GitHub. I sometimes do pull requests for it because I change things in Arch ISO that affect his tooling of course. This is basically how all of our install media gets deployed and built and we want to automate this of course so that not one single person has to go through the monthly ordeal to provide this and run this on the clock. Rather long-term we probably move to Drakut or something that is not init CPIO based anymore but for this first we have to split out init CPIO anyways. A rather long-standing issue that we're trying to resolve currently is how to provide secure boot support out of the box for something like arch ISO and also how to sign our kernels. On fly spray we have a ticket that has been open for quite some time outlining many many many of the ideas and things that can be done basically to achieve this. One of which was provided by Jonas Vitschel who is one of our trusted users is to get a signed shim which is a preloader basically for the boot loader that we use which in this case is a system debut but could also be grub or whatever. This is also already packaged in community but we require a signed EFI binary. We currently have an unsigned one of course because we do not have a signed certificate yet. The EFI's the shims can basically be signed then also via redhead boots project. This is all rather complicated and also time consuming and anyone who works on this of course always does that on the free time so bear with us until this is solved. I don't really have a fixed estimate or timeline for this yet so we'll see where this leads us. The changes introduced by us to arch ISO of course also have an effect on derivative projects however I believe that the changes that were introduced are actually for the better. They allow people to much more easily create profiles than before and only require make arch ISO to actually build an image. Branding and modification is basically being done in a profile by now and therefore build.sh and also make arch ISO do not need to be modified anymore. Closing I would like to add please get involved because there's still lots left to do and the more eyes the better. You can reach me on XMPP or IRC or via email and if you have questions shoot. That's basically it. Have fun using arch ISO. Thank you so much David for your talk. We're here now to do the questions and hopefully some answers sections as well and my name is Thor and I'll be taking you through the questions today. I think we'll just jump straight into it. The first question is from well I hope I'm not neutering the pronunciation. Energy creator wondering why host the arch ISO project on GitLab instead of GitHub? Well it's to be fair it's our own GitLab instance of course that we host on our own infrastructure. GitHub surely has some nice integration but at the same time it also ties us to this platform basically that we would rather be independent of and to also move into future directions of potentially auto creating images in secure built environments. We actually want to be able to self host that of course and GitLab has very nice integration for many things actually nicer than GitHub in some regards I would say. That's maybe like a personal reference or something but yeah it definitely has to do with us being in charge of our own infrastructure. Ask a one. We have another one from who I asked you for. Wondering if there were any specific reasons why the GNU public license version three or later was chosen I imagine specifically for our arch ISO and wondering if something like the MIT style license would encourage more adoption? I mean the question is I guess why would you not use a GPL license for this? MIT of course offers fairly great integration also if you're maybe a company etc. If you really want to retain certain privileges on the code base but I have the feeling that for Linux distribution it makes much more sense to be as open as possible and to be able to share all of that knowledge as openly as possible plus it allows us also to maybe code share and basically knowledge transfer from all of these sub projects that have been trying to do their own thing but yeah it makes much more sense to me actually to be able to get some of that knowledge back and retain features in arch ISO itself again. Yeah there was a discussion about copy left versus the other alternatives. We have a question from Sven Slöru asking will arch ISO stay shell or will archer remain a project that uses bash as its primary programming language or shell script? Do you have any rough plans to change that? We have behind the scenes probably discussed or I kind of talked about this sometimes more than a jokingly fashion as in ah we need to rewrite everything in Rust but it also I mean it increases the barrier of people being able to to basically work on this. The Rust knowledge in general I would say is probably lower than average people people's bash skills I would say or Python skills so Python might even be a more suitable thing to use here because it's also easier to do tests and sustainability checks and stuff like that. Bash can be really really bizarre and very indirect and dangerous also so I understand that people would like to change and I'm actually open to that but at the same time currently we're still trying to kind of streamline the entire thing because it drifted apart and became this kind of feature creep that had like lots of open-ended things going on in many directions and we need to clean that up first I think to really understand what the core features of that thing is and should be and then we can think of doing something else I think. We have a question from Macalano how is it determined what software is included in the ISO? I think I skipped over that actually in the talk there is a package file packages.x64.86 that defines what packages will be installed for that profile. I didn't look into that file during the presentation I think but there we list all the packages that will be installed to that ISO basically that would be the place where you define like okay I want I don't know KDE do you know whatever installed on that thing and then not modifying it more. Yeah I have to admit I've I've accidentally forgotten to install the Wi-Fi software requirements more than once when I've set up laptops. Oh nice yeah that happens. We have a question from Lirst I think it's been answered on RSC as well but it's basically asking how one can install run ArchISO which is the script in the ArchISO package. Yeah exactly it's included in the ArchISO package I think I mentioned this quite early on but it's maybe a bit indirectly mentioned so if you install ArchISO you will have run ArchISO as well. So Pacman's your friend there yeah this is a bit a longer one but I think it's a good one. From Diabunas are there any plans or ideas to run automated installation profiles such as full disk encryption logical volume management or LBM rate in the continuous integration pipeline to make sure that not only the generated disk image is correct but that the included software works is expected as well. Yeah that's a big one and a very good question actually thanks for asking that. It ties into actually two things first of all we of course need to actually have a proper CI pipeline to to build this thing and that's fairly non-trivial currently still but I'm working on that due to us requiring root and a bunch of mounts for all of this arch packstrap calls to install packages into this chroot and that that makes things a little complicated but as soon as we have that basically we will be able to use some sort of artifact that we create there the ISO basically and use that to install Perthon and I think that's something we should do. I'm not entirely sure yet how this will be done in the future probably it has a bit of like a disconnected pipeline thing going on where we just spit out daily images and then try to A. reproduce them properly and B. boot into them install things maybe into a fake environment and just run some tests but this is really far ahead in the future I would say. One thing that might help that's the second part of it actually is stuff like arch ISO or arch install being basically shipped with arch ISO would allow this to be more straight straightforward or easier to do basically because you don't need to script all the things anymore because it's already there but this is still up for discussion and an ongoing topic because there are still lots of problems with an approach like this as well. We've got a bit of an interesting question I think wondering specifically on how things are with regards to Drakut as you mentioned in your talk and collaboration with Red Hat and it seems like this person asking a key factor tends to be a Red Hat employee and wonders if you need any help. Well yes I'm quite sure we would need help there because I for myself I don't really have a Drakut based system yet and we currently actually have issues. I'm not entirely sure if that's arch ISO related or that person's specific system related where people running a system with Drakut on the host basically have an issue building the image and yeah so yeah help was much appreciated we actually we are considering this because we have a lot of init CPIO scripts of course that we currently also look into splitting out into a proper its own proper project to yeah separate it out from from the entire thing because it's not required during during runtime you only need it in the image itself right. We have a question here from you 1106 wondering did I miss where ArtsLinux bootstrap tarp comes from? Is it the same project or package or is it something unrelated? Ah the bootstrap yeah no that that stuff is generated by the things that Pierre Schmitz provides basically on GitHub there's a link during the presentation I will share the PDF shortly that also creates this basically and there it's mentioned how this is done basically just an extraction of the created image basically yeah I hope that answers it. I think it does. There is another one from Oak wondering is there a way to run scripts automatically I imagine in arch ISO? Well there we would probably have to distinguish between during build time so when you build the ISO image there currently is still our customization script that is allowed to be run but we are soon deprecating that I'm not entirely sure how because it makes things unreproducible but I'm not entirely sure yet how we would allow this in the future maybe we provide a directory within a profile that you can put your scripts into and these will be run automatically within the creation step of the ISO with of course the downside that this will probably render your stuff non-reproducible this is nothing that we would use then for the installation ISO and otherwise during runtime of the image itself you can do whatever you want with systemd services or scripts that you basically provision into the root file system there. And I mean yeah I started to wonder like who could be doing this but I guess this is probably more related to the internal use inside wondering if anyone uses or tests the pixie booting of the arch ISO result. I do sometimes mainly when I'm breaking it it's due to the fact that we for instance now changed the naming of the of the images that are being created by Megan in CPIO in the image they used to be called arch ISO.img etc and so we're very specific to the ISO and now we rely on the names that are given by the kernel package itself so you could now install several kernels side-by-side or you can install any kernel you like it will always generate that specific kernel for you. The thing is just that pixie booting is a bit tricky because it involves also modifying arch web due to the certificates basically validating the images and so on that you download. So that's slightly non-trivial and I hope we can improve on that a bit in the future because that would be much appreciated. We've got a few more questions I think and here's one from I think a if not already a potential future collaborator. Alcazar79 wonders if it's a good time to rebase on the latest version of arch ISO or if there are still a lot of disruptive changes coming up. Oof, yeah good question. I guess rebasing definitely makes sense now already we currently finished on more or less deprecating build.sh scripts for the conflicts of the profiles so we just remove them but they're not being really used anymore anyways. So that's basically the biggest step just get rid of your build.sh script and you should be fine. If you have questions I mean you can always get in touch if you're running to issues with specific use cases then yeah please do get in touch on the mailing list for instance. I think we have time one more and then we'll yeah sure probably round up. This question from Jerry Xiao I'm sure that's the right pronunciation so my apologies but it says arch ISO recently added a service called reflector which is a package because you can also download normally from pacman if you like which connects to internet and sorts mirrors by speed without user permission. Is it against the keep against the kiss principle? Keep it simple stupid. Exactly. I'm not entirely sure I mean I guess it is not against that principle because it actually allows you to have a functioning system right from the get go and as we I mean we can use reflector we could also use something else if you want to do this manually you can of course build your own ISO but I have the feeling that it makes more sense to actually provide something that is functional from the get go instead of having too many user interventions until you can actually install things and that felt it made more sense to me but yeah if you have other I don't know incentives about that please let me know. I'm open for suggestions. Well with that thank you for answering all of those questions and thank you. Thanks for the talk.