 Good afternoon. I am Andrej. And I am working as a software engineer at Red Hat. And I am working on a project called ImageBuilder. And today, I would like to show you what's new and how you can use ImageBuilder to build some nice, up-to-date, customized, and shiny images. And yeah, by the way, I am going to skip over some of the details. So feel free to ask if something is unclear. And also, we have a website. So if you want to know more details, go there. There are some guides. And you can follow them basically to do exactly what I am going to show you here. So let's not waste any time, any more time. And go directly to do important bits. So what's ImageBuilder? ImageBuilder is a modern tool for building operating system images. We will go into the more depth, which exact images we can build, because images, it's kind of a generic term. So we will explore this more later. But the basic facts are that we are building them from scratch. So basically, we will download all the repositories on RPM from the repositories and then somehow combine them in order to create an image of an operating system. And it's important to know that there are no VMs involved, which is quite nice, because a lot of modern cloud environments don't have support for running nested virtualization. So it's kind of nice that you don't need to spawn a VM. But of course, then the other method that you can use is to use containers, and namesplacing. So it needs root. So no VMs, but root required. The important fact is that ImageBuilder never boots the image. So it makes really sure that it's pristine. And we will get to this later why this is interesting that the image is never booted. And yeah, this talk will focus on ImageBuilder that you can install on your machine. We have also some other options where we will host ImageBuilder for you. But yeah, I will mention them at the end of the first part of my talk. And yeah, so that's ImageBuilder, modern tool for running operating system images. Now, what images can it build? And I will start with distributions because that's the first thing I want to mention. So we can build Fedora and its children, which means CentOS Stream and REL. And for Fedora, we of course support all the supported versions for CentOS Stream and REL. That's eight plus. So yeah, but the more interesting part is for which environment ImageBuilder can build images. And there's plenty of them. So I actually put them on two slides. The first slide is about all the cloud images that we can build. The first blood point is called KVM. And that's what I like to use for these environments where it's pretty much just KVM without any specialties, which means libvirt. So you can just virt install the images. OpenStack or kubvirt, you know, openshift virtualization, there was a very nice talk earlier today. So if you are eager to try kubvirt after this conference, you can also use ImageBuilder to use a custom image there. And yeah, then we support also all the major cloud players, for example, AWS, Azure, Google Cloud, Oracle Cloud, and VMware vSphere. And from my experience, the KVM image, if the cloud provider uses KVM underneath, the image usually works. So I, for example, use the images from ImageBuilder at Hatsner because you can just upload it there and it works because they use KVM. So it's pretty cool. So these are the cloud environments, but we can even build more. And these ones are, yeah, more interesting, I would say. We can build installer ISOs. So basically that's, you know, Fedora has the installer and ImageBuilder can build a customizable, we will get to that later, but it can build a customizable installer that you can then use to install as many bare metal machines as you want or even virtual machines, but please use images for virtual machines. It can build containers like the OCI one, so Portman. Just a word of caution. We don't support custom container files or Docker files. We are really meant as a tool for building the base. So, you know, if you want to build something on top of Fedora, you use specify from Fedora 37 and ImageBuilder can build this base. And the last thing that we can build are OS3 artifacts. Currently, that means Fedora IoT or REL4H in the REL world. And we can build all of the artifacts, so comments, installers, raw disks, simplify these containers, that's pretty much it. And that's pretty cool, but I won't dig into this deeper because I think that during this conference there were like free talks about Fedora IoT and REL4H and each of them mentioned ImageBuilder. So, if you are interested, just watch the recordings. All of the talks, I was there. They were really amazing. And just the last word of warning. Some combinations are not supported yet. So, for example, you cannot build a container of REL current, I think. It's not implemented just because no one had the time to do it. But if you are interested, please contact us. And usually it's pretty easy to fill the matrix. But yeah, there might be some gaps in the support. So, that's what images can it build. But of course, there is one more thing because base images are nice, but base images are also boring, right? You can also very easily build customized images. And that's where ImageBuilder really shines, I think. We can do plenty of customizations on the image. And let's go over them. I picked the most interesting ones. And then at the end I have the new ones which are pretty important and interesting. First thing first, we can do custom partitioning. Custom partitioning, that's where ImageBuilder really shines, I think. Because, for example, Pecker is an alternative, but Pecker boots the image and then does something to configure the image then it takes a snapshot. But when you want a custom partitioning, for example, when you want to separate slash user or slash var on a booted image, it's kinda, well, it's possible, but it's really hacky and just don't do this in production, please. But ImageBuilder doesn't do it. Like everything happens on, you know, not booted system. So it can do whatever partitioning you want, which is pretty nice. And you can even use it, yeah, we will even use it later on this slide. And of course, install extra packages because we believe that the base operating system should just carry enough stuff so it boots in the target environment and nothing much more. But that's, you know, pretty useless. You want something like HTTP server or, I don't know what. So there are 50K packages or even more, so there's plenty of stuff to install them there and ImageBuilder can install extra packages. Also, it can do, it can add to users, which also, you can do it, for example, in cloud. You can do it via cloud in it, which is a good tool, but maybe in certain cases, you want predefined users, so all of your deployments are the same and you have, for example, one admin user with the same SSH key everywhere. So maybe it's good to put it into the image itself. Why not? It can simplify your deployments if you need this. It can configure firewall, of course. Once again, that's pretty nice because the firewall is configured immediately after you firstly boot the instance, which feels kinda secure. It can manage system units, which is, you know, it ties nicely to the extra packages because web server is cool, but if you cannot enable it, then it's kinda doing anything, it doesn't do anything. So that's think, and yeah. Now let's go to the new stuff that we introduced in past few quarters. One of it is hardening, which is, yeah, based on OpenScap. Basically, how this works. So OpenScap is a tool, you give it a profile and it scans your whole system and applies some remediations. And these profiles are basically based on certifications. So if you need a certification ABC, you apply a profile ABC and OpenScap will try to do as much as possible so your system is certifiable by the ABC certification. And ImageBuilder can apply such remediations on the build time. So once again, the system is hardened from the very beginning, from the very first boot, which helps, you know, to establish a secure pipeline. Also, a lot of the certifications require a specific partitioning. And it kinda ties together, right? You can use ImageBuilder to do the custom partitioning and also immediately harden the file system, like the system. So these two features are very often used together. We can inject extra files into the image and that's also very useful because, you know, you install a web server, you can enable it and you can also configure it so you can have an image that immediately starts to behave as a reverse proxy, for example. And the feature that I like maybe most from this list is embed containers. What this means, when you tell ImageBuilder to embed a container during build time, it will download the container from the container registry and put it into the right directories. So when you run Portman or MicroShift, it doesn't need to pull that container but it immediately has it on the system. And this is great if you have a disconnected environment because then of course you don't have any container registries or for the IoT slash edge use case because your device might be on top of a hill and there is no good internet connection and you don't want to wait 10 minutes for a container to download for a registry. So that's quite useful for these kinds of scenarios. And yeah, I just wanted to mention that yeah, OpenScape by the way, so you can, so we know what it does. So for example, it can do some small changes to the configuration like changing, configuration auditing, SSH configs, OpenSSL configs. I mean, base distributions, I don't know what is pretty secure but these certifications require more and we don't want to put this config in the base system because sometimes the system might behave weird because users want to use kind of, you know, older hashes and things like this. So it's always about trade-offs. Anyway, there's one more thing that ImageBuilder can do and it can immediately take care of your uploads to the clouds, which is pretty cool because if you've ever tried to do multi-cloud image uploads, it's a pain. Like I uploaded an image to all of these and IBM cloud actually also and OpenStack and all of these CLIs are different. The process is different. You need different compartments. That's my favorite word from Oracle Cloud. You need different storage groups, storage accounts, storage containers, whatever, it's crazy. And of course you need to have all of these CLIs installed on your machine and it's annoying. But ImageBuilder abstract this away. You just give it a config and it will build an image and immediately uploads it to a cloud, which is very cool because to be honest, we believe that this should be more, like more tools should do this because building an Azure image locally is fine but what are you going to do with it? Yeah, you can boot it with QMU but it's an image meant for Azure. So it really should end up in Azure. So we are trying to tie these two processes together so because it really makes sense. Anyway, yeah. So the ImageBuilder that you can install locally, it's just both GUI and CLI. And at least my understanding or my vision is that you can play with the GUI and see what ImageBuilder can do and kind of click around and add customizations, define everything nice and visually. And then when you are done, you can just export the blueprint and then you can just use the CLI and set up an automated CI pipeline. So yeah, I consider this as a two-step process every time. Yeah, and if you want to build an image or start building images, it's very simple to install just DNF install two packages. One is the CLI, one is the GUI and then enable the service that runs on the background and builds images and you can start building. It's that easy. Yeah, good. Now I promise that I will talk also about other options how to consume ImageBuilder and it can be also run as a web service. And currently the web service has two clients, one, well, three. Yeah, one builds golden rel images and one builds a golden currently federal IoT images. So that means that ImageBuilder can be integrated with Koji and it is, it builds images every day. The second client is our ImageBuilding service in Red Hat Hybrid Cloud Console where you can just grab the free rel subscription. It's fine, go to console, there's a link here, console.redhat.com slash inside slash ImageBuilder and Red Hat will build the image for you and you don't need to maintain an infra, you will just send you a link or share the image into your account and it will just work. So my message here is that ImageBuilder is run in a production environment and it's suitable for every day use. So just use it, it's stable, it's fine. Good, is that that? Yeah, and I should also mention that the service has also an API so you can also automate your CI pipeline using the hosted service. Which is pretty neat because you can just call it from a container in a good lab CI and it will just get to have actions and it will build you an image. There should have been an image of the service, but yeah. Anyway, and just two items for the future. We would like to collaborate more on using ImageBuilder to build more federal artifacts. Currently the project is with the installer team. Hello. Because we are talking together to building federal installers because we would like to modernize the ImageBuilding stack. And also our big dream is to enable the community to easily build and share customized federal images because we think that the process currently, you know there's Koji, Banji, Rallying and it's like it's pretty, it's pretty a big process. So we want to make something that would simplify this process by a lot. I also forgot to mention at this slide, the ImageBuilding service, I just like how everything ties together in this conference like with KubeVert, I mentioned, I mentioned, what did I mention? Rally for Edge and Federal IoT. And the service, it's actually tied to the keynote because it's an open source service. And yeah, it's one of the, we try to follow as much open service principles as possible. And Simon here is going to have a talk right after this talk about open services. So you can go there. Anyway, that's a bit of promo. It's demo time. Good. So this will be the part which can go wrong at any time. So I have this small shell script. I will show you it, I will show you the script in a bit. But I will build a config called demo for this demo and it will be a QCOW too. So we will boot it in a bit with KVM with just QEMO. So I will start it and now I have about five minutes to explain you what's going on because the image will be ready in five minutes. No uploads will be done because of the wifi here. Yeah, I don't want to risk it. So it will be local, but whatever. So the image is building. By the way, I will quickly show you the script. Yeah, it's pretty short. I have some debug stuff in there, but basically I'm just calling Composer CLI with a bunch of commands and then parsing it. It's pretty simple. I made this script so I don't need to go over all the steps because they can go wrong, so I automated it a bit. This script, by the way, will be available at the archive of my talk after the talk probably. Anyway, let's go to the configuration. We call this the blueprint and this is what ImageBuilder consumes. It's a Toml file and you can put a lot of stuff in it. So let's go over it and see what our image, when it's done being built, what it will contain. So a good, a nice feature of ImageBuilder is that it can do cross-distro builds or cross-distributions builds quite easily. So my system is actually running Fedora 38, but for some reason I buy more different distribution, so the image will actually be Fedora 37. So it's completely possible to do that with ImageBuilder. You can also, for example, build Central Stream 8 or Central Stream 9 and yeah, that's it. The next thing that I want to have in my image is Portman because I want to demo the embedded container thing and I need a container, how do you call it? Container engine. So I installed Portman and some optional dependencies. I'm not really sure why they are actually needed but that's a story for another day. Not sure if that's a Portman bug or I don't know. So yeah, this will install Portman inside of the image. Then I want to embed a container. So I can just tell ImageBuilder, hey, download me this container. And yeah, I give it the source, which is my repository at gitlab.com. It actually contains the slides of this talk in a web server and yeah, it will just save it locally as a talk container, just talk, it will be named talk. And that's it. But you know, download the container, that's boring. So let's run it. And for this I am using the new Portman Quadlet Generator. So let's explain it a bit. It's very cool. Yeah, it's cool, right? So basically it looks like a system unit and the system D can consume it via its generator thingy thing that Portman implemented. And so it is a system unit, right? It has a unit thing and it apparently runs after network targets. So after the networking is up and it can be even installed. So like when I install it, it will run or it will be required by the multi user or default target. So it's a system unit, but there is no service but there is a weird container section which basically the generator will translate it to a service and it will translate this to a running Portman container. So I can just tell it, hey, take this talk image which I downloaded in the previous step and you can do the usual stuff that Portman can do. So I can just tell it, hey, publish me these three ports because it's an HTTP server. So I want 80 and 4443. And of course, the container is embedded so I can just tell Portman, hey, don't ever pull it. You already have it in the store, yeah. So that will save me some bandwidth potentially. So I also expect maybe with my image that I will need to store more data. So I put there an extra var partition with 10 gigs of size and yeah, it will be there. So this is the example of custom partitioning. Yeah, there is just a quick note but we have a support not yet in image builder but we are getting there slowly. So this will be just X4 but I mean for the cloud I don't really care so it's fine I think. Then let's harden the image which is this section basically this tells the OpenSCAP tool that's run during the build time to apply from this file. This is like a real file that you can install on your system. This profile and then OpenSCAP will run anything what's defined in this file and you know the scanning as I told you previously it can I don't know configure SSH to not allow password keys. Well, that's disabled by default in Fedora but I don't know some other stuff like enable USB guard and yeah, whatever is needed in terms of security. And there are different profiles and on our website there are some examples of other profiles that you can use and you can also see the OpenSCAP documentation for more details. Anyway, let's go to the next customization we can add at firewall. So just install firewall D and immediately configure it so we want to allow HTTP and SSH and that means that all the other ports should be closed. And the last thing is adding a user as I promised you before image builder can add users and yeah, I can just create an admin user I can add my key this is my public SSH key and I can add it to the real group but the issue is that so the real group is for accessing sudo and the issue is that Fedora by default doesn't allow passwordless sudo but that's fixable by the first customizations because I can just add a drop in and we will be passwordless so that's possible. Good, let me just quickly open the front end. This is actually the front end so you can see that the demo blueprint that I put into image builder via the script is there so it's the same thing as I did on the CLI just visualized here and you can see that everything that we talked about is here so there is the OpenSCAP thing OpenSCAP remediation, there are the file system options firewall users, if I go to packages there are some extra packages, you know there are also versions which is pretty nice custom files are currently not there we still need to add them but otherwise everything is there and you can just click create image here you know select that you want QCOW2 and it will build an image for you and you didn't need to touch the CLI but he likes CLI, right? Anyway, so yeah, while I was talking it took about five minutes to build the image and I downloaded it as an image as a file and I will just use a simple script that I have to deploy it this just calls QEMU and yeah forward some ports, it's nothing super amazing you can also use LibVirt if you like LibVirt and install it, it doesn't really matter it's there's no difference and the image is now booting and I should be very quickly able to SSH into it this is the scary part of the demo because it worked like 20 times in a row good, it worked so this is the image and we can go over the customization so Federa 37, let's quickly verify it it is Federa 37 let's see if the container is running so the container is there yeah, it's the image that I had in the in the blueprint I can even like you can even see that the image is installed I can even curl to there yeah, there is something I can even show you in the browser that if I go to the localhost 8080 it's there I just forwarded the part so that works so everything is good and what's there, what's next next was this was shown oh, let's show the partitioning so if I do df-dash-h you can see that var is there it has 10 gigs or 9.8 oh no, it has 10 gigs, right so the custom partition is there and yeah, what else? firewall so firewall let's see if I can do it on the first attempt list all no good, so by default Federa enables the HTTP V6 client and MDNS and we edit HTTP and SSH so yeah, this also worked and the firewall is actually enabled and I think, oh, I edit the user, yeah and the user, yeah, sure, I am logged in as admin and it didn't ask for a password so apparently my key is there and I used sudo without any password and yeah, that's my demo and I think that I am almost out of slides yeah, I am totally out of slides so yeah, image builder I think that it's a great tool and not only because I work on it but also because I like it and I think that with these new options it can get you pretty far if you need a customized image there is a lot that you can do to basically build your image push it into a cloud launch an instance and it will run its workload as intended so yeah, this is image builder and I'm glad that you were here for this talk and if you have any questions please I'm here to answer them that's a great question so I will repeat it, thank you so the question was how does the environments I showed KUKAO but or QEMU but there were also Azure, AWS and other ones and how do they differ so there are some slight differences for example, on Azure you want to install their Azure client, it can do some more integration agent, ah, agent on Google that's the same thing Google has their own agent so you want to install some extra packages and yeah, also sometimes the configuration is a bit different for example it's not done actually for Fedora but maybe we should do it by the way, CloudSync EC2 really recommends that all the distributions has EC2 user as the default user, Fedora has Fedora so this would be something to configure just for EC2 images which would be, I don't know, quite cool yeah, and the package says kernel arguments can be different I think that for CentOS 3 we are disabling some drivers because they are causing some issues on some instance types so the differences aren't huge but I think that making images or tailored to the target environment it makes sense oh, sometimes you need to take care also about bootloaders a bit yeah, hope that answers the question no, all right, so the question was whether we always do QCOW 2 no, it depends for example, for AWS we will build a raw image because they accept only raw images for OpenStack it's QCOW and for Azure they need stream-optimized VHD images so we will provide that and of course then for the non-clouds ISOs are produced as ISOs and yeah, that's it yeah Neil? Yeah so the question was why we don't have a customization that would simply allow you to alter the Sudoverse drop-in to allow a passwordless configuration yeah, that's a great suggestion and actually a funny story when we introduced the Etsy customization yeah, I got a report from a colleague that it doesn't work it screwed up his environment yeah, he forgot like something in the file, white space so I think that's a great suggestion and our team is trying to actively seek for input so we can integrate more stuff that would help and this is a great input, thank you indeed indeed next question, there was, yeah well, Red Hat pays me and we use only RPMs but if a community member comes and does some stuff, for example the guy next to you did some work for Arch Linux then we are not opposed, we want to build everything especially Arch especially Arch no David? sorry, you mean okay, we are out of time I can answer this after the talk okay, good good, okay, we are done, thank you