 My name is Lato Sibiru and today I'll be talking about building Fedora layered container images. So let me give you a quick background on me. So I'm a barely active Fedora package maintainer, you know, all those jokes about day job, you know, time, whatever. And I'm also a senior software engineer at Red Hat and I am currently working on the OpenShift build service and related container image build tooling. I'll get there soon. All right. So that said, let's go. So when I talk about layered images here, I'm talking about something a bit different from the Docker concept where each line on a Docker file is a layer, but I'm talking towards applications. So we have, like, let's say you have a layer with a base image. Let's say it is the Fedora base image. And then I want to ship an application. Let's say I want to ship Apache in here. Then what I'll do is that I'll build a container with the Fedora base image layer and then I'll add another layer on top of it, which will be the Apache one. And I'll ship that application. And then if me or someone else wants to ship Apache with some additional module installed, then all we'll have to do is to get that on their image with the Apache layer on it and add another layer on that with the application I want to ship. So this is what I'm talking about. It's actually quite similar concept of what it's done in Docker. But what we do in our tooling is that we squash some layers to make sure that each layer actually means something for the end user. So that's it. And but wait, why do we want to ship layered images through Fedora or end through a Fedora registry if we do have Docker Hub? Why not just shipping images through Docker Hub? Well, I'm not sure if you guys remember, but November last year, Docker Hub decided to reduce rate limits on image pools. Well, that definitely broke my page of CI. I'm not sure if it broke yours too, but well, you know, anyway. But then you could say, yeah, but why not just moving to another registry? Let's say, why not using Kueio, for instance, or any other registry? Well, wait, what about all the cool features that Fedora can give you and to the core images that you're shipping? For instance, let's say we were using the master builds in the registers as well as we do for RPMs. Well, then we would have a four less dependency updates every six months. How cool is that? Also, we have all the tooling set to provide automatic rebuilds. So let's say that there is a CVE found in some component in one of the images in the registry. Let's say it was in the Fedora base image. Let's say sudo had a CVE. Well, then all the images that are based on any image that had sudo are affected by the CVE, right? Well, no problem. We could rebuild the base image with all the images that had sudo, like in the base, and then all the childs, all the images that are on top of those can also be automatically rebuilt. Problem solved. I'm not sure how people can deal with this sort of issues in Docker Hub. And I think that's a huge issue, like lots of images with CVEs being shipped around because they are just not updated, right? All right. So for the ones of you who are not aware, we do have a registry in Fedora Project. It's registry.fedoraproject.org. Moreover, if you don't know, from Fedora 33, if you try to pull an image, let's say Podman Pool Fedora or Podman Pool HTTPD, the first place that your tooling will look for an image is in the Fedora registry. It's not in Docker I.O. From there, it will look to the Red Hat registry, then the CentOS registry. And only then, if it cannot find the image, it will look for it in Docker I.O. In raw hide and in X Fedora versions, we will also remove the CentOS registry from there and add Quay in the end after Docker I.O. So what can you ship in Fedora-leared images? You can ship RPM packages. You can ship configurations. You can ship documentation. You can ship modules from modularity. And you can also ship flatbacks. No, the Fedora registry is not a shortcut for people who don't want to package RPMs. You have to package your RPMs and then you install them in the images and then you ship them. It's just another meme to ship Fedora content, right? And how can I ship Fedora-leared images? Well, the process is quite similar to the one that we use for RPM packages. You're going to submit your Docker file and all your configuration files and documentation through a package review process. And after that's approved, you'll get a diskit repository and then you're going to push your code in there and then you're going to use Fed package to make a request to Koji. Then Koji will delegate the execution to OSPS and then OSPS will push the final built image to a candidate registry. After that, you can submit updates to Bodhi and then Bodhi will transfer that image from the candidate registry to the final registry, the registry.fedoraproject.io. Okay, so what is OSPS that I've been talking about? OSPS is the OpenShift build service. OSPS is a set of tools to build and to release layered container images. As I said, it's not a single program, it's like a set of them and you can find the code if you're two years in github.com slash container build system. And you can also read the documentation in OSPS.read.soleio if you're two years as well. Okay, so how does that work? So when you run Fed package from your diskit repository containing your container sources, Fed package will send a request to Koji and Koji will delegate that build to a special Koji builder that has this Koji container build plugin installed. This plugin will then make some calls to an OpenShift cluster and then this OpenShift build will start a custom build in one or many architectures and then it will build your image for all the architectures that were requested. For now in February, I think only the x86 cluster is activated. And after that, it will push all the images to the container registries. In this case, we're pushing to the candidate registry so Bodhi can pick it up later. Okay, so let's see, let's go hands on here. Let me know if you guys cannot see my terminal window, okay? Okay, so I package a container just for this session here. We did it this week. So this is a diskit repository for the container. If I were to get remote here, you see that it's in diskit and there's a special namespace in there in Baguio just for containers. So it's under the container namespace. And what we have here is a container, a Docker file, which is the most important file here. And then we may have a container.yaml file with some additional configurations for your container. And also it's nice to have a help.md in there as well. The readme file just came with the diskit. I just forgot to remove this one. We don't need this here since we have the documentation in help.md. Of course, this is a markdown file, but during the build process, OSBS will pick this file and we'll transform it into a main page in the root of your container image. I'll show you later if we have time. So let's take a look in the Docker file here. So we'll start with a from, we're starting from the base image, right? Which is the FedoraProject.org, Fedora Latest. So we're getting whatever is latest, whatever stack is latest in that registry. And then we have a set of labels and some of this you must add to your Docker file. For instance, this ConradHat component here is the name of the package in Koji, right? So this is where we'll find your builds in Koji. So this is the name of the package. Then you have the name and this is the name that is going to be pushed, the name of the image that's going to be pushed to the container registry. Then you have a version. I'll explain why this is set to zero later. And that's all the things you must add here. You see that we were talking about Koji, right? So we have an NVR. Here I just added a name and a version. That's because for the release, OSBS has a special code path where it will set the release for you automatically. Given that you never set your release here before, right? So it will start from one and then every time you rebuild the image, it will bump the release for you automatically. So you don't have to deal with it here. And then there's some other levels, but they're optional. And then I'm just like installing RPMs and preparing the container image, exposing ports, setting entry point, whatever just small container image, right? Other than this, we have the container.demo file. Here you can specify among other things that you can see in the OSBS documentation if you're curious, but additional tags for your image. So here I want to set a tag that's called latest. And I also add a tag called that conf.cz, right? All right. This is it. So if I want to build this image, I will just have to come here and fat package container build. And that's it. Just like an RPM build, but instead of just fat package build, container build, let's not wait for this because it may take a while. We can sort of go back in there later. I already built this image before anyway, so we can find it in the registry. So let's go for this, podman, pull regis3.fitora, turn the second guys. Oh, of course. Yeah, I didn't push an update your body. So it's candidate. There's definitely something out here. I do have like just second guys. I do have it here. So take a look at the tags. So reg is a two query container registries. It's in GitHub as well. If you're curious, we can talk about this later. Yeah, so let's try now podman, pull. I did type something wrong there. You guys probably know and I don't. So I won't keep looking at it. I can see the video later and see what was wrong with the URL, but that's it. Of course, I didn't specify a label. So it's pulling the latest. No worries because the latest is also the one that is DevConf. So it's all good. I should probably have downloaded this image before, but well, now we have to wait. All right. So while this happens here, I can talk about that version. Label in the image for the CodeGNVR and for the image as well. So why is that set to zero? This is interesting. This is like an ongoing discussion. We still have to learn how to deal with that in the field of infrastructure. But the thing is that if you paid attention to my Dockerfile, I wasn't specifying a version for the package that I installed. It means that every time I will build the image, it will pick whatever is released for the Fedora version I am building. So I would have to keep changing the version in the Dockerfile according to whatever is released in Fedora. And that's extremely error prone. So for now, people decided that the version level will be set to zero. There are some alternatives to deal with that, like trying to fetch that during build time, but it's not that simple. If you want to make part of the discussions, just take a look at the container, SIG in Fedora, which I'll talk about later. And anyway, that's it. The other thing is that as I said, one of the cool things here is that after OSBS finishes the build, it pushes, I'm not sure if you guys are familiar with how Docker and Manifest work, but we push Manifest lists to the registry so we can actually have multiple images for each tag, right? Meaning that we can have images built for several architectures. I'll probably use RAG soon to show you guys some of these. Let's just finish downloading this guy. I blame my bad internet connection. I can't all stop in there because otherwise you guys will see, like, anyway. Are there any questions in the question and answers? We may have time for one right now. So we have one question. Let's say I have two images where first one depends to the second. Is there any plans for chain rebuilds in federal land? Plans for what, sorry? Plans for chain rebuilds in federal land. Oh, I see. So that's the thing. We have the tooling in place, right? There are no plans because that's another problem I'll talk about later, but the content is currently quite inactive. So what we do need is people leading these efforts. We don't, and I think everybody that could be involved or that is barely involved, is also like quite busy with lots of stuff. So that's the thing. But yeah, that would be cool. That's something I would love to pursue. Okay, so we finished downloading it. So these are image, right? So that's right. I will get stuff from here because, you know, better this way. Okay, so I have my personal blog here and I will just run a little here with the image. You remember that there was a level called run, the image in the dog file. Usually we want to add the comments that the user are going to use to run the image there just to make their lives easier because, you know, so there you go. There's a server running here. Not going to go in there, but it did make the build for me, whatever. If you guys want to see the blog, you can bring me later. All right, so the image works. There's a cool thing I want to show. Let's see if this works. It's this guy here. Remember that I told you that we're shipping a homepage in the root of the image. If you can see this bit here, I'm trying to redirect the file to my machine. So there you go. So just a tutorial on how to use the image and the boards that's going to bind to whatever, whatever. Okay, as I said, all you need is a help.md file. You see that it's a markdown file, but it does get converted to a main page in the root of the image. All right, now, do we still have five minutes or should I stop for questions and answers now? We still have five minutes, so you can go ahead. Okay. We don't have any more questions in the Q&A so far. Okay, so I'll keep showing things. You see, if I take a look at the manifest list for that image, oh, this guy's cool. I love this too. Okay, so if we take a look at the manifest list for the image, we can see that it was only built for MD64 for x86, right? And then we could also take a look at the manifest of the image. You'll see that, where is that? Okay, it has only two layers here, you see? So the first layer is the Fedora base image, as I said, and then the second layer is the Ugo image. But as you can see in the Docker file, there are several lines here. It was supposed to have more layers, but it doesn't because we squashed the images in Fedora, right? What else? Let me see if I had anything else for the demo. Oh yeah, this is cool as well. If we take a look at the root directory, I'm not, oh, let's just run this one. There'll always be a built info file in there and a built info directory in there. And it always shows the Docker file of your image. It usually shows the Docker files for all the layers. We don't have the Docker file for the base image because the base image in Fedora specifically is not built in LPS due to some hardware constraints that we have in there, but that's it. If I show you the Docker file that's shipped in the image, you'll see something quite interesting as well. It's different from my Docker file. We perform some substitutions in there. For instance, we set the release level, as I said, and some other levels. We add that main page in the image and we make some substitutions in the front line just to make it easier to pull the image from the internal registry to interview it. All right, this is it. Let me go back here. We still have two minutes. So, let me talk about the container Sieg. This is the page for the issues of the container Sieg in Fedora. So if you want to interact, if you want to lead some efforts in there, if you want to ask questions, just go to this page and open that file and issue. The tooling for and the processes still need a lot of improvements. For instance, do we want, and can we make those automated version levels? So we have, for each application, we have the container labeled in the registry with the same level, with the same version as we have them in Fedora in the RPMs. How about these tags for the Koji builds? And also, for now, we are building the container images in the same Koji package as the packages that are going to be shipped, of the main package that's going to be shipped in the image. For instance, like if you go to the Google package in Koji now, you'll see that there's a mix of container image builds and RPM builds. And also the container review process still needs a lot of love. So we're still like trying to figure out how to do these things. So everybody's much welcome to help. Oh, you guys have any questions? We still have like a couple of minutes and here are some useful URLs for the guidelines of the container sig, for the container sig and the OSBS docs.