 Hi everyone, I'm Andrew Jettalo and today I'm going to be talking about Fedora CoreOS build tooling, so how we actually build the Fedora CoreOS artifacts. A little bit about me, I used to work for CoreOS in San Francisco and when I was there I worked mostly on container Linux and Ignition. As you know about a year ago Red Hat acquired CoreOS and so now I work for Red Hat and I now work on Fedora CoreOS, Ignition and still Container Linux. So today I'm going to be talking about a little bit of the background in terms of how we've built Fedora Tomic host and Container Linux since those are the two projects that came together to make Fedora CoreOS. We're going to talk about what goals we set out when we were talking about what we wanted out of our build system for Fedora CoreOS and then where we are now for actually implementing that and what we want to do next. So Container Linux is a Gentoo-based, it's actually a fork of Google Chromium OS and this means that we compile everything from source ourselves. There's no bin packages, binary packages like RPMs we can use and we have an SDK which has all of the tools we need to build Container Linux in it. This is basically just a tarball that you download and shrewd into. You've got a little bit of tooling around that to make it much easier but when you download that tarball it has everything you need. There aren't very many dependencies on the host so as long as you're running a reasonably recent kernel you can download this on any distro and build Container Linux. This is also self-hosting so every time we do a release we also build a new SDK that'll be used to build the next release and this means that we end up building our tools to build our tools to build our tools and even more than that because we go through a full Gentoo bootstrap process every time we build the SDK so we end up starting with a small subset of tools and that we then use to build themselves to verify that they actually can build themselves in the artifact self-hosting and then build the actual OS from that. This has a lot of advantages. Gentoo is really flexible. They have something called use flags and if you're not familiar with them they're a way of abstracting away compile time options so a lot of packages might have like optional x11 support and with Gentoo there's one common flag for saying I want or I don't want x11 support so for building a really minimal OS like Container Linux we can globally say we don't want any x11 support in any of our packages we don't want any Python or anything like that. Gentoo's packages are also very close to upstream. They try not to patch except for when they really need to to get something to work so we avoided a lot of like distrovisms with Container Linux and this SDK that we have means that we're not tied to any one distro for building it so like when I was working at CoreOS we had people running Gentoo, people running Fedora, people running Nixos, our releases themselves also use the SDK and so that was happening on Container Linux itself and this means that our release pipeline that we had is identical to the like developer build process. You know when you're building it on your laptop it's not going to be different than when it actually goes out to be built for release. Finally Gentoo makes it really easy to hack on like software on your own system. They have a method for just adding patches that you have locally when you go to build your software and then the SDK took it a step further and this is something we inherited from Google Chrome OS where you can tell it hey instead of going out to the internet and downloading a tarball of my source or getting it from Git I have it in this directory here build that instead. So if I'm working on a project like Ignition I can have the source cloned locally and then build from that directly and build the packages that way. This also has some disadvantages. You're going to be building everything from source every release it's really slow and we also don't have an army of packages to help us like Gentoo upstream is pretty good but there's a lot of combinations of packages and use flags and other configuration which makes it much harder for them to catch all the bugs and so a lot of times you end up having to fix like build failures or selves. This means there's a lot of maintenance for keeping container Linux running and it's made worse by the fact that we manually pull in packages one by one rather than taking the entire Gentoo tree at once and so when you have to update one package we may end up actually needing to update that one package in 20 of its dependencies all at once. We also cross-compiled instead of building everything natively and we ran into a lot of issues where packages would build differently when they're cross-compiling and when they're building natively and we introduced bugs like that. So this is just to give you an idea of what I mean by when we say we have slow builds. These are pretty fast by container Linux standards. This is after we've dropped ARM supports we don't need to build like the ARM backends for GCC and LLVN and all those tools. This is doing like a single build like not a whole bunch concurrently so yeah. The Fedora Automate Coastside is a lot more similar to what people who are familiar with Fedora would expect. It's built from Fedora RPMs. It's built on the Fedora infrastructure. The RPMs come from Koji like they do the same RPMs that Fedora uses. You get your images and your installers from Pungie and that under the hood is using Anaconda to like build the cloud images by basically installing it to a virtual disk. This has also some advantages and disadvantages. First of all there's already infrastructure for this that's great and people maintain it and you don't need to rebuild all of your packages every release. You already have like binary RPMs you can consume. On the other hand it makes it a lot harder to know that like what you're testing locally on your own like developer machine is exactly what's going to get released and because it doesn't have something where you can just build an RPM locally from a directory and pull that into your build process it makes testing a small change in a project that you're working on a lot harder when you're testing it integrated into the rest of the OS. So when we set out to see like what do we want out of the build process for Fedora Core OS we kind of wanted the best of both worlds to you know the largest degree possible. We really wanted to preserve the distro independence because people running containers are the target for Fedora Core OS. Part of the whole idea of containers is it doesn't matter like what operating system you're on and so we didn't want to have people developing Fedora Core OS need to be running Fedora. We also want to preserve that idea that your release process is just your developer process happening like in an official release pipeline on some Jenkins somewhere and we didn't want to duplicate all of the hard work that goes into packaging things for Fedora and similarly we didn't want to have to rebuild everything all the time and we want to preserve the idea that like it should be easy to hack on projects in Fedora Core OS as someone who works on ignition that's very tightly coupled to the OS it runs on and so you have to frequently test it integrate with the whole OS and so we want to make it really easy to hack on projects within Fedora Core OS and we want to do a much better job of supporting multiple architectures. This is something that Container Linux never really did well. I think at most ARM users we ever have as four so at this point it's probably a good idea to talk about what we're actually building and this will be a little bit of review if you were in the overview talk before this but Fedora Core OS is the release artifact as an image that is directly bootable like you can DD it to a disk and boot that. We have one standard partition layout and you can actually change that with ignition but that happens on first boot rather than at install time. This means that our installer is basically just DD and if you're installing a bare metal injecting an ignition config but other than that that's it. This is very similar to Container Linux and there we have the Core OS install script that's just a like 200 line bash script and about 80% of it is a GPG key. So and then we also want to have just one release image and this means one image that supports BIOS and UEFI turns out you can do them together. Container Linux has done this for years and we also want to have just one image for all the different platforms so like one image that works with GCE and AWS and on bare metal and you can't quite do that you have to have a little bit in there that's different just tell hey I'm on this platform you know admission should go fetch it's config from AWS's metadata or GCE's but other than that we want to be the same. So right now we have the Core OS assembler project and this is just like you can run it with Docker or Podman and all it does is compose it a OS tree from RPMs using RPM OS tree and then builds disk images out of that with Anaconda and we actually want to move away from using Anaconda for that but I'll get into that more later. So this is a rough diagram of what the build process looks like. We have a config that has your tree file for RPM OS tree which defines like what packages you want. You have kickstart file for Anaconda and again we're looking to remove that but more on that later and there's some configs for yum repos and other miscellaneous bits and then we also pull in all the Dora RPMs that we need and this all gets pulled in by a container which is built from the Core OS assembler repo and that's mostly just a Docker file and a bunch of scripts and then out of it we get disk images and an OS tree repo with a commit for the release we made and these build scripts are pretty simple like all the there's two main scripts you need to run to build an image it's just init which clones in the config you specified and downloads Anaconda and then when you go to build it uses RPM OS tree to compose an OS tree from the RPMs and then you create a disk image with Anaconda and there you have the little bit that goes in and injects the correct platform ID for whatever platform we're building for and that's about it but we also wanted to make it easy to work on the build process itself so with that we have a little script called bottle cap and it's called bottle cap because the old SDK for container Linux is called cork and so if you have your core OS assembler repo cloned locally and you've got some changes to the build scripts there you can run the bottle cap script with the developer mode option and that will bind in your scripts to the container and then install them into the container so we're looking forward where you know we might not necessarily have them all be scripts they might be compiled in the future so it'll actually build them reinstall them and then go about the build process as it would normally and this has been really useful I've been working on trying to remove Anaconda recently and being able to quickly iterate on this rather than having to rebuild a Docker container every time it's been really useful the other thing we wanted to avoid is needing to use privileged containers and if you're building a operating system this is somewhat hard. RPMOS tree uses bubble wrap for some isolation internally but you need to be running that in a privileged container because it can't create namespaces otherwise. As I've been trying to remove Anaconda my first thought was to do what container Linux does and use loop devices for virtual hard drives but they aren't namespaced and you need to bind them out in the DevFS for that and I also ran into an issue where on my local laptop I don't have a C Linux enabled in my kernel and if you do that then the image you build will not have the labels C Linux labels applied properly so when I finally got that working enough to boot everything failed because nothing was labeled properly. So to get around this there's a cool project called Superman and what Superman does is allows you to really quickly build a VM out of RPMs on your host which in this case is the container and then we can then boot that VM using Kimu with DevKVM mounted into container and RPMOS tree can use its bubble wrap we don't need loop devices anymore because we can just attach a disk to a VM and because we're using the Fedora kernel we've got a C Linux there and all those problems are solved. So looking forward we want to improve our build caching so right now when you make changes to the build scripts themselves then your build caching may be invalid it doesn't know that the scripts themselves are possibly a dependency. We want to integrate with a project called RPM distro get overlay which allows you to do that local development style so it'll allow you to build RPMs locally from source and give you a repo with that that you could then include in the build process. We want to do a better job supporting other architectures so we've got a couple bugs come in for both ARM64 and 64-bit power PC little endian where our core assembler container didn't quite work as well and still need to get the BIOS and UEFI working together and that goes into the next point of removing Anaconda so we don't need most of the functionality that Anaconda gives us like Anaconda is fantastic and does a million things but we actually need to do like four things so we need to partition the disk create file systems install the OS tree onto those file systems and injected error and edit the disk to save a platform it's on and Anaconda is overkill for this and actually makes it harder in some cases because Anaconda doesn't know how to install like two bootloaders at once and for doing BIOS and UEFI together you actually need to do that and if you're doing it just normally with grub install you can actually just run grub install twice one in UEFI mode and one in UEFI mode and one in BIOS mode and you'll get an image that works so yeah I've included a couple links here one is to the core assembler repo so that's for the project I've been talking about and then the other one is for the Fedora core OS tracker this is where we do most of our development on Fedora core OS itself this is where we do pretty much all of our planning and discussion so if you want to get involved I'd recommend to check these out any questions yes so the when we when we were talking about image the question was we want to replace kickstart and Anaconda in our build process because there's about four things that we need to do to build an image those four things are just partitioning creating file systems installing the OS tree and installing bootloaders so no so we're not gonna be using kickstart at all yeah so it's just gonna be we have a bash script basically and the actually the part where you tell it what platform if you're running on happens later we actually take our generic image and modify it using a guest fish it's just a kernel command line argument so we just go in and edit the probe config that's it other yes other plans to integrate all this with pungy I don't know of any at the moment but I'm not opposed to it because it's just in a container it's pretty easy to host anywhere you want right now we just have it running in Jenkins but I'm not super familiar with punchy so QA any other questions how big is that back script yeah it's not that large actually well yeah I can pull can't quite pull it up because my current one is horrifically broken so I've been working on trying to get the UEFI and bias together but it's not why it's like maybe two to three hundred lines and most of that's actually just argument handling so yeah yeah do we have any immediate needs that like we need help with so I've been working on trying to get by us in UEFI working together with Fedora CoroS in general like we have a million things like if you just go to the tracker repo there's hundreds of things that need to get done for it CoroS assembler we need to do a lot of like cleanup to because it's a project that's grown very organically question was what are the things that we're missing from Gen 2 by going with a Fedora base thing I probably say the biggest one is the use flags where we can separate out dependencies really cleanly and disable whole swabs of options on a mobile level especially for building something where we don't want to ship Python and we want to build as small an image as possible other questions the question was could these images be used with a USB stick to install on a laptop absolutely I've actually like it's basically the goal is to get to like where we are with container Linux where if you can get it on a disk and tell a computer to boot from that disk it will come up question was to what extent is this in production and where are images being built so that people can use them the first part is there are absolutely not production ready at all yet and the second question there's a CI pipeline remember where exactly it is but the actually the slide deck from the previous talk has that and if you want to try it out I highly encourage you actually just go to the core OS assemble a repo and build it it doesn't take that long because it's just doing a compose and like running anaconda right now you can build an image in like 10 15 minutes we're gonna be shipping the question was are we gonna be shipping images or just the OS tree and both because we will need the OS tree for updates but we are also shipping images with every release I just like we do with container Linux other question question was are the core OS releases going to be tied with the Fedora releases the answer to that is kind of so we're gonna have three different streams updates and they will be based on you know various Fedora releases but they will not include like the version number in the release streams and so if you're on a release channel at some point you will just flip from like Fedora 29 Fedora 30 but we're gonna have three release channels each one tracks farther ahead than the other one and so the idea is if you're running Fedora core OS in production you should be running a mix of all three channels so if we're gonna ship an update that's gonna break you you have a chance to see that update coming ahead of time have it take down like a node rather than all of your nodes and then you can submit a bug report and we can fix it before it actually comes and rolls out to the rest of your cluster question was the release cadence is going to be like that of atomic where we do releases every two weeks and yes