 So, welcome and thanks for joining. We'll be talking about upstreaming Fedora CoreOS. So my name is Clément Verna, I'm one of the engineering manager working with the CoreOS team. And today we'll talk about this with Ellen. So hi, I'm Ellen, I'm a product owner on the CoreOS team as well. So yeah, we'll get started. Yeah, so we're going to talk a little bit about what is Fedora CoreOS, how we're making it and testing it right now on future plans with where we're going with it. Yeah, so what is Fedora CoreOS? So we're now an official Fedora edition. So we became an official Fedora edition in Fedora 37, together with Workstation, IoT, Cloud and Server. So this means we're more integrated with the Fedora release process, including release blocker and go-no-go processes. So we focus on single-no clusters, in these cases, we're a successor of two container first OSs, which were CoreOS in container Linux and Fedora Atomic host. So we've incorporated ideas from both provisioning stack and cloud native expertise and Fedora foundation and upstack. So the philosophy of CoreOS comes from the merge between container Linux and Atomic host. So the idea was to take the best of each project, which resulted in the following key values. So the first of which was automatic update. So this was something that only Fedora CoreOS does in Fedora. The idea is for F cost users to not have to worry about the OS. These provisioning, so it makes it easier to have one or a thousand nodes that are the same and then a mutual infrastructure, which leverages the OS tree technology to make it easier for the update nodes. So our release model is different compared to other Fedora variants. So instead of major releases every six months, F cost releases every two weeks on three different systems. This differs from other Fedora editions. The model is linked to the automatic updates. So in order for user to use automatic updates, this has to be stable in order to not put pressure on the system. And this allows testing in advance of the stable stream to avoid issues. So we were supported on many platforms and a few of those are now integrated into our automated testing. So some of those are cloud or VR platforms and some of those are bare metal options. So some of the automatic testing happens during our release process on X86 or AR64. Wells have bills on S390X, but we don't have automated testing on those yet. So it's just some of the statistics on some of the nodes. So the red line you're seeing there is the transcendent line and the blue is the static line. So the transcendent are including systems that have been live for less than one week. The static systems that have been live for more than a week. The dip in the graph you're seeing there was due to an infrastructure issue that we had. The number of CoreOS instances are coming up and continuing to grow. So we're reaching over 40,000 instances out of which 30,000 of those are up for more than one week. So just some statistics on Fedora releases. So you can see the success of the automated updates. Majority of the ESCOF instances are running on the latest version of Fedora. So why can we see older versions of Fedora? So the OKD, so the community version of OpenShift is using FGOS. Updates are managed by the OKD admin and a specific version of OKD is tied to a major version of Fedora. For the architecture, some of these are very popular on X86 and AR-64. FGOS is the most popular Fedora variant on AR-64. Okay, so I'll hand you over to Clermont. Thanks. So I wanted to take a deep dive a little bit into how the Fedora CoreOS is released and the release engineering. Since we don't have like the same model, release model as other Fedora variants, we have a bit of a different process. So yeah, how does a package update runs in Fedora? So we'll see the beginning is very traditional. It all starts with a commit in this git where the package will update their software, code your build, body update, and it reaches the stable repose of Fedora. Then that's where the Fedora CoreOS magic source happens. In Fedora CoreOS, we have like this concept of log files. Talk a bit more about it. Then we do our Fedora CoreOS build, Fedora CoreOS test, and then release. So let's look at this a bit more visual way. So commit in this git, like I just took the example of OpenSSL. Packager will do this work, build it in CodeG, then push the update in body. As you can see, the update reaches the stable repose. So now it's available for every users in Fedora. The thing is that up to this point, this update has never seen a Fedora CoreOS system. We went through a lot of testing. People have been providing feedback on body, but this has not been seen by Fedora CoreOS. So that's a big point we want to try to work on is that really the Fedora CoreOS release cycle comes quite late in the packageer process. So to get this stability for Fedora CoreOS and to try to keep a user to use automated updates, we have this system of log files. And for each release of Fedora CoreOS, we're just going to fix the package versions. So for example, if we go back to the OpenSSL, we're going to say that in the next stable stream of Fedora CoreOS, we're going to ship with OpenSSL 3.084. And that's going to be the fixed version for two weeks. And then two weeks later, when you get a new automated update, you're going to get a new version. So that's really how it's done. Everything is stored in a Git repository on GitHub, so you can have a look at that. And you can really have all the versions of all the packages that are provided in Fedora CoreOS. So to do this, we have a robot file, a bot that's just that periodically. It will just look at the new builds that are pushed on the stable repository in Fedora and bump those, create the commit automatically. Once we have this, we have a scheduled test job that runs every day. And that's the first time, pretty much, that we're going to try to build a new version of Fedora CoreOS with the new content. So you can see it's a Jenkins pipeline, so we've got different stages. We fetch the new content, build the new OS3, signed OS3, and then we build the QMU image to start to do our testing. In Fedora CoreOS, most of the tests that we're doing, it's using Kola, which is just a test runner that we maintain and develop. And in the happy path cases, everything works. And sometimes it's just like we start to have tests that are failing. So at that case, there is quite a lot of investigation that needs to happen to understand, OK, with the new updates that we've consumed from Fedora stable, what has changed to cause the failure in the Fedora CoreOS test. So if we go back to the release process, you can see that the feedback loop comes quite late in the process. We really only start to test for AFCOS quite late. So if we have to make any change, or if an update in the package is actually breaking tests for Fedora CoreOS, we can only provide the feedback a long time after the package did the work. So what are our options in the release engineering? We have three options. So if there is a failure first, it's like the investigations. So we'll try to, we open a bug, say, OK, this test started to fail, start to look at it, try to do the diff, OK, which packages have changed, which package could be the cause of the failure in the test. If it's an upstream issue with the update, we can just then file a bug, bugzilla, or GitHub issue with the upstream. So this is a recent case where a change in system deep broke Fedora CoreOS. So we'll report the issue. It will get fixed and upstream and then flow down in Fedora. We have the possibility also with our test runner to snooze tests. So if we see that the test starts to fail, but it's not necessarily something very important that we don't consider is going to break the stability of the system, we can say, OK, let's snooze or ignore that test for the next week or so by the time to get it fixed. But we are not going to block the release. If it's a package or a bug that is a bit more important, for example, like the system D1, we have the ability to lock the package to a previous version. So if system D update is breaking Fedora CoreOS, say, OK, we haven't identified the bug there. We're waiting for upstream to fix it. But for the next release of Fedora CoreOS, we're going to release Fedora CoreOS with the previous version that was working of system D. So that's where the lock files that we have, the system of lock files is quite useful because we can then play with the versions that we release to users and keep that stability. We have also the ability of fast tracking an update. So if we know that a bug fix that we are waiting for is already in body, waiting for people to test it, and it has not reached the stable repose already, we can fast track that update and take it from body, which we can reference the body update and take the actual version and release this in Fedora CoreOS before it reaches Fedora state. So that's where also we have all those different streams for users. Users would be very familiar with the next testing and stable streams, but we have also development streams. So we have a row-eyed stream that really try to get us like an early feedback on the changes happening in row-eyed. So we try to test early and get to see what is going to break Fedora CoreOS in the future. And we have the next dev-all and testing dev-all, which is where a lot of those nightly builds are happening and where we run our CI and the test. I think we consider it a little bit like our main development bunch. It's where the development is happening. And for user, the next and testing streams are really where we encourage people to test and provide feedback and tell us in advance if something stopped working for their system or their workload. So what we would like to try to improve in the next Fedora releases, it's how we can try to upstream our testing. And it's all about feedback loop. I talked a bit about that. I quite like this thing. It's like when our feedback loop better, when they are shorter. So as we've seen previously, we were only testing quite late in all the process. And here the idea is to try to provide a very early feedback when the packageer is building his package in CoG. Trigger a Fedora CoreOS build, run a subset of our Fedora CoreOS test, and loop that feedback back on the body update. So the packageer would know when he creates the updates in body if his update has impacted the Fedora CoreOS build or not. The idea is really to try to reduce a lot of the effort that goes into investigating why the tests are failing late in the development cycle and have the feedback directly on the update. So instead of having to look at which package was the root cause of the issue or to do some deep and look at the differences, we would directly know in body. We would be able to see when system D or the latest kernel came, it actually broke that test for Fedora CoreOS. And there is a lot less investigation needed. So how would we be able to do that? Actually, it should be relatively easy. It's like a quote marks, but a lot of the infrastructure is already in place because we already have testing on updates. And it's about trying to plug our Fedora CoreOS release engineering testing to plug that into the rest of the testing infrastructure used by Fedora. So there is a system called ResultDB, which is really a database with all the results, with all the test results that are run on updates and on Compose. So for us, it would be interesting to look at updates. And pretty much when a Koji build is started or when a Koji build is successful, we would have a Fedora messages message that we would catch, trigger our Fedora CoreOS testing, and report the result of those tests into ResultsDB. And then body will be able to get the results through ResultDB and display that on the update. So this is already working for other tests. And you can see pretty much for each update, we already run quite a lot of tests for just other variants of Fedora. So instead of having, for example, update-based system logging, we could have something like update Fedora CoreOS like kernel tests or whatever. So the packageer would be able to see on their update if there is any failures or if there are any also required tests that failed and so on. So if you are not familiar with the tests that have a small star or aesthetics, our tests that the packageer marked as required for the update to be able to be pushed to stable. So that's like the gating. Yeah, Adam? Derek? OK. Those are passive-taster-wide policies. OK. So they are not marked by the packages, but they are like distro-wide policies where there are some gating required tests for updates. Thanks. So yeah, the big difference is that we would run all the Fedora CoreOS tests before the packages are committed in Fedora Repos. So if it breaks Fedora CoreOS, it would not reach stable. And it would let the packages get that early feedback and know how their update impacted Fedora CoreOS. And it avoids a lot of later work into communication, going back, saying, OK, this was already unstable, but it broke that build. Like, can we get it fixed, and so on. So it's really trying to save a lot of time for people working on packages. So how do we get packages that fail in CoreOS, the eye tests? So we enforce passing Fedora CoreOS tests before pushing updates. So it's critical to make RPM OS3 variants at first classes than in Fedora. This will benefit from RPM OS3-based variants, like CoreOS, IoT, Silver, Blue, et cetera. So future considerations, so building a common and minimal shared Fedora Core minimal OS3 image, either to use as a base for other variants to layer packages on top of, or could be used to create other OS3 commits. Thank you very much. And does anybody have any questions? Is it on? Yep, yep. So I have some more detailed things that I can talk to you after. But I'm curious, how long does your pipeline take? So it all depends how much testing we want to do. I think the idea for the original idea would be to run just a subset of core tests that we think are important. And it would probably, I don't know, be less than an hour, between 30 minutes to an hour. That's good. Thanks. Thank you for providing us insight into how Fedora CoreOS works. So since my background is providing feedback from downstream to upstream, I can see that's an improvement, what you proposed, to provide feedback when the build is done. But honestly, I don't think that's enough. I feel that in one or two years, we would be seeing presentation when you go even further. So I'm just thinking about if we could do something better, like for example, providing that feedback when the upstream list is created or even in pull requests or going that far. Yeah, I think that's definitely maybe a talk for the next two years. There is a saying that I like. It's like, how do you eat an elephant? It's like, one spoon at a time. So that's the small step forward to think we are in the situation where we really want to try to integrate a bit better. As I said, the infrastructure is there already. So this is maybe like a low effort, high impact, a step that we can take forward. But I definitely think that this shouldn't be the end of. If we put that in place, that shouldn't be the end of the effort and go even more, shortening even more the feedback loop. Definitely. So I have a question. So I don't maybe it's two in the weeds, but I feel free to say that I can talk to you guys after. But with the Podman desktop team, so we rely on Fcause for delivering Podman on Mac and Windows. And one of the issues we run into sometime with releases is Podman desktop has a release cycle. Podman does and Fcause does. So what ends up happening when Podman 4.6 came out, it seemed to just have missed by a day or two an Fcause release. So then we're waiting for it to go to Stable and Bodie for a week. And then the next Fcause release is another. And it's like we've made a release, and then our users will not get the latest Podman that release announcement went out for a month. And I'm just wondering, I'm intrigued by your new model of having an additional test build. And I'm wondering what are the implications there? Can that somehow help speeding this up, or is it orthogonal? I think what will help a lot is, so it's not necessarily tied to the feedback loop and doing our testing earlier. But it's more like having a minimal Fedora core OS3 image that could have a different lifecycle or have just followed the normal six month release of Fedora. And you get the updates just like normally. And this could be like a base for all OS3 systems, so Fedora IoT, Fedora core OS. It could become like what actually is consumed by the Podman desktop team. And yeah, I think that could be something to explore if it fits better. Lifecycles and release cadence is a tough topic, I think, to try to align everything. I think we have some ideas around trying to have that core set of packages that is useful in all the, and that could have a different lifecycle or consume the updates in a different way. I think that's a future consideration. Does that mean it will happen or? I think we definitely have the idea to do it. So it's still like the core OS group is still part of the change process and so on. So that would be something we have to propose to the community, to have discussions and see if there is an interest. But yeah, that's definitely something that has been before working very closely to go with mine in the last few years. Thanks for the presentation. I have two questions here. So first up, how would Fedora core be exactly different from Fedora minimal, except for the base that is going to be OS3 based? So would there be significant difference between what Fedora minimal today is and Fedora core would be? So when you mean Fedora minimal, you mean like the container image? First there will be the kernel. Since in the container image you don't have the kernel. But yeah, that could be a good start. It would differ from Fedora core OS in terms of like in Fedora core OS we have a lot of, so for example, we have Ponman, we have Docker, we have things that are very opinionated to run containers where in the Fedora core minimal OS3 we would not necessarily have this. I think we would look at what's the bare minimal to have an operating system that boots almost and then let users or different working group or editions customize that to their needs. So yeah, it could be like the package set that is in the container minimal image could be a good start for that. Okay, I have another question which is currently when Kola tests, they kind of create if you will from one particular stream to another particular stream like 38, 39, so on so forth. But with automatic updates you can go from 35 to 39 all the way, but in recent past that has failed. Like there was an error with migrating from 35 to 39. So is this going to fix it? I think for, so when we have like updates, like for example, as you mentioned, like from 35 to 38, so there is a mechanism that I didn't put in the slide, but we have like a barrier release. So we can force the system to go through a specific update, so we would have like, so you wouldn't jump directly from 35 to 39. We would first update the system from 35 maybe to 36 because it needs to happen, 36, 37, and so on. So we have that concept. In terms of having earlier feedback preventing that, I don't think that's like the, that would be like the end goal, but that might help. I think like, if we are able to catch like bugs earlier and prevent package to go to stables because they broke some of those updates, that would help. But I don't think that would be a catch hole. I think we could still have like cases where, where like, yeah, switching from a major to another major would bring some issues. All right, thank you. Anybody else have questions? A lot of questions. I walked in right during the count me thing which is relevant to my interests. Do you, and this connects to Podman Desktop, do you have a estimate of how many of the running CoreOS systems reporting in there are Podman Desktop? Nope. No, yeah, it's hard to, well, you know that. It's absolutely different. Way of knowing the Podman Desktop thing from the Podman Metro. No, because they just use like stock Fedora CoreOS image so they report as CoreOS. I wonder if we could try to do some, I'm just wondering now in which segment they sit, like are those systems that lives longer than one week or not? No idea. Yeah. I'm not so sure, we can, I think maybe like this we could try to get some kind of rough ideas but we don't necessarily have like a very... So if we move to that core minimal image, then could that be built as Podman machine with some kind of metadata that then you could track it? Any other questions? Give me a work out. Thanks very much. This question is a little tangentially related to the talk but there's a request for Silver Booth support on a new platform and I'm thinking if we started working on that today, it might be... On which platform? Silver Booth? It's specifically the Asahi platform but it could be any platform and that's just an example. So I've seen Colin do a couple of talks and Bootsy has the new way for OS tree based operating system. If you're porting some of this OS tree based stuff to a new platform and say you're thinking it might be ready for over 40 ish, should you bite the bullet and start looking into Bootsy or I know that's bleeding edge stuff. That's a big topic. So I don't know how many people are familiar with like the work that Colin Walters have been doing. There's two topics like there's Bootsy and OS tree native containers. I think what you're mentioning is more on like OS tree native containers which the rough idea is that you treat your operating system just as an OCI container. So you would have a base. So coming back to this minimal core OS tree definition. So this could be like a Fedora OS tree core image that we provide as a container image which is an OS tree system. It's just an OS tree. It's all the root file, the root FS, the root file system of your system. And you can just use that in a Docker file. So you would say from Fedora OS tree minimal and then run your command in your Docker file to customize that. So if you need a specific kernel for a specific platform, you could then like RPM OS tree replace the Fedora kernel with the kernel that you need for a specific platform and so on. So there's a lot of work happening now on this. It's still a bit fresh, but I think that's like the future of, that's definitely the future of how we want to release or at least Fedora core OS, but I think long-term all the OS tree variants. And Bootsy is related to this in a way where this is what allows you to install and deploy those container OS tree images. They are linked, but not necessarily like the same thing. So Bootsy would mean that you can start from any Fedora variant or editions. So you could even start from your Fedora workstation on the platform. Use Bootsy to convert that Fedora workstation into silver-blue for that platform. I think that's an overtalk. There is a lot of details to go into that. We can talk about it if you want after. Any other questions? Also, thanks everyone.