 So, I guess it's time. Let's get started. And thanks everyone for coming to the talk. Originally, I had a little bit more buzzwords inside, but then I figured everyone is already tired. This is the last day of the conference, so I replaced some of those with the pictures. Not cat pictures, but pictures. So, who am I? I'm not completely new to this audience. I was formerly a release engineer of Mirantis OpenStack Distribution in back times, and yeah, in times of OpenStack Metaca and Liberky and something like that. I'm currently a senior principal engineer in Red Hat, and I work primarily on continuous integration for Red Hat Enterprise Linux and for CentOS Stream. I was also a member of the Royal Nine Bootstrap teams. So, we were building the new major release of Royal, which has been just released last month ago. And I'm also a member of Fedora Console, so if you are interested in any of these topics, feel free to reach out, and I absolutely love to talk about those. And I also talk about CentOS Stream a lot, so there is one more recording which may be interesting for those who is involved in CentOS Stream and wants to know more about it. And yeah, let's get started and let's talk about Linux distributions and how we develop them. I mean, of course, when you work with Linux distributions, especially with Linux distributions with long-term support cycle, it is tempting to think about Linux distribution as like baseline which never changes and which you just put there and then you build all your fancy applications, services, and infrastructures where some interesting development happens and Linux distribution just sits there boring, nothing has to be done with it. But in fact, Linux distribution actually is being developed and if you're looking to build a project with a long-term, long life, you probably need to be interested in how Linux distribution life cycle works and how the process is organized around it. And when we look into Linux distribution development, the first thing we know about Linux distribution is that it is continuous and it was continuous before it became mainstream and it is continuous now and every update to Linux distribution is actually an atomic update of a package which gets delivered for the update process. So in majority of Linux distributions, we never start a new release from something from scratch. We are just continuously built the new version based on the state of the previous one with incremental steps updating packages one by one. And yeah, if the interest and feature of a Linux distribution which sets it apart from regular application, continuous integration is that Linux distribution has circular dependency on its own, on itself. Like we built tools to build the update of a Linux distribution, a part of that Linux distribution and we also need updates. This is a very interesting aspect of CI which brings a lot of complexity in the process but I'm not going to actually deep dive into this area in this talk. So it's just a note for generic audiences. And so of this pictures, I will do them a lot. This is representation of the continuous flow of updates in Fedora Linux as an example. So we have one continuous development branch which we call Fedora Rohide. It's not just branching the sources, it's also a binary artifacts which we built from the sources, these are actual binary RPM packages. And at a certain point we create snapshot of this repository and these sources and we create a branch and this branch becomes a stable release of Fedora. Now the interesting part here is like every Linux distribution is continuous in this way but there is a specific properties of Red Hat Enterprise Linux distribution from that point of view. So REL has major releases every several years and minor releases every several months. And the question here and like a REL in itself is not a complete continuous development project because every major release of REL actually starts by inheriting Fedora code instead of continuing the previous REL version code. So when we prepare REL 9, we don't take REL 8 code and build on top of it, we take Fedora code and build REL 9 from there. So the creation of a Red Hat Enterprise Linux looks can be expressed in this way. So we have a certain stable version of Fedora from which we take the state and then there is a complicated bootstrap magic, how I call it, which then leads to appearance of the Red Hat Enterprise Linux release and the CentOS Linux release which is a repository built of REL. And this kind of contradicts with the previous attempt to represent REL distribution as a continuous thing. And this is exactly the area which we try to address with CentOS team and with REL 9 change. We wanted to bridge this gap and to make it more clear how exactly Red Hat Enterprise Linux releases start and to make it more predictable and more understandable and more open as a process so that people can contribute to it, people can make plans about it, people can reason of what Red Hat Enterprise Linux will be in the next major version. This also creates possibility to, like bridging this gap would also create possibility to plan ahead and to request certain changes in REL while we're doing that. So to bridge this gap, we actually introduced two things. The CentOS team was just one of them but the first thing we added was Fedora ELN. And you may be not seeing it exactly the first moment but Fedora ELN is the additional line which I added to this Fedora Rohite development branch. So Fedora ELN is actually a rebuild of Fedora Rohite sources in the environment which resembles Red Hat Enterprise Linux. So we use compiler flags, we use options, we use macro which are natural for Red Hat Enterprise Linux but we take Fedora Rohite code and continuously build it into that environment. So this Fedora ELN creates a starting point for the Enterprise Linux development process. And then once we have the Fedora ELN project we added the CentOS stream which actually connects the dots here and the CentOS stream starts from a certain snapshot of a Fedora ELN at the moment of a branching of Fedora stable release. And then it continuously takes updates in which lead us to the actual Red Hat Enterprise Linux release. So with Red Hat Enterprise Linux 9 we introduced actually three different changes and unfortunately when news came about CentOS stream most of the conversations over the last year were focused on only one part of this change which is I marked by the number three here this is the change of CentOS stream and Red Hat Enterprise Linux relations post release. But I want to highlight in this talk that there were like two more things which we added and which may be more interesting for people who were not targeting CentOS or REL as just a platform to put your production workloads on but if you consider a platform to which you develop your own services or applications and you need to plan for a future developments of those these are the things which you might be want to be aware of and you might need to consider using them for your development purposes. So again, Fedora ELN is a subset of Fedora Rohite packages built into the Enterprise like build route. Currently it's a playground to propose and trust and try the most controversial changes in REL. So what we do for example right now we talk about bumping the CPU baseline for the entire distribution which means getting rid of support for older Intel CPUs and setting the expectations for the CPU baseline to have AVX2 instructions. I don't know what this exactly means but this is the conversation which we're having right now and it's not yet decided whether this will be applicable for the next version of REL whether it fits in Fedora but this is the kind of conversation which we're having in Fedora ELN right now and this is where you can already participate and provide feedback and say like three years for now are you going to have CPU with this Intel instruction supported? I do you want them to be supported? What's your interest in this and what you want REL to be with respect to this feature? And this is just an example. So Fedora ELN is the place where we can have the conversations like this. Fedora ELN also is a place where we choose which content will be in the next release. So Fedora Linux distribution has more than 20,000 packages. Enterprise Linux of course has a smaller footprint we just cannot support all to 20,000 packages on the same level so we need to choose a minimal subset which is good enough for running enterprise workloads. So currently for example, a Red Hat Enterprise Linux is 2000 slightly more than 2000 packages and not 20,000 as in Fedora. And the choice of these packages depends on the choices we do for Fedora ELN. So if you want a specific tool enabled in our next REL major release this is the place where you can come talk about it. So the second part which I marked here is it is also new thing and it also went almost completely unnoticed and this is what I could call pre-release development of REL. So of major release of REL. Previously when we prepared REL major release you would only see it once it is ready. When it's launched we had a beta program where there was some ways for partners to get early access to certain preview of a REL major release but it was harder and not as open as we wanted it to be. So now we've sent to a stream the entire flow from Fedora to Red Hat Enterprise Linux is done in the open. Sorry, I just need to keep my voice. Does anyone have any questions so far? Okay, oh good. So with obvious open development we can show you the exact development process and development decisions done by REL engineers when they adopting Fedora code for REL. This also provides you a place to integrate early with upcoming REL release. So you can start testing and adopting your own applications and services to the next major release much earlier than the launch of a release itself. As a direct benefit of this approach for example I can name the early start of EPEL repo. Like normally EPEL packages appear months after the launch of REL release because community needs time to prepare them. People need to figure out how to build the new environment and so on. This year this release EPEL repo has been opened before the launch of REL 9 so we were ready with EPEL packages on the day one of the REL lifecycle. So this moves a lot of this preparation work before the launch of REL rather than to the zero release of REL first year of REL which kind of extends your lifecycle support time because you don't need to wait a year before adopting a new version. You can jump right in and everything is going to be ready there. Of course like this thing isn't new so not everyone adopting it yet and we need to talk more about it. I hope that more services, products and partners would be operating in this mode. This means everyone will be ready on day one rather than wait for two years before jumping in. And yeah the third one is most beloved and most controversial topic of center stream change is what exactly happened between center stream and how it works now as opposed to how it worked before. So on this diagram you can see the minor releases of REL as they are released. When we create like major version of REL we start to create point releases and each point release of REL has a certain period of time when it gets updates within that minor release branch. CentOS Linux as it was previously built it tracks a certain minor release for some amount of time but the moment there's a new minor release available CentOS Linux would jump to that new minor release and get updates to that version. So if you were using CentOS Linux you would do this like steps in your updates flow you're supposed to do these steps and your updates flow not everyone does it of course but that was the expectation. Now the CentOS stream on this diagram is this continuous line which does just continuously go through the updates of the next minor release and there are no steps inside of it. You just keep going. You can create your own steps if you like that's the freedom of the CentOS stream approach that you can create your personal schedule of snapshots between which you want to move and you can create your own delivery process around this but generally CentOS stream just keeps going and the important highlight here is that again the controversial topic of EBI compatibility so that EBI compatibility covers all these updates. So CentOS stream cannot at a certain point run far away outside of the EBI compatibility promise and then return back to it for the around minor release. This is not how it works. Every update which we do every atomic update which we do when we do a real development actually goes within this policy guidelines and within the limits of the compatibility promise and as it is set for any real update. So these are like three major changes which we enabled in the real nine workflow and we are going to continue using the same approach for real 10. So since we're released real nine right now the FIDORA ELN is already looking for real 10. The CentOS stream 10 has not yet appeared but it will in the same way at a certain point once we branch the relevant FIDORA version and Red Hat Enterprise Linux will appear as the direct consequence of that. And yeah, the more detailed slide here is the describes the actual path of every update in the CentOS stream. So as we're doing the, as I shown on this slide CentOS stream is actually the development branch of real itself. It has not developed separately. It has not developed before real. It has not developed outside of real. The exact source code which you commit to the sources of the CentOS stream it is the code which you commit to the sources of real because these are the same sources. We literally have a service which synchronize the history of the CentOS stream this git to git history of real this git preserving the git of every commit. So how the contribution works in the CentOS stream way and where real is in this picture. So you start from merge request to CentOS stream GitLab where the sources are. This merge request runs goes through a series of checks which is now managed by Zoolci which was a really new development and the OpenStack RDO team actually helped a lot to set up Zoolci here. So all these merge requests pass the check and they get merged to the sources. And then we have a system which, and then the built is triggered in the build system which is called CentOS stream quadgy and the build goes in the gate tag on the system. The gate tag is like a label you put on a build or folder to which you put the build in. And once we built up here there we trigger the package checks which are like tests which use this binary RPM package in these scenarios. And then if these checks pass then we move the build to the pending tag and this is where this circle of dependency plays the role. The spending tag becomes the build environment for the next set of updates. So this is the environment which is used by next series of merge request artifacts to be built. And then there's a certain periodic job to create composes which is like repositories and images with snapshot of the current state of the distribution which then also goes through compose checks and then gets published to mirrors. And so as I said like CentOS stream is rel in the sense that sources of the CentOS stream and sources of rel development branch it's exactly the same thing. But we go also one step further the gating process of the CentOS stream the package level gating of the CentOS stream is goes in direct sync with gating process of a rel package. So this is a zoom in into that picture. We actually have a infrastructure which ensures that CentOS package update goes into the CentOS pending tag into gets promoted through the pipeline only in sync with rel package which gets promoted through the rel pipeline steps. So when we test the package we run test for rel package and only those tests for rel package show us the passing test result the CentOS package gets moved to the next step. So this quality assurance or happens in synchronously between CentOS stream and rel packages. And the things diverge yet the level of the compose checks. So this is where we say rel has its own compose mechanics and rel has its own nightly builds and the verification of them and rel has its own like release strategy with minor releases and things like that. Well CentOS project has just a publication schedule for this repository and images and applies its own compose checks. And we discussed this just like in the morning how we can make CentOS stream more suitable for as a development platform for layered projects and products. And this is where you have three possibilities to join and make sure that CentOS stream actually solves your tasks and your problems. So you can join as a third party CEI for merge request checks. You can join as a CEI for package level checks and you can join as a part of the CentOS community to be with your CEI for compose level checks. And we are actually open to any suggestions on which scenarios are worth being added into this environment. And for example, if OpenStack CEI uses CentOS stream as one of the base systems, it makes a lot of sense to put OpenStack sanity checks in one of those places, maybe on package level, maybe on compose level, maybe on all levels. This infrastructure is open and OpenStack is one use case, but we're open to any kind of applications and we're really looking into the third party CEI concept in the same level like OpenStack does. So for example, if your use case is very special, you have a hardware lab, which no one else in the world has and you want to ensure that CentOS stream works on that, you can contribute. We don't need to access to your hardware lab but you can set up your tests in a way that you contribute the test results and debug logs and then we may have a discussion about this but we can actually use this test to get the CentOS stream updates. So this is actually the conclusion slide of this talk. So again, what we created with the CentOS stream and we've rolled nine changes with Fedora ELN, we removed this bootstrap magic thing which was hidden inside the rail engineering and never been visible for any developer. So we created continuous path for all rail updates. So in CentOS stream, you can track the update path all the way to Fedora and see how it happened. We created it all in the open so the entire process, the entire development process is completely open and the way how redhead developers interact with red rail code is what I showed in this picture. So when rail engineers create any update to rail, they go through this process and you as a contributor can use all the same tools and go through the same process at the same time. And we created a flexibility which means like you still have your post-release rail the usual way but you also have this pre-release development part and you have the Fedora ELN part. So depending on the planning horizon you have, depending on the level of features you want to see implemented in the base platform, you can actually choose at which level you would like to join the development process or provide feedback on development process, provide the feedback on the way this whole thing is going. You can participate in Fedora ELN for ground breaking changes which change the way Linux is used in the world or you can watch the pre-release development through the CentOS stream and start integrating like three months before to ensure that your application actually works with the future release. And it is really important to understand the full value of this pre-release integration because if you only start adjusting your application to CentOS stream or Red Hand Enterprise Linux post-release, any feature you would request would be subject to the EBI compatibility promise. So we would not be able to change something which would break the EBI compatibility for that feature even if this feature is definitely something you must have for your application to function. It will be too late. So we would really encourage everyone to start integrating slightly earlier than that. Well, there is still possibility to update a certain library to introversion or roll back to a certain version which would be really important to you. And we would really want to know that earlier than after release. And so this is the content of the talk and just as a reminder, CentOS Stream and TEN and REL-TIEN has already been started and we are already planning future development in Fedora EL and right now, and everyone is invited. So thank you. Any questions? Okay, everyone understood everything? No questions? We're all excited, right? Again, thank you for coming. Next week there is the CentOS.Joy. It's a virtual event. If you have more questions, if you want to meet the community, come join, it's open, it's free. Thanks.