 I'm all down. Here we go. Welcome back. How's it going? I hope you enjoy the party. And I would like to remind you that there is a session at the end from today to the session. And I hope all of you get some prizes, as always. And now we have the first session here. We have Andrew and Guido here. And we will talk about middleware automation from the H2Card. Thank you. What an introduction for 9.30 in the morning on Sunday. Thank you. Good morning, everyone. Props to you for getting up early and actually getting here on time. This session today is middleware automation from the Edge to the Cloud. My name is Andrew Block. I'm a distinguished architect here at Red Hat. I am in the Global Services Office of Technology. What we do is we work with customers around the globe to develop common patterns and implement them at scale. I am one of the founders of the Ansible Middleware project that we'll talk about today. I am a big open source advocate. I have mainly these days in the cloud native space. I'm a maintainer on the Helm and Ores project and contribute to a security further called SIGSTORE, which attempts to provide tooling for securing your software supply chain. And I'm an author on a couple different books. Here, Guido. I'm a long time mercenary in banks, insurance, breaking monoliths, and working in automation. And then I became a redactor to make it freely available and to give back to the community. Thanks. OK. So as some of us have already discussed this morning, we're starting to see applications run in many, many different places. We've been running in the public cloud for a number of years. But another trend that was starting to come more recently is running applications at the Edge. And that is really because we're starting to see more demand at processing and managing assets closer to where they're being sourced. So look at when we got on the bus today, or the tram. You went ahead and you scanned your phone, maybe? You paid? That is another Edge device that is currently running out in the field. There may be some processing on there. There might be some other capabilities. That's just one example. Check out the Edge Store. You name it. We're seeing more and more demand where it's needed. So that's incredibly important because you need to manage it somehow. So one of the key tenants of applications running anywhere is a type of application called middleware. So middleware really is the connected tissue between many of your applications. That's OK. Let's go ahead. So middleware really is the connected tissue between many different components of enterprise architecture. It's going to be the plumbing. Our team loves to use the term plumbing because that's what we really see it as being. Without it, you're just going to have nothing because it provides so many different capabilities out there, and it really does power organizations throughout the world. So question for you. Not you, for them. Who here, OK, I have a little bit of survey time. Who here has built and deployed an application to a middleware server? Who here has managed a middleware server? Have you used any automation tools to do so? Or has it been manual when you were managing the middleware server or the application? And the question is, who here has never heard of middleware before walking in this door today? OK, we've got some new ones. Awesome. All right. I love being some new faces for that. OK, great. OK, next slide. So one thing that we've seen is that even though there's a lot of popularity in middleware, there's still most of it not being managed in an automated fashion. And that's a huge opportunity that our team sees is providing tooling to close that gap. Next slide. So what are some different ways to manage your middleware portfolio? Well, number one, it's got to run on a server usually. You could obviously manage the underlying libraries, dependencies that your middleware needs. You then have the lifecycle of the middleware server, everything from the installation, the upgrades, downgrading, you name it. The configuration of that server, so it's going to be all different unique properties for that specific environment, maybe some database URLs, credentials, messaging, use names, you name it. And finally, the application deployment and management layer on top of that. All the different ways that you can go ahead and automate your middleware portfolio. So the question is, what are the different ways to automate it? Well, we're going to start with scripts, bash, PowerShell, you name it. There are some underlying specific operating system tools. Sysprep, no, we're looking at the more unique ones for Windows or Sysprep. Linux has, oh, it's the one for Rails. I'm looking at it right now that helps. It stands up. I'm looking at it. Oh, whatever. The automation tool for the underlying for standing up, never mind. I'm going to forget it. That's our now. I'll remember, I promise. And then finally, an automation tool. That's a good one. No, not that one, I promise. And then finally, a specific tool design for automation. That's going to be something like a chef, a puppet. An Ansible. So, how many here has heard of Ansible? Please say yes. How many here has never heard of Ansible? OK, good. Yet you haven't? Good, OK, so we can go ahead and talk about what Ansible is. Ansible is a simple, powerful, and agentless automation tool for managing automation at scale. It's simple because it doesn't require a lot of special syntax that you need to know under the covers. The tasks themselves, as you define it, in a declarative fashion, are executed in order. And it's used by every team. Anyone from your developers or your operations team can use Ansible. It's powerful because it can do everything and make your breakfast, too. It could have made breakfast this morning. I don't know. Who knows what those machines are using. It can manage everything from application deployment, configuration management. It has a workflow engine as part of it. It can do network automation. And then finally, one of the most important parts of Ansible is it is agentless. We're looking at automation tools like Petfuppet and Chef. They are traditionally an agent-based solution. So this is agentless. All it needs to do is connect to an end system via a number of different connection plugins. And you can manage it as you need to see fit. And it uses out-of-the-box open SSH and winRM depending on what OS you're playing with. So why don't we go ahead and think about combining the power of the benefits that middleware provides plus the power of Ansible. It's that combination that amazing stuff starts to happen. And that's what our team has gone and designed is tooling around the ability to manage your middleware portfolio with Ansible. And that's what Ansible middleware is. It really is that set of tools that are specifically designed to manage Red Hat middleware and it's underlying runtime. It has support for multiple middleware products. It is configurable. This is one of the best parts about it is that every organization and every use case is different. There are a million, put a number of different ways that you can configure it, would you say? Oh, yes. That's true. Quite a large number. That's true. Yeah. And finally, this framework by Ansible middleware is built and run by a team that has not only engineered Red Hat run times in the past and current, but also those who have delivered it at customers globally. So you basically have a combination of the two of us. He builds it. I work with customers to roll it out at scale. So this is how easy it is. Simple, easy as pie. This is how easy it all takes really to deploy a JBoss enterprise application platform server. It's simple with 14 lines, Guido. Then we've got 14 lines of configuration. A little more yet to configure it in the back end, but still a lot easier than the thousands and thousands of lines of batch scripting you may have done in the past or currently are doing today. We use opinionated defaults, which provide a production-ready enterprise-grade deployment, even if you do not have the right configuration. Of course, it is opinionated, so it doesn't work one hundred percent of the times. And that's why you can override with the environment. So what are some of the benefits? Oh. It's all middleware. You have that consistent deployment to be able to manage and scale your applications as you see fit. Right once, deploy anywhere without manual intervention. We talked about being able to manage different systems at the cloud or the edge. You can go ahead and use the same baseline configuration and then have specific environments override that can make it so it's speed defined and used in these different environments. Maybe you want to specify your JBM with a smaller heap size in certain environments, or you have a different profile you want to go ahead and run out. Those can be overridden per environment. You can use the power of Ansible Automation Platform to then be able to then orchestrate at scale. We'll talk a little bit more about how you can integrate Ansible Automation Platform into this entire ecosystem. Has seamless integration for upgrades, downgrades, install, uninstall, configuration management, scaling and more, and then obviously you can save time by reducing the manual errors you have by using all their more traditional base approaches. So what kind of different technologies do we support? Well, think about all the different types of middleware that's out there. We have everything from managing a web server, to then manage systems that manage enterprise applications to messaging, caching, identity. All these different capabilities and technologies are available to be managed via Ansible middleware. So what does that translate to into projects in the upstream community? So for web, we have Tomcat. And for more enterprise applications, you have Wildfly, who is your application server. If you want to go more towards the caching set of things, you go ahead and use InfinisVan. Identity with Keeklook, one of the newest CNCF sandbox projects. Then finally, if you're looking on the messaging side, there are two flavors of messaging that we support, traditional JMS brokers with ActiveMQ and Apache Kafka for the streaming Kafka-based approach. But Agito and I are both work for Red Hat. We also support the enterprise version behind all these projects. So if you're looking on the Red Hat-supported side of things, you have everything from the website to the web server, J-Bus Enterprise Application Platform, Red Hat Single Sign-On, Red Hat Data Grid, and then finally, the two different flavors of Red Hat AMQ, broker, and strings. So this is basically a nice little technology project product chart. Just going to talk about all the different components that our team currently looks at managing and providing solutions for. So we do take a very community-first-driven approach. We're here at a community conference. So we want to hear from the community. We want to work with the community so we're not running in a silo. We have everything out there in the open. We have everything from our source code repositories, which allow you to not only look at the code that we have available for our automation solution but contribute. We have a consumable, so it's widely available and open for that one-touch, one-click installation. That's 11 lines of JINJA or of AMYAML. Yeah, you really don't need much more than the primitives that are provided by Ansible out of the box. There's no additional components stuck in Ansible Core. Then finally, we want to hear from the community, the roadmap. So we want to hear feature enhancements, bugs. So we want to gain those and get that information out there. So we really want to hear from you what's working well, what's not working well for the solutions that we're putting together. So how do you get started with Ansible Middleware? First, start with the source. Because how many of us here are developers or at least they were developers? How many of you probably would have gone to get, if I were to put a GitHub link out here, how many of you would have either gone to GitHub right after this talk or since we're all on our mobile devices right now, you might have actually gone to it right now. Raise your hand if you would have potentially done that. So start with the source. First, we have all of our Ansible Automation content collections, our documentation, our demos, labs, and feature requests all on GitHub in the GitHub organization and Ansible Dash Middleware. That is the first place I always tell people to look if you are more of a cooter. If you are more, and if you want to be able to consume the content, you can go to Ansible Galaxy. This is how you can consume Ansible content collections anywhere. It's a publicly hosted service, very much like Maven Central or PyPy or NPM. And I did not update the link at the bottom. It is not github.com. It's Ansible Middleware. It is the middleware underscore automation GitHub or Ansible Galaxy. That is my bad. Sorry. Nobody saw that. And then finally, I got that link so right. If you want to look at more of the documentation out there, we have it there too. Check out the docs at ansiblemiddleware.com. And that is, right now, it has basically just our documentation for our content collection. We are relaunching the new website in the next month or two, which is going to be totally refreshed, rebranded, and have a nice look and feel. So you will want to check it out as the project continues to evolve. And of course, try to demo. We have a number of different demos out there that you can look at leveraging. One from those that you build yourself to other entire workshops that you can also look at leveraging. If you were, well, since it was only a few number of people in the workshop yesterday, you could have gotten your hands on trying out Ansible Middleware yourself. We'll provide some links later on in the presentation where you can actually spin it up without doing much on your own laptop. You can use Instruct as our platform so you can do everything and learn about Ansible Middleware in your web browser. Our demo is providing only capabilities of just getting off the ground, but more complex use cases, multi-product integrations. We're going to show some of that today in our demo shortly, as well as some more advanced capabilities. And then finally, what is the roadmap? What's coming in the Ansible Middleware service? Number one, we want to improve the getting started experience. For many of you, this is your first experience in Ansible Middleware. We want to make sure that that initial experience is as crisp and seamless as possible. Finally, our next, we're looking at enhancing our documentation. We talked about looking at providing a new, refreshed way of our documentation and website. We want to then look more into the integration with open-to-virtualization. We see that being a nice target because it's kind of a mix between those who are really looking at adopting the new way of working on a container platform but still need to manage more traditional workloads, ones that you run in VMs. I work in many organizations around the globe and I'm sorry to say that there is still a world that still runs in VMs and will still always run in VMs for a very long time. How many of you know that most of the time you're using systems that are talking on mainframes still? Most of us took a train or plane today. It's a good chance that all those back-end ticketing systems are running on mainframes still or all these parts of it. I definitely know that Amadeus, that it runs most of the airlines out here in Europe, is running on mainframes still. You're a typical word against the area that's in a month. Is that the making of that one? Isn't that fun? It's other than. And finally, since we do focus very much on our Red Hat customers, we want to ensure that we have a great Red Hat customer experience for those who are looking at getting support at scale in an enterprise-supported way. So, looking at providing certified and supported content for all the different distribution channels that we look at providing. Okay, I'm turning this over to Guido. I've talked enough. Now, I know Guido. Oh, you're gonna go, okay, want me to do that? Okay. He's just gonna do the demo, I guess. Okay, we're gonna go ahead and we're gonna show Ansible and Red Hat Middleware in the wild with a very complex demo. We're gonna show how we can push technology to the edge by using a multi-product use case to make and define highly available environments. We're gonna go ahead and spin this up in multiple clouds, in the public cloud. We decided to not go ahead and, you know, attempt our luck and do cloud and edge of this demo because you never know what things are. This is just a multi-cloud demo. We're gonna use Ansible Automation Platform to be able to roll this out at scale. And most importantly, while, yes, we've got the initial installation and configuration, but after it's already running, how do you know it's still running? How do you know it's still maintainable? Well, guess what, it's fully observable as well. So we use a lot of monitoring and observability tools to show you how you can then not only stand that up, but use our tools to be able to manage and monitor itself. And then finally, every environment that we are deploying is gonna look at the following. So we consist of an intelligent load balancer based on JBoss, JBCS, what does that stand for? JBoss Core Services, wow, there we go, see? JBoss Core Services. Yeah, basically Apache. A JEE application server, a single sign-on server for providing identity for not only the environment, but also the application that we deployed as two cache instances for highly available single sign-on and two database instances. And it takes into account everything from things that our cloud provider will provide, everything from virtual networks, public and private subnets, state replication, failover, crossover, and multiple regions and peering and connectivity between those. All of those is fully automated. It's kind of scary when you think about it. That's one of the power bands that I could provide. Cool, so before we get into the demo, I'd like to share some resources because a lot of what we have will be cool to talk about after the demo, so I want to put this out here. If you want to learn more about Ansible Middleware, go to our GitHub organization, github.com, slash Ansible Middleware. Go to Ansible Galaxy, where you can look at our Ansible Content Collections. This is the correct link for that. So if you want to jot that down from the links from before, you can do that. We have the Ansible Middleware website, AnsibleMiddleware.com. Our Red Hat Certified Content Collections are on Automation Hub. And finally, if you want to get in touch with us, ansible-middleware-core-redhat.com. Gido, it's your time to shine. Time for the demo. Yeah, of course, an infrastructure like this requires a bit more than Ansible Collection for me. So in this case, we are using management resources in AWS and Azure. Both AWS and Azure provide Ansible Collections for generating resources, and we provide a repository of code, if you want, or YAML and GINZA that uses the third-party collection to make the resources, including this path, which is the important path that allows different clouds to give the abstraction for adhesion networks, which allows us, with the middleware collection, to generate clusters configuration. Like this caching service is composed by a cluster in a region, another cluster in another region of the same cloud, another cluster in a first region of the viewer, and another cluster in the fourth region. And then, all together, they form an additional cluster on top of that, the configuration of which is fully controlled by the variable you provide as parameters to the data collection, and the host names you use to run Ansible against it. And I'll show you quickly how it works. I don't have a point. I have a point. Okay. So this is an instance of Ansible Automation Platform Controller, which is a product of Redat which gives an abstraction wrapping up the Ansible command line. It's both an API and a web UI. In the first case, it allows to integrate Ansible as a process in a pipeline or in a workflow so it can be an integral part of every CICD you might have and can turn to be useful both for testing and also for staging and deploying production environments. And it also provides an UI which makes the first deployment more comfortable, in particular for people that are not so comfortable on the common line, and at the same time provides some additional features on top of Ansible workflows, namely the ability to auto synchronize the state of the project and jobs inside the UI with the backend SCM repository, which in turn allows to trust that repository as the proper single source of truth for the live deployment you will be executing by Ansible. It's an extremely important concept. If you repeatedly are able to run your Ansible deployment against an environment, Ansible is made so it reports changes only when the found configuration is not what you expect and what you expect is what you define in your Git repository. Conversely, it would report changes when a commit has been made in the Git repository. The third option is an unexpected change. It is something that you would look for to see what happened while Ansible reports so it is proactively fixing it, applying the change, thus turning the configuration to the expected one, meaning the code in the source code repository is the source of truth of what is live. It's an absolute beautiful, easy way for Ansible to solve a complex problem, which other automation tools they don't even take. Let's go to the data. So for an environment like that, we have a number of groups, which are collection of hosts. Any host can belong to multiple groups. Like one random is two machines could be in a region and be assigned a service role, like single sign-on or JBCS, be part of site one. And we have the hosts to be a long list. Belonging to those groups. These are the hosts against which we launch a job. We have really two jobs for this project. One creates all the infrastructure, all the cloud resources we have seen in the diagram earlier and the other one is meant to run the full deployment. Now the system is up. We are launching this job. We did a little bit of a boom show, so some of it is already running. We're just going ahead and enforcing configuration as part of this. Yeah. Going from zero to everything with a project like this is something that I won't lie to you, it would take 35 minutes, which is not long. No, because it spans up the entire cloud resources and then installs and configures everything. All we're doing right now is just enforcing state ensuring the deployments are as we see fit. One of the benefits of Ansible is that IDPotency being able to ensure drift configuration doesn't occur. And we take a lot of care that all of our collection are 100% important because reporting one change or 10 is the same. The good part is reporting zero changes when there are zero changes. And that's it. An important run against all those 24 hosts take around five minutes. So we have about six, seven minutes left right now. While it's only had cooking as I say, are there any questions that we can take in the meantime? Any questions that you may have had regarding any of the content that we've shared everything from what is Ansible middleware, what the tools that we provide, what capabilities that you can look at leveraging, whether it be from the edge to the cloud? Any questions? Answer all your questions. Or you just have a second. What are some of the answers? That's going. If you are aware of the Instruct tool, we have a workshop available that allows people to experiment with the Ansible collections. You can get your hands on this without even having to install anything on your laptop. So you get your own Ansible automation platform and you get everything stood up, resources. You basically get to learn how to do a demo similar to this but see how you can take the role of being not only an administrator of this tool, but also a developer. So you get to get your hands on playing in Visual Studio Code and committing and managing and modifying content. So you get a full hands-on experience of learning not only what Ansible middleware can provide, but some of the different ways you can look at configuring it. It is worth mentioning Validated Content. We are working with the Validated Content team so that a consumer of our middleware collection will be turned into another collection and provide a pre-baked, good scenario installation for instance services like single sign-on or caching. Yeah, so the Validated Content team under the Ansible business unit is looking at providing more validated and not just certified, I wouldn't say it's, I think it's certainly certified from their team, different demonstrations and use cases that has been known to have gone through the rigor of Red Hat testing and performance. And the good news is this is almost done in the four minutes, I think the way. Almost, almost. Insanient on the platform. We have got on EAP, right here, we see it's downloaded, good download in the patch. That's what we're trying to provide in EAP. Actually, it's already done, it's already done. And the patch is already there. It does some testing for the framework as a part of it. And it also calls the clean, we'll see what the clean was. We also, one of the cool things that our team is partnering with different teams in Red Hat to establish a new API for downloading content from Red Hat Customer Portal. Downloading content from middleware on times had been very challenging in the past. There's now a fully available API that you can, not only query for a real version, but also download content directly. So it's really cool. Now you need more. I wanted to mention that the reason for which it installs at least version 7.9 in EAP is because that version has a nice technology preview feature, which is the YAML configuration extension, which allow to provide declarative configuration to the bootstrap phase of JBoss. Turning JBoss into something configurable. Yeah, before JBoss configuration was a little more manual. You could script it up to an extent, but now, as you mentioned, the YAML configuration is actually you. You can do so in more of a declarative fashion, but aligns up very nicely with what our team fully in automation provides. So basically, you don't need to query the state runtime of JBoss to check its configuration. You can move that in the bootstrap phase. Of course, then you don't use the key much anymore. It's a very good thing because you can trust what is the configuration without checking. Please, the COI is good. Remember, this shouldn't get cut. The COI is good for one time. If we were to see the application to roll out, it did, great. Did you see there we are successful? Let's go and pop up the application. You know, can we do that? Let's show them something in the two minutes. Instable fast, is that it? Yes, it is. It's a simple address book, all integrated with a single sign-on when we establish it via Ansible Automation. And all we need to do is use Flange User. We have a bit of a joke. Remember, Ansible Middleware is helping address the plumbing of your middleware portfolio. So we have that. It then goes ahead and validates the user against it. The automation stood up the key call extension for JBoss EAP so you can now integrate single sign-on into your applications and we are able to manage that. This is once again, highly available, highly scalable. So we decided to kick off one of our regions which we've done in the past. It still works. Let's go through it, because we have these two minutes. So what we did, opening that page, is doing a query to the Route 53 DNS of AWS, being latency checked to see which region is the closest to us. The DNS forwarded us to the closer geographically HTTPD which reverse proxy our request to EAP which has a war file deployed in it which is configured for single sign-on. So we were redirected in the browser to single sign-on which has a database down here in this cluster picks our user, validates our authentication. We are redirected back to EAP and we see the HTML page. It's the HTML page. For a simple HTML page, there's a lot that we're going to put out front of the covers and that just all automated through the power of Ansible Middleware. We thank you all for joining us this morning. I know it's always hard getting up for the first session after the party but thank you very much, I appreciate it. And we're here to answer any questions that you have. Thanks everyone. To mention some community and release management stuff in the beginning and then talk a bit about various features and I tried to build some coherent story to this middle part but actually it's like a fully connected graph that everything matters for something else. So we keep producing releases. It's not a steady pace. We try to make releases every two or three months and we fail at this. Every once in a while, when I was preparing those slides we were still planning to make a release in June, July but we still haven't produced RC-1 for version 254 and we already had 2,500 patches and usually we merge another 1,000 to 2,000 before between the time when we tried preparing for RC-1 and the final release so it will be a huge release and I don't know. I still hope for an RC-1 in early July. The number of contributors varies but it's not going down visibly. And interesting development is that we have more and more stable releases. So we release for example 253 and then backport some patches 253.1, 0.2. We are currently at 253.5 and we are currently at I think 251.16 or something like that. So it's usually hundreds of patches backported to those stable releases and more and more distros are using those point releases to build their packages and there's quite a bit of demand for them. If we keep up current pace for this year we'll have 44 point releases between various branches and if you have some patch that you think should land in a stable release you can either market as needs stable backport in the upstream GitHub repository or you can file a pull request against the stable, system distable repo with a backport. Just make sure to use git commit minus x to get the hash of the original patch or sometimes it's enough to make a comment if you cannot do the other things in a pull request. And we have well about 2000 open issues and it is growing but in a manageable way. We actually have, I think we close about 80% of issues that are opened so this is just the diff. And there's a split half and half between bugs and RFEs. We would like to make this number go down but it's putting tough. But at least it's not bad news. In Fedora we made an effort over the last year. So those numbers are for, I mean the growth numbers are for a period of approximately last year. In Fedora we made an effort to reduce the number of open bugs and in particular David Targon and I have been working on going for the bugs and closing some stuff. And I think like this reduction of 50% it's pretty nice. We don't see that often. So and now let me talk about well features and being removed first and add it later. So this has been going on for the last 10 years approximately but there are those two related ideas that you have something called unmerged user. I mean the naming is terrible, right? That you have separate characters of bean and user bean and leave and user leave and so on for every other possible directory and you look in both places. And if you have that, you have the other thing called split user where you delay the mounting of the second half until late in boot. And most distributions have stopped doing that. Well, so Fedora did that maybe 12 years ago. Other distributions maybe a bit later but the point is that we have been maintaining this split, the unmerged user thing in system decode all the time. And it's, well it's, if you don't have this then this doesn't make any sense because you have parallel directories but you always use both. So this and I mean both things are going away. A patch is ready in system. The upstream just removes all this code. We will probably measure it a bit later after 254 has been released just to make the release of 254 faster. And I think that it's going away is support for C Groups V1. This is a more complex problem but the way that system deworks is that you have, you specify settings in the language of C Group V2 and system you will translate the settings to the, as much as possible for V1 if you're using V1 but this translation is not always possible in any meaningful way. And V1 is not hierarchical so you cannot do unprivileged delegation because if you delegate stuff on V1 the delegated can take more than the parent has so it's not very useful. And some features are missing on V1 so we want to get rid of this and simplify our code base quite a bit so sometimes next year. And this is a bit, so if you are using Fedora like there was a time where this warning popped up in various places so system de tools are being unhappy when they are called in a system without slash proc mounted. And the code continues but it doesn't like the situation and why are we making this warning? So we need access to proc self-fd sim links to work with file system descriptors and without slash proc we don't have this kernel API and we cannot do various things as file descriptors that we would like to do. So okay and now about some positive stuff. So we had this issue that users were reporting that when they use user units they specify some settings and those settings have no effect. And this is because system de in general works this way that if you have settings for system units and those settings cannot be applied because the system doesn't have the right architecture or it doesn't have or something was compiled without the right capabilities, settings are ignored. So I mean you can for example specify both as a Linux and app armor policy for a unit and the one that can be applied on a given system or maybe none of them would be applied the other one would be just ignored. And this meant that you could specify some settings like protect home for a system unit and it would work you would specify for a user unit and it would be silently ignored because the user manager does not have enough privileges to apply the setting. It could have the privileges if it used a username space for the unit but this wasn't on and you have to apply this explicitly. So this is something of a compact break but we are enabling it because there were many reports of people being negatively surprised and we'll just enable private users in many more cases now so there will be more sandboxing for user units. And in general, the number of various sandboxing options is growing all the time. I mean it would be really a separate talk about them. There's a very nice tool called Analyze Security. Actually there's two, there's Analyze Security that will give you hints about security settings and there is system de-analyze verify which will check the unit file for correctness and give warnings if something is wrong. They are both useful for hints and for checking. A small thing but useful is open file setting. So quite often units would use a shell just to redirect some file descriptor to some file from some file. This can be done now natively by the manager and of course the advantages that things are simplified and also that the unit can run sometimes with less privileges because it can get access to a file as a file descriptor and not be able to open the file otherwise. We have a new unit type. So type equals notify is an old thing. So it's a, when units are started, the unit system de-likes to make this operation synchronous so it wants to know when the unit is actually ready. And in type equals notify, the unit runs and sends a notification message using the SD notify protocol when it's ready. And for reloads, we usually use, I mean the easiest option for reload is just to let system to send a signal to the unit. But this is asynchronous, the signal by definition is without any communication in the other way. So with type equals notify reload, the unit is supposed to send a SD notify notification after a reload has been finished. So we get a convenient implementation of reloads with symphony city, so readiness notification. Another feature, it gets a separate slide because it's, I think it's pretty nice, is soft reboot. So it's a, you say, it's a new way to restart the machine that brings down all the user space and mounts things. And instead of actually rebooting, it really starts the execs system D and brings up the user space again. So we have like a normal reboot which goes through the bootloader and the new kernel. We have Kexec, which just skips the bootloader and goes directly to a new kernel. And we have now we have soft reboot which skips both of those steps and goes directly to a new system D. Another feature, this one even gets an icon is the IOCOS controller. So actually this is a kernel feature. Mostly, and this is quite common in system D stuff that we are just providing a thin layer around the kernel. So when you have a block device, you get some number that it has some bandwidth. I don't know if I have the megabytes per second read speed or write speed. And we know this is not true in general because the actual bandwidth depends on what the device is doing and how much writing you are doing at the same time. So initially you specify some blocks to write and it is very quick because it actually goes to some internal buffer. And then things slow down quite a bit and depending on whether you are reading at the same time, the write speed will vary and so on. So the idea is to do fairly extensive benchmarking of a specific model of a drive. Come up with a simplified model that describes how the drive behaves under various conditions. And then you can set a meaningful policy to divide the bandwidth between services and groups and so on. And where system D comes in is that this is implemented in the kernel but we need to provide the policy. And system D now has a UDEV rules to figure out what policy to set for a drive based on the model, firmware revision and so on. And in the hardware database, we are starting to grow a set of rules for different drive models. So if you have a drive model that is missing, you can do the benchmarking and submit the pull request and then it will be applied to all the drives in whatever people are using this. So this was actually contributed by Facebook folks. So, and now some sub features and here sub stands for subterranean. And those are large features, they're just not very visible. And so this is another kernel thing. When we use process numbers, PIDs to refer to processes, it is well known that a process can die, we wait a bit and another process gets born and it gets the same number. So PIDs are not a reliable way to refer to processes. And the kernel has gained APIs to refer to processing using file descriptors, PIDFDs. So this ambiguity goes away and various system D tools have been converted to use PIDFDs internally. And also stuff like system D and the D-Bus APIs are getting extended with a second set of calls that allow PIDFDs to be used instead of PIDs. So this is quite a bit of work but it's generally not visible if it works. And kind of in a similar vein, internally we are converting a lot of our code to use file descriptors to refer to iNodes instead of the path name. The most obvious thing is that this allows, removes the possibility of a time of check, time of use race between, I don't know, checking the file and executing the file. But also it makes it easier to write code which operates on sub-trees of the system hierarchy. So change routes and also disk images that you mount temporarily somewhere and then you do out in operation on this image. And I also wanted to mention kernel install, it was rarely seen, it used to be a bar script and this meant that, well first of all, all logic had to be implemented, like for example to find where the ESP is which is actually a very complex game of guessing had to be duplicated into, and this was very annoying. So in the end Iwata Nabe rewrote it and it also means that we can use FDs in kernel install and kernel install will now operate very nicely on, you can say kernel install dash dash image equals image name and then do an installation inside of a disk image. I mean I think not yet but soon. And so this brings me to disk images. So the idea has been around for a while but the name is new, so discoverable disk image is an image or a disk image that, this image that or an actual disk that follows the discoverable partition specification. So it has a GPT table and the GPT table, the role that the different partitions should be used for is specified by their partition type identifier and there's, you don't need other configuration. And system detysect has been around for a while but it has grown new capabilities. So for example it can give an image and it can recursively mount the image on some mount point. So the mounting is done in the same way that if you booted the image and you would auto discover that I don't know like slash var, a specific partition is slash var and another partition is slash home base on the DPS. People talk about supply and chain issues, so a bit in this topic, the M3 command makes like a recursive report of the contents of an image. And this is a system detysect example. I try to make it fit on a slide so this is like after surgery. So it opens the image and extracts the OS3 file so it knows what the image thinks about itself and prints some metadata and partition list but it also has this idea of using the same image for different things so this is where the DDI power is. So this particular image can be booted in UFI so as a real system or in QEMO, it can be used as a container but it's not suitable for the other things and so we have portable services. So a service that is like a normal system service but comes with its own file system. An int ID and a DDI can also be an extension for each of those things above. So we have the same format but depending on what partitions are inside and some metadata is used in different ways and this area in general has been, there have been a lot of development in system diff with capabilities for this, in particular how to apply the extensions and various places and how to check that the extensions are signed properly. I have a second talk in the afternoon where I talk more about this because it's just not enough time here but briefly, so we have extensions that allow us to add stuff to an image. For example, we have an immutable image that is signed, we boot it and we want to extend it and we have those extensions that can also be immutable and signed but there are also other mechanisms so a lot of work has been going into credentials. So credentials are the system, the idea that you have a blob of data. For example, it could be some configuration snippet or a certificate file and the manager will take this blob and when it is starting, it's stored somewhere and when it is starting a service, the service specifies that it wants a specific credential, the manager finds this credential and creates a file before the service is started and then the service can load the file. So this doesn't sound useful but the thing is that this storage can vary a lot so it can be a file on disk and the credential can be also encrypted and then system view will decrypt the credential before passing it to the service. It can come from a pipe or a socket so this means that credentials can be generated dynamically when requested and they can be stored in QMOS and BIOS fields and some other similar stuff which means that you can pass a credential to a virtual machine. They can be specially on the kernel command line, the boot loader will load them from, I mean as the boot will load them from the ESP partition and credentials are hierarchical in the sense that we can have a situation where we have a credential on disk, this is passed to the virtual machine manager and then this passes it to the virtual machine and the virtual machine passes it to, well, it's sending the virtual machine loads it and passes it to a service and so on and so on, right? And an example of how we make use of this is there's this specific credential called VMM Notify Socket and we put the VM, we pass the credential to the Tocumel and inside of the machine system debuts and sees that it has a credential, I mean it looks for a credential by this name and sends notification to this socket. So for example, it sends ready equals one when it has finished booting. So this allows us to cross the boundary between the host and the machine and this is done in a fairly nice way, there is no network involved, it's a fairly generic mechanism and so also it can send an exit status notification so basically you can have the situation where you make the virtual machine and the container behave in the same way and you can specify an exit status and have the machine fail. For example, very nice for unit tests where you do some tests in a virtual machine or in a container or both and you want to have this uniform. Another tool that has seen significant work is system demesure. So the idea is that you build a new kernel and before you boot the kernel, you build a kernel and unit ID and figure out some command line options and stuff like that and with all that before you boot it, you calculate what PCR values will be, what the PCR values will be after you have booted this combination of things and this means that you can, well you can predict those numbers and it means that you can sign policies for them. So this is all geared towards pre-calculating PCR values and signing policies and encrypting secrets in a way that they're bound to certificates, not specific PCR values. And a related topic is that we have a new idea of boot phase paths. So system D will write those strings at various points during the boot sequence which means that the PCR values change and define points during the boot time and this means that for example, you can have a key for the root-volume looks, for the root-looks-lux-volume that can be bound to the TPM and can only be decrypted in the INDRD because after you have exited from the INDRD, the PCRs change and you cannot access the same secret anymore. And so on. And there's also a number of stuff that is services that write information about the machine into various PCRs so that we can build more useful policies. So for example, like the machine ID and the information about disks that are mounted in various places. And again, this is about building PCR policies that actually are useful. And something that I have been working on is a helper to create unified kernel images. So this is like UQFI uses system D measure. So you have a kernel in INDRD and some command line settings and now you call system D measure to figure out what PCR values will be. You build a PCR policy, all of this, you sign it, you put it also into the unified kernel image, possibly multiple of those policies and then you sign this whole combined thing for secure boot with yet another key. So it's quite a bit of messiness and UQFI makes this easier. It has been rewritten to, now it doesn't, I mean system D measure doesn't require root privileges because it doesn't access the TPM anymore. And so UQFI also doesn't require root privileges which is just nicer and faster. And another tool that has seen a lot of work, is system D report. So you specify a set of partitions that you expect to see on the machine and when the report is executed, it will match those definitions to the partitions that are on disk and create any that are missing and maybe, for example, grow the ones that are too small and so on. And if everything matches, then it's just an idempotent operation. And so it's nice because it works atomically. So it first opens a device without having a partition in the partition table, goes to a specific offset, writes the contents. And after this has been done since the disk and then creates a partition entry at the beginning of the disk. So the partition appears with contents, I mean with a file system and files in the file system if you specify so already at once. And we used to do it in this way that we would use a loopback device to mount the temporary partition somewhere. But now the code has been reworked to use file system tools to write the file system contents including files directly at a specific offset. And this doesn't require root privileges so you can build file system images inside of a container and also as an unprivileged user. And it's also faster, it's nice. And there has been also quite a bit of work on the system boot loader and the system disk tab. So system deboot is the boot loader for UFI and system disk tab is the thing that is prepended to the kernel to create a unified kernel image. And we used to have a dependency on GNU UFI, it was quite a lot of code and it was annoying to have this. We wrote a bunch of stuff to get rid of the dependency so they are now smaller and we also like our code better. You can use as the boot to for direct kernel boots under QEMU. So you call QEMU dash dash kernel as the boot and then as the boot works as a kernel and it will actually load another kernel from inside of the image. This has been around for a while but it's kind of becoming more interesting with the whole work on unified kernel images that as the boot will do enrollment of secure boot keys if the machine is booted in setup mode. And I also wanted to mention that there's some improvements to pass the random seed to the kernel. So the kernel gets started already has the random pool populated so we don't need to delay waiting for randomness. This used to be a problem source of delays in the past. And Anaconda, the installer used for Fedora and RL systems is getting support for system debut. This pool request has been merged. It's not complete yet but hopefully we will be able to install systems using as the boot fairly soon. And I wanted to mention that we don't bite and like there's issues and stuff to work on and we merge pool requests quite often and I will be happy to see more contributors. And yeah, so four minutes for questions. Sorry, I didn't catch this. So the question was whether software to boot will work with OS3 systems. I don't think that there's any reason why not. I mean, what it does, after it has brought the user space down it calls the equivalent of like a switch or operation and you specify a new system binary and potentially some options. So we can figure out a way to boot OS3 the same one or a different one. Yes, the same kernel, yes. And actually processes can survive. And you can mark processes not to be killed. So it's not a complete replacement. Okay, so I'm so tired. I haven't gotten any sleep. Hello, let's welcome Eric Gustafsson, my dear colleague. This talk is going to be about converters with these distributions. The floor is yours. Perfect timing. Why is this not working now? Because of course the computer has to lock itself just as I'm about to start. But yeah, I can start a bit. I'm Eric, I'm the team lead. For now, the team lead of Converturell which is managing a protocol Converturell. It's a team of seven, something like that. And I'm, yeah, God, this is frustrating. There we go, all right. So I'm going to be talking about how you can convert a system, right? And it's going to be primarily about CentOS. So most of the systems that are similar to CentOS are based off of REL. So you have Alma, you have Rocky, you have Oracle Linux, CentOS as well. CentOS Stream is actually upstream so it doesn't really count. But that's what REL is based off of. So that's why it makes it quite easy to convert. So you have relatively simple process, everything's basically the same. But when it comes to something like Debian, it's a lot more difficult because then you have to deal with a completely different like architecture and like ground, right? Because it's not similar in that way. But converting is still difficult in a way. Like the actual process is easy but there's always going to be some like edge cases and other difficulties. So there's a few tools that people develop over time. I don't think Converturell was the first one. I think that was actually CentOS to Oracle Linux if I'm not correct, but yeah. There's a few tools, there's Elevate which allows you to upgrade systems. So like from CentOS going to Alma and some of that. They also have an Alma Linux deploy which is actually converting from CentOS or CentOS Stream to over to Alma. For Oracle there's CentOS to Oracle and they're a bit simpler. They have Converturell which is a relatively big product which converts you to RHEL and then for Rocky you have Migrate to Rocky. There's a few tools available anyway. So I'm from the North Pole, I come from Sweden. I'm currently making some kombucha, I'm trying to learn piano. I tried to do some mountain biking when I was living here in Brno but now I'm just, it's very flat where I am so I have to go quite a bit to get somewhere. I've been a developer for five years. Started like a front end kind of environment where I was mostly working by myself but I learned a lot and then I think since two, three years ago I started with Python and actually developing Converturell. Yeah and being a team leader it's something I'm gonna transition over because I can't really be a manager as well as being a team leader but yeah. And I'm also a crazy cat lady because I have some cats, right? There's Isolder on the left side and then you have Lisa on the right. Both men and coons, they're gonna be quite big. Oh and Mimi's a lose as well. So what are the differences between the tools? Well, first of converting like a system why would you even want to do that, right? Like it's, you can just redeploy and in most cases I would say redeploy is probably the better option because when you redeploy you get after a system you're not gonna have any like weird things that you didn't think of before. Like when you convert a system like you're gonna have something even if you don't know what you're gonna have something that is steal from the old systems. If you convert from Centus to Alma there's gonna be some Centus parts, right? But you're not gonna have any redeployments when you're converting place there's gonna be all the configurations and files and then when you have converted over you know what you haven't converted with the packages at least. So there's some security there and I would say most of the time like it still makes sense to redeploy but like it's if you can't redeploy then converting in place is the way to go. And for redeployments is probably the better option, right? You have better like safety margin if you can like everything you can make sure everything is working you have stage migration so like you can keep the old systems running when you redeploying the new ones and then after a while you shut it off, right? And that's usually what bigger companies do, right? I think Facebook did it where they went from REL to Centoest to Centoestream they also had like gradual staging migrations. So the differences, I don't know maybe it's gonna be a bit heathen but yeah I think converter REL folks is a bit more on like stability and testing so a lot of the tools have a lot of lines of code but I think the converter REL is probably the biggest project out there so far. If we exclude elevate which is kind of leaf which is really upgrades but I'll get to that soon. And then elevate which is a play on enterprise Linux which is why they have EL capitalized. They kind of treat Centoest 7 as like an upgrade path so they disregard that it centers all together and they say yeah well it's similar so we can just upgrade it to Alma to Rocky to Centoestream anything like that. And then Alma Linux deploy CentOS to Oracle and Migrate to Rocky they're basically just a bash script so they're very, they're automating the tedious parts but there's gonna be some manual steps to fix things. So the general execution flow of like converting a system is kind of easy, right? I mentioned that before. You just have to clear the cache you have to update the factory to the latest version so to make sure that you're on the latest at least. You change the repository to the new version or a new distro by keeping the same major version. So if you're on Centoest 7 and you want to go to REL 7 then keep the same major version. You don't go to Centoest 7 to REL 8 because it's more complexity so it's more difficult to know what's going on. And then you just replace the packages with their new distro repo counterparts, right? So if you have Apache on CentOS repos you just reinstall that so it's now pointing to it is now the package from REL. And that's it. That's the whole process of converting. Talk over, right? But there's always edge cases like UFI is one of them. We're like on CentOS, there are some edge cases that doesn't really work when you just try to install a new kernel and so on. You have different custom scenarios where if you're doing something with your company there might be some things there that you will have to work around. Bad packages which is more of like, well, for example, CentOS logos is one of them, right? That's very CentOS specific so you want to have to try and mitigate that somehow, right? And then package depends on mismatches, et cetera. The package between mismatches probably won the biggest issue, I would say. Like there will be something and a lot of the tools try to just like, oh well, either we just ignore it if it doesn't work or we try to mitigate it somehow or just like say fix it before you convert depending on the tool. Yeah, there's endless configurations, right? Every system is gonna be unique. There's gonna be packages that don't work, right? For example, model that this one in packages that you can see on the right side that just does not work when you try to convert it from CentOS to REL because there's some dependency things that they can't really resolve. So those things we're gonna have to ignore. And then it's in general easier to convert to Alman Rocky from CentOS than, for example, from CentOS to REL because they are, I mean there's a successor to CentOS so they have some like, they're pretty much similar. I would say it's almost the same system. So it just makes it easier but REL actually packages their own systems and all our packages and then it makes it a bit easier to convert to Alman Rocky. And then CentOS Stream, that's like the big scary one because CentOS Stream, since it's an upstream from REL, there's gonna be downgrades when you go from CentOS Stream to REL or something else because you have so many, you have so many different downgrades. So either you just preset in place for a long while which is not secure and not something you should do and then you just hope for the best but it's still a scary subject to like try to do it yourself. And the reason for that is like that you're gonna have downgrades and they're not usually tested. So package maintainers, they usually test that upgrading a package is fine but when it comes to downgrading they don't usually do that which in general it's not gonna cause an issue but if let's say a package updates on configuration file like a format or something then they can't really go back to that previous format. So let's delve into some more facts about how to actually do it. Yeah, as I mentioned, it's all based off of REL which is in turn based off, well it's based off CentOS Stream which is based on Fedora, right? But almost the same thing. Packages are mostly fine when upgrading since it's tested, you have like some security there. I don't think it's gonna be niche if you go from CentOS to REL, CentOS to Alma and have to upgrade them because the version difference, it kinda works out but the downgrades, we just don't know. We can never tell. It's like, okay, we have to try, see what breaks. If nothing breaks, it's probably fine. And as I mentioned, they don't usually test that downgrades work. But CentOS Stream is still something that you can convert. There are some tools that allow it. Unbounded Explorer has an option for it where if you put in dash dash downgrade, I think, then you will be able to convert CentOS Stream. And then convert to REL is trying to do it as well but we, since we're trying to do resilience and make sure everything is going to get smoothly, it's still, we have a semi-working draft that's been up for quite a while but it's such a big topic that we have not really tackled like how we should actually do it yet. And then configuration migrations, they will happen, they can't really be reversed. Like when you have a breaking change for a configuration file, yeah. You're probably gonna have to fix it manually. At least that's what I believe, right? Or just talk to the package maintenance if they actually have some script that you can just reverse engineer, maybe. So bootloader is another thing that's a bit edge casing. So bootloader, if you don't know, you just have a bunch of options. It has an order to go through for the booting and this is from my machine. So I have running a Fedora silver blue or the Kinoite, which is kitty. For some reason I see rail there but I don't know why, because it's a Fedora machine. But it's booting from Fedora if you're trying to convert this to something else you would change the boot order, you would change, add a new entry and it would show a different file. And that may cause some issues in some system as I mentioned, right? Secure boot is another thing that will not work when you convert because Secure Boot wants things to stay as they are. You should not modify things, right? So you're gonna have to turn that off when you're converting. Then for CentOS 7, there will be some edge cases if you're using UEFI because when we're creating a new system converting to rail, for example, when we create a new boot order it's not really gonna work because there's gonna be a difference with the actual, I'm gonna go back. In the actual file pass there's gonna be an issue because we're not gonna be pointing to the same to the correct files, we have to fix it manually and create our own entry. And then you may be naive in thinking, okay, well I can use LSBOK to get all the block devices and then I see the minor thing there and then it's gonna be the minor number that they use so I can just use that and do it automated. Which without worked until we got some customer complaints that it doesn't work and as you can see in the lower picture you can see that it says SDI1 which is the first partition of SDI but the minor version or the minor block number is 129 which is not the partition number so that's getting that was actually quite difficult. In the end we managed to find this command here, BLKID and we just get the part entry number from there and that actually worked. And another thing that I don't think anyone on our team realized boot number entries, they're actually hexadecimal so you can get something that says boot 0 0 1 A and that's just gonna break everything you just automate this like assuming it's all numbers because why would you need like 9,999 configurations, right? I mean it's, I don't think I've ever seen that so but they use hexadecimal so that's why I have to think that's an edge case. And then with the actual tools, I mean there's gonna be more variants, right? You have the normal AMD Intel X86 which is the like easy part, you have ARCH which is getting, oh I'm sorry, which is getting up into like popularity because it's usually a bit better than X86 in a lot of regards. You have IBM power systems, you have different major and minor versions and then you have different life cycles and non-minor versions and major versions, right? So you have like long-term supporter extended up the upgrade support, I think that was the answer, something like that. And that means even more packages to test so if you want to like make sure things are good then you have to go through the best way to test is just convert it, right? You can just convert like a test system or like spin up a mirror and try to convert that and see if anything breaks but there will be some package specific or architecture specific versions for each package unless they're using like no ARC which works on everything. Like if they have a Python package then as long as the system can have Python it should work so it's just no ARC or no ARCH maybe. And then kernel packages. I haven't looked into this but they assume that kernel package is gonna be different on like ARM versus X86. So that's another thing because packages from kernel is gonna be, I mean that's the most important part, right? Because when you convert it you want to make sure that the kernel is actually from the new repository, the new distribution. So that's one of the things. And then if we want to make it more complicated we have cloud conversions, right? So for subscription-based services like Oracle Linux or REL they have like two options. You have bring your own subscription, B-A-Y-O-S and pay G, pay S-E-Go. And B-A-Y-O-S is like images that the cloud providers provide and then pay S-E-Go is like well you don't pay for the subscription right away you pay for the use end so it's like add an on top of the hourly rate that machine server like the minute rate. B-Y-O-S is similar, like it's similar to normal conversion it's gonna be quite easy. Everything should work as normal. There might be some subscription issues but in general it shouldn't be. And then pay G, that one is a bit more of a difficult thing because actual cloud providers they hook the billing to the actual machine so when you're converting that from Oracle over to REL for example you're gonna get double bills. So you're gonna get the bill from Oracle and you're gonna get a bill from REL, Red Hat. Which it's not ideal and the cloud providers are trying to solve that. I know AWS, I think AWS is the one they're trying to fix this somehow. And then there can also be slight differences in ISOs provided by the cloud providers. I think that Google they provide their own centers Linux builds because they want packs around, they want to introduce something. And then AWS, they actually use secured out and images from a third party that's partnered with AWS. And without knowing the difference between these and the actual images from the official repositories it's difficult to know whether it will pass or not. You have to test that as well. And then you have normally images from non-enterprise Linux like Alma, Rocky, so on. All of those should be available and you can just convert and we kind of treat them as like a normal conversion because that's usually easier part. But the enterprise ones are more difficult. Oh, and actually testing the conversions. I mean, this is a big topic. Every OS, every architecture, every version is different. So it's gonna be huge test suite of all kinds of different environments. We want to stay as close to the environment of the tool that you're trying to run. So if you're converting century seven that's using some IDM server, you kind of want to reproduce that in a test environment first and then see if it converts. Ideally, you just want to like spin up a backup and then see, okay, can I convert? If not, then something needs to be changed, right? Containers, they are actually great for converting. I know that Elevate, Elevate is actually going into some boot management options so Container doesn't really work there because they spin up an in-a-trom FS and kind of boot entry where it finishes the conversion in and that doesn't really work with Containers. Correct me if I'm wrong, Heather, but I assume you don't really run it in Containers, all right? So, but most of the time you can use a unit test with Containers. That's like an easy part. But it's not a good picture. You can just test the actual unit but it's not gonna test like the whole conversion. So you have to actually use VMs or some kind of bare metal machine to try to convert. And I know that Converturell extensively uses it so we have a whole, we're using testing from to test a whole array of machines and then we also have our own VMs. We try to use Vagrant if you want something simple and we can spin up pretty much any of the systems that Converturell can support. And then you just try to install this many packages as you can. If possible, install all of the packages that are from the distribution from the repositories because then you will know if there are any packages that are not gonna be converted if there's any issues and so on. And ideally just call the system because otherwise it's just gonna be, it's not gonna be the complete mirror of what you're actually trying to convert. And make sure that you have backups. I didn't put it in here, but backups are like important because if there's an issue and you're into production and it's like, oh, well, my production system is down. I didn't do a backup. Shit. Then it's kind of ruined, right? And then you have to try and fix the actual machine instead of it's restoring from backup and so on. So, God, I was so fast. But yeah. So, any questions? Because I feel like there's gonna be quite a few when it comes to converting. Yeah, okay. Right. So the question is when you're converting center is stream to row and you have to like freeze all the packages. Can you just allow security patches to go through? Basically, yeah. I'm, I don't think so. I think there's still gonna be some issues with the downgrades. I mean, it's gonna make the system more secure, right? Like we need some security, but when you're actually converting the system over from something, there will be downgrades. And if it's a security related package, then you will have to solve it. But it depends on how much resources you have. Like if you have enough resources that you can investigate, okay, these packages can be need you. How can you mitigate that? How can you fix it and you just create some custom scripts and like fix that? It's fine. But there might be a lot and that's not ideal. So it's, you have to try and see. Yeah. Yeah. All right. Any questions? I think that's it. You can always reach me afterwards. I'm available on master.nl as well as on Twitter. It's just spitech. So you can find me there. And I also go by Augustus on Red Hat. Yeah. Thank you. Yeah, I think we're expecting more questions, right? Okay. Well, time is up. Therefore, I would like to welcome all of you on a presentation related to the interesting topic. I'm going to call it a Contribution Topic, or Jularity. My name is Jeroz Lomaracic and I'm from Red Hat and the Software Management Team. Well, oh. What I will talk about, well, there will be some introduction. Then there will be something related to the, how to move modular rules to the RPM. And then it will be follows how to hack modules. And let's talk about how to list alternative software and what about stream defaults. And at the end, well, there will be the section related to the testing of alternative streams. Well, before we start to talk about modularity and how to move rules about modularity, let's talk about feature of the modularity because this is sometimes overlooked. Therefore, what is modularity and what it provides? Well, modularity provides virtual repositories with dependencies. And it means these repositories has a request and the conflicts. Also, it allows you to set for the alternatives different life cycle. For the sum distribution, this is one of the key feature. Therefore, how to deliver software to the users with the explicit stating and visible stating, okay, it has a limited life cycle in comparison to the main part of the distribution. Well, another feature is that it allows you to create alternative content without modifying a spec because it will not change the name of the RPMs. Modularity, it's not required because modularity creates a conflict for you and hides everything what can conflict with your packages. Well, another feature is better user experience with alternatives. You can easily manage your content because you can list alternatives. You can pick what you want to install, which profiles and so on. And well, you can also build dependent modules. Therefore, you can build on top of existing modules other content. And the last but not least, well, default software streams. This is quite important because not all features is available for all of your alternatives. But I will talk about this later. Well, here's just a scheme, just a small example. Sorry for making it a spiral. I just use it as an example name. Well, well, Parallel is distributed in Fedora and in RHEL with alternative versions. Therefore, for the one Parallel, you have several alternative streams and one of the streams is marked as a default. Therefore, if you install your system by default, you will start to consume if you need this stream. And also, on top of that, you can build dependent modules. In this example is a Parallel DBI that could be built for all of the version of the Parallel. Modularity, what it does. If you want to install Parallel DBI by default, it will automatically pick in the background the default stream. If you want to use the alternative, if you already have enabled alternative version, it means, for example, Parallel 5.32, then it will automatically pick the right content for your system. Therefore, it's quite easy to use from the user perspective. Well, you know, everything has a price. Let's talk about the price. We said that you don't need to modify spec. Therefore, even all alternative Parallels still build package called Parallel. Therefore, to keep upgrade path, you need to do something with your content. Therefore, first of all, none of the modular rules are written in RPM. Therefore, you need external metadata that will say, okay, this is part of the, this team, this is part of another stream, and so on. Therefore, there is a modular YAML, okay. But also, as I said, yeah, it needs to hide things that are not related to your system. Therefore, it's example of the Parallel. If you enable, or if you have a default, you will only see one stream. The other streams and other content from the other stream is hidden for user. Therefore, it requires modular filtering. This is implemented in DNF. Also, well, there are some protection layers. Therefore, we know that there are some safety measurements that if you install modular RPM, then, and you don't have a description for this modular RPM in the modular YAML, then DNF will refuse to install it. They will say, okay, well, sorry, I cannot do it because I don't know nothing about this RPM. To resolve all of these relationships, okay, it requires also additional solver, which will say, okay, these content is valid for your system, and I will only provide and show you the valid content for your system. And of course, well, there is a modular build system. Therefore, the system where you can build these modular alternatives. Well, let's just go with the alternative scenarios. Well, to first, I would like to say a few things about how to move modular rules to the RPM. And then, well, let's talk about alternative stream, which are modules and how we can, for example, use the groups to deliver alternative modules. Well, one of the features that I mentioned at the beginning is that, well, you can have one spec to build multiple modules. Therefore, I will try to use an example where you will have one module spec that will be able to be used for this alternative approach without modifying for the multiple streams. But this is just optional. Yeah, you don't need to use that. Many things are optional. That's the problem. Well, in the original spec, well, you only define the name of the package. Again, parallel, sorry. And well, with alternatives to ensure that there will be right upgrade path, you should rename package for each stream because you want to ensure that there will be right upgrade path. Therefore, if, for example, in this example, you use that you will use a parallel and with some suffix, then it's unique name and if you install this name, it will always keep the right upgrade path for your system and it will not jump from one alternative stream to another one. Additional thing, yeah, this is just a prerequirement. But as you can see, well, we start to modify spec. It's not enough because if you have two names, two different names and it doesn't matter whether inside it use the string parallel or not, by definition, they are normally installed in parallel. Therefore, if they are not installed, these alternatives in parallel, you have to set up conflicts. Therefore, you have to conflict with parallel because, well, if you will provide parallel, then it will work. Therefore, it will ensure that only one provider of the parallel will be on your system. If you will not do that, you will get again conflict, but after when the transaction will start to perform, therefore after you download everything and then RPM will tell you, you know, there is a fire conflict, you cannot do that. It's too late and people don't like it and don't do that, please. Well, if you want to achieve that your users will type DNF install parallel and you want to install any PR parallel, including with these suffix. Therefore, these packages must provide parallel and this is also prerequisition for the conflict. And if it's arc specific package, you have to also provide parallel with ISA macro. Otherwise, you will break other things. Well, all of these things are optional. You have to think, if you are there, you will get a feature, if it's not there, the feature is not there. It's not a requirement, everything is optional, but some of these are required if you have a conflict with the files. Well, because we move from, we would like to move also identifiers of the module, then we can set up some virtual provides. Well, I used some examples, but well, this is just an example of how you can write information using virtual provides. But please be careful or try to use it as less as possible. Because, you know, anyone, any new virtual provides, increase the metadata, any new provides, increase the requirements for the solver. And you know, our distribution is still growing in this direction. Your provides is also growing because no one is stripping these former provides and so on. And well, it will require more resources, disk, RAM and so on. Therefore, well, please be careful. As a package manager, please look what you have in your spec and periodically clean it, but it's not needed. It's good approach how to sustain or how to maintain our distribution in the right shape. Well, I also mentioned the Parody BI as a dependent module. Therefore, if I have a package that depends on top of alternative, then I have to specify the build requires. If, in this example, I use a build that requires this package.conf macro. Well, with this stuff, all of your packages will provide package.conf.parry. Therefore, you have to modify it because otherwise, you will be unable to pick the right parry provider. Therefore, if you want to build your package with a certain alternative stream, then you have to modify the require. And you can use these that you specify the full name for the demo package. Well, be careful. This is not according to the guidelines. If you don't read the guidelines, they prefer to use the package.conf. But if you provide alternatives, you have two options. All of them will provide the same macro or satisfy the same macro or you will rename it inside your code. It means it's the name of the one file and again, this is extra work. And this is up to you. This is a definition or this is option. Well, if we have such a package, if you have these alternatives, well, we can start to think about what we need from the modules. And as I said, yeah, there is a module of filtering. If you rename your packages for each stream and you set up correctly a conflict, you don't need that feature because upgrade path is ensued from this point by rules inside the RPM. Therefore, you don't need a modular filtering, but you can use it. Now, from this point, it's optional. Why? Because, well, sometimes you don't want to confuse your users by providing thousands and thousands of packages. Therefore, sometimes it's much better to say, okay, well, here's one alternative and you see all of the packages and here is the list of alternatives and you can switch. It's optional. It's up to the end user whether you want to use it or not. And if you want to disable modular filtering and still use modular YAML, then you can specify name of the packages in the section of the demodulars and RPM and then you will list all of the names of the packages and then you will see them, yeah. VNF start to ignore them. Therefore, nothing will be hide from other repositories and from non-modular stuff, but it's usually not needed if you use these compact packages. Well, also, you don't need to file safe because you set up all of the rules into RPM. It's intact. Therefore, you don't need any additional garter. How to disable modular versus system for the things that are described in the modular YAML? Easy. Don't build them with the modular label. If the modular label is not present in the RPM header, then again, the rule is not applied. Well, another topic is default stream and maybe, well, before because we are, well, closely to the middle, well, let's try something different. Let's try a small exercise if you don't mind. Well, let's play a game related to the solver, yeah. Therefore, solver is an algorithm that tries to pick the right version. Therefore, you set rules and according to the rule, you will get the solution. And let's just say, okay, well, I will ask you a question and please according to my prerequisition, please answer. Yeah, therefore, I'm trying to pick random person. Well, may I ask you for cooperation? Yeah, please, well, pick anyone in this room who is in front of you. Yeah, me. Okay, well, can we repeat it once again? Frank. Yeah, if you will repeat it one other time, will I get a consistent answer? Great. Okay, let's write, I know. Okay, one with you. Same question, same rule. Well, will I get a pick anyone in the room who is in front of you? In front of me. Yes, thank you, yeah. Okay, then, and we will test it. Thousand times we'll guess the consequences. Yes, I tested independently in the two examples. Yeah, and it works. That's how we test our, let's say, our theory. Yeah, we have, we set the rules and it works. Well, you know, time is passing. We have a more alternative content and then I will ask someone from the last line. Okay, they are sleeping. Ah, Neil, please, please pick anyone in the room who is in front of you. Okay, well, try again. There. Yeah, okay, and once again. There. Okay, that's a regression on the solver, yeah? Yeah, because at the beginning, we test that we have consistent results and we test it well. 100 times in multiple, into two different environments and it works, yeah? And that's how it works. Therefore, at the beginning of the distribution, you set your environment. You think about that you have a right approach how to deliver a software and you test it. You set up multiple environments and it works. Suddenly, after a few years, when more and more version of the software appears, then you get consistent reasons. And what happened? Because I tested well, solver is wrong, yeah? It's regression because I tested and it worked correctly, yeah? And that's what we have normal reports for our site. Therefore, if you want to set up the distribution, please think about the future. Therefore, if something works right now, even if I test it well, then it doesn't mean that it will work in future because environment change. Our distributions are like organisms. Therefore, they involve. They are moving from the one state to another state, but you are unable to change the rules. Rules are usually fixed, baked, unchangeable and you have to leave it out. Therefore, thank you very much for cooperation and let's continue. Well, and we have here our default streams and it's quite related to the exercise that we have, yeah? Therefore, well, if you start thinking about defaults and why I need that, I want that or everything works fine, yeah? But, well, in future, you can get a problem because your infrastructure, your customers start to pick alternatives because they have a better version, for example. They are the better place in the repository. They have other advantages. They have a small, let's say, install size and so on. Therefore, there are reasons why you can get a problem. How defaults were mentioned or handled in modules. You have a special document in the module Yammer that defines, okay, this stream is a default and if a user doesn't specify by another way, it is the only visible content. Well, if we use our RPMs with the alternatives, I set up three examples what people might or wants to have, yeah? Therefore, if I run, you know, DNF InstaPero, I expect that I will get a para package, default package because I would like to use a default. Why default? Because default is the most supported, the length of the lifetime is usually longer. Most of the software or the best, all of the software use that stream. Therefore, I will not get the trouble with imports or with additional features and so on. Therefore, if I want to install a para and I use a compact packages, then any of these provider of the para can be installed. Therefore, let's think what we can do. Well, if one of the stream is built without a suffix or without original name, then DNF will pick a para because first it starts to search for the names. Therefore, if something is named para, then you will get the install para and nothing else. Well, what happens if you have a package that require para, yeah? And or you want to install alternative version of the para video package. Then you can specify in the requires. It requires built, or it requires the alternative para or it requires original para. The last thing that I, if I have a package that requires any para, you know, my package does the most important part of the work. It asks for the version of this dot, yeah? And then you want to get any one. Then it's a problem because your package requires only para and all of these alternatives provide the para and for the solver, I can pick any of these. Therefore, I can use the para that provides the highest number, but it's usually not the default or it is default at the beginning, but not later. Or, well, that's, there's no restriction for the solver. Any solution is valid. And it's, of course, it's difficult. You can use, you can use a hint that you can hint that okay, well, I suggest that you will install the default version of the para. I will set something. It's a weak rule. It's a hint, therefore, it only helps the solver, but it's a weak rule. What about weak rules? You can break them. Or, if multiple weak rules appears, it happens, you know? No one is restricted to not use such as para for not default, yeah? Anyone can do that. Therefore, it's weak in comparison to the modular system where rules are much more stronger. Well, another topic. I am a user. I'm developer, yeah? I would like to see how many alternates I have for the para. Okay, with the modules, it was quite easy. Just write DNF module is a para, or I would like to list all the alternatives. That's even more easy. No needs any argument, and I will find what I have interest in. It's easy. Well, if my rules are strictly written only in RPMs, then, well, I have to play with the DNF repo query, for example, and I have to try a few commands. First, okay, the logical way, if I know how it is constructed, there is a para with the suffix. Then I say, okay, perl and white card. Well, you will see thousands of packaging stone. It doesn't work. Too many packages. Well, then, more clever way, if I know that all alternatives provide a para, then I can use what provides, but it will give you all of the versions with the rel. It will be quite painful because, you know, each of the version of the para has a tons of the versions therefore. Well, if I will also use a query format and I will ask for the name, then it will help because I will only get the unique names for the unique providers of the package, of packages that provides a para. Well, with the Python, that's even more funny part. Yeah, again, well, if I run repo query with the Python three white card, well, you will get a five tons of packages plus. Yeah, oh, yeah, even more. Well, then, again, well, let's think about what it provides. Yeah, only one package provides Python three. Therefore, it's not exactly. Therefore, you can use what provides and then you can write dot and then you specify version. If you will not specify the alternative provides, you will again get quite a lot of mess. And or you can use, you can start to think about using just searching for the names. And again, you have to use a specific pattern. Well, what is output or what do you think? I would like, there is outcome from these two examples. Any idea? I need to use reverse engineering skills. First of all, I have to look inside for the packages. I have to find what it provides. And each of these needs another approach, how to get a list of the alternatives. And that's the problem, yeah? You cannot use a unique approach. You cannot use, anyway, this is quite expensive anyway, these searches. You have to find what you have. Thank you. You have to find what you have. You have to look what it provides and then according to your finding set some rules. It means this is not what you want to expect at this time when users wants to get user experience. Yeah, you don't want to do such a searching and your research just to know how many versions of that software or how many major streams I have in my distribution. Yeah, that's something that is, you know, it's from the last century. Well, I also mentioned that we can use comms group to deliver a definition of the alternative content. It's quite similar to the modular model. It's, well, there are some not nice features of the groups, like updating of the definition and, well, it doesn't support default. Therefore, if you want to use, for example, mark default then you have to name it. And if you compare how it can look like for the parallel and parallel DBI combination in comparison to the module you can get such a combination. It will help you to pick the right packages. You don't need to care about the suffix or the renaming because you can use any pattern. And it's somehow human readable but anyway you have to read it and pick the right content. You see more alternatives for the groups because there is no logic that will pick the right content for you like with the modules. And again, well, the combination with the defaults is getting more and more ugly for the depend stream that for the original providers. Okay, let's just look for the testing. Well, if I have one application and it requires two different packages, well, the testing is quite clear. It's one setup, I run all of the tests and I have a result, it works or it doesn't. Who knows. If I will provide alternative provider, even if it's not default, what expectation from our users that it will work? You know, you provide that, you can install that together and so on. Therefore, if you want to test it correctly, then you need to do it double test. Therefore, with one alternative and the second alternative. Well, that's not the end. You can have also alternative for the provider B. Well, there are already four combinations and we are just starting, you know, that's the problem of alternatives. Therefore, there are so many combinations. If you provide any alternative software, our users expect that it works and it's dusted and we've marked that, you know, our distribution is dusted, but you know, not this combination or this one, but you don't know, therefore, this is expected. You already have four combinations, you have only two streams from the two providers and, well, you can look how many alternatives you can have in well or in Fedora or look to the past. They have multiple alternatives. You have also things that people built on top of that and they can use, they can select certain set of the combination or they can have, they allow all of the combinations. Therefore, the matrix for the test is increasing dramatically each time when you increase, when you arise or when you provide new alternative. And as I said, well, right now, there are completely different expectations from our users. Our users expect that everything will work. They expect that it is dusted, it's easy to use and that our distribution is not the set of the packages, but it makes together sense. Therefore, there are different expectations that in pass and we have to count, we don't have to count on that, that, you know, that they will be tolerated and you're gonna tell them, you know, you install alternative, that's your fault, yeah? Use a default, what's the default, yeah? Difficult to say. Well, and let's come to the end. Let's talk about the summary of this. Therefore, the first part of the summary is just focused on you as a developer who wants to provide alternative features, alternative content. The short summary would be, don't do that. No, but you usually need or you won't. That's the small difference. If you need, that's fine, but you have to calculate the price. You will, doesn't matter whether you will use modules, whether you will use, well, compact packages, whether you will use a vendor label like in SUSE, everything has its own price, yeah? You will receive new problems, new combinations, unexpected combinations because, you know, users are pretty, pretty, you know, productive in using your products in the way that no one expects that you will, they will be used, but it's valid, you know? We provide tooling, therefore, and they use it as a tool. Well, therefore, not only think about maintenance, testing, please test all your combinations. I know it's impossible, but think about that. It is expected from our users that we provide tested solutions, not a random one, yeah? Think about that or alternatives we provide the problems with the package builds. As I said, important are defaults. If you pick the wrong default, then, you know, your user experience with your packages will conflict with the rest of the distribution, and think about user experience. Therefore, if you would want to use modules, please try to somehow make it easy, your users to find your alternatives, to find the recommendation, and the best way, think how to do it on the distribution level. There are, right now, I know, 30, 40 groups that might want to provide alternative software. If each of them will invade the solution, how to list all of the alternatives, then each user needs to know all these 30 combinations, how to get the list, you know, you can even create your own app, and say, okay, list all of my alternatives. It works, yeah? And everyone say, it works. And users will say, you know, you are crazy, yeah? Because I have to know how I find your solution to listing all alternatives. The problem is there are tons of ways how you can deploy alternatives. Therefore, you cannot find similar patterns between each teams. Therefore, please open the discussion on the distribution level, how to make life of our users much easier. Well, if you have modules without modules, doesn't mean that you cannot sell them as a module. You can still ship non-modular packages with a modular YAML. Therefore, and you can disable, as I already mentioned, feature. Therefore, modularity is not only about the built system, it's a complex stuff that can be interfered with multiple things, with multiple signs. And additional things, if you think that the best way is to introduce a new type of the metadata, well, be careful because any new metadata cause a lot of problems in infrastructure, in satellite. And many distributions will drop them because they don't know that and they will not sync your content. And if you will decide to go this way, or your distribution, then please unify it and don't expect that in five years, you will again want to introduce another way how to list your alternatives. Well, thank you very much for the listening and let's open a discussion. Do you have any questions? Go on. Me? No. Okay, the question is, can we just kill it? And, well, and then additional note modules. Maybe let's not answer this question, let's ask another one. Doesn't matter, it can sleep. Well, don't ask that question. Ask, do we need alternatives? That's the question number one. Do we need alternatives? Okay, the answer from Daniel is that, well, no. Because any way how you will do alternatives, it will make you crazy. And that's maybe what even I said, yeah. Therefore, in definition, alternatives makes no problems. Doesn't matter how you will lend them, you will only experience different types of the problems. Therefore, it's a trade. I don't want this problem. No problem, you will have this one. It's just what you want to. And okay, well. Or even a bulk problem. Yes. But the, yeah. And mostly you don't see them both at the one time. You experience one by another one, especially during the life cycle and so on, therefore, yeah. Okay, another question. Go on. What's the current state of the application? Deprecation of modules? You want, you want to see them in the realm. Okay, well, with the answer is that, you know, there are some attempt how to get rid of the modules. And we have to talk about whether we want alternatives. This is the, for my side, the most important questions. The, well, you can ask what is worse. Yeah, to have modules or you have alternatives without the definition. Therefore, you will not see any list of the alternatives. You will be maybe, you know, you will sleep calm and so on, but sooner or later you will get all of the problems. Or, you know, you will get different set of the problems. That's a funny part. You can't. Okay, well, the question is whether we can get rid of modules from realm eight and nine. The answer is no, you can't. Because for one good reason, what is expected from our users of the realm? That will not change. Yeah, okay, I am out of time, but please don't hesitate to ask me any questions. I will open any discussion after the talk. Thank you very much for listening and thank you for the questions. Okay, so let's go. So this talk is about MKSI InterD, which is an alternative way to build interdies. And the official title had in Fedora, but the Fedora part kind of shrunk over time. So, you know, it's like MKSI InterD and a bit of mention of Fedora at the end. And so let me start with a justification. And this justification will just use, refer to the talks on Friday. So we had some excellent talks about conditional VMs. And conditional VMs can be summarized as you are not allowed to run any code which has not been signed and verified to be the right thing delivered by the vendor or the owner of the machine, whatever. So we need to have everything imitable and signed and checked. So if we, and this is certainly the case for conditional VMs, but it's also the case for other things, like for example, you have a device somewhere on the edge and you just want it to do the thing that it's supposed to do. And if somebody comes in and swaps the disk, this shouldn't allow them to do anything with the device. So if we start at this point and say that we have to run the sign only and go back from there, well, currently we have only one way to deliver kernels that are checked, right? It's to create a unified kernel image which combines the kernel and the interd and the whole thing is signed together. Okay, it's not done anyway, that's not true, but it's, I think I would say the best way. And so if we are supposed to sign the interd, it must be built by the distro because we need to build it first before we can sign it and the keys are somewhere in a vault somewhere. So local modifications are not possible, they're not just useful in this model. So the current system that we have that has been built around local modifications and doing things at the end machine, it doesn't, it cannot be used anymore. And so let's say that we design the system from scratch. What we would do instead of this fairly complicated thing that we have right now is, well, do things in the simplest way possible and this means that we take AppSkin packages and distro packages which already have our software in a form that is ready to be deployed anywhere and just build an interd out of this and this is what MkSI interd is. So Martin, you have questions? Because if anyone has questions, then please raise a hand. I think it's, if I'm talking nonsense, then it's better to clear it up during the presentation. So, right, we have, in Fedora and Dora we have track code but other distros have similar systems that take files from the host FS, use some of, do some magic to resolve what dependencies should be installed. So LDD on binaries and maybe some special hacks to figure out things. So essentially people, when you're building an interd, you are redoing the work that was done by packages previously just in a more hacky way. So it's like, I mean, essentially duplication of the packaging layer. You have a dependency mechanism and conditionals and a lot of that stuff. And of course this takes time on each end machine. So this is what happened before, I mean, while we are building the interd and before we install it. And after we have installed it, the interd is very special. It has, so with track code we have a need queue and we have special code. So for example in track code we have bash code that generates bash code that generates bash code. And so things are complicated, different than in the host system and it's, if you want to debug things you need to know both the thing you are debugging and the interd and it's just complexity. The environment is set up in a slightly different way. And what I want to underline is that not everybody's aware that our interd's use system D in the interd. So we start system D in the interd, it sets up the whole system in the way that it likes it to be set up. So all the slash props slash sys slash dev is mounted in exactly the same way as on a real system. And it starts services in the same way and so on. The difference is that the root of the file system is a temporary file system instead of a real file system. But the logic and the API is the same. But we add to this and we have, so we have the system D queue for jobs and we also have the drag queue for jobs and they kind of interact and play with one another in some semi-defined way. And another issue that I find with this system is that every distro does a different thing. So the alternative approach is to say, well, let's do less. Let's build images from distro packages and MKSI, it was kind of a script, a wrapper on package managers that was initially written to test system D. And it builds images, but it's actually fairly convenient to build interd archives with it because it has support for different package managers and conditionalization and stuff like that. But MKSI is not the important part. I mean, we would replace it with a different system and it would probably work quite as well. But anyway, so MKSI builds images from distro packages. So it's kind of like a fancy wrapper around change route and DNF. And for the last upcoming, well, for the upcoming MKSI release, we have done a lot of work. So RIP part has been written to write stuff directly to the file without using a loopback device, which allows for unprivileged operation, which is very nice if you are building images as a normal user or you're building images in a container because you don't need loop privileges. It has been converted to use DNF5. A lot of work has been done like on the way that the configuration files work. Now you can do profiles and conditional logic based on matching. So for example, you can have a shared config as one file and a subset that is specific to distro has a separate file with a match section and then you end up with a limited amount of duplication. So all those things, they're part of MKSI 15, which will be released as soon as done, says it should be released, so maybe next week. So we have MKSI and now I can say what MKSI ID is, right? It's just a few config files. So the main thing is the 20 line file that lists packages that should be installed in the ID 13. And this is, well, okay. So what are the benefits? So this is a really, really minimal system, right? We, MKSI itself is not, well, it's kind of complicated, but this is actually a wrap around DNF, so the heavy work is done by packages, and we use the package dependency resolution mechanism to figure out everything that should be installed. This means that our packages have to be packaged well, but we do that work anyway, so that's okay. Right, and we let the existing tools handle 98% of the work. We are independent of the host, so I want to build an MKSI image for Debian or for Gen2 or Arch on my Fedora system. This works without any problem and other way. So, I mean, this is not just a question of not being, I mean, it's kind of ugly to pull files from the file system because they might have modified locally or something might be off. But it's also important for stuff like build reproducibility, right? If you are supposed to sign something, if you take it from packages, you know that the package manager will verify the hash of which file is installed, and if you repeat the installation, you expect bit for bit identical result, so there's much less variation. Yeah, the images can be reproducible, and, well, fortunately or unfortunately, everyone gets the same image, which is a problem I will talk about later, and it makes sense to sign them. And we reuse system D, right? We get rid of those additional helpers splash stuff in the interd. System D does the setup, and the execution environment in the interd is exactly the same, well, but it's very similar to the host environment, right, you open a shell into the interd, and it's like debugging a normal system. The root file system is different. And also, I know, like you want to install SSHD into the interd, you specify an additional package on the list of packages to install, and SSHD pulls in the dependencies, and you don't really need to do anything else, because the packaging has already been done in this way that you can pull SSHD into any system, and it gets started as part of the normal system D transaction. Yes? So, I mean, reproducible in the sense that if we take the exact same of RPMs and install it, we expect the same result, because we installed them in a fixed order, and it's a bit hand-wavy, because it's possible that it doesn't actually work. We haven't really tested this properly, because, I mean, there are other problems, but in principle, there's really no reason, right? I mean, the set of RPMs is fixed, and the installation order is fixed, so we expect predictable results. And then the other side? Yes, we have to, yeah, so it's like it gets into the same problems as building of RPMs reproducibly. You have to do some adjustments to fix time stamps and so on. Yes? Maybe twice. So, I'll get to this, this is a valid problem, right? So, what do you mean? Bigger, but it's not terrible. So, first of all, I don't want to have specific numbers, but generally, it turns out that you would expect that the inter-D built with trackwood is so much smaller because it has a smaller set of files, but actually that's not true, because almost everything now is delivered as a binary and some libraries. If you follow the package dependencies, you get almost the exact same set of files because you just need the same libraries, and most of the space is used by the libraries for code. So, this is very little difference in this regard. Then we have some stuff like the hardware database, and I'm installing that because as part of the RPM, trackwood is not installing that, but I think it's a buggy trackwood. So, there's a little differences, and then the big difference is the stuff that you said that, well, what to do about kernel modules. And what I'm doing right now is I'm just installing the kernel modules core RPM that has all the modules. So, that's 30 megabytes or something like that. It will be possible to modify this, right? I mean, I'm saying that we want to install everything from packages, but it's not like we cannot. I mean, we already removed some files that we don't want, and we can also remove maybe, I mean, if there's no other choice, we can figure out the mechanism to select a subset of modules. This is doable. Okay. And another way to answer this is that, okay, so the idea is, let's say, less than 100 megabytes. For current machines, this is actually tiny, right? I mean, you can have go binaries, there are multiple size of that. And why is that such a problem to have any interd of this size? It's a problem because we load it at boot and decompress it and put it in memory. And there's an idea to use interd's that are not CPI or archives compressed, but they are, for example, EROFS with internal compression, so you get like a block device and only uncompressed the stuff that you use. And then, so this needs some small kernel work. And then you sidestep the problem because you have an interd that is a gigabyte and you wouldn't really care. I mean, depending on how much memory you have. So, I mean, right, because copying of 100 megabytes in memory, it's a fraction of a second, right? This is the uncompression that is the problem. Yes, the kernel modules. And also, what works in the interd's that are built this way is the stuff that is directly supported by packages, right? So, for example, LVM is not a problem and normal installations of ButterFS are not a problem and encrypted disks are not a problem. But iSCSI is currently implemented in this way that there's some very complicated logic in track hood that does string matching and builds the configuration of the fly from some other stuff. And I don't want to repeat that. I want to change the package so that it supports running in the interd natively and let the package managers or package authors deal with that and not to have special logic. Okay, so, let's say that we build the image in destroy infrastructure and everybody gets the same image. This is nice if it works, but it will not work for everybody because people need to do some local modifications. So, the side step answer is to just build multiple interd and we will certainly do that. So currently in Fedora, we build a unified kernel image for virtual machines, not with interd but with track hood but it has an interd that only supports booting in all types of cloud VMs. So it has enough kernel drivers and enough software to work on virtualized hardware and not on other things. We could and probably would do a bunch of variants, but we also need other ways to extend the, to provide local modifications and there's a bunch of answers. And over the last, well, year at least in the system project, a lot of work has been going into this functionality. So it is needed if you want to have, read only signed images that you run, for example, in the cloud, but it also is the same problem for the interd because you want to have an interd that is signed in this read only and then you need to extend it. So I want to talk about those approaches. So credentials, right? We have a row of data, we started somewhere, we have multiple ways to store the credential and I talked about this during my talk in the morning so I'm skipping over this. So the important part for confidentiality is that we can encrypt this. So we encrypt the credential with, they have this here, we encrypt it with a file that is stored on a local disk that is protected with lax. So it's a secret. We also encrypt it using the TPM and either this, either that or both and both is the best answer. And then we have a credential which we can actually place in DSP where it's public, so to speak, but it is a secret because you cannot decrypt it and also the encryption works as an authentication layer but because if we decrypt something that was encrypted in the wrong way, I mean it was the wrong keys then the file will not be accepted. So this is one way that we can have staff, the local modifications, they're created on the running machine and then they are placed in DSP. Second mechanism is configuration extensions and this is kind of a new concept but it's a variant of something that was there already before. So we have system extensions and configuration extensions and they're kind of the same. So it's a discoverable disk image that has a partial content of the file system. That system you will load the runtime use overlay of S to make it visible in the file system and system extensions are for holders and slash user and slash opt and configuration extensions are for slash ETC. So you can make, you can drop those images, those extensions into the right places, for example, in DSP and they will appear in the interd one system DSP and they, the discoverable disk images can contain the embedded data and the embedded data is protected by a root hash and this root hash can be signed and also embedded in the DDI. So system view will tell the kernel to use the kernel key ring to authenticate the contents of the extensions before they're being used. Yes, discoverable disk images, this is, right? So STEMD dissect will tell us that, okay, the thing is an extension for system images, for interd, for portable services depending on the metadata in the image. And so here is the example, we have, so this is a disk image and it has a user partition slash user partition and slash an ESP partition and verity data as a separate partition and a signature for the verity data. This is all separate partitions in the discoverable disk image. So GPT as a file system table. And we have yet another mechanism which is add-ons. So an add-on is a Uki like binary that has a section and the section has kernel parameters. So, I mean, this might sound crazy and I think it's a bit crazy, but the reason is that because it's a Uki, it means that it's a Microsoft PE binary. And this means that we can feed it to Shim and Shim will tell us using the secure boot infrastructure whether it has a valid signature. So we are putting a text file inside of a binary that is signed by some key and this way we can verify the whole thing. So returning to this list from before, we have the variants, we have credentials, we have extensions that are checked by the kernel hearing. So this ultimately also means the secure boot infrastructure and we have the add-ons. So it's, I mean, different options and the addition of those options is what I think makes the whole project viable because we will have ways to deal with just not having flexibility in the entity itself. Yes, so, I mean, the stuff that I would talk before was kind of like on the more on the system side and on the MKSI entity side, there has been work done to integrate with kernel install. So, kernel install, gained a few plugins for different things related to UKIs and MKSI interd. So we have a kernel plugin to invoke MKSI interd to build an interd. When you call kernel install, we have a plugin to, and for some reason I forgot to list it here. We have a kernel plugin to call UkiFi to generate an image and UkiFi gain support for config files. So if you put the right config file in the right place, this operation will create a signed image. And then we have a plugin to copy the unified kernel image into the right place and we put partition after everything has been ready. And this stuff is opt-in, so you need to specify some config, it's a very simple config in various places. Oh, because it's kind of experimental and we don't want to break systems yet. And also boot CTL kernel identify as a helper that will tell you what the kernel, what the type of the kernel is and kernel inspect will print details about unified kernel images and so on. So I'm to wrap this up. I mean, in some ways, not much has happened in the MQSI RD project over the last few months because there isn't really that much to do from the MQSI in the RD side. The work has been happening in other places. And I think we're kind of getting closer to the point where it becomes useful for real users. In principle, we have a change for Fedora 39 to make MQSI in RD, in RDs available. I'm pretty sure that this will not happen or maybe for Fedora 40. There's some outlook how to build such images in Koji. Graph2 might get support for unified kernel images. I purchased our out there, they are being reviewed. There has been progress on making kernel modules here to install. We don't need the whole, I mean, it's still a single package with a big set of modules, but it doesn't contain the kernel anymore. And this was done because of the change of introduction of unified kernel images for cloud DMs. And I mentioned that MQSI works unprivileged and there has been some work on integration. And links and, well, questions, comments. So I was, I guess I, this is, so if we, so the interd is a CPIO archive and actually we can build it without any privileges. So the stuff that I mentioned about running without privileges when we build that DDI, so like, for example, a system extension would be a DDI and then we need a fight system. The interd is actually don't have a fight system and they are built unprivileged without any issue. So the building of interd is locally using this mechanism. It's like a supposed to be temporary step, right? In the sense that, yes, we do it and we also need it for development and we also always need to do it for development but for end users, we want to build the interd's ones and deliver them as a package content. So basically you have a, like with the kernel package you have an interd package and it's just, you get the interd there. Or even better, you get a, you have a kernel package that has a unified kernel image with the interd already embedded. So I have to admit that that was my idea in the beginning but maybe it's actually not that useful. Maybe we want to do, I mean, the fact that the modules have been split out from the Linux binary, this is nice because they are both big and we don't want the Linux binary at all but the selection of specific modules, it could be done this way or it could be done using some filtering mechanism. I'm leaning towards the second option now, either way. Yes, that will be one option, yes. So okay, I forgot to repeat the question. So the question was if we have some special hardware or file system and we want to deliver the code to open up the file system in the interd as a system extension, how does it work? So the way that extensions work is that very early when system D is booting, it starts a service, the service locates any extensions that are present and signed properly and they're at some point just appear in the file system. So this happens in early boot. So if you have this extension, this extension would essentially be overlaid and then the code inside of it could be used at a later point during the boot to mount the root file system or the whatever storage. So there is a mechanism to match extensions to the running system or the image. Okay, so the question was if we have multiple kernel versions and how do we match extensions to the right kernel version and the answer is that for each kernel we have a specific place for extensions for this kernel and we also have other places for extensions for our kernels. So there is a mechanism to do the matching. So the question was whether this has to care about the installation of the kernel and whether it will be installed by the RPM packages. Yes? I'm not sure. I'm essentially maybe copying a file a second time so not a big issue, but. So I'm out of time. So let me do a demo. So let me show the config for MQSI InterD. Sorry, MQSI.com. So this is my config file and it lists packages and it says that the output is CPIO and now we call MQSI and since this is without a demo, I am doing without network. Oh, it failed because it exists already. So let me do it with minus F now. And we do the package installation and we do a setup and then we remove some files so that the InterD is smaller. Here, those packages are needed for installation and then they get dropped. And we have a needRD somewhere here. It also comes with a manifest and a changelog so that we know what is inside or like hash of stuff. Okay, thank you. Okay. Are you feeling fine? So good afternoon, I am Andrei and I am working as a software engineer at Red Hat and I am working on a project called ImageBuilder and today I would like to show you what's new and how you can use ImageBuilder to build some nice up-to-date customized and shiny images. And yeah, by the way, I am going to skip over some of the details so feel free to ask if something is unclear and also we have a website so if you want to know more details, go there, there are some guides and you can follow them basically to do exactly what I am going to show you here. So let's not waste any time, any more time and go directly to do important bits. So what's ImageBuilder? ImageBuilder is a modern tool for building operating system images. We will go into the more depth which exact images we can build because images, it's kind of a generic term. So we will explore this more later but the basic facts are that we are building them from scratch so basically we will download all the repositories, sorry RPM from the repositories and then somehow combine them in order to create an image of an operating system and it's important to know that there are no VMs involved which is quite nice because a lot of modern cloud environments don't have support for running a nested virtualization so it's kind of nice that you don't need to spawn a VM but of course then the other method that you can use is to use containers and namesplacing so it needs root, so no VMs but root required. The important fact is that ImageBuilder never boots the image so it makes really sure that it's pristine and we will get to this later why this is interesting that the image is never booted and yeah, this talk will focus on ImageBuilder that you can install on your machine. We have also some other options where we will host ImageBuilder for you but yeah, I will mention them at the end of the first part of my talk and yeah, so that's ImageBuilder, modern tool for running operating system images. Now, what images can it build? And I will start with distributions because that's the first thing I'm going to mention. So we can build Fedora and it's children which means CentOS Stream and REL and for Fedora, we of course support all the supported versions for CentOS Stream and REL that's eight plus, so yeah. But the more interesting part is for which environments ImageBuilder can build images and there's plenty of them. So I actually put them on two slides. The first slide is about all the cloud images that we can build. The first blood point is called KVM and that's what I like to use for these environments where it's pretty much just KVM without any specialties which means LibVirt, so you can just vert install the images, OpenStack or KubeVirt, you know, OpenJet Virtualization, there was a very nice talk earlier today so if you are eager to try KubeVirt after this conference, you can also use ImageBuilder to use a custom image there and yeah, then we support also all the major cloud players. For example, AWS, Azure, Google Cloud, Oracle Cloud and VMware vSphere and from my experience, the KVM image, if the cloud provider uses KVM underneath the image usually works. So I, for example, use the images from ImageBuilder at Hatsner because you can just upload it there and it works because they use KVM, so it's pretty cool. So these are the cloud environments but we can even build more and these ones are, yeah, more interesting I would say. We can build installer ISOs. So basically that's, you know, Fedora has the installer and ImageBuilder can build a customizable, we will get to that later, but it can build a customizable installer that you can then use to install as many bare metal machines as you want or even virtual machines, but please use images for virtual machines. It can build containers like the OCI one, so Portman, just a word of caution. We don't support custom container files or Docker files. We are really meant as a tool for building the base. So, you know, if you want to build something on top of Fedora, you specify from Fedora 37 and ImageBuilder can build this base. And the last thing that we can build are OS3 artifacts. Currently that means Fedora IoT or REL4H in the REL world. And we can build all of the artifacts, so comments, installers, raw disks, simplify these containers, that's pretty much it. And that's pretty cool, but I won't dig into this deeper because I think that during this conference there were like free talks about Fedora IoT and REL4H and each of them mentioned ImageBuilder, so if you are interested just watch the recordings, all of the talks, I was there, they were really amazing. And just the last word of warning, some combinations are not supported yet. So, for example, you cannot build a container of REL current, I think it's not implemented with just because no one had the time to do it. But if you are interested, please contact us and usually it's pretty easy to fill the matrix. But yeah, there might be some gaps in the support. So, that's what images can it build. But of course there is one more thing because base images are nice, but base images are also boring, right? You can also very easily build customized images and that's where ImageBuilder really shines, I think. We can do plenty of customizations on the image and let's go over them. I picked the most interesting ones and then at the end I have the new ones which are pretty important and interesting. First thing first, we can do custom partitioning. Custom partitioning, that's where ImageBuilder really shines, I think, because for example, Packer is an alternative, but Packer boots the image and then does something to configure the image then it takes a snapshot. But when you want a custom partitioning, for example, when you want a separate slash user or slash var on a booted image, it's possible, but it's really hacky and just don't do this in production, please. But ImageBuilder doesn't do it, like everything happens on a not booted system. So it can do whatever partitioning you want, which is pretty nice and you can even use it later on this slide. Of course, install extra packages because we believe that the base operating system should just carry enough stuff so it boots in the target environment and nothing much more. But that's pretty useless. You want something like HTTP server or I don't know what. Federal has 550K packages or even more so there's plenty of stuff to install them there and ImageBuilder can install extra packages. Also, it can add to users, which also you can do it, for example, in cloud. You can do it via cloud init, which is a good tool but maybe in certain cases you won't predefined users so all of your deployments are the same and you have, for example, one admin user with the same SSH key everywhere. So maybe it's good to put it into the image itself. Why not? It can simplify your deployments if you need this. It can configure firewall, of course. Once again, that's pretty nice because the firewall is configured immediately after you firstly boot the instance, which feels kinda secure. It can manage system units, which is, it ties nicely to the extra packages because web server is cool but if you cannot enable it, then it's kinda doing anything, it doesn't do anything. So let's think and yeah. Now let's go to the new stuff that we introduced in the past few quarters. One of it is hardening, which is yeah, based on Openscape. Basically, how this works. So Openscape is a tool, you give it a profile and it scans your whole system and applies some remediations and these profiles are basically based on certifications. So if you need certification ABC, you apply a profile ABC and Openscape will try to do as much as possible so your system is certifiable by the ABC certification. And ImageBuilder can apply such remediations on the build time. So once again, the system is hardened from the very beginning, from the very first boot, which helps to establish a secure pipeline. Also, a lot of the certifications require a specific partitioning and it kinda ties together, right? You can use ImageBuilder to do the custom partitioning and also immediately harden the files like the system. So these two features are very often used together. We can inject extra files into the image and that's also very useful because you install a web server, you can enable it and you can also configure it so you can have an image that immediately starts to behave as a reverse proxy, for example. And the feature that I like maybe most from this list is embed containers. What this means, when you tell ImageBuilder to embed a container during build time, it will download the container from the container registry and put it into the right directories. So when you run Portman or MicroShift, it doesn't need to pull that container but it immediately has it on the system. And this is great if you have a disconnected environment because then of course you don't have any container registries or for the IoT slash Edge use case because your device might be on top of a hail and there is no good internet connection and you don't want to wait 10 minutes for a container to download for a registry. So that's quite useful for these kinds of scenarios. And yeah, I just wanted to mention that OpenScape, by the way, and so you can, so you know what it does. So for example, it can do some small changes to the configuration like changing, configuration auditing, SSH configs, OpenSSL configs. I mean, base distributions are nobody's pretty secure but these certifications require more and we don't want to put this config in the base system because sometimes the system might behave weird because users want to use Kaina, you know, older hashes and things like this. So it's always about trade-offs. Anyway, there is one more thing that ImageBuilder can do and it can immediately take care of your uploads to the clouds. Which is pretty cool because because if you've ever tried to do multi-cloud image uploads, it's a pain. Like I uploaded an image to all of these and IBM cloud actually also and OpenStack. And all of these CLIs are different. The process is different. You need different compartments. That's my favorite word from Oracle Cloud. You need different storage groups, storage accounts, storage containers, whatever. It's crazy. And of course you need to have all of these CLIs installed on your machine and it's annoying. But ImageBuilder abstracted this away. You just give it a config and it will build an image and immediately uploads it to a cloud. Which is very cool because to be honest, we believe that this should be more, like more tools should do this because building an Azure image locally is fine. But what are you going to do with it? Yeah, you can boot it with QM, but it's an image meant for Azure. So it really should end up in Azure. So we are trying to tie these two processes together. Because it really makes sense. Anyway, yeah. So the ImageBuilder that you can install locally, it's just both GUI and CLI. And at least my understanding or my vision is that you can play with the GUI and see what ImageBuilder can do and kind of click around and add customizations, define everything nice and visually. And then when you are done, you can just export the blueprint. And then you can just use the CLI and set up an automated CI pipeline. So yeah, I consider this as a two-step process every time. Yeah, and if you want to build an image or start building images, it's very simple to install, just DNF install two packages. One is the CLI, one is the GUI, and then enable the service that runs on the background and builds images. And you can start building. It's that easy. Yeah, good. Now I promise that I will talk also about other options how to consume ImageBuilder. And it can be also run as a web service. And currently the web service has two clients, one, well, three. Yeah. One builds golden rel images and one builds a golden currently federal IoT images. So that means that ImageBuilder can be integrated with Koji. And it is, it builds images every day. The second client is our ImageBuilding service in Red Hat Hybrid Cloud Console, where you can just grab the free rel subscription. It's fine, go to console, there's a link here, console.redhat.com slash inside slash ImageBuilder. And Red Hat will build the image for you and you don't need to maintain an infra, you will just send you a link or share the image into your account and it will just work. So my message here is that ImageBuilder is run in a production environment and it's suitable for everyday use. So just use it, it's stable, it's fine. Good. Is that that? Yeah, and I should also mention that that the service has also an API so you can also automate your CI pipeline using the hosted service. Which is pretty neat because you can just call it from a container in GitLab CI and it will just get up actions and it will build you an image. There should have been an image of the service, but yeah. Anyway, and just two items for the future. We would like to collaborate more on using ImageBuilder to build more federal artifacts. Currently the project is with the installer team. Hello, because we are talking together to building Federa installers because we would like to modernize the ImageBuilding stack. And also our big dream is to enable the community to easily build and share customized Federa images because we think that the process currently, you know there's Koji, Banji, Relang and it's like, it's pretty, it's pretty a big process. So we want to make something that would simplify this process by a lot. I also forgot to mention at this slide, the ImageBuilding service, I just like how everything ties together in this conference like with Kubeware, I mentioned, I mentioned, what did I mention, REL4H and Federa IoT and the service, it's actually tied to the keynote because it's an open source service. And yeah, it's one of the, we try to follow as much open service principles as possible and Simon here is going to have a talk right after this talk about open services. So you can go there. Anyway, that's a bit of promo. It's demo time. Good, so this will be the part which can go wrong at any time. So I have this small shell script. I will show you the script in a bit, but I will build a config called demo for this demo and it will be a QCOW too. So we will boot it in a bit with KVM with just QMO. So I will start it and now I have about five minutes to explain you what's going on because the image will be ready in five minutes. No uploads will be done because of the Wi-Fi here. Yeah, I don't want to risk it. So it will be local, but whatever. So the image is building. By the way, I will quickly show you the script. Yeah, it's pretty short. I have some debug stuff in there, but basically I'm just calling Composer CLI with a bunch of commands and then parsing it. It's pretty simple. I made this script so I don't need to go over all the steps because they can go wrong. So I automated it a bit. My script, by the way, will be available at the archive of my talk after the talk, probably. Anyway, let's go to the configuration. We call this the blueprint and this is what ImageBuilder consumes. It's a Tomo file and you can put a lot of stuff in it. So let's go over it and see what our image, when it's done being built, what it will contain. So a nice feature of ImageBuilder is that it can do cross distribution builds quite easily. So my system is actually running Fedora 38, but for some reason I buy more different distribution. So the image will actually be Fedora 37. So it's completely possible to do that with ImageBuilder. You can also, for example, build CentOStreme 8 or CentOStreme 9 and yeah, that's it. The next thing that I want to have in my image is Portman because I want to demo the embedded container thing and I need a container, how do you call it? Container engine. So I install Portman and some optional dependencies. I'm not really sure why they are actually needed, but that's a story for another day. Not sure if that's a Portman bug or I don't know. So yeah, this will install Portman inside of the image. Then I want to embed a container. So I can just tell ImageBuilder, hey download me this container and yeah, I give it the source which is my repository at gitlab.com. It actually contains the slides of this talk in web server and yeah, it will just save it locally as a talk container, just talk. It will be named talk and that's it. But you know, download the container, that's boring. So let's run it. And for this I am using the new Portman quad-let generator. So let's explain it a bit. It's very cool. Yeah, it's cool, right? So basically, it looks like a SystemD unit and SystemD can consume it via its generator thingy thing that Portman implemented. And so it is a SystemD unit, right? It has a unit thing and it apparently runs after network target. So after the networking is up and it can be even installed. So like when I install it, it will run or it will be required by the multi-user or default target. So it's a SystemD unit, but there is no service but there is a weird container section which basically the generator will translate it to a service and it will translate this to a Portman running Portman container. So I can just tell it, hey, take this talk image which I downloaded in the previous step and you can do the usual stuff that Portman can do. So I can just tell it, hey, publish me these three ports because it's an HTTP server. So I want 80 and 4443. And of course, the container is embedded. So I can just tell Portman, hey, don't ever pull it. You already have it in the store. Yeah, so that will save me some bandwidth potentially. So I also expect maybe with my image that I will need to store more data. So I put there an extra bar partition with 10 gigs of size and yeah, it will be there. So this is the example of custom partitioning. Yeah, there is just a quick note, but we have a support not yet in our image builder, but we are getting there slowly. So this will be just X4, but I mean for the cloud, I don't really care. So it's fine, I think. Then let's harden the image, which is this section. Basically, this tells the OpenScape tool that's run during the build time to apply from this file. This is like a real file that you can install on your system. This profile and then OpenScape will run anything was defined in this file and just scanning. As I told you previously, it can, I don't know, configure SSH to not allow password keys. Well, that's disabled by default in Fedora, but I don't know. Some other stuff like enable USB guard and yeah, whatever is needed in terms of security. And there are different profiles and on our website there are some examples of other profiles that you can use and you can also see the OpenScape documentation for more details. Anyway, let's go to the next customization we can add at firewall. So just install firewall D and immediately configure it. So we want to allow HTTP and SSH and that means that all the other ports should be closed. And the last thing is adding a user. As I promised you before, image builder can add users. And yeah, I can just create an admin user. I can add my key, this is my public SSH key and I can add it to the real group. But the issue is that, so the real group is for accessing sudo. And the issue is that Fedora by default doesn't allow passwordless sudo, but that's fixable by the first customization because I can just add a drop-in and real will be passwordless. So that's possible. Good, let me just quickly open the front end. This is actually the front end. So you can see that the demo blueprint that I put into image builder via the script it's there. So it's the same thing as I did on the CLI just visualize here. And you can see that everything that we talked about is here. So there is the openscape thing, openscape remediation, there are the file system options, firewall users, if I go to packages, there are some extra packages there. They are also with versions, which is pretty nice. Custom files are currently not there. We still need to add them, but otherwise everything is there. And you can just click create image here, select that you want QCOW2 and it will build an image for you and you didn't need to touch the CLI, but you like CLI, right? Anyway, so yeah, while I was talking, it took about five minutes to build the image and I downloaded it as an image, as an file and I will just use a simple script that I have to deploy it. It's just calls QEMU and yeah, forward some ports, it's nothing super amazing. You can also use LibVirt if you like LibVirt and when you install it, it doesn't really matter, it's there's no difference. And the image is now booting and I should be very quickly able to SSH into it. This is the scary part of the demo because it worked like 20 times in a row. Good, it worked. So this is the image and we can go over the customization. So Fedora 37, let's quickly verify it. It is Fedora 37, let's see if the container is running. So the container is there, yeah, it's the image that I had in the blueprint. You can even see that the image is installed. I can even curl to there. Yeah, there is something. I can even show you in the browser that if I go to the localhost 8080, it's there. I just forwarded the part. So that works, so everything is good. And what's there, what's next? Next was, this was shown. Oh, let's show the partitioning. So if I do df-h, you can see that var is there. It has 10 gigs or 9.8. Oh no, it has 10 gigs, right. So the custom partition is there. And yeah, what else? Firewall, so firewall. Let's see if I can do it on the first attempt, list all. No. Oh, good. So by default Fedora enables the HTTP v6 client and MDNS and we edit HTTP and SSH. So yeah, this also worked and the firewall is actually enabled. And I think, oh, I edit the user, yeah. And the user, yeah, sure, I am logged in as admin and it didn't ask for a password. So apparently my key is there and I used to do without any password. And yeah, that's my demo. And I think that I am almost out of slides. Yeah, I am totally out of slides. So yeah, image builder. I think that it's a great tool and not only because I work on it, but also because I like it. And I think that with these new options it can get you pretty far if you need a customized image. There is a lot that you can do to basically build your image, push it into a cloud, launch an instance and it will run its workload as intended. So yeah, this is image builder and I'm glad that you were here for this talk. And if you have any questions, please I'm here to answer them. That's a great question. So I will repeat it. Thank you. So the question was how does the environments I showed KUKAW or QEMU, but there were also Azure, AWS and other ones and how do they differ? So there are some slight differences. For example, on Azure, you want to install their Azure client. It can do some more integration. Agent, ah, agent. On Google that's the same thing. Google has their own agent. So you want to install some extra packages. And yeah, also sometimes the configuration is a bit different, for example. It's not done actually for Fedora, but maybe we should do it by the way, Cloudsync. EC2 really recommends that all the distributions as the default user of Fedora has Fedora. So this would be something to configure just for EC2 images, which would be, I don't know, quite cool. Yeah, and the package that's my, kernel arguments can be different. I think that for CentOS 3, we are disabling some drivers because they are causing some issues on some instant types. So the differences aren't huge, but I think that making like images pair, you know, suit or tailor to the target environment, it makes sense. Oh, sometimes you need to take care also about like bootloaders a bit. Yeah, hope that answers the question. No, yeah, all right. So the question was whether we always do QCOW 2. No, it depends. For example, for AWS, we will build a raw image because they accept only raw images. For OpenStack, it's QCOW. And for Azure, they need stream optimized VHD images. So we will provide that. And of course, then for the nonplots, you know, ISOs are produced as ISOs and yeah, that's it. Yeah. Neil? Yeah. So the question was why we don't have a customization that would simply allow you to alter the Sudoverse drop in to allow a password as configuration. Yeah, that's a great suggestion. And actually a funny part story when we introduced the Etsy customization. Yeah, I got a report from a colleague that it doesn't work. It's grew up his environment and yeah, he forgot like something in the file, white space. So I think that's a great suggestion. And our team is trying to actively seek for input so we can integrate more stuff that would help. And this is a great input. Thank you. Indeed. Next question, there was, yeah. Well, redhead pays me. And we use only RPMs, but if a community member comes and does some stuff, for example, the guy next to you did some work for Arch Linux, then we are not opposed. We want to build everything. Especially Arch. No. David, sorry, you mean, okay, we are out of time. I can answer this after the talk. Okay, good. Good, okay, we are done. Thank you. Very good. All right, hello, everyone. My name is Davide. I'm a Production Engineer at Meta on the Linux User Space Team. This talk is about what we're doing with ELN and how we're using tools to help us use our tools. ELN and how we're using to test upgrades on the production fleet at Meta. The agenda for today, we'll start with a quick introduction about what ELN is and how it's made. We'll talk about what we're specifically doing with it at Meta, what we got out of it, and how you cannot get involved and maybe get some value out of it as well. So without further ado, ELN stands kind of our enterprise Linux next. There is an explanation of the puns in the name on the website that I will not try to repeat. It is a continuously variable of raw height using the centos, macros, and toolchain. So the idea is that every day we take the raw height as it is today. We rebuild it using the same macros that would be used as if we were cutting a new centos train train today. And we put the compose out. So you can take it and use it. And you get a preview of what centos train would look like if we were starting today. It is part of the development process for stream and for rel. And it is used for this ongoing process. That's the link to the documentation. I'll add the slides on the website later so you don't need to worry about links. So here's how the sausage is made, more or less. Development work happens in raw height all the time. Raw height is continuously rebuilt into ELN. ELN is used to test how these changes would look like in the future centos train release at some point next month-ish, I believe. The first branching for stream will occur. And at that point, we will have a new set of branches and a new set of repos and everything. And that will be stream 10. And then stream 10 will be stabilized. And then eventually, we'll have rel 10 and so on. By the way, if you're wondering about the colors, this slide comes from an older presentation. I gave the orange boxes, because those are all things you can actually contribute to and work on if you would like. You can't contribute to rel unless you actually work at rel.at, which I do not. So how this actually works is that the composers are made using something called the ODCS, which is the on-demand compose service, which is the thing that is able to make distribution composes and put them online so people can consume them. The package itself is defined in the content resolver, which was an offshoot of the phenomenonization project that is still developed. If you go on tiny distro builders, you can see the actual package set and play with it and slice and dice it. That's a picture I took of yesterday that gives you an idea. You can also see kind of the absent flows in this. As we get closer to the branch point, you can notice this is getting lower, because I guess folks are realizing they don't really want to maintain stuff long-term for 10 years, so they're chopping it out. Which seems reasonable to me. Now, this is just for ELN itself, which is what will end up in CentroStream NRL. Most people, I would say, do not run just that. They need additional packages that they generally get from Apple. To test those, we came up with something called the ELN extras. The idea behind the ELN extras is an additional set of packages that are composed together with ELN, but are not going to be part of ELN itself. It's still kind of influx how we're going to do this, but the general idea is that we will use this set of packages to bootstrap Apple 10 so we can get a head start there and try to make life a bit easier, both for users and for packages. This is also defined in Contra Resolver. You can look it up there. You can actually contribute your definitions here, so if you maintain packages in Apple and would like to get started or have them ready for Apple 10, you can put up a PR against that GitHub project. There's a little YAML file you have to fill in, and your packages will end up getting continuously rebuilt in ELN. Now, of course, this means you sign up for maintaining them, so if they break, you will need to fix them. But you'll need to fix them eventually anyway. And again, that's the text set. This work is coordinated by the ELN SIG, which is a Fedora SIG. We hang out in Matrix and in IRC. While ELN is primarily a thing that is happening on Fedora, I read that infrastructure uses a lot of internal components. Work on the project is by no means restricted to folks that work at Red Hat. You're welcome to join and hang out there. There's regular meetings on Friday, where it's discussed. These meetings are also good to hang out if you're curious on what's going to happen in Strain 10 down the road. I'll talk about this later, but this has been very useful, for example, to get ideas, features that might be coming down the road that you'll want to know about if you actually run this in production. And here's a link on details about the SIG itself. So much for the intro. Let's talk about what we're doing with this in Meta. Before I do that, a quick primer on the infrared Meta. As you might have heard, Meta has a lot of machines. We have millions of servers. These are all physical machines running in data centers across the world. They all run CentOS Stream right now. We always run CentOS. We started with CentOS Linux. Well, I started in 2012, and we were running CentOS Linux 5 back then. We went through several major release transitions. So we went from 5 to 6, 6 to 7. Linux 7 to Strain 8. We just about finished in now Strain 8 to Strain 9. I was very happy to discover last month that we got rid, finally, of the last CentOS Linux 7 host. So we don't have those anymore in production. So that's nice. So I could put 2022 on that line. I expect we'll probably have a long tail of 8 stuff at the end of the year, but we should be mostly done with it by the end of the year. Right now, we're at 86% on 9. This is some containers on 8. Actually, 8 is the majority on containers, but containers are comparatively easier, I would say. And for contests, whenever we do this migration, we already provision the whole fleet. We don't do in-place updates for production systems. So this is wiping the machinery, installing, which is fairly expensive, as you might imagine. Now, our customers don't love these upgrades, because they're kind of expensive, because you're wiping their machines. They have to spend time qualifying the distribution. Usually, the way this has worked in the past is that we get the new CentOS 3 release drops, we start working on it, we put it out, we will play with it, we find some major issue that delays everything by six months, because that's how it works. Like, it's absolutely normal. When you have a large deployment, it's pretty common to end up having to deal with these things. With 9 in particular, the SHA-1 deprecation ended up being a fairly sizable annoyance, because we had a lot of packages that predated that, that had to be rebuilt. We also had a lot of packages coming from external vendors that, shall we say, questionable packaging practices that required a lot of work. So in general, we had been looking at ways to make this process better and more streamlined. When we did 9, I actually started working on these, started working from the very first beta that came out, and that helped quite a bit, because we could get a head start, since the, basically, since branching time. But you still get kind of surprises, because while if you follow Fedora, you kind of know what will probably end up, because you know where it's branching from, it's hard to tell until it drops. The way we do this better is if we could start this earlier, and that's kind of where ELN comes in for us. The idea with ELN is using it so that we can streamline bring up of major rest releases. Turn this from a thing that we have to do at qualification time when a new major release drops to a thing that we do all the time. So we can continuously validate what's gonna come down the pipe, find issues that they come up. When something comes up, we can either figure out internally if it's we that we fucked up and fix it, or if it's something that we have to discuss with the community, because maybe there's a better solution that can help everyone. It also allows us to identify policy changes early on. Things that will come in the distribution that might impact us, since there have been deprecated, new things that are coming up. And because this is a continuous effort, and because we started long before, it's a much more pleasant experience for customers, because instead of having to deal with, oh, all my packages don't install anymore, and I have a month to fix it, now all my packages don't install anymore, but this isn't a production system. Maybe I have a year to fix it. It's much nicer. People are a lot happier when you tell them they have a year to fix it, and they can sort of sort it out to their own leisure. And long at home we like to have an actual CI platform, where we continuously roll out ELN on production systems for some value of production, so that every customer with different workloads can get a sense of what's gonna come in down the pipe and do this validation at their own pace. Now, to do this, we started actually bringing up ELN in our infrastructure, and bringing up ELN is about the same effort as bringing up a new center stream release. So this was broken down into getting the repos, hooking them up into our update pipeline, adding them to our configuration system, which is Chef, actually provisioning and building provisioning, medium provisioning machine, and then deploying it and qualifying. And pretty much every step here involved fixing bugs, mostly on our end. So starting from repos, the repos come from all DCS, as I mentioned. There's the daily-ish snapshot or weekly-ish, depending on when it's got the shows up there. We didn't particularly want to scrape that with WGAD-R, because that seemed kind of bad. So we worked with Infra, folks at Fedora, to get an Arsync endpoint exposed from there, so we could mirror it using Arsync. And now we have these exposed on our public mirror, so if you want to access that and you don't want to hammer the Fedora servers, you can use the mirror on the face with the net. I believe we see that daily-ish, so it should be reasonably up to date. But we don't consume that directly internally. We actually snapshot it via another process that I talked about in a minute called runningOS updates. Now, this is only for the base repos, but of course there's also a Pell. So for a Pell, we use the ELN extras that I mentioned before. So we started populating two workloads in ELN extras, one for packages that we maintain as part of the CentOS Hyperscale SIG, which is where we do most of our upstream work within the CentOS project. And another workload for packages that are really specific to Meta that we felt it would be easy to have all in one place to keep track of. You can look this up. I put the screenshot here, so you can see roughly the volume. The giant jump was when I started testing development systems, which have a lot of random packages on them. So I ended up adding a few dozens there. Oh, that's what this looks like for. And these are all packages that we'll be maintaining then in FL-10, or at least assisting to maintain, because we're not the primary maintainers on all of these, obviously. Now, once we have the repos, we have to hook them up into our base pipeline. I won't talk about rolling-glass updates in detail here. I talked about this in other talks, and it's not terribly interesting, because it's fairly internal to Meta. But the idea is that for CentOS Trim, we snap short the production repos of stream every two weeks. We roll them out on the fleet across two weeks. We do this via Chef. And effectively, Chef will update the DNF.com from the machine around DNF upgrade. And it just works, more or less. When it doesn't work, people get tickets and fix it. In the case of ELN, this process is much easier to some extent, because we just made a really strange called ELN, which is going to be continuously updated. We snap short it every day initially, because that made it easy to eat very quickly and later every week. We don't really care about in-place updates for ELN, because what we really care about testing is the provisioning and the initial bring up. We don't particularly care about testing updates from one ELN from yesterday to ELN from today. So we just stopped out of the test for that and automatically promote all the snapshots. And this basically works. The only caveat here is that ELN composites can be broken, because we publish all composites regardless of state. So we need to filter for only composites that are tagged as finished. Otherwise, you end up trying to roll out something that is missing three quarters of the distribution. Also, ELN extras workflows can fail. They can fail, for example, if you add a package that doesn't build. When the workflow failed, the entire workflow is excised from the compose, which is obviously bad. So right now, we just keep an eye on it, but I need to actually write monitoring. So we do the same thing and validate them before pulling them in. Okay, now we have very close. We need to do the config management side. The config management side is fairly straightforward with one caveat. ELN is odd in that it really is Fedora. It identifies itself as FedoraOI. If you cat, et cetera, what's really success FedoraOI. The only way you can tell it to ELN is because it has variants set to ELN. However, everything else about it is sent us and we really want to validate it as if it were sent us. So what we ended up doing was something horrible, which was adding logic to detect ELN based on the variant ID and then monkey patching it so that it actually looked like sent us. So in our Chef code, we have methods that would like, if not sent us and those we return through on ELN. Do not do this. This is a terrible idea. And like, we did this because it worked for what we were trying to do, but not something I would recommend in general. After that, it was just the usual loop of like testing it on a machine iterating, finding there's all these packages missing and adding it to ELN extras, or all this logic is hard coding sent nine, but we can actually just flip the convert the gate and make it hard code eight because it's just a negative thing, things like that. Our open source code books are there. There isn't much that's ELN specific in there, but I put the link in case you're interested. If you use Puppet or Ansible or anything else, I expect it would be roughly similar. Frankly, I expected this to be a major pain and it turned out to be very straightforward. Okay, now we have conflict management that we want to do provisioning. Provisioning and meta is complicated. And to size that the issue, and because there's a circular dependency that you need Chef to have provisioning working and vice versa because the provisioning image is built with Chef, but you need machines to be able to test that Chef is working. We did another horrible thing which was converting in place existing CentOS M9 host to ELN with ENF Easter sync. To my surprise, this actually works. Not only it works, but it produces systems that still boot. Initially I expected to do this and then throw the machines away, but they still booted afterwards with minor, minor caveat. And I would not do this in production. It is a terrible idea, but it works very well for development because I was able to take development systems, convert them in place, iterate quickly on the Chef stuff, and then later when the Chef stuff was sorted out, get the provisioning images going, and then we could provision machines from scratch. The provisioning system uses an internal thing that is in particular useful outside of meta, but basically they're just images that get dropped onto machines, as you might expect. The images are continuously delivered and tested. So the provisioning system will build the image, try to install a bunch of machines with it and then tag it if it's good or not good. Right now we're at the point where we're effectively in feature parity with nine. We actually were able to refector quite a bit of this logic to make it saner and better support future distributions. After that is just a matter of testing. So we started using, we have a small set of machines we call kernel test that we use for doing kernel and hardware enablement work, which is very handy for these things because it doesn't run anything. So you just have to get the base OS working. So we got that going and that took care of most of the basic Chef stuff. Then we switched to doing the development servers. The development servers are great because nobody cares if your own development server is broken. There's no services running on it except your editor. But they have a ton of packages on it because they have to support all possible work cases from building the Android operating system to doing God knows what. So these are very useful finding packages that were missing. We didn't need it to add to Elenxtras. We didn't quite know. Also, we were able to work with the folks that maintain these systems to discuss, oh, these things you're doing here, maybe you should rethink it because it will probably break in the future. So this is kind of where we're at now. Right now, I'm working to expanding this to the container platform systems, which is the one that I was actually into the entire thing since the start because that's where most of production actually runs. And those will give us a good measure of whether this will actually work long term and also the container system has extensive unit tests that should be able to tell us when things break very quickly. So let's talk about what we got out of this. Took about four months to go from nothing to having provisioning working. This isn't four months of head down work. This is four months of spending time on this but also doing other work. I suspect if you were speed running these, you could probably do it in half time. We now have some dozens of systems running on Elen. I can't say exact numbers for reasons, but it's not a huge amount. I expect this will expand in probably low thousands at some point. I don't think we'll ever have a large deployment of this because it doesn't really make sense. And because these machines will effectively not be doing useful production work. But we have started conversational wider deployments because at this stage what I would like to see is different product owners at Facebook to start trying this out and evaluating, so say the database folks can use this to help the qualification for the databases in the future and so on and so forth. We've also began preparing for sensor in 10 because the work we were able to do with Elen made us aware of changes that might be coming in 10 that we should start thinking about. We are also able to start testing things like the NF5 because Elen is built from Fedora, it has the NF5 so we can actually install it and play with it and test it and start integrating it. Changes that have been deployed and discussed as part of Elen can also be integrated and discussed internally. There's discussions ongoing if you look at the meeting meetings for previous Elen meetings. There were discussions about for example dropping 32 bit packages. That's something that I was very happy to know I had time because I can start talking to the people that use them. And then of course we found bugs and we fixed bugs and I will give you a few examples. So this was the very first thing we found that when I put a machine up immediately after converting it to an RPM-QA and I had a shit ton of errors and they were all that weird invalid OpenGPG signature. So you might know that RPM for ATN ships with Sequoia which swapped the entire OpenGPG implementation that we want that actually works. Notably these validates packages and make sure the packages are actually compliant. It turns out our internal signing service was using a Golang library and the Golang library was generating invalid signatures by design. Literally if you read through the comments it says we know this is broken, we don't care. So that wasn't ideal. Luckily we found out that the Protomail folks at a fork of this library that was actually compliant. So we swapped it in. This fixed our signing service but of course we had all of the existing packages that now had to be fixed. So we had to wait for a few mass rebuild cycles. We still have a long tail of this because not everything is auto rebuild but it's at the point that it's manageable at least. If you have your own signing implementation I highly recommend making sure this is not happening. Also maybe don't use Golang. Now this isn't all the fun we had with Sequoia. After we had the packages we installed a system for scratch and which our key wasn't importing. Now you might remember I mentioned in stream nine the policy disallows sha1. Turns out RPM never validated the key at all. So when you imported the signing key on nine it imported just fine even if it was signed with sha1. Probably even if it was signed with MD5 I suspect. So this is a problem. We turns out the key was not actually valid. We were using a very old, well it was finally actually wasn't an old key. It was a key that was kept updated but they kept signing with sha1 for reasons. So we fixed that and then it was fine. There's actually a very handy tool that I got packaged into a Pell for this. That was already in Fedora that's part of Sequoia and I branched for a Pell because as you hear in Linter you can feed it a key and it will tell you in human readable form what is wrong with it. It also has a fixed option where if you have the private key for the key and it's an actual private key not an HSM or something you can just fix it and spit out a fixed key. That can be handy if you're your own key. If you have an HSM you will need to work with your security folks on how to fix this. You may want to talk to your vendors about this. I audited hours. We, I would say at least 50% of repos that we were consuming were invalid. And most of these folks when I talked to them they had no idea what I was talking about and it took quite a bit of effort to make people aware that this was actually an issue and they should do something about it. Also if you're re-signing your key make sure you don't actually invalidate your key because otherwise upgrades will be very painful for users. Which actually happened with at least one vendor from what I can tell. Okay, enough about RPM. The other fun thing was Utilinus. So ROI ships with the very latest Utilinus 239 which is a lot of fun features. It also had a couple of regressions that we managed to hit. One was easy, they changed the option parsing for NS Center. NS Center has some bespoke logic for option parsing. That changed lightly where you had to pass equal something instead of just doing option value. Okay, we fixed it. That wasn't a big deal, I would say. That was also easy to work around. The more fun one was the mount API. There's the blog post that explains the mount API. We discovered this when I started provisioning machines that they never came up and I would go on console and see a slew of error coming from system D. And then when I finally managed to find where the console password is, I would get in and see that the root file system was read only. And if you passed RW on boot, it would find but it would, for the like of me, I couldn't get it to remount and if you try to remount manually, you'll get inbound back. Turns out there is a bug in Utilinus that if you're running an old kernel, it doesn't quite detect that the mount API is fucked so it should use the old one. Finally enough, I was talking to Adan the other day and this was actually independently discovered in Rohite thanks to OpenQA. But we don't run OpenQA for ELN so on ELN this lived through. This is fixed in Utilinus by the way. I think I put up the RF2 back ported in Fedora. I don't know if that got merged but I'm sure it will get sorted out quickly there anyway. But this is a good example of something that we were able to catch quickly and get sorted out in a way that is hopefully beneficial for everyone. There's also a couple of other things I wanted to mention. By the nature of how ELN is built, it is like 98% like CentOS but not exactly like CentOS. So first of all, sometimes just people playing fuck up and it happens. In this case, we found that ODD was installed using the FedoraRollSet and not the RallyRollSet because there just wasn't any logic in the package for this. I guess when it was branched for nine, the maintainer probably branched it and then just updated the CentOS side of it and never back ported the changes. So that was easy. Put a PR to the conditional. We only noticed it because in Chef we disable ODD and the way you disable it is different depending on whether it's the Fedora or the CentOS RollSet. The more interesting one was SystemD. So you might know the SystemD SD thing called presets that defines what services will start up on boot. On first boot when you install the package. So it will tell you when you're installing this package should the service be enabled or disabled by default. The presets from Fedora and the presets from CentOS are very different because the use cases for the distributions are different. ELN uses the Fedora preset. So you might wanna keep an eye on this if you deploy this. The way we found this out is because Fedora is explicitly disabled systemD networkD, which we want to explicitly enable instead and we never actually did that. We just, because there was no preset for it, it would get enabled by default in our case. So that's something that caught up by surprise. I don't think there's, we've discussed this with the SIG, there's not quite clarity right now on how this is gonna be handled on the CentOS claim side after branching. I expect this will probably keep deviating because that's not really a good way to manage them. But that's something else you should probably be aware of. All right, so what can you do with this? Well as I said, CentOS will branch in soon. There is a thread on the belt that I would encourage you to read that talks about the plan, how the branching is gonna happen, when it's gonna happen, how this will work out. You also have about two weeks to get a system by changing Fedora if you would like and getting a system by changing Fedora now is a higher chance of it maybe ending up in CentOS stream. Definitely a higher chance of having to do less paperwork than later because once this branch is a lot harder to get changes into stream for obvious reasons. But also if you maintain anything that runs on stream or on rel and you would like to get a head start, this is a good time to start playing with ELM and trying it out. It's also a good time because once stream 10 is fully branched, ELM will transition to 11. And we'll be able to start doing this all over again. As I said, you can join the SIG, the SIG and Gausson matrix and on IRC. The channels are bridged. If you maintain packages in a Pell, you can contribute them to extras. That's the link to the ELM website in general that has information and details about the project. The documentation itself is on GitHub, as is most of the code that controls the build itself. And the Fedora minimization work and counter-resolver is what actually controls the set of packages that you can find there. And I would be happy to answer any questions. Yes? Yes, that would be very helpful. Oh, and the question was, we found this out because we didn't run OpenQlN ELM and whether that would be helpful. Yes, it would be very helpful. So I think in general testing composites would be useful. Which set of tests I think is something we would need to, yeah, we would need to sit down and figure out. I don't know if doing the same test that we do for ROI is useful. I expect a lot would fail just because it's a very different system. So maybe looking at what kind of tests are done as part of qualifying, stream, and rel on the other side of the house and seeing if he makes sense to pour those to OpenQa, maybe, that could be interesting to look at. Any other questions? Yes? Oh, yes, I meant to add this but I didn't know where to slot it in. Yes, there is support. There are shows for ELM. There is also support for Packet to build against the ELM. Yeah, so what we actually do is for our open source projects we use Packet to get continuous builds and signals so that when folks internally land changes they won't break the open source builds and we build this in copper for all the releases including ELM for packages to have help. That works really well. Neil. So it depends. There are ISOs. How do you install ELM with the question? So there are ISOs that you could theoretically use. I have never tried them. What I would do is the DNF install route process that you use normally to build images or you can use your favorite image builder. Then who is there, landed support for building ELM images in MKOSI the other week. So that's an option. I believe they should work with any other image builder. It should work with you, we probably do. Yeah. But yeah, in general I think this will depend a lot on what you want to do with it. Any other questions? So that was the question I actually had and that's where the discussion started. The question is whether we could fix the preset issue by providing a Fedora ELM release package with the REL preset. The problem is that we don't know what the REL presets will be because you can't just blank use the ones from nine because it's different enough. And my understanding is that the way these presets have developed as part of stream is that they start from the right ones and then over time they get more into the new ones. Yeah, I'll go to see if you have any questions. Yep. I don't think that the quality of the new ones is just start with the ones that are nine and then they get a job coming over. I would be in favor of having something because it would make my life easier. It's definitely, I think it's definitely something we can revisit. This would be a good argument to discuss in one of the next sick meetings. Yeah. I think that we have a proposal with Martin where he said we can look at what can't be changed. But perhaps you could have somebody that or a board that every time that we set change in the Fedora or I expected to open an issue on the ELM package so that people actually take a look at the changes and decide what to do. I think that's a great idea. Yeah, we, and to repeat for the stream, the proposal was that we have a process where whenever there's changes to the package in Roight an issue is filed to see if the corresponding changes meaningful for ELM and for the future stream as well. I think that's a great idea and it's probably something we should look at. Yes, Neil says that's probably meaningful only if the package is fourth. I don't think so, because you might have packages where you want a service to be enabled in Fedora but not in stream because it's not supported because it's not part of the default set, you know. You had a question back there? So, I haven't seen breakage from this just yet. I expect I will at some point. I knew about this change because it came up in the last meeting and to repeat for the stream there's a change in ELM where it will use X86 64 V3 as the baseline for X86 and this will cause breakage because anything other than as well I believe, don't quote me on that, we'll stop working. Anything, any hardware that is older than a certain GPU generation which is probably as well we'll stop working. So, we haven't seen breakage from this in production yet because the systems I was testing this on were all fairly new. I did start inventoring stuff. I am, I don't have hard data on this but I am fairly sure we would see some breakage from this and this is something we will need to deal with. Yeah, there's, I think there's a long enough window of between now and when stream 10 and real time would be released that I expected it should be less painful than it looks like now, hopefully. Yes. It might be changing. That should be. It should be changing sooner or later. Certainly it will change in ELM. I hope it will also change in Elmite. Yeah. But I'm pretty sure it will change in ELM and also it will change that even if you by default until you have the calendar two, there wasn't any support or there was no way to do it with the time generator. But now it's the manager. So, right now, if you don't use KVM, you can, or you cannot do it for KVM. You can actually run ELM because it supports ELM. Yep. And to summarize for the stream, one area that's impacted by these is VMs because KVM right now, by default, uses Kimu uses an unsupported asset that's older than B3 but that is getting addressed and should get backported to our right as well, hopefully. Yeah, but I mean, we should have passed for a lot of things. But yes, I don't disagree there. I haven't done much testing on the RecreationStack so I don't have signal on my end on this. I was going to talk to the Recreation folks on my side to see if they could put this up and whatever they do. Sure, we'll do, thank you. Anything else? Going once, going twice? Alrighty, thank you very much.