 So what we want to talk today, maybe I should, I wanted to list what tooling we have actually created for our things is. So ELN means there is a Koji tag, Koji build route where we build packages, but to make it work, we added the automation. So what we added is I think one of the ELN effort is a content resolver, which is driven by Adam here. And content resolver allows us to define certain sets of packages, which we want to have in the build route, but then content resolver creates the whole dependent from this list of packages and allows us to create a list of packages needed to complete the build route. So all dependencies are resolved in there. So maybe Adam, you can talk more about it. I can even share the links so it's like people can see what I'm talking about. But basically content resolver was initially developed. I wrote it for federal minimization. That's the link. And it was basically, hey, I want to see how HTTPD installation is big, right? So I can say give me the HTTPD package and all the dependencies and show me how big it is. And I can experiment with HTTPD on different base images or environments. They were like, hey, we want to define this thing called ELN. We have no capacity to rebuild whole row height. So we need to figure out the subset. So we just ask various teams just define what you actually want in ELN. Not all the dependencies, not all the build dependencies, just literally what you want. And they define 3,500 packages. And we just put it in content resolver. It resolves all the runtime dependencies. And that's basically ELN. And then we also resolve all the build dependencies for that and also build dependencies for the build dependencies. So it grew. And then we have this huge blob of packages that can build itself. And it's complete, but it's no more than we actually need. And that's what content resolver does for ELN. Cool. Thank you. And if I understand correctly, content resolver can be also used not only for ELN, but for different, I don't know, if someone else wants to create their own subset of Fedora packages, can they approach content resolver with this task? Yeah. Anyone can send a full request. If you open the URL, there's like at the bottom configs. And there should be plenty of things. So you can just contact me if you have question. And yeah, you can send in workloads. They called workloads these installations if you want to monitor them. And yeah, you can even submit a view, which is basically what we use for ELN to make a group. Resolving the build route is kind of trickier. We would need to figure out how to do that, because that's then actually by a different service, because we need to rebuild all the source RPMs on all the architectures because they're different. So that's beyond content resolver. So we're doing that on the side and then pushing it back. But yeah, the tool is available to basically anyone in Fedora who wants to do things. We already have CentOS Stream 8, you workloads just to text it. So you can find it there as well. Do you think it can be used also by Spins, for example, in Fedora? If I don't rebuild stuff, but if I'm creating a new spin, for example, can I do that? Yeah, sure. Yeah, this is actually... So the primary thing was to see the sizes of things that we don't ship as an artifact. So for example, we ship Fedora container image of Fedora ISO. We already know how big it is because we ship it as a thing, right? But we don't ship, like, hey, for example, someone installs a web server on the base image of someone installs these five things in this environment. Like, we don't ship that as a unit, right? So we can see what that would look like when people install it. So yeah, whatever you can install from the repo, this can monitor for you. Awesome. Yeah. So in the current ELN workloads, I think this is worth maybe mentioning that current ELN workloads are specifically managed by Red Hat teams, because the content resolver itself can be used by community, can be used by anyone to build different views of Fedora packages, and you can use it to build your own collections. But the ELN part would specifically want to have it mapped on what Red Hat wants in the downstream of the, in the inter-world enterprise Linux downstream. So the content of this ELN view in the content resolver is driven by Red Hat teams rather than generic community. So the process of changing, for example, the content of ELN parts, it can be suggested by the community that you change a certain workload, but then it should be reviewed by like Red Hat representatives and confirmed. This one thing which is important for ELN to be useful to Red Hat. Hi, Steven. Good morning. Cannot even imagine how much it takes for you to join this call. It took a loud alarm clock. Just one. It was a very annoying one. The one that just like exposed me to a simple lift before it just stops. Okay. Well, I wanted to just quickly respond to one of the things that you were just saying, because I know that's going to be a point of contention with forming a community around ELN. And I don't want people to think that this is going to be an extremely onerous process, that I don't think that I've worked at Red Hat for a long time. And one of the things that I've always found funny is the number of people in the Red Hat, sorry, not Red Hat, in the Fedora community who think that Red Hat is this corporate behemoth that works in lockstep and that every Red Hatter thinks the same way. It's, I honestly wish that we could every once in a while just at random show off an internal meeting. You know, they just surreptitiously record an internal meeting and share it, because we never ever agree on anything. But any more, honestly, we agree on things internally less often than we do when chatting about them in public. So the important thing I want to mention here is that if you make a proposal and you suggest something, you set a pull request to change something in ELN, the fact that it's going to be reviewed and approved by a Red Hatter should not, basically, the reality of things is that it's going to be approved unless it is clearly antithetical to the goals of REL. We don't want ELN to be a case of, oh, well, let's just push everything that's in Fedora into REL because realistically, we can't support everything for 10 years. There are packages in Fedora that I have regretted supporting for two years. Yes, I think it's important to understand that there will be a conversation, there's always a conversation. There's not like some hive mind in Red Hat who says, no, no, no, there are people there and there are some same people, many of them are same people who work with Fedora and we work together for a long time already and it's just the same kind of conversation. But we do need to preserve a certain aspect here because, as I said, we want this environment to be useful for Red Hat so that Red Hat can pull in resources. We try to keep all this negotiation around so that we can make it useful for as many people as possible, but keeping Red Hat in the loop for this. And also, I don't know if it was in your email, Stephen, there's like two things for basically building ELN as like the thing, which is like useful to Red Hat and then there's the infrastructure that just enable us to build ELN, even the way that you can build an alternative build route that you can define an alternative content set, right, like from Fedora. So that part is supposed to be useful to anyone and like go build your own ELNs, if that's useful, right? But like the actual set that we maintain, this is, yeah, that's probably where like some reviews will happen, but am I saying that right? I absolutely agree. And that's what I actually wanted to go through by listing what components we have in ELN. So we started with Content Resolver. And as we discussed, Content Resolver contains a view for ELN, but it can also be used like to create views for spins to create your custom views. So if you are thinking about some use case where you want to do this dependency resolution of certain workloads for container images for whatever reasons, you can go talk to Adam and you can reuse the stuff he did for Content Resolver in ELN for this particular purpose. And it's under Fedora minimization umbrella also can be useful for certain tasks and efforts. And also, sorry if I could just interrupt, one thing you mentioned was go talk to Adam. I would love if there's anybody at all in the community or who's listening on this call who would like to do that, who's a manager of a spin or similar. I would really love it if we had some people that came and were interested in having Adam teach them how to do this rather than putting Adam on the spot for every spin. I think the process is relatively simple. So I think it would be great. And that's a fully open source project. So the Content Resolver is on GitHub. It's published on the tiny distra things. And Adam, I think you have presentations for this already in Fedora Flock. It's a huge topic of its own. And it's definitely worth in the working from a spin owner's point of view, from remix owner's point of view, we can look into this further. And if it's something confusing or whatever, I'm not trying to build a generic tool for everyone, for everything. I'm mostly like, I see a problem, so I solve it. So if you make something just talk to me, we can figure it out. But there will not be features in case someone needs them. It's just me developing it. I don't have time for that. So if someone actually needs something, we can figure it out. But it only does what you actually need to do. So if you find there's something that you would like to see, it might be actually easy to add. I'm very open to discuss it. It's open source. Yeah. And it's open source. Yeah. I'm just defining open source for some reason. I think this particular crowd probably understands that concept. Actually, one thing I did want to note. This session isn't necessarily meant to be a just a round table or a panel talk. Anyone who has questions or wants to engage, please go ahead and request permission to turn on your video or post in the chat and we'll make sure we get you in here. Yeah. One of those corners, I don't know how. One of those corners, that's a button. Yeah, Steven, you may have maybe missed the first few minutes. Yeah. They ever pad link. So I have some draft of what we can talk about. But as you may see on the top of that ever pad as well, question and answers is what the top priority for this meetup. So please stop us at any moment and we will be gladly answering questions from the audience and just talking between ourselves. It would be more interesting. Please stop us. Save us from ourselves. Yeah, but maybe just to go through that components part to completeness. So the content resolver story is a huge one, interesting one. What are other things which we have? They are like smaller, but maybe also interesting to some people. What we have is we have the automation part. The automation part is quite simple Jenkins jobs, which are deployed to Fedora CI infrastructure. So we like I here work with two heads on like I'm Fedora CI SIG member and then Fedora ELN SIG member. And we kind of use Fedora CI infrastructure to do some ELN automations. And we have Jenkins pipelines, which the code is on GitHub, which gets triggered on certain events in Fedora Rohide. And then whenever built in Fedora Rohide finishes and gets into the Fedora Rohide repository, we trigger the ELN rebuilt of the same package. It's a very simplistic approach right now. And it has many issues, especially the painful one is the site tag thing. But even in this current setup, it's something you can reuse. And if you ever look into how to make an automation attached to Fedora Rohide build, go look at Fedora ELN pipelines, and you can reuse this this part in your flow and go talk with Fedora CI SIG and we can reuse some Fedora CI infrastructure to host your automation if it's something relevant to other Fedora projects, well, like other Fedora needs or your needs as Fedora maintainer. So this part is, as I said, it's on GitHub. Jenkins pipelines are Jenkins files. And like the current idea of this Jenkins pipeline is that it only calls a Python script, which does all the logic. So if you're interested, you can contact me, I can talk more about this or like whoever it's on Fedora CI SIG or Fedora ELN SIG. The other part of the automation which we did is compose creation. This is done through the same services as Fedora infrastructure uses for main Fedora composers. So it's also like something other people can maybe reuse, it requires more conversation because it uses the infrastructure resources and the ODCS service, which is shared service between, which is, as I said, the main service for Fedora composers. But it's also like some step towards making compose building more available for community members. So if you're interested in specific topics of building specific composers, we've continued images with whatever for certain use cases, you can also talk to us and we can share the experience and find ways how to help you and how to consolidate our tools around this. And yeah, the one more thing is the comparison thing, which I think is also interesting on its own. It's just a comparison script between state of two code tags or which checks the differences between the components available in these tags and like the versions and it can show you the status. We use it to understand the current status of ELN, like how many packages we have, how many have been synced with Fedora and so on. It also kind of can be reused for other use cases if you need to compare tags and we can look into it and make it more generic than it is right now because currently it's kind of hard-coded ELN parts the other, but it can be moved into better tool in the future. Other questions maybe about that somewhere? I'm checking the discord as well, but I guess we don't have a huge crowd there. Yeah, so one thing I want to talk about, oh actually we do have a question, is the plan to continue building periodic composers between now and RHEL 10? Absolutely. In fact, I think since ELN started tracking RHEL 10 we've had what, some 100, 200 composers or test composers since then? Yeah, the one goal, like the RHEL nine story is done from a point of view of ELN, but we do want, I think we proved for Red Hat that the ELN is valuable and so we now consider inside Red Hat ELN is considered as an important tool for next release and there is reasons to support building it, so as ELN said, we also want to keep going and maintain at least the stable composers and that's like the very least of what we want to keep on all the way until RHEL 10 and beyond RHEL 10 actually. And not only that, we've actually had subsystem teams within Red Hat approach us and ask us, okay, now that RHEL 9 is finished from ELN's perspective, can we start doing some experimentation towards RHEL 10 in here and the answer is absolutely yes. This is the perfect place and the perfect time because there's two, three years of lead time between now and the next release, there's plenty of time to break things and then fix them. Whereas history has shown us that if we wait until the Fedora release that is going to become a new RHEL and then start introducing features, it doesn't go well and we end up having to do all of our fixing in RHEL, which is not a good behavior, which is not good but open source behavior either, so. Okay, so answering Phil, when you say. Could you reset the question for the recording? Okay, so the question is, is this effectively replacing internal staging and just moves the manging of Fedora into a L-ish looking thing into the public view where people can contribute and see and use. First of all, there wasn't internal staging before ELN, so it is a new thing for Red Hat as well because previously when we tried to do a bootstrap, it was something like, when you have the whole thing in Fedora and you just need to turn it into RHEL in some rather short amount of time and we've no understanding how to do it, it was really a heroic effort of a couple of people to build this thing together. So ELN is our attempt to create that staging and we started to do it in public directly, so it's not a replacement of something, it's a new thing to continuously preview the RHEL before so that we're ready to branch at some point. And yeah, it's important to keep it in public, so public and contribute and preview, but it's also important to keep it in Fedora and close to Fedora so that we have insensitive for RHEL engineers to work with Fedora closely, not just forget about Fedora for three years, then look at Fedora and try to do something about it, but rather continuously look into the state of Fedora and work with Fedora maintainers on the ways how Fedora is developed and how RHEL should be adjusted to it. And alongside the content resolver now where people and subsystem teams have a much clearer picture of what their dependency chains look like, we're hoping that that will also translate into subsystem teams maintaining their dependencies in Fedora between releases, whereas right now, or prior to now, we've had a lot of surprises when people, when things went through a bootstrap in RHEL and then suddenly subsystem teams started the proverbial horse trading to get rid of maintaining some of the things they really didn't know about or want. So being able to do that in a rolling way between now and RHEL 10 and RHEL 11 and RHEL 12, one hopes is a good thing. Yeah, I also prepared some picture, which is not yet merged, but it will be eventually at the ELN dock site to give you a better vision where is the place of ELN with regards to Centro Stream and these ongoing discussions in the community of Centro Stream status right now. So ELN is basically the tool, the vehicle, which is used to bootstrap Centro Stream and RHEL, but it's not overlapping in the use cases. So ELN tracks Rohide, it's a Rohide story, and at some point, some snapshot of ELN becomes the origin point of the Centro Stream and RHEL, but ELN goes further with Rohide in the meantime. Other questions? I see a comment from David. Yeah, so continuous integration and validation. This is all about continuous integration, and this is all about doing it in public, right? So doing it together with the community and trying to get closer to the community in the entire line of changes on the enterprise Linux level. And it's about living our open source roots too. It's about making sure that we maintain we maintain that culture as well internally. As RHEL continues to grow, we don't want to, and people have worried too, since we were acquired by IBM, that we were going to become this more traditional tech giant in the world. And we really want to stay close to our roots, because that is the reason for our success, and it's the reason, frankly, for the success of many of our customers too. And if it's not broken, let's not fix it. Okay. Yeah, thank you, Phil. We try. We try to keep it and, yeah, to make this continuous open story. Let's hope it will work and let's see how it develops. It would be interesting, of course, both for Fedora Community and for Centro Stream Community, I think, and for RHEL. Okay. Other questions? If there aren't questions at this particular moment, we can talk a little bit about, I think, actually, we're almost out of time, aren't we? But we can talk a little bit about our plans for expanding the ELN SIG, or for really kicking it off, because up to now it's been mostly the people currently staring out of your computer screen doing this. But we would really like to open it to get involved, or to open it up to get more people involved. And I sent out an announcement last Friday on the Fedora developer list. Some of you probably saw it. Oh, I'm sorry, Miro. I didn't see the topics in the... So, yeah, actually, why don't we do that? Mostly, I was just going to comment that probably on Monday, I'm going to send out a follow-up on that to get together when it's good to pick our first formal meeting and then, as noted in that thread, basically, whoever shows up at that first formal meeting gets to be on the SIG. And then after that, they get to decide who else joins the SIG. So, if you're at all interested in helping with this effort, please keep an eye out for that email. So, Miro, would you actually like to... I think we can invite him into the session if you would like. Oh, good. Yeah, I think anyone can join if they want to. Oh, yeah. Hello, Miro. Hello, Miro. It's good to see you. What time is it there, actually? 4.21 a.m. Oh, we're so brave. Hello, everybody else as well. So, I just put my topics to read, but because there are something that I really worry about, but I also would be interested to discuss the technical issues above later. One thing that I worry about is that there is a big internal pressure to reduce dependencies in RHEL and to do it in ELN because that's why it's going to preserve itself for all 10, 11, and whatnot. And to do it with conditionals in the Fedora spec files. And to disable tests or disable something else, whatever. The problem is that then if we receive a contribution, the check will only run on Fedora. It will never run on ELN. So, we no longer know whether the ELN built functions properly. But we have this excellent tools for CI, either the Jenkins CI or the Zoo CI. So, we can run some tests on the CI instead to get rid of the dependencies. The problem is that both the CI's only run on Fedora. None of them runs on ELN. So, if I disable check in the spec file of if RHEL, I never run the tests on ELN. And whether I need to verify a change, I need to re-enable the build conditional and build it in ELN build route to be able to tell whether the tests actually pass on ELN, which is really inconvenient. So, at the end, what's more convenient for me is to have the tests enabled in ELN as well and then wait for the internal forking to actually disable them in RHEL. But that means I will have to do it every RHEL release again and again. Would, in Fedora ELN-based CI that runs on Dureqvess, I think this was something that was mentioned early in the ELN proposal, but I had never seen anything about it. Is there any plan about that? So, sorry, Steven. You're the CI guru, please. Okay. Regarding Fedora ELN CI, I guess we kind of postponed any kind of verification with ELN of Fedora height contributions, because also there was not explicit request from Fedora maintainers, and we also were busy just proving that we can build Compose out of this. But I really like your request, and I think what we can do is create a pipeline, maybe with Zool, which, for example, builds an ELN scratch build on pull request to Fedora Rohide is an additional test case. And then maintainers can use this result optionally to verify their contributions or to do some work with the code there. So, do you think the scratch build is enough or you're looking into like more testing? Yeah, I don't really care about scratch builds. I mean, as long as the check runs, then the ELN build will fail in the ELN. I would know, and I would know it's not just a dependency issue because of broken dependencies, but it's an actual failure, then I will know that there is a problem. But the problem is that I need to disable check in the ELN because there's internal pressure to get rid of the testing dependencies from RHEL through ELN. So the official or official, the omnipresent thing to do is to say if RHEL no tests at all, if Fedora run the tests. That's the way how you get rid of the dependencies, and you need to push it into Fedora spec file. Hence, ELN automation rebuilds the package, but without tests, and the only way to run the test is either on the CI or manually if I change to become back and run a mock build. Okay. Can I take a step back? Yeah, go on Adam. Just like want to take a step back and basically like, yeah, you presented like multiple problems. And so with ELN, right, you said like, yeah, there's internal pressure to remove dependencies, so yeah, we can either do it like internally and then just like fork the package and let like RHEL maintainers maintain RHEL packages and don't touch Fedora, that would be horrible. Right. So like with ELN, we're trying to actually put it like in the same spec file, so it like runs in both, right? So like whoever works on RHEL can actually do work in Fedora, right? Then about testing, like, okay, so you do a Rohit build, right? And you want to test it in Rohit and both in ELN. And this is kind of conflicting requests. Sometimes like, okay, as a RHEL maintainer, you want to test both, definitely, right? But then just Fedora maintainers who just want to maintain packages in Fedora, like gating them or just failing their builds on ELN that we didn't want to do. Okay, I don't want to gait anything. I just want to get the information. I think I understand the situation here is because when we use test in the check of RPM, then we kind of, we bundle tests into the RPM package itself and we cannot build the package without having all the test dependencies also packaged and installed. So what I think could be an option here is that if we define tests not in the check section of RPM, but rather in the tests in this git, and if we move the test there, then we can run this test no matter of the test dependencies because we can, on the test environment, we can install Fedora-Rohite dependencies easily to test the ELN build, but it won't be dependency of that ELN package and it won't be a dependency of a build to build this ELN package. So if we move tests from the check section to the disk git test environment, then we can add a pipeline which will test ELN builds with the disk git tests which are regular Fedora disk git tests. And then we will, like, this will not be impacting the dependency tree, it will be just testing with external dependencies of, but on these particular builds. I think this is what we do in RHEL right now, mostly because test packages don't follow the RHEL support cycle and we don't provide them as supported things. So we, in many cases, use test code from master branches from Fedora or from whatever places. So if we decouple test from spec file, but put tests in ELN, then I can create a pipeline which will run them for ELN builds and provide it in BOTHE as a feedback. Yeah. Rather have the feedback in the pull request, but BOTHE is also... Yeah. Yeah. Okay. I'll write it down in the items as a... One of my pet... Something that's been on my wish list for a while and I've been talking with Fedora QA about for a long time is whether or not we could actually extract the check section from RPM and actually just automatically run it outside of the... We have packages that have tests that don't work offline in Koji. So we have disabled the test conditional in the spec file and on the Fedora CI, we basically just say, yes, with Internet or with tests and build it with RPM build. It rebuilds the entire package again, which might produce a little different artifacts, but usually this is Python, so it's just copy-pasting Python files and it runs the check on the CI. I can send a link for a package that does that. I mean, that's an interesting idea. Actually, one thing that we might want to look into, I think, again, now that we've moved to ELN on to REL10 is early on we had intended and hoped for setting up a set of conventions with conditionals. It's similar to the way that... Well, drawing for some of the design choices around Gen2's use flags, but just essentially having a set of specific conditionals that everyone agrees to use so that if I have with tests, I make sure that all of my tests are under an IF has this and so on so that we can know that at any time if we just do a RPM build dash dash without tests, it will exclude the dependencies and so on. So I'll basically move it from instead of in the spec file, try to figure out all the distros and everything, if Fedora, if REL or whatever, you would just say, hey, I have this capability that you can turn on and off and this one and this one and this one, and then on the build side, you would say, okay, I want to build the package with these three, maybe do these two features and like this test or something. The one problem which we have with migrating checktas outside of Koji, I generally very much want to move all tests from Koji because then we can free Koji resources for doing actual builds and we can scale testing into Amazon cloud, which it is easier to scale the task decoupled from a built artifacts from infrastructure point of view. But the one problem which prevents it right now is the multi-arch support, unfortunately. That's why GCC runs the whole test suite of GCC during the Koji build and we post the whole log of this test suite as a like three gigabyte artifact, something and this out of three hours of GCC build, like there is one hour of build and two hours of testing which is three hours. It's less than an hour. Yeah, yeah. But unfortunately, we cannot do it right now because we don't have a possibility to scale into tests into multi-arch clouds. We have only x86 right now and ARM will come soon, but like the other architectures are hard. But generally, I think that decoupling tests from builds will be a good help and that also serves the minimization purposes. So this with test conditional at least could be a good step in this direction to for downstream developers or for whoever developed like revixes on top of Fedora or doing some additional combinations of Fedora packages, containers or something. Okay. Thank you for a comment. This is indeed what Miro brought about. It's not like completely removing them, but we need to weigh, disable them and so on. Miro, do you want to continue with your topics? I don't really care about the order. Feel free to go for topics presented by Contek. I really do like all of them. I actually wanted to talk a bit about the communication channels really. Okay, go ahead. I also am not sure if I feel that ELN on Fedora develop doesn't really work, but I'm not sure which options we should consider here. I recently started to use the discussions Fedora project or more and I find it easy to use and easy to make some discussions which are, I mean, it is easy to navigate discussions there for me. I feel like it's more open because you can just like anyone can just go there and you can link and then like at any point of time and even if you join late, you won't miss all the conversations and you can just see it. I like it much more than mailing lists, personally. But I also think that we want to keep compatibility of sorts with the mailing lists and people who use just them. For example, Miro, what's your opinion? Could discussions Fedora project or be a place for these kind of discussions or would it generally work? I have two different kinds of opinions. Personal opinion is that the discussion Fedora or whatever is horrible and I prefer mailing lists, but that's just my opinion and I know other people have different opinions and that's fine. And then there is a general opinion and I think that should be shared is that whatever the discussion channel is, you people should be there, listen and answer. So I tried to initiate several discussions on the today communication channel, which is ELN on DevL, whether it was the ELN CI I've asked right now, whether it was build order issues or whether it was something else and I never got them replied from any of them. And this is really frustrating because it's like a sense of message like, yeah, ELN is a Fedora project, we really want to be engaged in the community, but when people ask questions then when we are not there, I know you have a lot of that was a failure. That was absolutely a failure on our part. And mine in specific, because I had initially taken on the role of communicator in chief and then dropped it on the floor when things got busy and I did not properly take ownership of that. So I apologize for that. That was bad behavior on my part. That's fine. I mean, if this course would work better for you, go ahead. Don't hesitate to choose what works for you as long as you are there. I think it's really important to be able to communicate and discuss this openly. Thank you. I do think that a regular ELN CIG meetings will help here because they kind of get you checkpoints and we should review I think the issues and the mail mails at the meetings. So at least we will have this there, but yeah, we need to do something. Maybe this whole meeting is how attempt to formalize it. So we just know, oh yeah, ELN CIG. So we need to be looking into this place because it sort of started and everyone got busy. And then we need to make this work and there were technical issues and it's sometimes easy and it's horrible, I know, but it's sometimes just in your head. You have a technical issue. You need to fix and unblock people and then have communication. And I know how I'm behind an email and it's just sometimes hard. So having a formal meeting like what we actually forced to just talk to each other will definitely help. And this is like the attempt to fix this. I think we're not going to have time for any technical discussions today. We're quite late already. Oh, about seven minutes, right? Yeah, yeah. So let's maybe take a look what else we can discuss with the audience currently if there are other questions which we should cover. Or maybe let's figure out what's next, like where to continue this, like when the time runs out in seven minutes, like so everyone knows where exactly to go. So the next thing is first ELNC meeting. And I guess for that one we will send a notification and the status still on FedoraDevelop. And then on that meeting we may decide if we want to change communication channel or not. But the next one will still be FedoraDevelop and right after this meeting we should go and check what other ELN topics I left unanswered on that FedoraDevelop main list. The other action is I want to file an issue for this additional CI job for pull requests or not pull requests which would run this Git test on ELN build. It would be easier for us right now I think to run this Git test on ELN builds when they are done rather than on pull request because on pull request we will need to do additional step of building a scratch build before we can test it. But I will file just issue tracker for both of these tasks and we will see. I will discuss with FedoraCIC what can be done in this area and maybe Zoolfolks can help set up easier. In copper as well. Maybe we don't worry about the multi-arch in the initial pull request maybe we just do the you know just take care of x86 as a good enough for ELN for the first pass and then yeah probably okay but if it tells you tells and then like you can catch it earlier so at least that's better than nothing yeah right. We have one question regarding S390 containers I don't know the latest status of this Steven have you heard anything? Honestly that kind of that pull off my my radar a little bit but I suspect that the answer to that is we didn't have access to S390x resources until basically this week so I will bump that up on my priority list. Thank you for question yeah so I think we disabled S390 in the very old early days because we really needed the compose working because content resolver uses compose to analyze the dependencies but yeah we should probably revisit this and then take a look if we can fix it now if it's better now. Other questions okay so I think one of the craziest tasks right now still the side tech support this is the top priority for ELN to make to be resilient how do you this is the right word I'm not sure so to continue over next years we really need support for side techs and handling them in the right order. Yeah it's not so much the side tag for this is for the audience not for you obviously what the real problem is that right now we just trigger on tagging when the side tag is tagged back in we do a rebuild but the rebuild does not necessarily occur in the same order that the things were built in the side tag and as a result they they tend to fail or they are built if they're built in the wrong order sometimes they succeed but don't have the right options built into them or depending on the right version of the of another package so what we need to do is we need to build some tooling that will ensure ensure that we rebuild the content in a side tag in the same order as the content as it is built by the maintainer because we really right now it's a very manual process we've been just trying to keep track of whenever this happens and one of the SIG members has been manually rebuilding things in order this is definitely automatable but it's a complicated problem and we determined that it was too complicated for us to attempt to solve before we split off rail 9 and rail 10 but now that now that we've done that we have a little more leeway to play around with it and this reminds me of the other important question which goes beyond Elon SIG is it as they build an ID in the NVR I think we have two years to actually solve it and I had the fedora see I meet up yesterday at the same conference and the same question appeared it's like one of the most important infrastructure changes which we need to do which would allow us to rebuild package without changing the sources of a package only when build environment changes we would like to be able to rebuild the same sources and get new NVR for that without bumping release or version filled in the spec file and this is important for ELN because it opens abilities to fix the wrong order of builds and so on without touching the sources but it's also critically important for fedora see I it's basically it's important everywhere and I think we need to start some conversation about it more widely on fedora develop and like get more people to join and finally find a way to do it yes Phil that's a great comment indeed like we do talk about real eight real nine and real 10 in the open way right now and we've sent to a stream it becomes even more open and easier so it's definitely a huge change from what we had previously and we like we'll probably need to close up so as a closing statement from me I really really see that ELN brought many real developers like triggered many real developers to go look into fedora more and I think this is a very good outcome for fedora while there were some issues of course you know when you trigger people to work with a community project it's not always unicorns and rainbows yeah you need to also teach people how to work with open source project and sometimes you need to you have an errors and issues in in the process which you need to then resolve so it's not that fedora suddenly have a burst of awesome solutions and good energy we also need to help from fedora side to kind of navigate and to help people to integrate with fedora better but at least it gives fedora the possibility to have this conversation and to drive this conversation and to direct this like downstream into the right area of the upstream and then have it more useful for everyone so this openness and this communication is driven by this link we created from fedora to rel and by by the fact that we made it more obvious for everyone I think it's a good outcome in the office initiative to you Stephen I don't actually really have anything to add you you pretty much covered it so yes we are we're in a better place I think as far as the sustainability of rel as well like like I said earlier the one of the most important things to me for years and I've been advocating for in red hat is doing much more of our rel prep and rel you're talking about rel getting getting people involved earlier and now with eln as the basis for the next major release of rel and sent to a stream coming out as the as the stabilization and minor release a train of rel I think we finally have a good proper you know beginning to end story of of open source for red hat enterprise linux and for fedora and for sent to a stream and I think they I think they fit together a lot more cleanly than the well the house of cards that they were up until this point so it's like nothing really often really early like just like having smaller bumps is better than than just like large fire and screen sorry sorry philt when I said stabilization for rel I'm specifically talking about stream for the dot zero not stream for subsequent releases