 Good afternoon. How's everyone doing on day two of Flock? So we're going to go ahead and get started up here on the main stage. We've got the, I should have had my notes up here from the full name of the panel, but we have the upstream collaboration and cooperation in the enterprise Linux ecosystem. We have our four panelists and our moderator up here who will, Matthew Miller, who will do some introductions in a moment. Facilitator. That's our facilitator. I just wanted to acknowledge first, you know, generally with Flock we really try to prioritize the diversity, equate, inclusion side of things and I realize that we try to avoid the manholes, but due to a number of circumstances this year with travel and the logistics, not the ideal, but you know, something I just wanted to acknowledge and point out that it's something that we were aware of and I hope that we can continue to bring more diversity to some of our panels that we have at Flock, but I'm really excited for the session and for what we have planned and I'm going to go ahead and hand it over to our Fedora project leader to kick us off. Thanks, Matt. Hi, everybody. So everybody in the room knows me. I'm Matthew. I am the Fedora project leader for the benefit of anybody watching this for drama online. I think we're planning to not have drama, so you're going to be disappointed. Sorry. But I hope people are walking out of the room now. I thought I'd start a little bit by just talking about myself and why I'm here, how I got to this because it's actually a little bit relevant. I've been the project leader for 10 years, but I have been involved in Fedora for a long before that and I got involved there by working in a building something called BU Linux, which was a downstream distribution of first Red Hat Linux and then of Fedora and of CentOS for some of our server systems. So I've been a rebuilder, so I kind of have, you know, I know what goes into this and I also really care a lot about Fedora and it seemed like an important conversation at the time we're at right now. This conference is a community conference. It's not a big corporate event, so this is in some ways a little bit of a different thing because people here on stage are kind of speaking for their projects and normally we are all individuals. So I wanted to start with asking everybody to introduce themselves as who they are, how they got involved in open source, separate from the project that they're working on, and then a second round of, you know, kind of what hat you're wearing on stage and what project, you know, how you're involved in the project that you're involved with. So go ahead and start. Wait, I also want to say thank you, Neil, for proposing this as a session here. Yeah, absolutely. I'm super happy that we're able to have these conversations and this while a little bit off, you know, kilter I guess for Flock as a usual subject, I think it's an important conversation to have. So anyways, I'm Neil Hanlon. I got into open source actually with Greg with the Forman project many moons ago writing a DHCP plugin for the company I was working for and trying to open source that. And I think that's part of the process of trying to open source things in the company I was working for at the time was new to them. And it was something that I was trying to pave the way for. And eventually that led to my involvement with Rocky and eventually, you know, being hired by CIQ to work on Rocky and so that's the kind of journey I've taken in the open source world. And it's been, you know, more of a dream more or less to be able to do this in a full-time capacity to enable the community and also with myself be able to contribute back to the communities that I've benefited from for so long. Hi, I'm Brian Exelbeard. Thinking about open source, I first encountered open source in a role that I had as a developer and the idea that I could get access to everything in that operating system and extend it was new to me. I wound up writing some drivers for PAM, that's what it was, PAM. We were long story short, I extended PAM. Y'all don't want to know the story. At the time, my management was opposed to us releasing this code because they thought that somehow this would mystically cause all of the competition to be better, which was not true because this was a terrible idea. But that being said, after that I wandered for a while in the desert of academia and wound up at Red Hat. And while at Red Hat, I got to work on documentation in the Fedora project. I got to work in contributions around calendaring and some other stuff that Red Hat was doing in the Centos project and then became the F-Cake as I mentioned yesterday. And after that wound up in the RELL business unit, which is why I'm here on behalf of RELL. And I'm, like Neil, I'm grateful that you proposed this. I'm actually really excited by this. And even though it's a little quirky, I hope this sets the basis for something we can do more regularly. I've been excited about the co-location of Centos Connect here and I think this is pretty awesome. Hi, I'm Davide Cavalca. I started using Linux in like 98 or something. I think my first significant foray in open source was I was, I started using it and contributing it and eventually helping maintain a small embedded Linux distribution for media center systems. Like back then where you'd like have like a six megabyte ISO, you'd burn a lot of your movie to watch movies called Geeksbos. That doesn't really exist anymore. And then I was fortunate enough, I kind of was able to spend most of my career working on open source and around the open source base. So I used to do small business consulting back when I lived in Italy and like setting up mail servers, those kinds of things. And then these days I work in Meta on a Linux user space team where one of the things I get to do is engage with upstream projects and upstream communities to try and like take the work we do internally in Meta and bring it alongside the world and there. And that's also what eventually brought me to working closely with Fedora with Centos without these communities. Hey, my name is Jonathan Wright. I got into Linux early 2000s. My dad had a box set of Red Hat 9 and I believe it was SUSE7 and I wanted to host websites and TeamSpeak servers for some video games I played. So that's how I got my start in it. And then, you know, from there it just continued on. I got into CentOS probably 2005, 2006-ish. I've been in web hosting for about 15 years now and being in web hosting, it made it a pretty easy, a pretty natural fit to get involved with Alma Linux as it started up because, you know, web hosting is very heavily or has been traditionally very heavily reliant on CentOS. So I got involved with Alma Linux and that was my first opportunity to kind of contribute back to open source. From there I met Carl George at open source summit a couple years ago and he encouraged me to become a packager and later sponsored me. So I got into that, working on Apple and then naturally ended up on the Fedora side and yeah. Cool. Yeah, so now let's do the, what hat are you wearing here today thing a little bit? So talk about your role in your project and how you're representing the project here today very briefly. I'm not wearing a hat. No hats. I will be representing Rocky Linux today. I'm the infrastructure team lead. I head up all of our operations basically for building and running and releasing Rocky on wherever it is. I'm representing well today. I just didn't get the memo about shirts and I only packed one red hat shirt for this trip and I wore it yesterday and it's not clean. So I spared you all by wearing a better shirt. And I am on the stage today because I am a technical business strategist in the Red Hat Enterprise Linux business unit and my area of focus includes our upstream communities. I represent the Santos project. I serve on the board of directors for Santos as of a year and change with Amy, who's somewhere there and Sean, who's the community architect and other folks of course. I also co-founded the Santos Hyperscale SIG within Santos and generally contribute to the project where I can. I'm representing Alma Linux. I'm like Neil, the infrastructure lead. So mirrors, internet, presents, stuff like that. Pretty good fit for web hosting. Cool. So I don't, my days are, it was yesterday. Brian Becks gave a speech, gave a talk, it's not a speech, I'm not a weird, a talk. It's what does Red Hat want, which is a traditional talk. We've had it flock for a while and so people who might not be familiar with Fedora and flock might think, oh, yeah, Red Hat's a big sponsor. Of course they get a sponsored talk. That's actually not where this talk comes from. This is something that, so Fedora has been sponsored by Red Hat since the beginning and sometimes communications between people in the community outside of Red Hat and Red Hat the company and people actually working on Fedora inside Red Hat and Red Hat's like business once and whatever have been really confusing and unclear. So this is actually an invited talk, the Fedora community together asked Red Hat, hey, could you just kind of tell us what you want instead of doing weird hints? There's a thing inside Red Hat where the business unit complained about how they always get pizza that they didn't order. And from a Fedora perspective it's kind of like, so there's these weird people in suits like outside our pizza shop and they look kind of hungry and they're sniffing around and like, I guess so, you know, sometimes we feel sorry for them and give them some pizza and then they complain about it. So instead of that, we thought, why don't you come tell us what you want from the community? So that's kind of the origin of that talk. So having had that though, that seemed like it'd be in good spirit to ask all the other projects, you know, what does your project want? So this is very much a with a hat of your project on like, what are you trying to do? What are you looking for? And kind of it, I want to kind of maybe do two rounds because one is kind of like a general like here, so the project is all about. And then I want to talk more about Fedora in specific and kind of a second round of things. I think that for that it's sort of two pieces. And one aside of it is what the Rocky Linux community and our users are wanting and or expecting of us and also what we want as a project of, you know, the Rocky Linux enterprise software foundation, which has the Rocky Linux project underneath it, which has a board. And we ultimately are set up to act with the best interest of our community. So what we've heard from our community is that they are still wanting what CentOS was and what Rocky Linux and all the Linux are, you know, CentOS, a downstream rebuild basically of Red Hat that can be used for free. What we want as a team, a group, a board is to collaborate with all of our upstreams, CentOS, Fedora, et cetera, and the broader enterprise Linux community, which involves more than just the Red Hat derivatives, but SUSE and everything else that comes in that van. I mean, Troy Dawson and the Apple, state of Apple yesterday, I was talking about all these weird Linux distributions that ended up showing up in mirror stats and such that may or may not be real things or may or may not be real at all. So I think from that perspective, we very much want to collaborate and figure out what the future looks like and not try to define it by ourselves because that's not what our project is about. It's founded in a way right to not be beholden to any corporation, and we're still as an organization figuring out what that means, and we obviously can't do that in a bubble either. So we want to collaborate for not only the good of our project, but for the good of all of the projects that have been spawned in the past few years or even before that. I want to also point out that Troy Dawson not only works on Apple, but also made this amazing shirt that I'm wearing here. And you cannot get the Fedora logo on Etsy, but you can get the, a bunch of other ones there. So let's plug his store. That's great. Bex, you want to do a abbreviated version or do a spacer? We want a double pepperoni pizza, and we want you to remember we haven't worn a suit in basically eight years. No, I won't repeat my whole talk from yesterday. Well, the way I would summarize it is our ultimate goal is vibrant communities that are advancing open source, that are a place where we can meet everyone and contribute and collaborate together. Our upstream first philosophy is one where everybody benefits if we all go as far upstream as we can with these collaborations. And I think that, you know, this stage represents an opportunity for us to continue that collaboration. I think for Centus, the main thing that we want, that we think would be important to develop is a community where people can feel comfortable contributing and advancing the state of the project and the ecosystem as they see fit within, within the boundaries that are available. I think for Centus specifically, there are various intrigue points that one can use to that end, like their Centus trying proper, but there's also the six, and I think the six are a great example of a space where if people have a specific niche that they would like to focus on their specific thing that isn't currently well-adressed by the project, by the ecosystem, where they can work on develop that, and then use that as kind of a launching pad to see if their idea is viable, see how that will can work out, develop it, deploy it, see if there's interest and see their customers, customers in the broader sense, not necessarily paying customers, but people that have an interest and want to contribute, but not just be users, but also be contributors and be able to, be able to work on it together. I think that by far that's the main thing we would like, we would like to see in this environment. So the, the mission of AlmaLinux is the same now as it was when we originally started that the users that came to AlmaLinux want a downstream rebuild of Red Hat. They want rail-like features, but what's been changing over the past month is how is the value to the entire ecosystem defined, and I'm sure we'll get into it more later in the talk, but we're, we're shifting a little bit with the community as, as we announced a few weeks ago, and I look forward to, you know, at first it's easy to get hung up in negativity, and only looking at the downsides of things, but as we've had time to think about it more, we're very excited for the future of how we can collaborate and contribute upstream in different ways that add diversified value to the whole ecosystem. Cool, thank you everyone. Yeah, so part of the Fedora mission, which I cannot say word for word offhand, which is terrible, but is to make a platform that others can build on to do cool things for their own users. So we want to make nice things within the Fedora project solutions for our own users and for ourselves, things we care about. We also want to make something that, you know, can be built on. That's one of the things with the Asahi Linux announcement yesterday. We want to be a great place where people can come and do really interesting projects like that, and on your downstreams, both RHEL and everybody else here, like this is part of what Fedora, we're here for that really. So I guess this is kind of a two-part question of, you know, what can Fedora do for you, and you know, conversely, what can you do for Fedora, in addition to your generous donations to this conference. So obviously, thank you. So for what Fedora can do for us, you've already done a lot of it. We've built Rocky and used a lot of the open source code and infrastructure as code and standard operating procedures and a lot of the really great stuff from Fedora. There's much great stuff in Fedora. So we've used a lot of that and taken it and modified it slightly for our purposes, but not in a way that, I don't want to say it requires, but it facilitates upstream collaboration for that. But we want to continue building off of that with the tooling and everything that we built upon, because there is such a great history and tight integration with all of the Fedora tooling, and even more than that, a community of developers that already know all of the Fedora tooling, specifically for packaging and such. But we also, as a organization for our special interest groups where we are trying to add our own diversify whatever on top of Rocky, we always encourage and have from the beginning collaboration with upstream special interest groups and parties. Our Wiki has said, and I mean, I think on our main page there for how you can get started with Rocky, it's pretty much from writing that page, we're like, if you want to collaborate with the overall Enterprise Linux ecosystem, you can also just go and collaborate with the Enterprise Linux ecosystem in Central West Stream and Apple and the existing places there. So for as for what we can do for Fedora and what we plan to and are doing for Fedora is, you know, I personally have been involved with Apple and Fedora for the past few months now, a similar kind of story from Jonathan, getting some mentorship from people like Carl and some other folks, and I'm aiming and working to bring more of the Rocky Linux community into that ecosystem and help collaborate and build those relationships with the packages and all of the different things that we're trying to do. I don't mean to repeat you, Neil, but it's a great answer. But yeah, Fedora is actually doing a lot of what we need. You have sat at that intersection, as I was talking about yesterday, between integration and innovation, and I am grateful for the way that you push things forward in that space. Candidly, this is partly me, but I really like it when Fedora pushes back on Red Hat a bit. I love the fact that ButterFS is in parts of the Fedora editions. This is good. We want to see that kind of stuff proven out. We want to see that demand shown to us, and sometimes we just literally need to see it in front of our faces, and I want to continue to encourage that. That's where System D came from. Yeah, everybody is like Red Hat's forcing System D on us. Conspiracy theory. It was Fedora forced it on Red Hat, which is great. That's how this is supposed to work. You've been a great home for ELN, where we do the next major release of Rails development. I'd love to see more like that kind of collaboration. Obviously, we provide a lot of sponsorship to the project, lots of this event, Candidly, and other things, and I don't see any of that changing and going away in a direct sense. We're certainly going to shift around how we do it at times, based on what's going on in the universe. But yeah, you're doing it. Keep on doing amazing things, and maybe one last echo point. Continue to fill that white space. Maybe that's the place that I should mention this. I had the privilege of sitting in on the Fedora server conversation with Peter Boye earlier, and he mentioned the server group's desire to work on single-board ARM NAS devices. That's not a red hat objective, but that's really cool, and I want to see that come to fruition. I'm also happy that Bader Fess is in Fedora. But kidding aside, Fedora is the ultimate downstream for everybody on this stage, and the thing that I think is important to stress is that when people are looking at where to contribute, it can be difficult on time to figure out what at what level do you want to contribute on the level stage and what you're interested in, and contributing at the level of Fedora versus at the level of CentoStrain versus at the level of, well, contributing to realm, meaning like filing back to realm, unless you're a customer or contributing to your Alma. They're all different layers of kind of the same cake here, and like that's something that I have personally been struggling with, like, communicating to people and making sure people understand where they want to sit here, because, and that is something where I think we could all do probably a better job in making it clear how this works, because generally speaking, if what you are really interested in shaping the future of the ecosystem and distribution, I would say the work you want to do is in Fedora. Fedora is where the leading edge is, is where this work happens, is where all the new technologies get integrated first, that is by far where you want to be. If what you're interested is in working on things that you want to see land in the next minor release of realm or in stream, you probably want to work in stream. If what you want to see is working at the level of the rebits of the RRL projects, that's probably where you want to work. And I think figuring out where to sit there is something that is important. But yeah, Fedora has been absolutely fantastic, and I think when you look at things, things like Epel provide tremendous value for the entire ecosystem, like in you're trying to say something. This panel is great. I would like to have another round of people saying nice things about Fedora after this one. That's, that's all right. Yeah, I mean, I'm serious, like it's, I personally think, generally speaking, where you're setting up a system with any of our distributions on the stage, there are very few instances where it makes sense to use it without Epel enabled. And that is something that provides tremendous value, both in the immediate and in the potential, in that it makes it possible to get any package that is in Fedora now potentially available for the wider enterprise ecosystem if the leader rises. So that's a great example. An ELN that you mentioned is another great example of something that in the past used to be nebulous and invisible, which was how does the new URL major release appear from the ether? That now is a thing where you can actually sit on a meeting on IRC every Friday and discuss how this is working and look at it and get your packages in a workload so you can get notified when they break and actually fix them ahead of time. Like all of this is fantastic. I think you took on my talking points. No, I was I was going to lead with Epel. I mean, it's especially over the years as Red Hat has gotten more behind it officially. It's a huge resource to anybody running anything in the EL ecosystem. And I mean, that's largely in part thanks to Fedora contributors who otherwise don't have a direct hand in the development of EL. And prior to Sino S stream, you know, changing and becoming a little bit more open to contributions getting into the EL product sooner, Epel was the best way to for the community to contribute. And in turn, one of I feel like the most valuable ways that rebuilds can contribute back is to contribute to Epel. I so full disclosure, Mike McGrath is my boss's boss. So but he's also a long time Fedora contributor and actually is the primary person or at least one of the main people who started the Epel project back a long time ago. So we have I know Mike spent the villain of the open source internet for a little while. So let's let's give him some credit as well for like really starting that project. And actually, this was actually another thing that Red Hat was like, can you? I don't I don't think you should do that. And Fedora did it anyways. And look how it's been a lot of benefit to everybody here, obviously. For people who don't know, rel and the downstream distributions are generally much smaller than the package set in Fedora Linux, which is in the tens of thousands of some amount. But it turns out that so, you know, when you want to commercially support a bunch of software, that amount of that's it's pretty untenable to do a really good job across all of that. So it's a more restricted set. But all of that software is actually useful. That's why someone put it into Fedora in the first place. So Apple is rebuilds of that Fedora software against Enterprise Linux base so it can be used in a much wider audience. And in fact, if you some of my previous charts and graphs talks, I haven't done it this time, but the number of systems running Apple is order of magnitude higher than those running Fedora Linux. But, you know, it isn't all isn't all a numbers game. But it's also one of the things where people working on things in Fedora have a really big impact beyond just the things they care about in Fedora Linux itself, which is sorry, I'm speechifying a little bit, but I think it's and it fits into all of this. And since everybody mentioned it seems fair. So those are all the questions that I had. I guess, do people in the audience here have questions and is somebody else a mic runner or am I the mic runner here? Awesome. There's one back there actually. Cool. Yeah, that's fair. Let's even have questions specifically for Jonathan. Was there a question? No questions at all. Everything is settled. Mike dropped. They don't have to just be for Jonathan. Yeah, yeah, that's true. That was. Yeah, yeah. And they can be about pizza. Hi, I'm Shaza. I'm an intern at the hot and I work in the stormy project. I'm new to the open source contribution and projects and all of these. So I just have a quick question about if I would like to contribute to any of these projects after my internship as a student. I'm a computer science student, by the way. Would there be any level of difficulty in between these, I'd say, four projects or five projects or all the projects that we listen to throughout these two days, in your opinion. So, like, you know, more like I'd say beginner stage that I can start from. Is there any of this project that is easier than the other or are they all in the same level? I'm going to let someone else answer that and hope they all save the door. I was going to say for Dora Docs, so I think I'll go first, I guess. But all the projects have varying degrees of where you can enter into them based on your commitment level, your experience level. And I think the most powerful thing about all of the community is being, you know, facilitated or part of or adjacent to them, watching them go on. Like, they're all very, very inclusive and welcoming to new people, whatever that level is. And none of them push you away or anything like that. So I would say something like low, there are parts of the projects that don't get a lot of attention like documentation that can be tremendously valuable to participate in. And there's there's tons of other ones besides documentation. I just can't think of them right now. I was just going to very front say I think your best bet is to consider starting an either the Fedora project or the Centos project, primarily because rel is developed in Centos stream and then major releases in Fedora and directly contributing code to the rel product is basically impossible without the internship that you already have right now. But on the other hand, these projects are the entry points for that. And I like the way that you expanded the envelope of contribution there like contribution is a lot of things. And so I think it really comes back to what are you looking to contribute to is as a computer science student, I could see you wanting to take a specific sub system apart because it has some area of interest that you have. And that may actually be better in Centos stream because it's a more slowly moving code base where you could look at very specific kinds of tweaks and changes and then get advice from the existing contributors and engineers in that project of where upstream that really needs to be socialized because it may not be directly appropriate, but it's actually a great entry point for direction. On the other hand, you may also be interested in some new language stack that's being used and and bringing that into the Fedora project is going to also bring it into the Apple project, which is going to get it wide range of of impact and visibility. And of course, like to give you to give you a small assist there, there's also like art and other design. And I will tell you right now, I think all of the projects would really love some help with promotion and those kinds of things. But those may or may not be your interest and you shouldn't feel pushed to them. Yeah, thank you so much. I was going to add to that just a little bit. From a technical standpoint, if you want to get involved, the I mean, you got to think Fedora has been doing this with community involvement for what over 20 years now. The documentation is, you know, almost unmatched. So to get in from a packaging standpoint, for example, Fedora is absolutely the place to start. Recently, I sent some of my first merge requests into CentOS Stream. And while the documentation there is lacking a bit, I was able to do that with a little bit of outside help from some friends. But the process is so similar to Fedora because that's where it all starts. So if you want to contribute packages as a, you know, be a package or something like that, absolutely start in Fedora because everything is really structured around how Fedora has been doing it for so long anyway. My personal recommendation is to focus on something you find personally interesting that you that you would like to see to see in the distribution. And then, yeah, I think I agree that Fedora probably has the widest breadth of well-developed opportunities for newcomers. Fedora also has things like the Joint SIG that is explicitly designed to help newcomers get started on the project, find, I believe you still have a thing that I find, get paired with a mentor or like something like that. That's definitely helpful for people new to the project, new to the space, new to the environment. A lot of these, a lot of distributions, like that from distributions here, but distributions in general, in addition to the code itself, there's a lot of process around things. And there's a lot of ways that people do things within the space where people have strong opinions on the way to do things. And I think it's always fantastic when there's new cameras that point out, no, this is stupid. There are much better ways to do this. We should think about it. And I'm very much radish when that happens. No, this is stupid isn't necessarily the best way to introduce yourself. But yes, I agree. Yes, it's a fairly, it's a fairly common one. And some of the people who started out that way, you know, actually come around and become valuable parts of the community. And then later have somebody tell them that what they're doing is stupid in a nice virtuous cycle. But yeah, like finding something that is frustrating and you think it ought to be better is actually a really good place to start because you know what your pain is and getting rid of it is, you know, satisfying, I guess. David. OK, so that's a great way to start with. I want to I want to accept Fedora and rel from this. I'm familiar with those products. I have a personal server that used to run CentOS. I like to run an EL distribution on it, but I don't want to deal with rel on this personal server because it doesn't matter. Why would I choose Alma, CentOS stream or Rocky or how do you differ from each other and which one would make the best choice for that? You're welcome. There's there's the fighting. Let's go. Thank you, David. Well, CentOS streams always broke. It depends. I hate it, but that's it's the answer in everything. It's when I was doing network engineering, it's the answer there. It's the answer in computer science. It's ultimately, you know, at this point, CentOS stream is just as fine as anything else to deploy. It's it really ends up just being a matter of preference, I think, right? And if you if you exclude rel, like let's say rel for some reason somehow didn't exist and source RPM is just a period of the ether, I think it's really just a toss-up of like what name you like or what community you like. And I think the the silver lining of this entire situation that's going over the past month or so is that the enterprise Linux community as a whole has been brought together like forcibly smushed maybe, but it's a silver lining that I am personally really grateful for because it's enabled us to have conversations between projects like Rocky and Alma and Sousa and CentOS stream and all of these different projects that are part of the enterprise Linux space. The the EL community IRC channels grown massively and there's conversations good and bad that happen there. But ultimately, I think it's a net benefit for the overall community. And that's why we're all sitting on the stage right now. I would say from the point of view of stream. So full disclosure, I ran stream at home on the router in my home, which is just a Linux box and it's fine. I would say running stream is roughly comparable to if you're used to running CentOS Linux is running CentOS Linux with the proposed updates or candidate updates repo enabled with the difference that on stream all the all the updates that go out to go through the the gating and the testing they go through. So assuming the gating and the testing on the rail side is set appropriately there they should be higher quality when they go out. I would say stream gives you slightly faster updates because you don't have to wait for things to go to point releases and it means you don't have to deal with point releases, which is something I personally like a lot because I don't care about point releases. On the other hand, if you were running software that is specifically tested against the rail 9.1 or whatever and you want to be sure you're running, well, if you're running software that needs to be certified, you should probably be running a rail. But if you're running software that for whatever reason you don't care about running rail and you still want to be pinned to a specific point release, then I would recommend not running stream but running one of the rebuilds or running a rail. The other thing I would say is with stream, you probably have a slightly faster cycle if you want to get changes that you eventually want to get into the rail as well, because if you contribute to stream directly, that's where the development for rail happens so that's where the changes will eventually go into for rail as well. And again, assuming the process works the way it's supposed to, it is reasonably, reasonably quick. The thing with stream is that every package has different maintainers like in Fedora. So the response you would get depends on what the maintainer does, what the priorities are and all of that. So whenever I do this quite a lot, because for my day job, I end up doing quite a bit of work, like by reposting contributions to stream. So you would have cases where you point out, hey, this package should really be rebased. And it gets rebased within a week. And you have cases where you need to have a conversation and a bit more of justification why you care about that. Just a little bit further, you said you want to run an enterprise distribution, enterprise Linux distribution. My question back to you would be, how do you define what enterprise Linux is? Yeah, I realized I was thinking about that while you were talking. So to me, for a personal server, I want to run an EL distribution because I want the slow moving pace of the system. I don't want it to change out from under me. I want it to run the web server, whatever I'm running on it. I don't want to have to change it at the rate we change things in Fedora. So basically I want to avoid home IT day like twice a month. Seriously. And that's my definition of it for a personal server. Not for my laptop. I want to sneak in another possibility, which is you can run Fedora CoroS on that and not worry about it and have your workloads in a container running one of these, or running UBI, which is a rel based or magically turns into rel if you buy a license. I think that's the way it works. I know that's an option, but I'm trying to pick between these three. I'm just throwing out another one. You have four choices, sorry. So far we've had Rocky Linux recommend CentOS Dream. We've had CentOS Dream recommend CentOS Dream. So what does Alma recommend? Alma Linux. Oh, okay. You haven't said anything that rules out anybody up here. So the things that start looking at then is going to be support lifespan. Do you need support? I don't guess you need support. You're doing it at home. You're messing with it yourself. I'm on the supply side of bugs. So which community like the best? Which community do you like the best? David said he's on the supply side of bugs. I just wanted to call that out. Pick a community, get involved, go with it. Okay. Thank you. It is also fairly trivial to build your own CoroS like spin with whatever distribution you want or mix them all together. Yeah. And if you package for Apple by chance, you can use any of the distributions to build your Apple packages against using the Fed package utility. And they should all work. I will say to add on to David's point, there is movement in CentWest Stream and stuff happens. Like there is breakages that occur. They don't happen all the time, but it's also a very different situation than it was three years ago when if something was broken when it came out in rel, you couldn't do anything about it except open a bug. Now if there's something broken, there's recourse and there's gating and there's policy so that you as a consumer of that package can also go and contribute directly to help make those tests better in some way to prevent that from happening in the future, which I know was a frustration of me as a consumer of CentOS in the past just because of inheriting bugs downstream, right? And the ability to look into that and kind of help contribute is new and fascinating. Yeah, I would say depending on what your level of the kind of SLA you expect to have for your home server, I would probably not put a cron job that runs the DNF update every six hours on any of these systems but on streaming particular I probably wouldn't do that just because sometimes updates would go out necessarily in exactly the right order so you might notice some breakage if you run it at exactly the wrong time. That said so the way I mitigate this at work for what is worse is that we just take snapshots of the repos every week or every so often and then we're all out this snapshot of the repo as a unit so that's one way to mitigate that. I don't think this is a concern for a home deployment to be honest and I don't think I've ever hit this at home and at home with a machine so it's definitely not an ideal well managed scenario I definitely agree you need to be running DNF a lot on a lot of different systems to be able to see the signal of possible breakages that occur because they don't happen that often you see them in CI more than you see them anywhere else I think or maybe in your deployment of systems you have enough to see who if things break maybe well yeah the other thing is stream doesn't have body so it's not like you can look at body and you can group updates in bodies it's designing a very different way from that point of view and it has benefitted in a Starbucks in that regard so if your expectation is something like Fedora it definitely works a bit differently there you had something I was only going to pitch Fedora IoT at you because I use it personally and then you took it off the table and then I was going to admit that I lied a little yesterday because it turns out I run another OS because I run a home assistant server and I was going to tell you that you really need to think about your use case because like while I love all these folks up here these may not be the OS as you need and then I was reminded of the fact that you're on the supply side of bugs and like David you don't write you don't call you won't run row yet you work for us where do these bugs come from sir there's another thing that I want to bring up because I think it's not obvious but it's it's a defining difference in that if you have any of these systems generally speaking updates between major releases can be tricky or at least aren't necessarily officially well supported that doesn't mean they don't work and you can generally make it work in a home case this may not be a problem on the other hand if you use Fedora generally speaking if you use grants like Fedora server you can do system upgrade from release to another easily the downside or the flip side shall we say necessarily like forklift upgrade your whole system every every time Fedora releases also in the in the shorter window Fedora releases way more often seems like there's kernel updates a lot more often you may not want to reboot your server to get a kernel update whenever there's a kernel update there so that's another consideration to keep in mind one more thing I think is a little bit unique to Alma up here is we a couple of weeks ago we've changed we haven't changed directions path to where sometimes we can do builds of our own do bug fixes of our own and release those that aren't necessarily a direct clone of what rail is so with rail you're getting rail with I mean correct me if I'm wrong with rocky you're getting a one to one sorry if that's the wrong terminology you're getting a clone of rail and with Sinoa stream you're getting what's coming up next I'm really excited to see with the changes that we are making in the future what extra we might could bring to the table and you know that might be a value to you you know depending on the direction that basically our community takes it and where people choose to contribute and where that goes and just quick yeah we've got a queue of questions here too sorry yeah so I'm a bit of an enterprise Linux new so apologies for any ignorance in this question and it's a two part question the first part is for everyone up on stage you know what would you say is your major differentiating factor that you're providing with your distribution my second question is you know we have many sort of very similar distributions up on stage do we ever feel like we're kind of over differentiating from each other by creating these separate distributions I'm curious about your thoughts in a ah shucks so I think that as Jonathan was saying my Rocky's differentiator right now is that we are continuing to do what we promise to do for our community in terms of rebuilding one-to-one with Red Hat I will add on that we recently proposed a fast track SIG kind of a callback to the CentOS fast track SIG where we're looking to be able to bring in new contributors and also collaborate more with the Enterprise Linux community like with all the Linux and the patches that they're planning on adding to be able to bring those as well as other bug fixes and packages that our community finds helpful into the operating system as an optional thing to bring on but at its core we'll always stay that kind of one-to-one for the promises that we've made to our community in terms of the over diversification of this panel right now to begin figuring out what that all means for the future. With RHEL we have built very carefully an ecosystem of vendors and partners where we try to provide an OS platform for them to help our shared customers execute workloads in a safe and reliable way with certifications and all of the things that those kinds of RHEL spaced holes exist to fill and so that's like that's a significant differentiator for us from say Fodor Lennox or Centaur Stream I mean you know Alma is going with a mixed approach as I understand it and our friends at Rocky are going to attempt to copy us and that's that's fair I think that's a reasonable statement it is not meant as a slide the thing that I think is interesting and your over differentiation knowledge at its core all of our operating systems including folks who aren't on stage our friends at Canonical our friends at Oracle our friends at SUSE they're all the same thing they're all made from the same upstream components you can sit down and do basically all of the same stuff on all of them sure some of the commands are different DNF versus APT I know which one I like to type but you know sure whatever some of the files are in different locations but they are because it also calls back to David's question around that you have a specific need and one of us needs to fill that and so like I said yesterday I'm really excited about the white space in Enterprise Linux being filled by other projects and other people because not every problem is a well shaped problem I would say on the on the center side there's a few there's a few size to these on one end is what I mentioned earlier that because trainees continuously deliver information you effectively only have a continuous stream of updates you don't have to pick minor releases or think about that at all and that has its benefit and has its drawbacks so it's something to consider the other thing is the centers isn't just stream but it is the ecosystem of the six so if there is something that you want to do that fits into what one of the six is trying to do that can be very useful that can be very beneficial could be a valuable way to go ahead on the differentiation point I would say I don't think differentiations per se is a problem I think where it can become a problem is if we end up with silos that don't communicate to each other and I think the fact that we're all sitting here is a good sign that that isn't happening right now but what I would personally like to see is a way for all of us here to contribute back to the same part which ultimately is better at the end of the day but in a way there are specific issues that one of our projects is solving and addressing that isn't well addressed by the others we can figure out what is the root cause there together and get it addressed where it makes sense and where it makes sense might be in Fedora it might be in Santos it might be a process around how these distributions are composed together it might be something that has to be rethought on the outside but I think that is something where the most value can come out of the differentiation around the players on this stage what was the term you what was the term you used for all of us yeah I wouldn't say that over differentiation is bad as long as everybody at the table is bringing their own value that's kind of the beauty of open source when you have projects that don't agree on every tiny little thing you end up with two projects and going back to what David said those different projects meet different needs so it brings more to the table overall because then that all goes back upstream and if somebody you know wants a combination of both of those they can go take that and you know run with it as well so it adds more building blocks for everybody to build with great so I just want to let you know people have been raising their hands so I've got Justin here I've got the gentleman beside David there and I've got you next alright maybe we could skip this because I feel like it was kind of covered for Jonathan but I was just curious what you think about not having to be one to one bug for bug compatible with RHEL anymore I guess you did kind of just answer that but I don't know anything else you want to add to that I was just curious like what that looks like for Alma and if there's anything else you want to add to what you just said I am was that an online question or is that from you okay I'm beyond excited because it allows us to do something you know things of everything from ButterFS support in a kernel to you know putting raid card modules back in the kernel and everything in between are things that have been mentioned and prior to now it was things that we would simply have to dismiss and say well you know RHEL's not doing that and we couldn't you know that was what we were doing to help contribute them and kind of see where it goes from there and going back to earlier some of the value that Fedora provides to us you know this is all new this whole path has happened within the past what 6, 8 weeks so one of the things that we're strongly looking towards is how Fedora handles you know making decisions and changes within the operating system and the way that Fedora has it all set up to take take a lot of the ideas and you know turn that into something positive you know as a differentiation for ALMA that can meet different needs so my question is kind of in the same vein because if I want to contribute to center stream I contribute to center stream if I want to contribute to change something in RHEL I contribute to center stream but what happens if I want to change something in ROCKY or in ALMA where do I go I guess I'll go first if you want to I'll let Neal cover ROCKY I don't want to speak for him if you want to change something in ALMA it could go either way we're still using center stream as our primary upstream so anything that does get in stream you will see in ALMA Linux but we can do things in addition to that so it could be in stream or talk to the maintainer of a package in stream or want to add another package and there might not be a pathway to do that in stream and subsequently RHEL that's something that we can still look at doing and that's kind of the difference that we can take with the RHEL compatibility but not RHEL clone model basically same thing for ROCKY outside of having the core what you get when you install ROCKY and update your RHEL we're adding improving and facilitating collaboration in our SIGs with the upstreams as well as just internally to help folks that are using ROCKY to add to change what they want to and add the value to ROCKY in that way without having to necessarily go through the process of contributing in different upstreams it's I can speak for myself that getting involved in the core and CentOS Stream communities and collaborating in those personally from my background and coming from the ROCKY side wasn't a super inks-free experience for me and I know that that's also true for other contributors from ROCKY and from Alma and other distributions who may feel disenfranchised in some way and I think that's something that ROCKY and Alma and CentOS Stream bring those people back and bring those people back into the fold under the fold into the ecosystem I would disagree with your comment about feeling disenfranchised I feel like I was when I wanted to get involved with the door I was welcomed with open arms let me rephrase I was definitely welcomed into both communities with open arms I just felt personally from where I was coming from that there was resentment or something coming from that side which may or may not have been there and probably wasn't but I know that that's not an uncommon theme for folks that are coming from that background coming from the maybe more emotional side and I think as humans we are all very emotional it can be hard for them to feel accepted into those communities even when those communities are literally saying we're welcoming you okay I see the thing that I would I would stress again on the stream side specifically is because stream is what's used to effectively develop rail there's going to be a set of changes that aren't going to be acceptable for stream but that doesn't mean there is no space for those changes in the project so the example is for example the one of the things we do in the Centros Hyperscale SIG is maintain the latest version of system D as something that can be used on Centro stream and that's something that obviously wouldn't be acceptable for stream proper every time there's a release of system E although they have re-based system D a few times now in the nine cycle which I was personally very pleasantly surprised so that's an example of something that can be developed within the ecosystem and it's also something that then is available to other projects in the area that want to leverage if they so desire thank you so my question mainly for Red Hat and Centro stream at this point regarding the contribution model between Red Hat and Centro stream you know finds a bug in a code that they are pushing upstream what will happen if the developer upstream didn't want to accept that code would it still be available on rail or not and what will happen in few months time when Centros 8 stream goes end of life and Red Hat still continue with rail had 8 there is no way to push upstream unless you will back port the patches so somebody can shed some light on this sorry if I can ask a clarifying question because you set upstream multiple times and it's a loaded word I mean Centro stream do you mean upstream as Centro stream or upstream as the upstream project no as Centro stream okay thank you if I understood the question correctly and please correct me here if I wonder of course now I'm a little worried with this clarification I can split it like two parts yeah split it into two parts at the moment how Red Hat is contributing Centro stream are bugs fixed in Centro stream first and then merge it back to rail or rail fixes the bug and then submit it to and release it to their customer and subscription and developers and then push the patch to Centro stream this is part one so if you look around the room carefully while I'm answering this they're all about to blanch but as someone who does not regularly push MRs through either process I'm going to give this the college track the general process for almost everything that Red Hat does in rail except for rail seven because there is no Centro stream seven is to do that work in Centro stream that work goes through an MR process it is discussed and it is merged that primary case for those exceptions is like an embargoed CVE where that work cannot be done publicly and in that case we must do it to the side we then ship it to our customers in rail and then we put it into Centro stream there is no artificial delay in this process and my understanding is that depending on the engineering team involved they even have the MR ready they're just literally waiting for everything to come out the ship to go and then they're pushing it in stream like it's not even a now go start over problem they've been keeping it up because if you think about doing that kind of work like you're already in there with a needle picking up drop stitches to begin with you're not going to want to set this up to be something you do twice this actually also happens with Fedora packages with Red Hat maintainers where there is an embargoed issue that Red Hat is aware of that to build it internally and have that ready to go when they can but it has to wait for the embargo to be done Justin's got a clarification here to clarify that build it internally like I said look for the people who are blanching right now that's the person you really want to talk to not to clarify that especially with embargoed TVs build it internally yes we can build it internally I build embargoed I can't make sure it works but there is no system in Fedora for me to build and stage something that would be an official build so I can't start the build process until I can commit to disk it and then get it through Koji and that's all once the embargo is lifted that's the same in stream so that's basically the cause of the delay is not having secret places to do things and I've gotten yelled at when I proposed the second question that I believe you had was what is going to happen with CentOS stream 8 at the end of its 5 year life that the CentOS project has announced and that Brett had obviously supports because I didn't deny consensus we don't know yet we're in the process of figuring out what our development strategy is going to look like and then we're going to move forward with letting the world know what our answer to that is we will of course do that in consultation with the CentOS board who embraced the idea that CentOS stream from their perspective is a 5 year thing and the USS Enterprise will launch on its new mission so I had to work that in there right so I think that that's the perspective and I do want to leave room for Davada if he wants from a board perspective or a project perspective to add something yeah I would say from a in a general standpoint if you go on giddlub.com right at CentOS stream and you look at MRs on the group you can see very very clearly how the activity flows you can also see that oftentimes the people making the MRs are still learning how to work through the process so you might see a variety levels of quality in the comments shall we say that I think is proof among other things that this is really where the development is happening and there is a secret cabal on the side but it is understandable that for things like embargoed CDs and things like that there has to be a separate process there's no way around it and like if you've I've been in environments where you get the email that oh yeah tomorrow at 4 a.m. expect something to happen to open SSL I'm sure you might have been in environments similar to that there's really not much you can do in those cases the good thing though is that at least in every example I can think of what has happened as soon as the vulnerability was public like the the commit was published almost immediately on GitLab and then the build went out within a reasonable time I think there's only a couple of example I can think of where this took slightly longer because people were still learning the process effectively and I apologize I don't like chaining but I your question reminded me of something and I want to publicly acknowledge it when Centaur Stream was first announced it flowed backwards oh yes we fixed that we fixed that a long time ago and non-Red Hat employees were not the only people upset about everybody trying to swim in the wrong direction so we did at the very beginning of time have this weird system where things went around the curve and then kind of pretended they flowed down that has long been gone and I really encourage people to do as Davada has suggested and go take a look at GitLab that is your proof like you don't need to believe me if you if you find it useful I wrote like a short blog post from blog.census.org a year or maybe more I don't know it's weird we covered it but like some time ago I wrote a blog post that kind of tried to explain how this process works because it was something that I wanted to understand and I found that fighting it was a lot easier and that people at work needed to understand so maybe that helps is that from Alexandra? no no that was something I wrote Alexandra also gives a talk and I think had also write up that was very helpful we should probably write up part two of these that put the two together because I think that would be valuable she couldn't make it because of visa things but would be in the audience correcting us on everything we say here yes if she was here correctly too so just wondering what the collaboration workflow process is for Embargoord CBEs with regard to RHEL, CentOStreme, ALMA and Rocky because you know I imagine that if we get another spectre meltdown type situation it kind of behooves us in the sense of partner and openness to include the distros as early as possible and of course as early as possible is when they get read in and when is that but I'd like to what is that process today or does it need to be hammered out? I will speak at least from my perspective and from what I know and obviously I can't speak to what happens internally at Red Hat but I do know that a lot of the CBEs are circulated on a private openwall.org mailing list pre-release in the Embargo period where those patches are discussed Red Hat, Canonical a number of other organizations I think Metta might be one of them I don't remember I don't know if I can talk about this properly okay well anyways there's a list it's public but that's kind of where that process happens and there are requirements to be part of that group and so I would personally love for more collaborate for us distributions like Alma and Rocky to be able to participate that and I think that these sorts of changes that are being made specifically with Alma being able to vary and build off of rel in a diversified way that there's definitely reason for more organizations I think to be part of that and to collaborate I am not an expert in this space and so I am only going to say that Red Hat fully cooperates within this space where it where it exists and I honestly don't know the status of Alma Linux and Rocky Linux with regards to participation in this process I encourage everybody who is building code to pay attention to where that code is coming from and to be able to have the engineering wherewithal to manage all of the pieces of code that matter to them beyond that my understanding is that we do all of the CVE work on behalf of CentOS project and on behalf of Fedora project my suspicion is with a focus on Fedora Linux and a focus on CentOS stream however within the Fedora project my understanding is that it is still true that everything that Fedora produces is derived from Fedora Linux and that everything that is currently being built by Fedora Linux is derived from CentOS stream so indirectly all of those other pieces will benefit from that work I think the only caveat here is there is no formal CV process for content coming from CentOS 6 so it's up to the new CentOS 6 how to handle that I would say most of the SIGs end up building content that's derived as you mentioned from either Fedora or CentOS stream so it's pretty easy to do because it involves certified metadata and things like that you are probably not going to get that there maybe you can start a SIG to look at that in that mirroring everybody else up here I'm not the best one to talk about that but internally some of the questions we have at Alma right now like when we created our Orata for example that was done in a way to mirror rail and recently they basically called us to ask ourselves a question of we've got to rethink how we do Orata now because we pushed that first and at the time we had no idea if that was going to make it into to stream and rail or not it turns out it did so now we'll pull that same Orata but we've got we've still got some rethinking to do and it's worth saying that none of this is 100% figured out and trying to get involved in the development more and contributing to stream is helping to force our hand on figuring that out it's forcing us to stop and ask ourselves these questions I think we're running out of time way over time way over time so I guess if there's more questions well this will be a continuing conversation I think I'm really glad for everybody came up here it is a little bit of a scary thing so it's good I think everybody who wanted drama should have come to my talk where we talked about discourse and the mailing list there you could have had it I think we did a good job of disappointing everybody who was looking for it here so thank you everybody and I think we're going to go on a scavenger hunt and dinner so Justin's got some stuff to talk about so first we'll have a applause for our panelists and our moderator so quick housekeeping for tonight we are going to go to a community dinner everyone is invited tonight we will have shuttles leaving at 5.30 so I would suggest being in the lobby at 5.15 so take some time to refresh or we also have three workshops that have another half hour well we ran over a little bit so they're probably gonna 20 minutes but meet us in the lobby at 5.15 we're going to go do the scavenger hunt through Cork City from the restaurant so I hope to see you all at the dinner and we do have to leave this room because there is another event in 20 they're gonna set up for the event and take everything down here now so we will have to we are gonna have to do a quick exit here but thank you all for coming for day 2 of Flock and hope to see you at the social tonight