 All right, let's let's go ahead and get started. So the The topic for the next 50 minutes is rel versus fedora. Well, where really diverged and why? so as Denise mentioned previously rel is derived from fedora primarily and That that generally means that we do things in fedora first and then we inherit them in rel and In some cases we end up doing things in rel first and then we try to sort out what to do in fedora after So this is that latter case that we're talking about So first of all, this is a panel. So I would like to introduce our panelists. We have Denise Dumas who presented just moments ago Paul frields rock star former product project lead and also mr. rel 7 and Alexandra fedora part of fesco and see I hacker extraordinaire and I'm the new Paul frields for really and Really, it's definitely not a stale distribution Neil. So Let's let's talk about what we're talking about today. So there's basically four things we're going to cover Background for anybody who hasn't been through a new rel release before The things that we actually changed inside when we were making the new rel release How we're doing updates like how we keep it stable and then what the opportunities are for reconciling those last two things with What happens in fedora? So let's talk about like the rel 8 origin story First of all really alpha was derived from fedora 27. It's kind of like the first pancake. It tastes pretty good but it's also sort of funny looking and Generally, you just throw that one away and that's what we did. So really rel 8 is fundamentally derived from fedora 28 That's where the rel 8 beta came from it. It's delicious and 8 ga will become will move to 8 1 beta a 1 beta will become a 1 ga a 1 ga will become a 2 beta And basically that's just the case of us taking upstream patches applying them to the sources We already have and releasing it. So we we get a nice stack that way And it it has the unfortunate side effect though that when we do this we actually stop inheriting from fedora most of the time So before rel 8 came out a couple months ago. We hadn't really done a mass import from fedora since 2014 so it had been a while and We we've actually taken some measures inside 8 to make it conceivably possible that we could like interact more often and that's what we were looking for so a few other things Well previously the schedule for it was pretty sporadic we didn't know when the next release is going to be until the previous one was done and We've actually moved we're converging with fedora on the six-month schedule. This is like a brand new thing for us We're still learning how to do that Basically fedora has a head start on us on this one And we've also decided that we're and announced that we're going to try to do a new major release every three years So rel 9 is about two years and ten months away now So it's surprisingly not a lot of time left for us to figure out what we want to do next together and get it done So let's talk about Where where we actually started diverging from from fedora and there's just a few a few fundamentals that are different and It has consequences. The first one is that the fedora infrastructure is completely separate from the rel infrastructure different Koji builders different Koji build system different storage different test equipment different disk it all those things are completely separate What that means is when we're doing that bootstrap? Every single time we end up having to rebuild the world with whatever changes we do and it's it's kind of like a Laborious activity for us and it has very little value other than we have this separate infrastructure So where the divergence really begins though is in package selection fedora is huge It has an enormous number of packages rel has a very small number of packages by comparison And there are a few things where rel is actually not a subset of fedora like the kernel We take the spec file, but then we just pull from kernel.org because that's where most of our developers actually Are living on the kernel side and then there's a few partner packages for For enabling weird hardware that are still open source, but overall a lot of it is pretty common and This is kind of corporate looking all right then pancake so Fedora exclusive content it's pretty big Apple comes from this we love Apple and Just overall though when we take a subset of the fedora content We also have to change the dependency tree because everything can depend on everything and when we When we bootstrap we actually have to solve all those build failures We end up usually doing that internally first and then pushing them back out later hopefully the fact that we have a public schedule now and and Greater greater ability to even admit that we're going to do a rel 9. It's totally going to happen It means that we can do things like put EL 9 disk tag comparison things and spec files so the other big change was When we did rel 7 and earlier we had they compose for server for workstation sounds kind of familiar We've actually done away with that. We decided you know, it's better to just have kind of the everything compose But we have two composers and the foundation for that is We wanted to separate the operating system from the applications Because nobody actually boots rel or fedora or anything just so they can boot it. It's not it boots ship it It's it boots. I want to run my application. I can run my application. That's my purpose so By having separate OS and application compose. It gives us the ability in theory the ability to Update the operating system independent of the applications. So this is kind of the the the early premise is What's that set things up so that we can get out of just the everything depends on everything? Position we're in which sounds that's especially like there were rings, right? Like we Kind of did rings, but it's not I wouldn't say it's done yet. It's a good starting point. So the For just going a little bit deeper on this one The rule that we ended up with to make this practical was that base OS cannot depend on Appstream content except at build time Appstream can depend on at build time and run time on base OS content. So this is the way we found Attractable way to split things up, but it's not perfect things like Pearl are still in base OS. So we want to keep working on that But this is this is the basic idea and it's not a new idea. It's just one that we got to work on a little bit more than that had been done before The other place is that in some cases as I mentioned Fedora 28 was the basis of rel 8 well But we're 19 was the basis of rel 7 so can you imagine like During Fedora 19 that you do a YUM update near for door 28 It would not go well, but that's kind of what our customers are we're going to experience if we didn't do a few things to address incompatible changes so inside Fedora, there's a most excellent change process and going from 19 to 20 to 21 Little gradual change is pretty reasonable. It works. Well, it seems modest But if you are not on that update train if you're just going from six to seven to eight These are giant jumps. And so we had to make a few changes a few reverts if you will and I just want to cover three of them really quickly first one is DNF so DNF is really cool technology the the soft dependencies weak dependencies pulling dependencies are outstanding It opens up all kinds of our opportunities The ability to do modules and have parallel availability is outstanding We still had to call the package YUM because that is it was fundamental to customer experience to just be able to say Install a package called YUM run the command name YUM and have that YUM behave like YUM 3 Which was Ralph 7's YUM There were other just minor things that happened in Ralph 7 that we had to revert like no default syslog That made a lot of sense for Fedora didn't make sense for us Because of existing customer expectation then finally we added this thing called platform Python. So Python 2 is going to go away someday. We've known this for a while Python 3 is is pretty standard now three or four years ago it wasn't and The challenge we had was we need to get everybody over to Python 3 But which Python 3? Python 2 is like there's only one version of Python 2 anybody cares about it's the last one Python 3 there's a new version coming out on a very regular basis So we added a thing called platform Python that is the one that is used for rel 8 that all of our applications use It is not user bin Python. It's not user bin Python 3. It doesn't exist anywhere But like user live exec and by doing this we let our the people that want to build exactly for rel 8 i.e. the rel developers to use the platform Python and Customers who wanted to use Python for their actual application to Install the version of Python that worked for them so user bin Python could be anything in Rel 8 we don't know that's up to the customers to use and That's that's the kind of direction we want to go is giving customers the option to use the thing They want to use and not force them to use a thing that we chose to use for our own internal pragmatic reasons so With that I want to hand it over to Sandra but updates okay, so first of all I would say Testing updates and stability of a base is not a new concept we introduced in rel 8 is Kind of was we are from a very start and this is what redhead does actually, but what's interesting in our rel 8 is that Given by the new plan for a shorter release cycle and for regular release cycle We introduced the new concept, which is ci and gating and I hope you heard about ci and gating before and in Fedora as well So basically what we were targeting for is to bring the testing which redhead does Bring it into earlier stages and like do it more often. So if you The release early release often mantra then like here's our variant for this is test early and test often as well So this goes together with the effort of upstream first for the tests In as well. So we here in Fedora know about upstream first It's the concept which we use we know the benefits, but the fact is that it also works for test code the same way it works for your regular applications the possibility to bring code test code to upstream gives you the opportunity to For better maintenance for easier maintenance and for this test early test often possibility so this was one of the effort which was introduced in rel 8 cycle where Quee teams of a rel distribution were bringing tests to Fedora and to upstream and This was just the beginning of it, but we are looking forward to continuing that buff and Also like CI in gating essentially is all about fast feedback. So we are trying to Provide feedback to those people who can act on it. So you learn about the result of your change with feedback of your change When you do the change and not like several months later when someone else tries to figure out whose change caused the regression And we also provide feedback while there is still time to act on it So the earlier the better and To implement this we introduced the gating rel 8 development process Well, we talk a lot about gating and we will be talking about the gating for Fedora People always like associate gating with a picture of closed doors in front of them I would like to give you a different Mental picture of the gate. So please don't consider it as a closed door in front of you consider it more like the airport gate is the Place where you kind of regroup in the review the changes or review What's here you're going to do here? And this this gives you the opportunity like if you forget your luggage or if you forget to Present for someone you can go to duty free shop and get something this Like and imagine like if you check your boarding pass only after you got to your destination place Yeah, you probably don't want that. That's why we need gate and this is the kind of gate we built in rel and You probably heard that we are going to try to build something for Fedora as well I like the idea that gate because duty free means liquor. So that's that's great Okay, so it was a very brief primer on the things that were that we changed and we'd had a few ideas for things that we Like to kind of come back into the community and and kind of work together on but This is like a starting point not These are the things that we will do so we're looking for some audience participation here Especially if we don't cover something you think this would be a good idea Please like raise your hand and add in so Just got four things basically and the first one is we're doing real nine development We'd like to be able to do more of that in the public space if you're a redheader You know that there are things like don't even put yell eight in your disc tag Like in your spec file That's silly. We know there's gonna be a nine and and that's fine to do but wouldn't it be cool Yeah, I wanted to talk to this one in specifics. So I don't know if Matthew's in here Is he in here? He's not okay He has many things to do in which I very much sympathize with as a former Fedor project leader And so this is a topic that I feel very strongly about it is probably one of the most Uncomfortable things about being the Fedor project leader is having a lot of knowledge about what's going on Internally at Red Hat with our product development and not being able to share that with the community It's a very painful place to be because you want to be open and transparent with everybody about what's going on And yet there's this huge block of knowledge that you can't share which would make people's jobs and their and their Their contribution so much easier It's so much easier to get people to work together if they know where they're going and when they need to get there, right? It's a lot harder to do that when you don't have those imperatives to share And so you know once again that we kind of breezed by this a little bit earlier And I wanted to call really strong attention to it Just once more real quick if I could how many people in here do not work for red hat This is a it is a helpful thing to know great. It's it's yet again It's like it's maybe not quite half and half, but it's still quite a few of you I would really like to call on everybody here whether you're employed by red hat or not and you you care about fedora Just try to keep this in mind. So we just released rel rel 8 in May Right, and so the public commitment by red hat is that we are going to try and turn out rel 9 in three years Now we don't have a hard date for that yet. There's you know, there's no published calendar But the commitment is three years. So let's just think ahead and say well by spring or summer of 2022 You would expect to see rel 9 Premier and so I would like everybody to just be thinking about that as we have conversations in public You know on fedora develop list you're gonna hear people talk about rel 9 and I would hope that that's you know This we could kind of share this knowledge Around with our fellow contributors If I were Matthew, this is one of the things that would make me most happy about the coming few years Because I don't have to keep a secret about whether or not there is an el 9 or rel 9 that we start thinking about How the things that we're committing into fedora affect rel 9 the fact that we can have those those Conversations in public is a huge huge step forward And I'm kind of jealous of Matthew right now because I never had this and I hope that doesn't really come through too badly Yeah, I just I just want to cry a little tear of happiness It really makes me happy But but again, you know just keep these things in mind as we're having those conversations on the develop list You know that that some folks obviously are going to be driven by their the fact that they're thinking about What does this mean to rel 9? Well if rel 9 comes out in the spring or summer of 2022 That means that at some point prior to that, you know, we've got to do freezes There's got to be testing that's happening inside red hats. Some of those things are going to be a lot of those things We're trying to push out into upstream first like Alexander is talking about then There are always going to be a few things that are going to be red hat value ads that are going to happen internally Of course, I think everybody would expect that but we're trying to you know kind of constrain What those things look like right and the fact that there is you know some time ahead of that means that you know From right now, we probably have I don't know Maybe two years at most to get things into fedora to make sure that they make it into rel 9 So hopefully people are thinking about that when you see somebody introduce a change You'll be thinking about the fact that they're they're trying to also do you know right by upstream They're also trying to do right by what's happening internally to respond to customer needs for rel because remember fedora is the upstream for Rel and we want that to to continue being the case for you know the time time to come So I think that's all I had to say I'm just really happy that we can be more open about it, right because for a long time We weren't you well as you say we weren't able to even talk about what the schedule is going to be for the next row and now It's good. We're out there, right and I we want to be more open. We want to be more transparent I'm really excited to see more of the tests moving upstream. I think that's huge because that benefits fedora Yeah, I'm just rounding that out like we know about Usually there's about six months between beta and general availability It's usually about six months between alpha and beta So if you if you just assume that we're gonna do a cookie cutter repeat of eight You could probably guess the right fedora version But you know depending on how things go with with gating if like if rawhide is suddenly a very stable source of Trees because we have most excellent gating maybe it's not really What fedora release are we derived from it's like what day of rawhide are we derived from we don't know We're gonna figure that out together One thing that is certain is that we like to do more of it in public more of it shared And if we could somehow construct a way that we don't have to like redo a bootstrap of alpha Throw it away a bootstrap of beta in private and actually just have all that happen up front that would mean that you get earlier insight and we'd also have Like Apple immediately because those packages would have already been built. So there's a lot of cool potential there We want to explore it So that's the first one second one is let's Let's do Venn diagram pancake brings 2.0 9,000 so that When we I'm gonna dwell on this so When we split the operating system from the application such as we have We were trying to address the too fast too slow problem. Matthew talked about it yesterday. We've been talking about it for years for our customer base usually what what people want is a much slower smaller steadier operating system and then really rapid application deployment like from upstream into to their BNF repo or container host It is quickly as possible and that's what a lot of the development went into eight is about Modules are fundamentally about parallel availability getting people the option to choose this or that and it's a great thing to have for your applications to give customers the option and One of the parts of this was that we assumed that if we separated these two things that we would be able to draw from Fedora more often for the application side because that is Dora is constantly refreshing. It is a fantastic machine at that and if we can find better ways to Distinguish these two buckets then we can find better ways to pull from Fedora and deliver upstream Cause things to go through Fedora first and otherwise just kind of have a greater hesitate to say synergy because it's one of those words, but Everybody gets some benefit from it So this is this is a thing. We'd like to pursue further That's the meaning for us the meaning for For community could be different We need to figure that out together so that we get something that is mutually agreeable and Then update policy and structure and whatnot to match We think probably the minimization objective is the place where all of this pans out because when you're switching your packages up when you're doing a subset the foundation of it is just Dependencies change and the interrelationship has to be codified in the spec files We can't do that if we can't do it with you or we have to do it on the side and it's very inefficient So that's the second thing we're looking to do And you guess what? We're going to have more see I so I mentioned that in rail development We'd like to shorten the gap and Create a fast feedback loop from making a change and testing that but while we're doing that in rail like why stop at rail level, why don't we close the gap between making a change in upstream and Testing it and integrating it on a rail level. So why don't we talk about Fedora? And why don't we talk even about further going upstream? We actually want all those things. So this is one of the places where we can improve and increase our Calibration not just between Fedora and redhead and rail but also between Fedora in upstream projects, so this they related work like the related objectives for this kind of thing is The fast lane for straightforward updates like what I call straightforward update if something if upstream project is Like simple enough or introduced and change which is simple enough which goes You you get it merged upstream. You will consume it in Fedora directly We've no changes and then you consume it in a rail with more or less no changes And then for this kind of workflow where the update can be Like just doesn't require a lot of work on integration side and just you need to consume the upstream We are going to work on Setting up a fast lane for these updates. This is going to be packet service. I Not sure if Thomas is here. There will be talk about this today later on please see and This is a very interesting effort. So we basically you go from top upstream to Bottom in the downstream in one go by one service This is an awesome effort to participate in to simplify your daily work as a package maintainer And the second part is distro tests can go upstream now again with the packet service with a possibility to package Upstream changes immediately into the package which can be then installed on Fedora or on the rail You can actually use this opportunity to test upstream changes in upstream but test them against Fedora environment So again, you can use Fedora packet service To implement this kind of test in upstream changes So that you don't have to deal with upstream breaking your Fedora stuff But you inform upstream as they do the change that they are going to break Fedora stuff And you have the opportunity to fix that before it lands in the upstream even and We'd like to also provide more possibilities for teamwork. So well the Usually historically the maintainer concept in Distributions in rail and in Fedora is about person owning the component and our Processes are focused on updating the components We want to extend this and we want to extend gating to handle Multi-component updates as a unit of work. So you can focus rather on Feature you want to implement and then discover which components are involved in this feature and how many work You need to do on those components and then you can group these changes in several components into one update Which you can test as it is one thing you which you can Implement as one thing. So I hope this will increase the possibility for our maintainers to communicate and collaborate and work together focused on a larger scope not just the folk of the scope of one individual Component and these maps into Fedora CI objectives If you have been following we currently are Working on the multi-package gate, which will allow such things So you will be able to group your changes in several components into one unit of work from the gating point of view All right, so we have about 20 minutes for questions, and I hope you have some so Any questions? So the question was why didn't we use the young bindings? That were offered in Fedora 21 21 22 I Think So this was this was a a deep debate within the organization Can we just like offer another package that that kind of wraps on top and It actually came down to our customer support organization and our product management organization and saying that this is such a deep Fundamental that the package name has to remain the same the same command line arguments need to produce roughly the same result you know plus or minus what the Resolver came up with so it was it was really about Maintaining continuity of customer experience, and you can't quite get that with a wrapper Even though in some sense it seems like you can if you look at like the full end-to-end picture. It's not quite sufficient They were terribly they were terribly concerned about Customer scripts right because at least at one point in time I mean we're trying more and more now to make rel easier to manage Easier to to install easier to deploy easier to manage, but many of our customers have antique Infrastructure and scripts that they developed themselves and yet yeah legacy and no customer of Rail runs only the newest rail right so the day rail 8 came out It wasn't like it was like like City Bank was gonna suddenly take rel 8 and convert their entire infrastructure to rel 8 So they have rel 5 and 6 and 7 and 8 and every one of our customers is like that And so maintaining consistency for the things that they count upon is really important If you see us working on things like Ansible system roles, that's another reason for that stuff, right? They need really a system management API, and we haven't really had one across the distro things change, right? But but customer infrastructure Increasingly relies on automation and so we have to give them something that they can run their automation against reliably Yeah, and as it works out the Technical correctness is not quite as important as Consistency so compatibility really meant looking and feeling more like rail 7 yet also offering all these new things at the same time So it was kind of walking a tightrope Other questions Jim So everything that's in rel 8 is open source. We we build it internally if it is Going into fedora. We generally pull it from fedora But if nobody has actually stepped up and said I want to maintain this in fedora It's not compulsory either So I would think if it's actually a problem as a BZ been filed have the maintainers been contacted Is there like an actual request for it because if there is we should address it but if it just so happens that it's not something that's interesting to The fedora community to have it doesn't need to be there. I personally believe when that happens It is a bug de facto That's just me But is it a bug file that particular case? I happen to know is a bug But I would I believe in all cases it is a bug So we have dozens of bugs that need filed then So yeah, so yeah kernel configs is a good example, right? I mean in turn internal to red hat the kernel team goes through. I mean there are there. Oh, sorry So how do we rationalize things like like differences in kernel configurations? probably KVM configuration stuff like that where some things are appropriate for fedora and You know where the hardware environment is totally different, right? And and some things are more appropriate for the enterprise class big honkin hardware That you typically find as you go out and have to do things more at scale the kernel team reviews the configurations and There are hundreds of kernel options that you can choose right and every one of them that you choose Causes changes large and small in what that kernel turns out looking like and so there's an exhaustive review Conducted internal to red hat by the kernel team They have to look at what they think is ready upstream what they think is still buggy Things that so they also they review like is this crufty old hardware that we don't want to support anymore So hardware that are or crufty old hardware that our customers still demand Yeah, and oftentimes the hardware support that you would want in fedora Where a lot of it's going to be deployed on laptops on workstations is not at all what you want to deploy on a superdome, right? And so the kernel team does a big review Working with the performance team and they figure out they run lots and lots of performance testing and Figure out what is going to be the best compromise and then they put together tunedy profiles To help customers get to a working configuration for themselves as well if I may add the actual Where is the effort in the kernel? I talked about a brief Closing the gap between upstream and rail in testing and NCI, but this also works for kernel So the actually working now to bringing rail based Kernel tests to the upstream and so we are looking forward for Making more impact on upstream with those testing with those Infrastructure and we hope that we will be less divergent from the upstream and from Fedora in the rail in rail kernel as well So there's going to be an effort to actually minimize the set of differences between Fedora and rail kernel So we don't maintain like completely different trees for that this is also like nicely tied to CI and gating story and it actually is in development in progress right now and probably Laura Abbott is the best person to talk about it. So Yeah, so she's given to talk today in a after the lunch So try to find it if you're interested in this in more detail And just rounding that one out Sometimes there is a good reason to have one build that does one thing and one bill that does another thing Like we have we have system-wide compiler flags that we use for for every architecture What if you want variation on that basically if you only have one build if you only have one compose Then you only have one option, but if there is some way that we can develop Rel and a shared space we could do experiments where we have a second build with a second set of options and People could try one try the other do what works for them. So it's definitely an area. We'd like to explore and Clearly that is something that you can only do when you have a really good suite of automated tests Go Alexandra You know all about robots Okay, so that wasn't exactly a question, but I will try and paraphrase like Neil is Neil Gumpo is saying that It will be helpful to have More avenues with with greater transparency in the rel development process. It would be helpful to have more transparency and input from people who are able to Test and in advance to to provide feedback on changes Obviously, you know well has a public beta program for every release that already exists But are there ways to build on that improve on it offer more in advance? I think so do I know exactly how that would work not at the moment, but I would say stay tuned Right where this is definitely of interest. I think to the product folks. Oh Yeah, sorry. I yeah, I was thinking about it and didn't vocalize it Which is would there be a way for people to actually contribute patches contribute? Specific fixes or inputs back back into the and tests back into the process. Yeah, I think that would be very useful Right the that's kind of the spirit of open source And I think if you're gonna have a more open rel process, I think it makes sense to build that kind of That kind of a procedure into it as well Did just before we move on just want to make sure I didn't all right I didn't get fired for that so that's great That's it's always good like I I'll know right away like I'll get kicked under the table And that's how I'll know and then yeah my my resume is ready. So it's okay Dominic go ahead So let me all right I'll paraphrase again for the for the cameras Dominic pointed out that part of the reason for having More of the testing moving upstream is that it gives people an immediate route that doesn't really Lock on fedora as a community or rel as a product that really moving those things upstream is a benefit to the entire open source Community and that's something that Fedora holds dear right? I mean this is Exactly what we've been saying for many years is that we try and drive changes upstream And I think the the expanded upstream testing and CI that is an expression a better expression of that Then maybe what we have been doing for for many years So I think it's something we should keep an eye on is that you know Fedora doesn't become sort of its own concentration of Value that it doesn't need to hold in other words that we're not maybe Inadvertently hoarding change or inadvertently hoarding value into Fedora that we don't need to so driving that work upstream is really important Just to add on top of it So is from strategic point of view It's better to have the initial verification as early as possible because for when you try to provide feedback to rel It's might be too too late in the process So I the main like idea which you need to keep in mind is that the first verifications is Happens in upstream or in Fedora and the goal of rel verification is mostly to make sure that we That things still work. So it's not the first time we check that things work We when rel with the check that things still are working the same way it's supposed to be So this is our major goal. Well, just checking of new things is going to be more upstream more Community project based and and so on we are we still have some gaps like when we diverge too much when we can Introduce something but we are going to be closing this and this is one of the efforts that we are using upstream testing testing rel to recheck that we didn't break all the things upstream did while we were Integrating it in rel. So this is going to be still working. Yeah It may be if I could add just one bit of color This is just maybe a philosophical thing. I'm sure how valuable it is But I think when you look at at how rel diverges from upstream I wouldn't say that over the years. We've gotten particularly better from major release to major release about that Like there is a gap there continues to be a gap in each product I would say that it has not gotten particularly worse, but it also hasn't gotten particularly better but I do feel that as As a development organization We continue to apply pressure on that gap and that's why it has not gotten worse because I think the tendency When an organization is not keeping open source principles in mind would be that that gap is going to get wider Naturally over time and the fact that we've kept it about where it is You know just just in this sort of you know, maybe meta philosophical way is I think evidence of the fact that a lot of people Think about this in inside red hats So I'm really grateful to work in an organization that that thinks about that But I think again, we're always trying to do better Whether we succeed in the end or not at at closing that gap down I think if we do if we do know better than keeping it the same that is evidence of a lot of effort Hopefully we can though, you know, we can close that gap more over the over the next two years All right, anybody else Then just one follow-up on Jim's question observing that there are some packages that are in rel but not in fedora Or for those packages if we wanted to have rel develop more and open more Transparently more visibly it would definitely be a bug because how can you have it be a subset of fedora? If it's not all there, so please file bugs All right, anybody else? Well, thank you all for coming and thank you panel. Thank you panel