 Okay, we're going to start talking now. So people who want to pay attention should start paying attention. And people who don't want to pay attention should go to a different room. So that's how it goes here. This is the Jim and Matthew show. It's part of our series of talks about things that we want to do that are different. This one is very specific and less hand wavy still. It's kind of like a funnel of specificity. That's right. Is that an actual word? Yes. You play Scrabble a lot. Not to win though. Anyway, this is a very specific proposal with a pretty concrete thing, which is also a big exciting change. Uh oh. Will that work? Oh my goodness. This operating system is terrible. What operating system are we? What operating system is this running? Fedora. You walked into that one. That's pretty true. Okay. So anyway. Diskit. Diskit, yes. How many people in this room need a background on what Diskit is? Are we all in this room because we know what Diskit is? Background and Diskit. Awesome. So Diskit is basically our shared repository for RPM spec file build information. And so both CentOS and Fedora use Git to keep our spec files. We use them slightly differently, which I think is in further slides, but we basically keep our spec files and patches that RPMs are built in in the system called Diskit. I forget exactly what the disk stands for, but basic distribution, not distributed, right? It's a bunch of repositories. So each package has its own repository in this gigantic 20,000 repository thing. What? All right. So. I clicked the button and it works now. Okay. Awesome. Are we going to the next slide? We did. Yeah. So, right. Both CentOS and Fedora have these. We are talking about ways that which we could collaborate better. And this one jumped out at us as hugely obvious because it turns out that we both have repositories named things like HTTPD and Grub 2 and call it at least the what 6,700 packages that are in the base sent distribution. That makes up a pretty decent subset of the 21,000 or whatever we made up, whatever number that was. So not only are they overlapping, they're actually things with a direct descendancy. So that's pretty awesome. And so not only can we combine and share resources and combine effort, but it can enable a lot of interesting things like actually compare using Git to compare between branches across Fedora and CentOS. A lot of possible things that can be enabled, which we are going to talk about further. So after a bunch of brainstorming between Matthew and I over several drinks, we came up with this idea entirely on our own without talking to anybody from the community at all. Right. So that was a lie. A lot of people think this is a good idea. I don't think Kevin is here, which is a shame. This is something that was on the Fedora development mailing list, sort of independent of us talking about the infrastructure things. I know Peter Robinson also had this idea. It's kind of an in the air kind of thing, but it is a very big change. So we wanted to actually make a presentation of it as well. So as Matthew brought up, a lot of us have the same packages. There's a descendancy setup. We talked or we heard earlier from Brendan and Josh about how the waterfall method kind of trickles down. We saw the packages coming from Fedora into rail. And we decided to take a step back and not look at system specific, but just how the packages work. And so if we take the Fedora instance and we start with the raw hide packages and you get the absolute latest and greatest, this is the stuff and I'll use the phrase that Matthew hates and say the bleeding edge packages. It's raw hide, so okay. Kevin says no. If you guys don't know me, I enjoy trolling my own coworkers and the people on my team about this. So it's a lot of fun to just poke. And then so from raw hide, we branch into Fedora releases and we branch at given time. I'm not putting numbers on these dots. It could be Fedora 27. It could be Fedora 8. It doesn't matter. This is the typical flow for how it works. And of course, some people have their own different package. Terry picking patches back and forth, so there should be a whole crazy network of lines. I was going for the simplified model. And then occasionally out of this, we have the Red Hat model or the rail model where Red Hat looks at a particular release. They branch off of that. Rail comes out of that. And then at some point down the road, in the case of what is this, 7.3, I think, there was a new rebase in rail of the no modules. So periodically they'll pull again from Fedora and dump that back into a later Y-stream release when using the right terminology for that. But yeah, as previously noted in talks, that happens very infrequently. Like the last time was 2014 that the big thing happened. And then some stuff basically pulled from there. And I was going to say cherry picked, but I want to be very clear, actually. In this case, it can't be cherry picked because they're just copied over to completely different repos. And so from the rail release, we have the Sentos base and the updates are built. And that's basically how the package set flows at a simplified level. And I look at this as the waterfall because you're coming from Fedora, from Rawhide, into the Fedora release, into the rail release, into the Sentos release. And there's that just direct descendant. And I hate the waterfall model. We have definitely moved beyond it. It's done great things for us, and it's time to let it go. So we need a way to show some feedback in this. We need a way to show what Brendan and Josh pitched in their slides. We need a way to get this moving. And so what I came up with, or we came up with as part of this discussion, was yellow dots. I feel like yellow dots just bring everything out here. And essentially what this is, and I don't have full dots because I couldn't come up with a decent working example for this, but it is something that we want to get to. So right here for the yellow dots in this case, we have patches from Sentos going back up into the rail fixes. Whether or not Sentos accepts them on the mailing list right now, we have people about, call it, twice a week, sending patches into the Sentos develop mailing list for various things that are in the base distribution that we've never fixed. We've always looked at it, said that's a real problem file that upstream. And they keep sending us patches. If we're doing this right, this gives us a way to actually give some feedback. And we can look at Git and look at a pull request model and say, all right, fine, submit this. Red Hat can at least acknowledge that a patch exists. And they have told me that they will at least acknowledge that a patch exists. They may not accept the patch, but they will at least pretend that it's there. And so I've tried to show that in the diagram where some patches get merged, but not all of them. So there may be some things that Red Hat looks at and says, yes, we need to solve that, but we're not going to do it that way. So would that involve sending to us then carrying patches that are not in rail? In some instances, that could be a possibility. In some cases, we may drop that patch later on. In the arms side right now, we do carry patches that are not in rail anyhow. So we are maintaining some branches that have patches that are not upstream. We're not in the rail as an upstream. You've made Josh put his hand up. Actually, you did. Really? It was a thing I was thinking about that I thought I would prompt Jim on. So in that very, very simple diagram, I want to highlight just how simple that is because I ignored all of the stream branching. I ignored Apple and how that feeds in. I ignored all of the layered projects that we deal with for RDO, for Over, for Gluster. All of that stuff I left off because in the initial attempts at packaging those diagrams into that thing, it looked like a big web of spaghetti and it was too confusing. So I understand that is a really simplified diagram. And these are some of the most exciting things, actually, though. So problem diagrams. So turns out Fedora actually has two different instances of Git right now. There's pegger.io and then there's sourcedupfedoraproject.org and that is entirely different from how CentOS uses Git. And so Matthew and I tried to highlight some of the differences that we have here. I can't read them because you... Yes, I have designed an I-Chart for this and I probably should not have done it. But pegger is traditionally where the Fesco tickets live. It's where the council makes some decisions. Where the infrastructure team has a lot of their tasks and issues assigned. There's not a whole lot of source code there that is specific to the distribution packages itself. You might find some scripts for infra, you might find some relinch stuff, but you're not going to go to pegger.io and find Apache source code for 2.4.9 or whatever. It's just not going to be there. So this gets to be a little tricky for how we can actually combine this stuff when we use Git in different ways. And so Matthew actually came up with... Wait, the button stopped. Make the button go Matthew, there we go. All right, so if we look at the common uses for what we actually can combine, turns out we accidentally named things kind of similarly because we were solving the same problem just slightly slower at a different scale and level. So we both use slash RPM slash source RPM name as the title for it. Sentels primarily ignored the master branch. We just don't use it. We branch for EL7, we branch for EL6, we have that sort of stuff in place. So that lines up with the current F27, F28, F29, and we can either continue to let Rawhide be master branch or we can put a document in there that actually describes what the hell we're doing with all of this and where you find different things. This kind of hurts me conceptually because I don't like it when Git branches have meanings other than this is a branch of the code and when people use like a docs branch to put their docs in, oh, it drives me crazy. So using master branch to document things kind of hurts from that point of view. On the other hand, from a workflow point of view it makes a lot of sense, particularly because the way Pegger works where it presents the read me and everything, like that's what you see first is the master branch. So making the master branch explain what's going on with the package and you as a packageer can talk about your package and the way things work there. Or if we've got different people working on different branches there, if they're stream branches and things, like having the master describe that might be actually useful. So I might be able to live with my shutter reaction for that reason. No, that decision is there isn't an actual decision here for that. These are just the ideas that we are kicking around. And some of the problems out of this, once we started talking about this, then we started looking at, wait, okay, so do we use the Fedora Git for this? Because that'll anger the Sentos side. Well, no, we should use the Sentos Git for this, but that'll anger the Fedora. All right, you know, it's Git, it's on disk, it doesn't matter. We can mirror this stuff between both of them. So now we have to figure out how to do that and now we've been looking at the tooling for doing that and I've been harassing Patrick about looking at the tooling for doing that and it's been fantastic for interesting definitions of fantastic. Turns out that permissions become kind of a big deal with that because on the Fedora side, we really don't want people from the Sentos community just arbitrarily rewriting stuff that Fedora has active. We used to have ACLs and then we decided we didn't need ACLs. Jokes on us. Maybe we still don't, but we need to make sure that doesn't happen in some way. It still may be a social problem. On the other side, on the reverse, we have to make sure, because it's Red Hat specifically pushing the code into Git.Sentos and that cannot change. This may be getting ahead of our slides because we didn't actually write this down. So the actual proposal here is right now there's Git.Sentos where Red Hat dumps code and then Sentos works from there. So this proposal would be for Red Hat to dump code into the Fedora, the new shared Git instructor, which is a big difference because right now, we as Fedora own everything in Git there and there aren't any things which we're not supposed to mess with. This would at least have the starts of branches that are basically pushed from an external thing. That makes me a little bit squeamish because I don't want to go to the, there are Red Hat special branches here, but I think we think of this as just a special source and I think the advantages of it are worth that. And that kind of touches on the last bullet with there are going to be things that, if we move forward with this, there will be things that get pushed into Git that Fedora doesn't necessarily like. Some of the software collections, things that land in rel that Fedora hasn't necessarily gone along with yet. It doesn't have to be there, but just because it's in Git doesn't necessarily mean it has to get built or approved. It just means it's in Git. Which can be the case today anyways. Josh. If you're doing mirror, can you structure it so that it always just fits for one particular... Yes. Yes, so the question is can we mirror it so that they're always forced pushed from whatever internal or whatever, or have Git hooks that enforce that, and the answer is yes, we can definitely do that. It's more of a policy shock thing than a thing that we... It's trying to avoid having somebody with a Fedora project.org account try to push something into an EL7 branch thinking that it's Apple when it's not and making sure that they cannot do that. Yeah, the point is we don't really need articles to solve this particular problem. Too much talking. Alright, fine. Thank you. You're welcome. Yeah, is that everything? Again, cannot read the slides. Alright, so for all of this as we've had these crazy ideas, I've been relying heavily on Fabian who's in the back hiding, basically explaining what I want reinforcing to him that I swear I haven't started drinking that early in the morning and then asking him to start testing at a significant level whether or not this is possible. And he has gone back and forth with the teams, with Patrick, with Pingu, trying to keep me honest about what is and is not possible with this, trying to keep Patrick honest about what is and is not possible with RepostBanner. We've gone back and forth on this a lot. So this isn't just a wild idea in presentation. We've actually got a little bit of code behind this to test stuff out. And I have to say thank you to Fabian for what he's been doing. And I want to reinforce that this is an initial step. There are a lot of things that we can do as we attempt to collaborate between the two distributions and sharing a little bit of get initially is one of them. We would like to get to a place where we're doing more than just copying and syncing Repost back and forth. We're able to get that, but once we are a little further down the road with a common authentication scheme that allows for federation between Fedora and Centos, that opens up the ability to share a bit more. And we can start moving forward a bit down the road for sharing other things where it makes a little bit of sense. So allowing for authentication into the CI platform across both that isn't requiring an additional account or an additional account type or anything else like that. We can just start using groups and say, okay, you're in the CI group for this particular project. Use this, go do something. We can start sharing tests back and forth because it turns out a lot of the tests for simple things like HTTPD are fairly consistent between the two projects and what we have in one we have in the other. And again, that goes back to a thank you to Patrick and Fabian for the secure boot testing for SHIMS which is making Peter do a happy dance off in the corner which is always fun to see. So it's stuff like that that's already starting to show a little bit of fruit for sharing between the distributions. We have to do this in as much isolation as we have in the past. So the branching of Git, the sharing of Git is just kind of an initial step to see what we can do and where we can go for the accomplishment. And now because there are some fairly opinionated people in the room. Peter! What's one really nice thing about having master not be master not be rawhide is the So you're in favor of this. Peter is in favor of not using master for rawhide. Alexander? So the question is what is the point of having this all set up on the Git remotes when you could as a package or set up different Git remotes and do cherry picking back and forth between those things? People don't do that right now. I think that's the other thing. So I think having this way encourages collaboration and it makes it the default that packages can have kind of a co-ownership between the Fedora and CentOS communities and it sort of enforces that there is a relationship between the two. If I can slander humanity as a whole people are inherently lazy. The more you make a road block or the more challenge you put in doing something like that the less frequently people will do it and you can absolutely do that right now but to do that you have to go to two different systems with two different accounts to do different things and it's a little more complicated. We do have people who are actively doing that in both communities and they hate it. This is trying to make their lives easy. We want to give people who are doing this in multiple places across multiple distributions one single point to go for everything. Josh, do you have a comment on that? Yeah, actually I'm sorry. I want to steal Steph's thunder but he made a good point when we talked about what you can do as an individual is different than what you can do as a project, right? So if you haven't shared it across the whole project you can do interesting things like figure out the dealt with but it also helps the data source of the projects as a whole That's a really poor summary of what you thought was going on. Dusty. The question was having basically two front ends to it and doing a mirror is that being too sensitive? What's the benefit of that? Why bother? Why not just subsume Centos into Fedora? That last part I added. Damn it Matthew! I'm sorry. That last part I added. Damn it Matthew! So to some extent it is being conscious that the communities are very separate in some instances and very opinionated. The other benefit that it gives us is across a couple of implementation details. Right now the CI instance as it's primarily stood up we forklifted the Fedora apps or Fedora project apps CI that was Jenkins that Fedora was running. That's dead. That's gone. We forklifted all of that into the Centos CI environment. Fedora's data centers primarily stood up in the Phoenix area. Centos is primarily based in Raleigh so one of the benefits that we get for doing this is actually geo-separation and we can use the repositories as we sync them for more localized testing that don't necessarily impact the other things that Fedora is doing as far as release generation or anything else. When we're done with this whole mess hopefully it will just be one system but then we've got some built-in redundancy so it's not just oh Phoenix is offline we're down we're screwed we can't do anything it's Phoenix is offline great I'll hit the other one because it's still up it's still there I can still get data I can still do my job. Let's go back in that we haven't talked about yet which is right well we went by it in the slide with the stream branching so and this is the slide I said was interesting but we didn't put on the chart and that is in addition to the release name branches for modules and for also for CentOS SIGs there's plenty of reason to have branches that are named other things what we previously called arbitrary branching but stream branching or and those can be built into modules for Fedora hopefully for Apple coming soon and those can be built into CentOS SIG packages so I think that the opportunity for sharing in the non-default named branches is really significant and I think that's where a lot of the interest and excitement will come out of this as well. So to answer your question there are implementation reasons why not to it helps some of the infraside and there are a few other interesting areas where doing this and keeping it kind of separate initially gives us some short-term benefit that we can roll into longer benefit once we get a few other pieces in place so Melan, you had a question? Sorry, Laura's been Basically we have our own Fedora has some of the system and CentOS will be having some of the system mostly with how it's going to work We're not talking about merging the build systems yet this is just git so it shouldn't impact the build system setup Fedora's Koji instance will still point to source.fedoraproject.org it's still pager it still pulls the code from the same place that doesn't change You could try to Fed package build CentOS packages on Fedora now they won't land anywhere unless we configure something to happen so yeah as so this is a down in the weeds implementation detail I don't want to get into why or why not yet we have reasons for why we are not using Koji to build CentOS as it exists currently I would like to get Koji to a place where we can use it to build the system but that has to happen before we can make that leap so right now we cannot share build systems we need a common authentication setup in place which Patrick is working on I keep throwing arbitrary dates at him to see if I can make him twitch and try to get him to work on it a little quicker or just to toy with him because it's fun to poke Patrick from time to time Laura had a question for long the question is whether Ackles could keep you from accidentally creating get branches you didn't mean to someone who knows more about get will have to answer that I'm sure there's some way to keep you from pushing those things that does actually bring up an interesting point in that as we move to a place where Red Hat can push code into an instance that is shared by both Fedora and Cent we're going to need a consistent naming convention one so that we have a thing that Red Hat doesn't stomp on and two so that we know what those branches are so that has to be something that we'll need to have in place if we're doing Ackles in a way that Cent can't stop on Fedora and Fedora can't stomp on Cent in theory we could solve that but again I would have to harass Kevin or some of the other folks on the info team and make sure that that's true there was yes currently as it exists right now they stay in the two different branches there isn't necessarily merging back and forth although we would be able to cherry pick from one to another and vice versa so if there's a common Josh so the versions in EL7 for example are significantly older version locked and patched for what is different than what's in Fedora 28 so they will stay in their their own respective branches there might be some fixes that cross pollinate between them but those branches will stay they won't be a merge back into one or the other in the traditional branch sense AppStream that may be different we may both be building from the same AppStream or same modular content and that just is how it is Josh if you have more than 30 seconds come up and use the mic because I don't want to repeat so the thing we keep in mind so everything is really hybrid and two it was designed where we would use a CVS and we just kind of carry it over with it in an APC kit yeah so Josh is this is not necessarily code development get use this is dist get use specifically so there is a bit of difference between code development get and dist get distribution style yes you're absolutely correct and Josh's explanation for that is that we've been doing this for a long time and it used to be dist CVS and that also that it is very oriented around the release model rather than about package names let's add them from over there before the gentleman with the red hat tattoo I'm curious if anybody considered or talked to a team to see if this is going to be applied in any way with arbitrary branching because arbitrary branching was a big push at the top of the conversation last clock I'm just curious what that entails with this plan the question is have we talked to the factory 2.0 team and about how this would intersect with arbitrary branching I'm just assuming that arbitrary branching is like from the fedora side of things all we're doing is adding several more arbitrary branches that happen to have sent to us and whatever labels on them so I don't see an issue is there something I'm missing no I don't know so arbitrary branching can't be that arbitrary but yes it should work within some basic guidelines we'll work just fine so yes someone has asked Ralph if this is going to work was the Peter you've had your hand up again so Peter has a technical comment basically that it's much faster if you have one shared get repo if there's a lot of files so that is one of the problems that we are hoping to attempt to address with this I am told that I cannot make promises for addressing this and I feel like that's accurate but redhead is at least willing to allow me to promise that they will review the stuff that lands there they may not accept it but they will at least look at okay we have this fix this fix came with tests it passes community CI that gets it a little more than I threw some random stuff in a bug zilla and we'll see what happens a year and a half down the road Josh am I overstepping no the conversation that was had will certainly be yes so to some extent we are attempting to already use sentos as a test bed for some packages that is kind of the point for like the rto sig is an upstream for the redhead open stack Josh and Brendan had that in their slides earlier we have in sentos released a 4 package as an additional repository that can be added that basically brings dnf functionality into sentos so people can play around with dnf and whatever the alias was to use it as yum to try and get some feedback that we could bring into the well business side for whoever it was that was responsible for that they put out a questionnaire on the sentos mailing list hey please come check this out please use this so we are actively trying to do some of this it's just a question of how much we can do and how successful we can be with it now advanced for everybody's question do we have any other outraged yelling really I thought this would be slightly more dramatic than this yes the gentleman with the tattoo again in the what is the a logo I'm not familiar with that project Adam Adam returns the ribbing by asking what the timeline is on sentos finally just becoming fedora LTS am I allowed to just drop this on his behalf didn't I already make that joke once in this talk no so I think in seriousness there are both pretty strong community brands here that mean a lot to a lot of people so it's something we want to be careful with and that's again one of the reasons we want to proceed carefully and have you know front ends that kind of look like they belong to each community so what's our timeline for this now that we don't have outraged yelling tomorrow can we turn it on Fabian what's our timeline for this you're the one building it soon the famous deadline I will translate soon as when it's ready I I am hoping we can have something in beta form the community can play around with hopefully around November I am hoping that we're able to pull that off for a get instance that we can at least show to the community and say look here this is the thing I want to wrap this up as kind of an early Christmas present and say here you go yeah before that would be good for just saying six weeks yes there that was the right answer you you sold yourself on the deadline I was pushing for later I'm not responsible for this anymore Adam yes why does Josh want it sooner so I would like it sooner because if we push it to November and we start running the holidays and then people go pick it up and play with it and then we don't really get feedback from the community right yeah Josh would like it before the holidays basically because the holidays are down time as we saw from my graphs earlier yeah holidays dev conf no one gets anything done that line of ribbing and you think I'm going to call on you again I already did I called him already Adam is trying to walk back he changed from the the joke into a compliment so we'll take it you should have stuck with the original man the original was fantastic well thank you everybody