 I'm Matthew, I hope you know me by now if you don't come introduce yourself to me later. This is Fesco Fedora Engineering Steering Committee, one of our, you know, the technical leadership body in Fedora, an all-elected panel, and we have most of the people here. I'm not on this panel, but I thought I'd, you know, introduce the general idea, and I will run around with the microphone if you have questions, because we're recording this. I'm sitting in that chair, right there. All right. And with that, here's Fesco. Should we do an introduction round? Do you want to know who actually those people are? So I'm Alexander Fedora, I'm book war on IRC, I'm book war everywhere, almost everywhere, not on Twitter. It's not me who's spamming Twitter from book war account, and yeah, Fedora is my real name, so if you're just wondering, and moving on. I'm Petr Szabata, content on IRC, been doing this for a while, you might know me. I'm also responsible for modularity in a way, so you can hate me. I'm Kevin Finzi, and I've been on Fesco, I think, since it started, so I've been around a long time. But I'm not really sure what I do, so. So I'm Igor, it's probably me who was touching your packages, and you don't like me, because I'm doing that over and over. I'm trying to do crazy things with modularity and whatnot, so, but I would love to talk to you. I'm Zbyszek, I work on SystemD, and also sometimes touch many packages. Justin Forbes, Fedora Kernel-Maintener, and been around for a bit, I suppose. All right, do you, anybody on the panel want to say something or make a statement, or should we go right to questions? Right to questions? That's all right, that's what I figured. Does anybody have a question for Fesco? Wait, oh, that me. Why do we do mass rebuilds? To take advantage of new compiler features, et cetera, changes in RPM itself for payload and such-not, and to also see that things still build correctly, because the best time to figure that out is at the mass rebuild time and not five minutes after the big exploit for your package was released. Actually, we've seen gating in place. We have plans to add the rebuild check for every change you do, so it shouldn't happen like a mass rebuild event just once in the release cycle. But there is a very controversial topic, I think, at least I think it's controversial. If we need to check that packages are possible, it is possible to rebuild packages after the change, or should we actually do the rebuild and update packages which were depending on this one? So I expect there will be heated conversation about this on Fedora development, so get ready to it. Merek, speaking, what's blocking Fedora as Windows Linux subsystem? Why we don't have Fedora there? That was a question for me. Why don't we have Windows subsystem for Linux? This is not a technical problem. This is a legal agreement problem, and it actually works fine. The problem is getting it into the Microsoft app store, and this is a model we're not prepared for. Basically, the Fedora model is we made some open source. It's available in open source license. Come and get it. There are things you can do with the license or things you can do with the trademark that we give you permission for or things you can ask for. In order to get into the Windows store, there is an app agreement you sign with Microsoft, and there are causes in that, which we as an open source community and Red Hat as the company ultimately responsible for that signing just couldn't agree to. So we're kind of stuck at that. I would love for Microsoft to come and say, hey, we're part of the open source world. We can just work in an open source way without having company legal agreements blocking everything, but we're not ready to do that yet. I guess that's the- It's stuck until we find some way around that particular legal problem, basically. Possibly forever. I mean, there are new people at Microsoft and new ways of thinking of things at Microsoft that might change this because really, right now, if they wanted to, Microsoft could take Fedora, put it in WSL, use the Fedora logo, and say, here's Fedora for WSL. They just don't want to do that. They want us to take on the risk and liability of that. Just to add to that, right, it's kind of mind blowing that anyone has agreed to this. Yeah, right. Langdon says that it is mind blowing that anyone has agreed to this, but that's not something we can do anything about. All right. Fesco topics. Yeah. A general question for everyone. What do you think the health of Fesco as an organizational body is in terms of things like getting new members on, productivity, things like that? Do you see it as being generally healthy or anything you need to improve? What's the general mood? We made some process improvements. Last year, the rules were changed to approve more tickets offline. And this year, we changed the nonresponsive maintainer process to actually put more work on Fesco, but less work on people who report nonresponsive maintainers. And I think that generally, our voting speed and processing speed is good enough. Fesco should not, in my opinion, should not be a body that sets the policy and decides. It should be that the community comes up with staff and Fesco basically rubber stamps, things that have been discussed to such an extent that it's technically clear what the right choice is. I mean, if there were multiple Fesco votes where we would have five to four votes, this would be a sign of staff being decided at the wrong level because Fesco members don't have enough technical knowledge to decide specific technical matters. It's, in my opinion, Fesco should work as a mechanism to have public discussion and public approval of changes that will happen. And I think we are doing okay in this regard. I think that's true. But also sometimes Fesco actually decides on the policies and defines the direction of the distribution on the technical level. And those tickets, they are open for years sometimes. And they are not really moving anywhere. I think we need actually to figure out how to deal with those. And I don't really have an answer to that. Generally, yes, I would say Fesco is healthy, but it's still, there's space for improvement. From my perspective, like, I mean, you be member of Fesco, basically. So it comes to change process and non-responsive maintenance process. Essentially, there are two things which Fesco is dealing most of the time. For change process, it feels like really when there is a help from community, when community has a proper discussion of topics on Fedora development list, then it makes the life of Fesco so much easier. You just go fetch the data community provides the feedback we have and you can actually make a decision on that. So we think we are stuck with usually things which are not discussed enough on the community level. And this is like really becomes painful because then it's Fesco needs to figure out like how to even discuss those things and how to think about them. So the more we have a feedback from the community on some changes, the easier life of Fesco is and usually varies quite good feedback before we even start voting on something. And based on this feedback, the life of Fesco is quite easy. Hi. My question is that when there's a policy change which is approved by Fesco or when there's a new policy which is approved by Fesco, would it be Fesco's responsibility to ensure that whoever is actually responsible for implementing the policy does actually implement it? Yes and no. Not really. What it comes down to is when you ask for a change, you've approved a change. We can approve that direction but we can't force people to do work. This is a community project. If Fesco doesn't have a budget, occasionally if we have something that we can justify, we might be able to get budget through a team that way. But for a lot of things, if you come up with a plan that you want to change, you also need to get the people behind it who would actually implement that change. We can ask nicely and a lot of times someone will step up and do that but we can't force it. I know you'll answer any question but what is the question you wish somebody would ask you? I have actually a topic which I wanted to discuss. The question is why do we need a change process and why do we have it? One of the concerns I have, sometimes I see the change coming when it says let's update this package to a new version. Then this change goes to Fesco. Honestly, there is no reason for me as a Fesco to say no to updating a package in Fedora. It's like this is what we do. We take packages, we update them. In this sense, this change approval process looks redundant but I think that it's just because it's approval of a change is only one small aspect of what change is supposed to be. I think we are not doing a good job in making sense out of this change description and out of this change process. What is important when you file a change is not that Fesco approves it. What's important is that everyone knows what you're going to do. You have a place to communicate with. You have a place to explain the impact of this change. You have a place for people to provide a feedback. It shouldn't be like, oh, I got a Fesco approval. I'm done with this change bureaucracy and I'm just going to keep working on my stuff. I really would like to ask people who work on changes in Fedora to understand that change is not for Fesco. Change is for you and for people who work with you. It's important to write down. We know that the change will be approved because it's an update to upstream version. No problem with that. But please still add the required information, add the impact, add the non-impact, because all of these things are important for current change, but they are also important as a historical document. And three years later you can go back to this change. We probably updated once this library before and then we had those impacts and so we keep this information and we reuse this information. So it's important also once you're done with change, you record what you forgot while you were filing this change in the beginning. So change is the ultimate project organization tool. It's not just something Fesco asked you to do so. Fesco can say yes or no. This is one of the things I really like to communicate to wider audience in Fedora. Have you talked to Ben Cotton? Were you at his talk about this? I was just going to say often the changes are also a source of documentation or inspiration for other distributions to do things. Ben here? Is he hiding? I hope he's okay. I saw him last night. So I'd like to piggyback on the change question and I'd like to have Fesco's opinion about this. A lot of the change proposals are actually regarding a very specific Fedora version so we want this change to land in Fedora 31, Fedora 32 and so on. And something we've been doing with the right-getting initiative is to also provide a change proposal to Fesco and the wider community as well about changes that are not directly linked to a certain distribution but changes that are happening in the way we do things in the community in general. And I was wondering what Fesco is on this. I actually think this is a good idea and we should build more on that but I don't know if I'm alone thinking this or if there are more people agreeing on that. So that probably wouldn't be a Fesco thing. That would be more like a council matter. It might be somehow part of the change process that Ben was talking about. Unfortunately he's not here. But I think Fesco really is about driving the technical changes for a specific release. What you are talking about is a little bit different. I agree and disagree on that. The change proposal was much about how to technically implement an objective that was already approved by council as something that is important for Fedora. So it's not about do we want to do that or not, but is this solution something that Fesco agrees with? So it was much more on how to implement it rather than if we should implement it. So there was discussion not too long ago about trying to extend a release cycle so we could do some longer term things. One of the things that came out of that discussion was it's okay to have things. We have goals that are not the next six months. That are going to be a year or two years to implement and go through. The way that the change process is kind of written, I think, or implemented seems poorly suited to that, but I don't believe that you have to think exactly within those lines. I think it's pretty easy to say we want to do a change and you could say I want to do a change for Fedora 32 that is the start of this implementation and then we expect to be finished by Fedora 33 or 34 and as long as you've got a plan and laid that out, I believe that that would work within the change process bounds fairly well. Probably the change is not the top level of hierarchy when you think about general Fedora development. So you can have something on a higher level which then splits into multiple changes over several releases. You don't have to try to fit everything in the one release and one change. It's okay to have initiatives which span several of those, yes? So a good example is the Python 3 rollover which has been going on and will be going on for a few more releases and it's split into multiple changes and I don't see this as a problem. I guess we don't have a fixed word for this. We have objectives but not like... So something which spans multiple changes. But anyway, it exists, yeah? Hello? Oh, my. So Fedora has always been a really great place to put somewhat disruptive changes so you mentioned Python 3 a little bit ago. There's lots of stuff that come on the horizon and we kind of have this side boat in the Copper Repos to do sort of, we'll call it odd ball Fedora stuff but it still feels like there's not really a strong engine to test changes before they go in. So like if somebody here said, hey, we should start doing a free BSD kernel, like we don't really have a way to do that without forcing people and we'll hide on a free BSD kernel for a while, right? So is there any plans or discussions or anything else about ways to do very disruptive operating system level changes that are not official Fedora? Because we kind of did this with CoroS and Atomic, right? It's a very different kind of thing but it was also very artisanal and white-gloved the way we did it. Is there some way to make Copper Repos more do more things or just if we want to change... If we want to test major changes without forcing them into raw hide, like the future of SC Linux or something like that, is there a way to do that? So I think that we cannot expect the core Fedora infrastructure to be the place to implement those changes. I expect that stuff like this is done on a separate system set up by one person or a group of people for a specific purpose. Like for example, with Python 3.8, now there is a copper that has 2,000 packages rebuilt with a different version of Python and this is possible because all stuff in Fedora is open source stuff that you can modify yourself as you see fit and as long as we keep this possible and not too hard, I think that this is the way to go. I mean, I have my own server on which I'll sometimes compile a few thousand packages if I want to test different things and it's not a big issue. There were discussions about modularity and proposals to have modules set up in a way that would be really hard to do stuff on your own and it will actually interfere pretty strongly with this kind of approach because you wouldn't be able to do stuff outside of the core infrastructure. So I think that as long as we avoid stuff like this, we should be good. Question was, if you avoid stuff like that, how do you evolve the distro? Well, so let's say that I want to switch to K3BSD kernel. I got there a bunch of machines, boot them with Fedora running on the latest 3BSD kernel. I show everybody that it's good and after I have shown that this is good, maybe fix a few bugs, then I say, okay, let's switch over and then we switch over, right? Well, so I have to, I mean, it depends on the scope of the change, right? If this is a small change, I can do it on my Raspberry Pi. If it is a big change, I will probably need to convince a bunch of people to work with me anyway. Well, and this is the type of thing that actually, so I mean, I'm sure everybody saw the development thread about changing the baseline compiler flags and basically making sure that everybody, oh, 90% of you need to buy new machines. You know, that was turned down, but there's actually a lot of merit to what they were asking for and a reason that we may want to consider something similar. So there are, there was some discussion about various ways we could implement that, but that's definitely one of those things that you don't just shove into Fedora, we're going to, or into Rawhide, we're going to have to side chain somehow and figure out how to implement that. And I don't know, I don't know that we're far along down the thought process of how we would do that, but I don't, we don't have an infrastructure that will make that easy to do with, you know, what we have now. But we are talking about extending the possibility, like you did it for IoT, currently the request is like for any kind of feature, I'm coming to Fedora and I want to try to do this. It definitely doesn't scale well the approach we did with IoT. We know that if you get hit by a bus, we're in trouble. I think we can agree that Fesco doesn't have a solution for this problem. I think we agree that we have a problem and we want to solve it somehow, but we don't have an answer inside Fesco. This answer should come from some other places and we need to just get into solving this. Yeah, so this is a valid use case, a reasonable problem, but like Fesco is not in position to solve this for you. It's like you who are in position to propose solutions for it. But on the other side, you can also not expect that you come to Fedora and say like I want to do Fedora for, I don't know, which kind of IoT you imagine and then like please Fedora provide me access to that. This is also not going to work this way. So we cannot just provide as Fedora provide on demand any type of hardware out there. So there should be some process how you can start with some smaller things which you can manage and then you get into Fedora where it gets supported properly by Fedora as a project. So I think part of it, like for IoT, we have a specific thing we did here which was you made a Fedora council level objective and so this is something that is at a top level. We have prioritized this. We have given a framework for it and we happen to know that you and the people you are working with can do this without actually needing to drag Fesco into it. So you are not just off doing a rogue operation and you are empowered to do that. And I think that if there are other things that fit into that, people can come to the council with that as well and try to figure out a way for that to happen. How long are you going to talk? Basically I just kind of fundamentally disagree with the whole discussion which is that I don't think it's Fesco's responsibility. I think it's the council's responsibility to figure out how to enable kind of a new thing. Fesco has a role but we have got to go and put a bunch of whatever will behind the fact that we want to do IoT, the fact that we want to do modularity. At least in my opinion, Fesco is about the operational aspects of the things that we have decided as Fedora to do. The people who make that decision sort of is the council, right? Again, going back to Justin's point earlier, we can't actually make anybody do anything but we can at least kind of guide where the approach is the right answer. So when Peter comes along and says, I want to do this IoT thing, we say, hey, well if you write it up in this objective we can kind of support that and figure out how to make it through there. The next person who comes along and has got that new thing, I don't think they need to figure out how the release engineering process works either, right? I think they need to come to the council, say I got this idea and then the council should be able to come back to them and say, okay, go follow these steps to go accomplish your goal and some of those steps might involve Fesco and some of them might not. I think there is still like the role of Fesco that we overview that all the initiatives which are currently implemented by implementing them we don't lose the possibility to offer for other like new spins or some derived work on top of Fedora. So the Fesco role is here is not to provide like resources for new initiatives but it is to verify that new initiatives still can come, can be worked on. I don't know if I explained it properly. So we need to keep the possibility for people to do things on top of Fedora and this is the Fesco role to make sure that these possibilities are still there. But when you come with a new thing you want to build on top of Fedora, this is a council responsibility to sponsor you kind of with resources and support to do that. I heard from Ben he's fine. Other questions? What do we have over here? Brendan again. So if somebody wanted to pursue a free BSD kernel or an alternate compiler flag set that was incompatible what is the right path to do that? Is it going to Fesco? Is it going to the council? Like if we have resources what's the responsible way to engage? I think you should initially go to the council definitely and get such a major change approved and then you can go to Fesco and get as much perspective as Matthew said. And after that we can definitely figure out like setting up a shadow instance of Koji and other builders and such and test those things out. It wouldn't be right to just go straight to Fesco and ask like I want a site tag where I will be testing a new kernel. I think that might be too disruptive and it's just really working. I mean with the compiler flag change changes for compiler flags every few releases usually around security and things like that so changing the baseline coming to Fesco at that first was fine except it was immediately shut down for obvious reasons. Like I said a lot of people have to replace hardware. So now it becomes a different effort of yes it's a great idea to be able to provide binaries that have these minimum baseline or in theory it's a good idea so we have to figure out a way to actually do that and test and now it becomes a much bigger effort than we're just changing our default compile flags. So that needs some additional involvement and it's not just a straight Fesco thing. Maybe to give an example in the FreeBSD case like if you need a change in current RPM packages and policies to support this effort when you go to Fesco to approve the change of this policy in the current state of Koji, state of packages and so on but like just to start working on it you go to console because it's a new effort which you built on top. I just want to say I'm seeing the Pheronics headline from the session. Fedora consider switched to FreeBSD kernel. No, it's been decided already. So returning to the question of flags I think that going either through the console or Fesco or from the top down is just the wrong approach. The right approach in my opinion would be to start from the bottom to show that it actually has not measurable improvement in benchmarks and to convince people and to start small with a subset of packages where this makes a difference and then after it is well known that this is a useful thing and it is well known that it can be done without breaking the world then maybe we can talk at the Fesco level to make this the default policy. That really depends on the nature of the change. It's much easier to actually to switch kernel or compiler flags than to change the infrastructure or the release model and things like that. So for those things you need to go to the bodies such as console first I would say. And yes, you said you need to convince people that it's absolutely true. You need to convince the council too. So if you have the data. Sure. But the council should not be setting detailed technical policy decisions. Absolutely not. It's just setting the direction of the situation. I also think the council should not be making decisions just in the council in our dinner party at flock situation. We should... The council decisions should reflect what the community is thinking. It's kind of our job to be leaders that listen and reflect the discussion that's already going on. So if it comes to the council first, we probably will say that sounds interesting. Can you make sure there's actually support for this thing that you want to do because we're not going to... Things can't be top down because it just doesn't work because again we can't tell people to do work. Maybe. Denise can but she says no. Maybe to add clarification, we are talking about different paths here. If you're going... If you choose wrong path and went to a wrong body, it's okay. We will direct you to the right one. So don't be afraid to come up with your ideas even if you're not sure who you're supposed to set these ideas for. Like for council, for FSCOR, just for contributor. We can start from something and then figure it out. So don't be afraid of it. Yeah and on the flip side of that, if you come to someone and then they say, wait, this is the wrong body, go here. It's actually trying to help. It's not just giving you the run around. I think it's also important to note that like for the council thing to use the example of the compiler flags, the council may see that, may see value in that and may say this is great but what should come out of that objective should be provide faster support for newer hardware or something like that. Not that the answer there shouldn't be dictating the solution. It shouldn't say we are going to rebuild Alphadora with these flags and have a separate distro and release. It should be the goal and then the how should be determined after that as the best way to get that goal. Do you have a new question or comment on this? Yeah, what I am writing is more of a, I think it's fine for things to go to FSCOR first because I think the change proposal in FSCOR is actually a much more open to community feedback and therefore something like the compiler flag I think. I think it's fine to actually propose that to FSCOR see how the community reacts to it and that's one of the, that's one of the points that the FSCOR members are sending into account when they decide on a change or not and in this case the community say, well we can't do that because of these and these reasons and at that time then if we want to provide faster support for new hardware then it is fine to bring it to the, to the console as with a different, okay, we tried the technical solution and that got shut down by the community for good reasons. Now that doesn't solve our problem to provide faster support for new hardware. So is FSCOR a console agreeing with this objective and then okay let's go back to, if we can't find the correct, if the solution doesn't work then we can come back and figure out what is option two of some series and so on and then work with FSCOR to figure out what is the correct way to actually fit that objective. Maybe do it like rejected change is not a bad thing. It's just like the first step, if you're changing, you like suggested the change, the change got rejected is, that doesn't mean you should stop everything you're doing, you probably just need to rethink and find a different way to reach your goal. So just rework it a bit and like people should be afraid of rejecting changes and don't treat this as a failure. It's just the first approach, the first method and then you go forward. Yeah, hi. As the person who put in the change request for secure baseline compiler flags and it got turned down in a couple of days, my question is that if I want to change, does FSCOR recommend that I should first go to Fedora Devil and get some traction on it and then come to FSCOR and ask for a change request? So there's general policy that the change is first sent to Fedora Devil and after a week I believe it is only opened in the FSCOR tracker. And this is on purpose, right? The change should always be discussed first at the mailing lists. But if you're doing some bigger change, for example some security compiler flags, it definitely would not hurt if before filing the change figure out all the details you would send an email to a Devil mailing list asking I would like to implement this security flags, I would like to do these things, what do you think about it? So do not create change proposal, I will put this, this and this flag and wait for feedback for it, in advance and I'm thinking about these 15 different flags, I'm not sure about this one, what do you think? I think it's based on whether or not you're sure what you're trying to change. Like if you have already a plan in mind, then it's better to discuss the plan when you write it as a change and start discussing, starting from there. But if you have idea which you need help to polish, like because writing the change is actually a hard job and you may need some help from community just to get there to write the first draft of this change, then it's perfectly reasonable to go and start rounds of initial discussions in various mailing lists before you get to actually writing the draft. So just mind the deadlines, but you definitely can ask for the community for help to even start with the change, like the beginning of it. Well and that's one of the things too that will help you polish. I have a list of security compiler flags I want to do. There is a specific reason that this one flag will cause problems for a large number of packages and a small tweak to that would actually be fine. But if all of this discussion happens before the change is pushed through, then you know that and you make that small tweak and you get mostly what you want or close to what you want instead of nothing at all. When you just come through with the list and then it gets discussed on the vell that this is going to work and why, and then the change is not modified to reflect that, the change does get turned down when chances are it could have gone through with some minor tweaks. Next question. I was told to ask this. I have no idea. If we look at Matthew's dinosaur diagrams from the beginning, from the early presentations around Fedora, the Apple usage is fairly drastic and it does seem like Apple doesn't necessarily get the love that it should from the Fedora community. What changes would Fesco like to see to help support the Apple user base that is there? So modularity is one of the things that can help here in my opinion as we actually introduce support for modules in Apple 8 that is upcoming still. Even though the beta was released just recently, we can build any content that is released for Rohit, for any version of Fedora. We can get it built automatically for Apple. That means that you need to opt out to provide Apple content. I think that will definitely help with making more content available and making sure that the package is actually built on Apple and so on. I think that's one way to do it. Besides that, Apple has its own steering committee, maybe we can work together somehow. So there have been various instances where staff was added to Fedora or RPM and it took a long time to trickle down to be available in Apple and this makes contributing to Apple unnecessarily hard in my opinion. I think there are many times where various powers that define how Apple is built could make it easier to contribute to it by pushing some changes that are fairly safe, like new macros, without delay. So is it the case with the modules that I get my magic wish of I have one stream and it just automatically gets built across all Fedora releases and Apple? That's all I wanted. Something that is built automatically doesn't mean that there is automatically support from the maintainers for any bugs that come in this newly built thing. I guess it's not only for the Fesco but that's the question, what do we foresee to help maintainers to actually live with that reality that they need to expand their forces to support Apple if there is automatic rebuild? So the feedback on packages in Fedora right now, 29 and 30 and Rohide, this is very consistent that when updates are released you get a karma for the package in the current release maybe within a day or sometimes even before the update hits testing and for Fedora 29 you get it maybe in a month or maybe not at all. So people who are testing stuff that we built in Fedora are also the people who are running the latest stuff and then the people who use the older versions they don't seem to contribute to Fedora either testing or development or anything else and I think it's even worse for Apple. I mean I have built packages for Apple and fired an update and I don't recall getting any feedback ever so I think this is the problem right? I mean nothing is for free and so I think the sad answer is that sorry the support will not happen without input from people who use Apple. Apple definitely has that problem as far as the usage is just very high but those people who use the packages from Apple don't really engage in the community or they're not really part of our community they're more consumers they're using this package and if it breaks they just say oh too bad and go on. If you even look at the number of bugs and bugzilla on Apple packages versus Fedora there's hardly any Apple bugs even and I'm sure that the packages are equally as buggy as Fedora's but you don't see that reflected over there so I think one thing we should look at or the Apple Steering Committee or something this entire community is how do we engage those people to know to get feedback from them to get bugs from them to have them join the community and like test things, that kind of thing is really difficult and I think Cintos has this problem also in that there's so many people who use it but don't participate in the community that it's very difficult to get that kind of feedback. So even though we don't get this that we will be building from the same source even though the artifact is different and it's tested in Fedora and we are starting to use OSCI more seriously now then I actually do hope that we will improve the quality the fact that it will be built automatically also means that the packages won't have to do an extra effort to actually get the content or maintain it or fix it and so on so I actually do hope that this will help. Yeah, I don't believe in that in my particular area because there are a lot of tiny but important details that prevent direct rebuild and reuse of the identity or authentication packages on CentOS as it is and we know about that because we have actually CentOS admins in the upstream community being very loud about wanting to have these new packages and not being able to actually run them there because of bugs and and the things and that actually goes much further than just automatic availability of packages. Nobody tests this delta between what is in Fedora and what is going through the rail back channel into CentOS let's say this way. So if we are saying for example that Fedora becomes an upstream for rail in real like there's no delta that big that justifies the ABI differences and semantic behavioral differences in various pieces of the infrastructure and the distributions themselves then we might get some benefit from automatically rebuilding in Apple and contributing to pre-testing of what goes in rail. So kind of this Apple Plus CentOS would become another upstream testbed for rail. But so far I don't see this happening and I'm really concerned how we get to solve those problems that we see particularly with ABI differences that inevitable coming from Fedora to CentOS even with the CentOS 8 who or let's say where is the forum that where we should be discussing this. Is it Fesca Plus or Apple Steering Committee or it's a I guess it's a wider question I think we should start some joint discussion between Fesca and Fesca together and see where it takes us. It probably would help a bit if we could it's not really easy if you want to get some new feature in packages which are in rail so RPM is my case I want to get some feature and let's say it's I cannot just send pull request and get some approvals from those people it's just you can file bug somebody could look on it and say we don't want probably to support it and basically we're evolving in Fedora a lot we have dating the change log we have rich depends we have so many new things but sometimes it's impossible to start using them for rail packages so even those we need to build them automatically but sometimes they will start failing because we started doing new features in Fedora and they will start to be again out of sync so if there would be some easy way how to contribute to rail packages then that could work I think what I was asking but I'm saying that the automation is not an answer it's one of helpful steps but it's not an answer to the real question and I think the modularity rebuild thing happens with certain modules not with every package so if I need a Rust stack or something like that that's going to be much better than any low level system packages or but we're not going to solve this problem it's like in the end it's initiative for the community I can see us trying to align the tool trying to help with infrastructure tasks but in the end it was a question to the IPL community do people want that do people invest time in that so there is no magic answer we can just provide to resolve this problem if there are people working on that they should have a tooling and a possibility this is what we need to provide but we cannot replace those people with ourselves we need to solve that problem for them I see infrastructure tooling improvements easy branching easy cherry picking things like that but in the end there is not that much we can do and honestly it's a different branch you need to maintain a different branch you're not going to change this we've got like five more minutes maybe one more topic does anybody else have anything IPv6 yeah I think Facebook people are going to help us with that I got one I look up here and I see five people who are employed by Red Hat and one ex-Red Hatter a lot of most of the Red Hatters are people who did awesome stuff in the Fedora community and were hired into Red Hat maybe all of you what is the problem it's a fully elected body there's no Red Hat appoints people but we end up with Red Hat dominated Fesco is it a problem is there something we should do about it what should we do I think more people who are not working should go for the elections and people when they're voting they probably should read the interviews and think is it makes sense or not I spoke to some number of people and what people say oh I know Kevin I'm going to vote for him because he was there forever he does awesome things in infra he does a lot of nice things people just need to think do I vote just because I know the person and he does nice things or do I vote for the person who wants to do this and this and this particular thing I would add to this maybe like I would be concerned that people consider Red Hat employee as different as normal Fedora contributors like honestly in my case for example like I had 10 years Fedora experience before and just one year of Red Hat the fact that I'm Red Hat employee now doesn't make me some evil person who just joins Fedora for some evil agenda is this like I'm still a Fedora person I've been here for 10 years you know so let's not differentiate people based on their employment status but as Igor said try to and also not differentiate on the years of experience let's differentiate on top of what we are talking about in the interviews like ask questions, engage with people we may consider like add in more interactive sessions to fast collections not just like free written questions if there is a community request for it because like we really need people initiative like choose your fast code wisely you need to understand what are your criteria and what you want to hear when you elect people to fast code and let's not focus like on the employment status here we are not here for employee tag yeah I'll add to that that in every election I've noted in my little questionnaire come ask me questions if you want to know what my opinion is on something not once has anyone ever done that so that kind of worries me I would think everyone should feel free to ask you know perspective electees questions what is your opinion on this what are you going to do about this what do you think of this and I think that will help you choose a good fesco so as someone who is obviously a long time contributor within the fedora community and not employed by red hat one of the things that has stopped me in the past from considering being on fesco or fbc or council or anywhere else is the mere fact that I don't know how I would actually be able to get rid of it and make the time commitments and the availability and things like that I think one of the underlying problems is that if you are a red hat and you become part of the fedora governance in some way it is a lot easier to be able to say I have to make time for this and be able to be there which is more or less that self-fulfilling prophecy that it winds up being red hatters even when it's not intended to be that way from my point of view it's kind of hard because it's hard to figure out how to justify to other people what is the value of being part of the fedora governance in some form or fashion even though maybe as a contributor it's obvious or someone who's deeply in the community but like it's hard when it's outside of that I'm not asking for solutions here I'm just saying that this is the legit problem I have even though I would very much like to be part of those things and I think that's going to be the last word here because we're running out of time one more okay make it click so originally I wanted to ask should red hat stop hiring awesome people from the fedora community but the answer is obvious so my question for thinking at home is should other companies hire awesome people from the fedora community and how do we make that happen something for facebook to think about