 Hi, I'm Steve Gallagher. I've been on Fesco on and off for about four years now. I work for Red Hat in Architects role, but really that means I heard cats and I try to ask I try to ask politely that you will not Okay, so Mainly that work related stuff and DNS DNS like and I've been in Fesco for one and a half You never have you ever not been in Fesco? No, I think I was in Fesco before it existed I was involved in Fesco since way back when but I did take a couple years about it I'm Josh. I've been in Fesco for a really long time I was out of it for like a release or two and then I got really irritated with Fesco. So I joined back up Yes Now you're part of the problem, right? Yes, exactly and For my actual job, I pretend to make cake now is the time where the audience asks questions Yeah, exactly doesn't crash horribly things just don't work without a crash. So When do we want to talk to Fesco versus the council? Oh, that's a great question If it's an engineering related item Talk to Fesco, it is a work group item So like cloud work group talk to the work group And if you want if you talk to the work group and you don't like their answer Do you want to escalate that you would go to Fesco and if you don't like what Fesco says, then you can talk to the council But we're probably If you wanted to merge all the secondary archcodies and the primary codies for building packages For instance, we would probably want to do a combined Fesco, cancel Every Fesco would need to be in charge of making sure it's done right But the council would probably need to be the ones that sets the guideline that this would be a really useful item Because it serves good purposes for the door But because of the structure of the council, you could do that separately You could talk to Fesco and get, I mean it would be more than just Fesco We're talking about merging the secondary architecture, Fesco, QA, you know All the engineering aspects of it Talk to them, get a general consensus and then as the engineering representative I can talk to the council for Fesco But certainly we'd invite whoever's proposing to that council Although our world is small enough that most of us are going to be aware of what's going on anyway The council is really the place where you want to go If you want to have an effect on the Fedora project's strategy And Fesco is more of a place where you want to go when you want to have an effect on their tactics of getting there Or if you have like trademark issues Like what versus how? Right, yes But then the council is also for policy, high level policy, governance, blah blah blah Triggers, trademarks, yeah Which architecture? All of them What's the... Stop having secondary architecture Change the definition of secondary archers from the way they're built To the way that we compose and ship them So the tooling would say we put this stuff in the secondary location We put this in primary Then we can do things like make, you know, like Make server for 32-bit X86 only Or we can promote AO64 to primary for server But leave it in secondary for cloud on Wednesday So essentially change the way Koji builds So that it builds all architectures at once And then if some of them fail it doesn't fail the whole build It will still fail the whole build And that will at least for stuff that's not running zero To just exclude arch and carry on So I'm going to have to exclude arch and MIPS on my packages now Is that potentially? If we do this, there's no kernel for MIPS At the moment the MIPS stuff is so sketchy and nonexistent That it's really not something that needs to be considered But it would get to a point where it's built completely enough That we would say okay we're going to put that in And we would build that as well Assuming we have the hardware Presumably it would be done in an incremental way But basically when you think about primary versus secondary Isn't that really like a fedora last kind of concept Fedora next solved that in a more elegant and fine-grained way I mean I agree with what you're saying But the reason we have primary and secondary architectures Is because they pissed off when they do a good All the architectures that are secondary now are faster Yes, right, but that wasn't always the case So to your point, right, it is a last concept But we haven't transitioned everybody to thinking that We can technically do this now Could we make it so the builds aren't failed? Excuse me, down-tides are too, but... I would love that Maybe if... Well, to make it so the builds don't fail But checks the body layer So maybe we allow stuff to be considered We could also move on to some other thing That doesn't need to be shared I think it's an interesting question in terms of rings Like maybe what you want to do is say Okay, these rings, everything has to build, these rings You got to be careful with that though too Because the kernel, I mean obviously I'm biased at that viewpoint But the kernel's always in the ring, being zero, right? But if there's somebody out there who's trying to bootstrap A new architecture like MIPS I really don't want my kernels failing For something that is legitimately not part of Fedora I think this would have to be still a process To add it in, to get to that point To be at the point where we consider bringing MIPS in And building that as part of what we build in Fedora It wouldn't need to be sufficiently complete They would have needed to... So they would have basically had to duplicate What a secondary arch already does So we're not moving, we're not taking away The concept of secondary architectures We're just making it even one level lower So they already have to do all the work To prove that they're going to continue to do it Which is what a secondary arch is Before they can become a... An add-on for Fedora, I guess Whatever we're going to call that A 1.5 arch What a co-dearch There you go Yeah, that's it Like in the kernel case I think, you know, I get for some reason PPC kernel gets some nasty button It's fast to build In a period of time You can go in and you can market That it's a no-build arch So that you don't build the VMNs and the modules You get the headers and the things That we have to have from the kernel to that arch And that's okay But the PPC people then need to go and fix the kernel Whatever they care about Yeah, it's going to mean that A build, co-build or fail But you've got a quick workaround That you can work to go Or we've got to fix this bug When your power fails Carry on and when they fix their stuff We're far better with power six machines In my office But two, nobody else cared, right? And it's totally not the case like that now So that's really good to see Those constituencies, those six Should still, yeah, we should have the tools That they can make them a Blocking feature for Yeah, it won't be a blocking feature For a release like it fails to compose I mean, the arch team is adding Getbooks, they file and you push a change And exclude arch or exclude Write something past its fingers Or they've added exclude arch, you know Fail builds that are allowed through E-Mays Like a fail and then our kingdom thing Failed, do you care? Ask about cocktail The official repository is because Some time ago I I wrote an article about The official repositories of Federal Arc slowing down the go But it was not the part of the article I don't actually see it as a bad thing What I wanted to point out Was that we've got like copper going But we don't only have any path For the copper contributors How to with some software Become very popular and it You know, starts making sense to Move this to federal repository We don't really have any clear path For the people and we don't really have Any motivation for them Because I've got the copper as well And like for me that was a thing I needed for myself So it built it, it works for me And that's where it ended And I haven't found a motivation And time to work on it You know to get it under The quality level of federal repositories So we actually have a concept That hasn't really gone into effect yet But the Empton stacks group was developing Something called the playground Which was basically going to be a curated set Of coppers that were You know internally consistent That were specifically for Useful applications and services That weren't quite ready to be Made for a repository and may never Because of bundling reasons or what not But we're still sufficiently useful To the users and the workstation group Has been doing some stuff with You know software that allows them to Read but not but prompt Before installing software from These playground repositories So I think that's where we want to go For some of the stuff is that The playground is kind of meant to be Hopefully someday we want these To my gradient or at least that's the Stated goal but for now These are sufficiently useful That we still want our users to have Access to them and not have the Barrier of the package policy Preventing our users from getting them Yeah I mean I think But I think kind of the underlying We have packaging guidelines that are Awesome that nobody understands or Wants to follow because they just don't care Right if it builds and it works That's good enough for them just like you said And I think copers is a great mechanism To get people into the Fedora project Even if their package doesn't get into What can we do to make this more official Or what can we do to lower that barrier And use copers as a good way to Promote that hey Fedora isn't all about Perfect engineering sometimes And actually there's some level of Where people are not using Fedora But they're just using their software They're unpacking novels and setting things up And then they just Well there's a whole project started Called Dober Dober Yeah That was essentially that It was copers and then Build a dober image out of it And I think we're looking at that at the council Or is it the council looking? Richard Fontana is looking to legal issues Around right now Oh actually I haven't had a chance To update the council on this But I did hear back from the doctor Upstream on some of the questions we had Also Steve Gallagher is looking into some stuff So I proposed to them that A better solution than having us Ship things as Fedora Would be tying doffering to Individuals for punctures on the registry That's allowing us to help them Take those steps without all the manual effort To do that we would need Token based authentication Upstream is very open to this In fact they have other constituencies That are asking for something similar So they're actually likely to put this on Their high priority list pretty soon That was one of the things Richard sounded skeptical Well I think even doctor 1.8 They just came out with You can sign your own registry Something that was lacking that we wanted In terms of distribution Not authentication It's part of the image format There's two different sets of work going on One is for the image format One is for the registry for the calls And they're related But the signing doesn't solve the authentication The rest of the thing this should Solve the copper versus coper thing Because copper versus coper Is kind of either way Dopper versus dopper Dopper sounds really bad So you can say dopper and coper Also in copper there are a lot of projects That are doing badly builds It's part of their CI There are some repos that are maybe not used But just used for the CI purposes So they see that it builds in Fedora Works So there may be also one of the reasons Why there are so many Another thing I'd like to pitch That may not be a popular statement Is I actually think that There's a whole lot of software in Fedora That really should start being Shunted out into copers I think in the repositories There's a certain amount of stuff That makes sense to be in the distro space And there's a whole bunch of stuff That really is It's there because years and years ago This is where you had to be In order to have your software installed So let's keep up with Demi Sure Is there any value in having World of Tanks In the official Fedora repository That's supposed to be a coper There's an awful lot of Peripheral software That I think would probably see Faster updates And more usage Probably if someone Maintain it without all of the The extra package cruft So I think there's perhaps some value In moving some stuff That wants to move fast out And also Strangely to say A major advantage of copers Is that it doesn't have to go through body There's an awful lot of software in Fedora That just because Unless it's Firefox or the kernel Hardly ever sees karma until it has So it just sits there for a week Or two weeks, even if it's a security update Kevin Kowler should put all of his stuff And it sits there for a couple of weeks Until finally you can hit the Go to stable button even though no one has bothered To test it We just had the whole Two month long Nagio's plugins issue Where there's a security bug That's open for two months and nobody fixed it And it got escalated where they were going to Orphan it and everything and so I went in And I built the updates and I filed them And they have not gotten a single Comment about karma at all I did actually get one comment On the Apple builds And it was somebody saying I have this Other plugin That breaks because it's Expecting that full version And I said well where did you Get this can you tell them to rebuild it And I haven't heard anything back in So I mean If I ever file a Fedora bug And I get it fixed in I'll give karma when I've tested that one fix And it remains the only karma On that update Until it's like I just set it to one The first one Well in the kernel space we just Disabled because it was to the point where Where we had like people that Fundamentally disagreed with how we Rolled the kernel in Fedora so they As soon as they would file they would Just go in minus one and it would never Get out even though we got a Festival exception to do what we do And so we just disable auto karma Kernel's pretty good, it usually gets karma But no votes doesn't necessarily mean No one has used it, it just means No one's noticed anything about it Well no one's bothered to go Oh you know what this update says Right, it's fine In voting 2 there's a whole bunch more Flexibility of what you can report there And say I tested the bug It didn't break my system booting Yeah Which I don't know if that makes it worse Or better But There's no benefit to a being in Fedora It should be, I'd say Cooper Oh right I think the one benefit of being In Fedora is the visibility When I'm looking for your application I see this in Fedora first before I just go searching random websites Right but again with the GNOME Software type thing where we can actually You can't search the coppers For what is available to install The difference between what's in Fedora And how can people allow it Well no We've said if it's in a copper Specific coppers We allowed any copper As long as somebody Is curing it So a working group Or a SIG has to say okay this won't be approved But as long as somebody Will stand up and say yeah I'm Saying this was acceptable It can go into GNOME software Or KGE app I'm sure people know that Do they know that? I know they're GNOME software people All the app data stuff Is still quite a lot of issues And I've talked with Richard For a dozen times over the last Three years to explain to him The things that need to happen To enable Generating So the problem The problem arrived Arrows last release But we got to a milestone I think it may have been beta But I can't remember Anyway one of the things We did to build one of the images That we make Didn't build, didn't compose We didn't compose We didn't even attempt to compose it Because we didn't know what it was supposed to Right and so the question then is Is that a failure of this Do we need this thing We don't know because we don't know what We don't have a list of exactly What we're supposed to be shipping And if that thing Is blocking or not If it isn't there Does that block the release until we get it fixed And there wasn't Any list like this It was basically just Well we have these things so we'll ship them Yeah So how Will be submitting the notes To sheet That's why we wanted the list And we were shipping what we had Was this enough We're not sure The thing that triggered it was The vagrant boxes stuff We Like as a release engineer I worked with the change owner And I implemented You know in talking with them Like oh we've got this kickstart And we implemented making the image I was under the impression that it was complete They never once actually Steped up and said oh hey We also need to add this extra thing We're doing not just the atomic Version of the vagrant image But we're also doing the cloud based Version of the vagrant image That was never Never got mentioned as it's deliverable Yeah it was in like In one little part of the change But it was kind of skipped over by everyone I mean I'd miss that it was Supposed to be there The change owners didn't They're either missing that it was Supposed to be there or they Didn't Assumably it just magically happened There's a bunch of miscommunications And in the end We We got to the point like Do we add this thing or not They're like oh hey We have it signed off on The cloud based image is not there I'm like what As far as we know We've delivered everything that is in this change And they're like oh see It's one little piece of text here that says Cloud based image as well There was a comedy of theirs And basically what we need now Is all the workgroups Should really be coming to Fesco and Rella Engine saying Whatever this is It could be multiple things It could even be a list of Everything we wanted last time But no 32 bit Just some kind of Idea of what the workgroups want to see For their additions And in such a form that QA can look at this list And when there's a release candidate Can go oh this one's missing And if it's a release And it's missing then We've got a problem here It's missing but it's not considered Release blocking you know like say The XFCE like CD fails to compose Because it's got a 4 gig root file system And some package changes Of course to just take up 4.1 gig And it didn't have the room And it's like oh well let's miss the vote And you know we just carry on Okay so The expectation is That Everybody knows That The idea was that we would have this list Before alpha Which of course is not the case now I don't think it has to be with every change Because there's a lot of changes that are like Update curl to the latest version Yeah it doesn't That's not going to be it Whether it's complex or not Just maybe like a field in the change request That says is this a new deliverable type Or something like that Will this modify the set Of things that is output by relinch And if so How Yep The question related to changes Are you And I sold it People start stuff and don't finish it That's why we have the Ready for field Categories just to make sure What is ready So maybe another category Releases Archived changes I also don't understand You mean like already approved changes Or changes that were not Approved to anyone There are some changes Which are already approved But never been implemented So they went back To the initial state And they are there like Yeah you should feel empowered To just delete them It might be nice to record The ones that were approved That didn't get completed somewhere That's why I was thinking archive might be nice Kind of useful to separate the ones That were approved and not completed From the ones that were just ideas That never got flushed out They all have email addresses attached to them too For those that were approved And then got strunted Maybe we could just ping them And just say hey is this Something you're still going to be working on Yeah that's fine I just wanted to know Whether the fiscal one to be involved It is somehow or not Unless you get a reply from somebody That's one of those that says Oh this should really Be ready or whatever it is If there is an obvious decision That needs to be made If they turn around and say I've already done all of that There's no need for changes Then maybe we want to Add to the talking points See if it's something That we should advertise The very secret of changes Is that a lot of stuff gets in and done With nobody ever filing a change Right? Sometimes that's fine Sometimes it's really bad For the most part people do Actually one question also Related to changes And approvals Now we have working groups Right? So what do you think Is there still need For FESCO to Approve changes Because then change owners Need to go to all If the system might change That's affecting also Workstation, cloud, server After FESCO approval They need to go to those working groups And discuss again And basically get their approval Before the very FESCO Ideally But it seems like An overhead to ask FESCO For approval when Working group Indians can say We really don't want this In our product or it doesn't make sense And some changes are When they're talking about the password problems I'm talking actually about the NSS But yeah I see where you're coming from I think it's a hard question to answer Because like Steven said If you want Ideally it should be It's going to impact everybody And then I come to FESCO and I can say What those other people said Sometimes change owners Want to skip that part Because they know that some work group Is not going to like it So that's kind of hard But then you also have the other case Where yes Some particular work group might not use that feature In the communication aspect of the change That everybody that's working on the district Should be aware that this feature is there So yeah it's kind of extra overhead If you already have all the answers to your questions But by taking it to FESCO And going through the change process Theoretically people still see that And they're aware of it And they can plan for it If it impacts their package And part of the change process Of course is that mandatory period Between the announcement of the intent But hopefully those other work groups Will see that and say We have these problems And discuss that publicly Actually I think the process Is pretty good like Discussing the change publicly Before any approval But my point is If it really makes sense for FESCO To make the decision or should it be like Base working group and then for Specific products should like Their working groups decide That the work group is not functional And I'm saying that as a member I would totally agree with you on the base working group It would be functional I mean it should So I think In terms of working groups The base working group Never really got stuck And I think it might be time to look At just pulling them back into FESCO And saying FESCO is going to basically Be irresponsible for whatever we thought Was going to be in the base working group For some minimal Installation environment And anacondas Needs So the thing that The base working group curates Is what goes into the minimal install And what goes into like The dog base image Making sure that the things The anaconda users continue To function or work Really it was supposed to be Base is almost like green zero The work groups for the additions Plus environment stacks And it never really Formed that way because I mean when I was on the base work group After it was really formed We came up with like a kind of tiny Well it wasn't tiny it was like 2000 packages but we came up with a ring zero But it was whatever anaconda needed Basically so it had a whole bunch of stuff It wasn't really Like runtime ring zero if you will And then I was like Okay we're not doing anything It sounds like I will say The one really good output From the base working group has been Significant production and competencies Yeah absolutely So I brought up with Harold a month ago The base working group Should be looking at What's in the minimal install And is there anything that needs adjusting And Back level Harold have kind of gone up into the weeds And Gaga land trying to Work out whether they you know What Coming up with these definitions like What's minimal I like what they're doing at the moment But I don't think it needs a whole working group to do You know what I mean And it's not function The base working group is very dysfunctional I guess I think it's useful to have somebody Working on defining what our ring one Thing is going to be once that definition Is done then the group is done Or if they're not working on that at all Then somebody else should Think For Red hat stuff They're talking about making a rail Raw hide that would correspond Roughly to this ring one thing In fedora And it would be nice For a person For that effort to be something that fedora could Possibly benefit from as well Because we're looking at some of the same problems If people at red hat are going to spend their time Looking at that let's have them Let's make a place to do that in fedora If that's the base working group or something else I don't care I think the base working group Need to step back and Do something and maybe it is That we just say hey you know what You're kind of up in the weeds Or maybe it's You know with fesco Just needs to take ownership of this Yeah At the very least maybe in the next time A specific set of outputs we want From the base design group The next company Well what we absolutely need is You know we want a complete list Of what is the base image Or you know just Some reasonable deliverable That we can give them as a milestone Maybe get them working together Quite so much So time is actually up to the session I'm happy to say here myself And answer more questions if we have any Otherwise One more like I would like to get back to the Merging process like If you think we should like revisit The changes approval process A little bit because it seems like It wasn't really updated To the working group things Like and the Fedora next So maybe stating that for System wide change like explicit Formal approval from working groups Before fesco discussion Is needed like make it public Do some public discussion then go to Each working group get approvals Then go to fesco so about The appropriate work If the new feature is A really cool scientific And probably will apply to the work Session Yeah But yeah, I think that's A really good suggestion Change process is just Yeah That's what I said for system wide Because these changes Are mostly in multiple Groups I think that's a good suggestion So like the DNS like change Should be An example of something More of a sign up And maybe like if it doesn't affect That specific product The working group can say like Yeah we don't think it's affecting us So go on and make your chance The change owner may think that I'm not affecting Workstation but they may So it's not like One person can be sent I think we should revisit that In the next meeting We have a quorum here today Any other questions Any comments? Questions About If There's any plans to speed up The updates a bit Because we have a program with that For example in Firefox team When there is some Zero day Thing that it takes Easily like couple of days Until the update So the last Firefox security update Came out Friday morning And A updates push finished About an hour before that Before the build was done And a new updates push was started Then at about 8 30 a.m. My time And it finished late that afternoon So it was the same day But you can Actually do a push with just one packet You can but that's the problem It does Yeah I guess that's true It helps very little So vody2 again Will help us with this This time there was no With karma But for example when there was Another exploit like two months ago When federal 20 was still supported And we didn't really get Enough karmas there And still Quite a lot of users using Federal 20 at that time And well we don't have Enough testers because it's For enthusiasts it's already To hold but I wonder If for you know really Critical fixes If we shouldn't have a defense Yeah we do actually there We were working on a proposal For a Fast track For critical security updates And I think the last thing In that is that I proposed some process And got no comments on It seemed fine So I should probably just say Alright nobody complain Let's just do it That's one There's some extra bits that we need Yeah there was some things we needed There's some funky things with implementation We pretty much need to make You're talking about the separate repulse We're pretty much about I need to make a whole separate distribution To enable it to happen But it only has The security update And it only has it for A week or two Until it goes into the main thing And also you can probably turn off The depth RPMs And the dummy for The security update So I mean the answer to your question Is yes it's just not there Yeah We know it's talking about it I'm just pretty much I understand that Yeah Any other questions Before I turn off the recording That doesn't need to be on the recording Stephen