 Hello everyone, it is very early on a Saturday, I agree So we're here to talk about development tools We did a bunch of this work actually inside red hat as well as upstream and fedora and You know as you might expect a lot of them are similar, but not always the same So but by way of introduction, I am Langdon White and this is Tomasz Tomacek This is reasonably close So I always like to do a little bit of intro slide So these are my three kids and as it says it's the only way you can get them in a picture together is you know manufacturing it So, you know That's them and then Tomasz you're gonna introduce yourself, but I'm an architect at Red Hat I work on platform and I've been the modularity objective lead for several years now and go ahead Yeah, so I'm Tomasz. I like containers. Sometimes they don't work. So it looks like this Yeah, so right now we are Trying to work on the development tools within red fedora and redhead. Yeah, sorry. I'm really sleepy and This is why we are doing this presentation today that we'd like to talk about it with you and maybe get some of the feedback and discussion going and And I also like to point out I have a cat and this is what she normally does It's just you know launches herself wherever I have never had a cat that was so much like a dog as this one It's quite amazing. So I thought you needed a cat picture just on principle So You know kind of what what are we doing here? Like what you know, this is not you know We could just come out here and talk to you right and make some stuff up But so we want to tell you a little bit about why we have You know we think some credibility on the subject and basically we did a bunch of work on this so we had a Developer survey and we've been doing one-on-one interviews and anecdotal experience of course and so we're going to kind of present you the results of the survey that Dimash run and ran and the interviews that I've done like one of but we did a lot of work with Mo Duffy who's not here but She did a lot of the interview work and then we've done some results from that too So if you want to talk about our survey Yeah, so So if you are already part of federal community or working for ad hat internally on our products You know that like our developing tooling is like not very up-to-date with how it works usually on github or in upstream communities So that's why we realized that we probably need to do some survey like ask the people who do the job every day and Talk about it them and get their data and then finally when we have all the list What like what really sucks then we can prioritize it and start working on it and this is what we'll be talking about today so Yeah, the only I wanted to point out was that my picture here for anecdotal evidence, you know Is the guy sticking his hand into a cactus? That's the picture is there so I thought that was appropriate And so yeah, so tomorrow you're gonna talk to tell us about some basically some of the highlights of the survey results Yeah, so we're running the survey internally within Red Hat in autumn And we got some pretty nice results, and I feel like that most of them are applicable in Fedora as well because Actually Fedora and Red Hat like they share bring like very similar tooling So if you are familiar with Fedora, there is Fed package for like interacting with the build system and with packaging can we have very similar thing in internally and Like all this shared code is open source, and there's just some slightly friends internally Okay, so let's get through like what's not so good about our tooling Okay, so first thing fix in upstream and backport to this gate Yeah, this is very common use case. So there's a new bug open for your component There's already fixed upstream or maybe there is not so that's even better for the maintainer because can do some coding so the first thing to do is to fix the bug send the patch upstream then Get a review and answer the patch is merged upstream. We can finally take it and backport it to our like either internally within Red Hat Enterprise Linux or in Fedora But the backporting like if the code changed so much like the backporting is like ton of work like you are pretty familiar with that and Like can we do something about it? Can we make it easier? So I think there was already similar attempts So there's one project called the rebase helper which like tries to help maintainers with that And I think it does like pretty good job, but a lot of people are not familiar with it so definitely if you are a Fedora package or check out the rebase helper and Next thing is rebasing patches. That's pretty much the like very similar thing when you are When you have several that downstream patches in Fedora or well, and there's a new upstream Release and you want to like take that new release you need to rebase your patches Like ideally drop them all when there were already merged upstream and they are part of the new release But if there are not you need to rebase them and that can be really painful and really prone to errors because you see all the This and like okay, so maybe this line is not so good At least that's actually my experience that when I'm rebasing my patches like sometimes I Forgot like just one line and then it makes it like completely Useless, but that's when testing comes right like if you have really good tests you can run your tests and see that Your rebase was bad and software doesn't work at all and and really good tests are very common in so yeah Well actually upstream projects have really good tests, but we just need to like run them more often right all right So Next slide yeah So does any like if you have any questions or comments like hey You can use this tool to rebase your patches or to ease your git workflow like we definitely want to hear them I actually brought a notebook to make all the notes. So please help us Okay, so what's next so Next on the item from the results from the survey was kernel build time I was actually checking it out yesterday. How long it takes to build kernel in Fedora and it takes six hours So imagine that you are like fixing something you have a patch you include it You build it and then in six hours you might get the results. So For me, that's like really like I wouldn't be able to work in in such environment So can we actually like improve it? Maybe we can get some more beefy Computers and run the bills there, but that's also very expensive So with like kernel build time There's also like slow build time from other projects So it's like that we have also other components software which takes very long to build or even test So for example, I know that system D maintainers have tests for system D which run for days Which is like wow, but again like it's like it's very it's really needed to have those But when they run for days, we should probably figure out how to make them like more fast or maybe run them in parallel or I don't know Yeah, I mean one of the big things that you're seeing a lot of organizations do right is they actually do, you know different sets of the tests right when you're doing development and compilation or whatever So that you don't have to run the whole Battery of tests every time so that you can get your your work cycle down faster You know one of the long mantras in programming is you know how fast can you make that that write code? You know test debug cycle the shorter you can make that right the more effective you can be and I think when we're Well, we're introducing CI as the through the CI objective We really need to think about how how we want to do that with it's it's kind of strategic, right? We can't we can't just say oh, it'll be fine if we just run all the tests all the time We have to be a little bit more particular about which things will run when Otherwise, we'll just be waiting for builds all the time Anything else you want to say about this slide or yeah, actually so would I actually do Like on my own when there when there are slow builds or slow test runs I usually like start working on something else So then I worked on two things parallel which like I feel like that is very productive for me because I Don't have any downtime But at the same time if you know Fernando Collione who is a gel practitioner at Red Hat like he's very Like he would tell me that like that's not a good idea Like you should like there take only one task focus on it and do only one job So but if your job is to look at CI looks for 20 minutes, I'm not sure that's fun or useful So actually into asking about it like what does he think about such situation? Yeah, I mean, I don't I don't know a programmer I don't know about you guys right but you know I don't know a programmer who doesn't do that right who doesn't kind of multi-task in this way and it does lead to problems Right, you know because you forget where you were or you forget what's going on And then sometimes it ends up being three different tasks or five different tasks because each one's taking so long So we really need to figure out how we're gonna improve You know basically allow you as much as possible to be single focused or have mechanics around You know one of the big things that's bothered me for a long time, right? It's like how can we save our context, right? So, you know, is there a way that you can kind of keep the context from that particular issue while the kernels building for six hours? You know while you work on something else so that you can jump right back into it because that context switching is what is really hard for a lot of programmers For people in general, but I think it's particularly bad with programming because you know You'll have some random number or whatever that you have to keep in your head because it's the you know The input that you need to test, you know So you have this weird context that changes all the time That is very very difficult to save off to the side So, you know, I don't know about all of you but like, you know for me like I end up taking lots of notes You know next to whatever bug I'm doing, you know, and then try to switch context and then come back and it's not very efficient You know, so maybe some of the ways we can do it is actually help programmers save their context, you know Yeah, I agree that's what I also like learned to do was whenever I'm working on something and I'm running like very specific long commands I try to like include them in documentation in comments So the next time someone works on it doesn't have to like reinvent it once again, right? Invariably, I'll have typos in there and make it worse for the next person. You know, that's just me learning experience Alright next. Yeah. Yeah, let's go to the next one Okay, so what's next? Ignoring broken content in Fedora Yeah, so everyone who works on redhead enterprise Linux know this very well when and this is actually like big issue We are trying to solve right now is that we have federal height So that's a place where all the new upstream content lands by default, but it's not gated So for example, there's a new upstream release of curl and we just put it to federal height build it and it's there And maybe it would break rest of the operating system And we would not be even able to compose it or even use it install it update it because of this change But right now we have no no mechanic to do such thing And if you imagine that like this happens for like tons of packages And then we try to take them into redhead enterprise Linux and they are like Coming broken so that a lot of people need to spend a lot of time on fixing them and then getting the changes back to Fedora and So yeah, so what we are trying to do right now is to gate on raw height Like this will be our like objective like for next couple of months, but I guess we'll discuss it later, right? Yeah So yeah getting broken content for Fedora. So this is like a real problem and Finally to hopefully be solved. Okay, so inconsistent result So I'm pretty sure that you are familiar with such thing that you have pull request you are working on it Then you are done it works locally just fine You push the changes see I'm runs and you get like completely weird errors or maybe There was a time out on getting metadata for package manager or or something like that And you're like, okay, so that's absolutely not related to my changes. Okay, so we test and maybe there'll be a different error Okay, so let's try again and maybe third time is the charm and it suddenly works So yeah, we need definitely to be better about this So yeah, I just wanted to comment, you know the pictures here I'm sorry, they're both, you know covering your eyes, but I thought both of them were so cute that it was important to keep them So when I started actually working on the modularity project This and the build route management were both things that completely horrified me So, you know that it is so common for Packagers to just say no just build it again. It'll probably work and I was like what? That's no, that's not right So that's one thing and then the fact that build routes drift Really blew my mind and is a it's kind of one of the drivers behind modularity is the fact that you know A build route is changing all the time underneath you is another thing that I find horrifying You know and so, you know, there's been much discussion of things like reproducible builds and that kind of stuff I don't even care about that. I just want things to stay still until I tell them to change, right? And so I think that's something else. We need to really keep looking at We've we've started to fix some of that with modularity But in my mind, at least it's it's not good enough because you know, I want predictable results Even if they're broken, I want them to break in the same way every time, you know So I find this particularly scary and something that we really need to be more consistent on basically Yeah, and this is especially problematic when you are as you said like the builder will change so often like you are probably trying to build Once again, and then there are changes already which are like broken and you are trying to integrate to something Which is already broken. So that's like impossible almost Yeah, so I'm very impressed and equally horrified with the you know, how this works today So maybe to the third point because it's actually not clear I actually had to think about what I meant by this. So poor builder management. It's about Koji So if you're a familiar with Koji build system When you are submit the build, there's already a build route Which is a set of packages which are used for your build and you are able to actually change them like okay So I want to swap this one for a new version. I want to change this one but it's like very manual so you need to like a lot of typing and it could be actually more smart or more easy and This is also something which we are working Or we're trying to figure out but we'll talk about it later when we get to actually how do we solve all of these? Okay, so what's next on the menu chasing failures? Yeah, that this also actually pretty fun Like when you there's a new bug report and like you look at the reports like okay So I have no idea what's going on and you spend days like trying to figure it out in your code or maybe figure out It's a dependency or it's a build system issue or maybe there's a CI environment is completely different and unstable So chasing failures. Yeah, it can take I don't know days weeks I heard stories about people who are maintaining some c packages and there were some Memory corruption and they spent even weeks or months and in the end The fix was one liner or something like that. So I'm not sure how rewarding or actually fun. This is actually never maintained any c package. So I didn't have pleasure Okay, yeah, so this one I wanted to come in kind of on the pictures here as well because what I think Like what I think these all four of these kind of reflect is that there's not a very good pattern for You know how you kind of go through the process, right? Everybody has kind of their own model and it's slightly different and there's there's not a lot of consistency there either And so as a result, we don't have strong tooling to make the things that are automatable more automated You know and obviously there are some things that will continue to be things that you have to do yourself, right? Like chasing down a straight-up bug. It's just what you got to do And But when you when you have to think about how the branching works if you have to think about how Bugzilla acts work or how you have to do, you know, whatever submitting Koji builds and knowing what all the right flags are Like all these things are there are things that you have to keep in the forefront of your mind You know because you have to do them a lot, but they're not, you know They are documented, but they don't really need to be documented as much as they need to be Here's a tool that just steps you through the process, right and says this is the next thing you need to do And then you can actually, you know, kind of keep your mind spare right for for actually chasing down legitimate bugs And I think this goes back to kind of the earlier point a little bit of you know, the more Inconsistency you have in your results the more, you know What's more often introduced right is problems that you aren't aware of right or you can't you know You think there's a bug but there isn't necessarily because it's just an inconsistent result So those are those are very hard on Efficiency right is like, you know chasing down a bug is hard enough But introducing a bunch of complexity to make it you know Makes it much much harder and then if we go back to the even earlier point of kind of the context if you can't keep That context, you know, it's it just kind of keeps adding and adding and adding and making it more and more inefficient So so are you saying that we need some kind of developer dashboard like perhaps it's velcro dashboard might do the trick Except so, you know, there is the existing developer dashboard that the challenge I have with that is that In some ways, it's more like a portal, right? So it kind of gives you an insight into a bunch of different places In the developer kind of workflow or life or whatever, but it doesn't really there's no real like kind of guidance Around the actual workflow you have to pass through right there's no You know going like I keep going back to context because it's a pet peeve of mine, but You know, there's no way to kind of say, okay, where was I on this issue? I was working on there's only here's its current state, right? Which are not the same thing all the time Okay, so to spend some more time on the other points So branching so again in Fedora branching is actually fairly similar like in this get you have one branch pair Federal release so it's like right now There are three it's Rohit Fedora 29 and Fedora 28 and like soon is still be Fedora 30 as well when we start working on it But internally we have like so many products. We have so many different releases so many different supported releases so for example When there's a bug or CV and I don't like Maria DB You need to fix it on 20 branches or something like that And like you literally need to do like 20 commits in 20 different branches, which again like why is why there is just not one So this is a big thing that was something we've really have been working hard to try to fix with modularity Is like why is there a correlation between the database my sequel and the product it ships in right whether that product? And I'm not supposed to say product, but whether that project be you know Fedora edition of 29 right or rel right? It's like it's still my sequel whatever 10 right? You like why do we why is there any sort of connection between the version of the software and the thing? It's going to land in when it ships right there like at least at the git level, you know Yes, somewhere along the way there should be a config file that says you know yes this you know My sequel 10 is in you know Fedora 29 But why is that represented in git in at all? It just seems ludicrous to me, right? So I really hope that as part of the modularity work, you know, we've introduced And we don't really have good term for this, but we've been calling it arbitrary branching But really it just means branching that's related to the thing that is being branched rather than some random thing That's over here at the end of the pipeline, and I think that really reduces the complexity of Like having to keep track of where things go But then also more so internally where now maybe God forbid we could have say a branch for 10 You know or maybe a branch for 10 and an LTS kind of branch for 10, but that's two not 30, right? So again, we we cross into pet peeves of mine quickly and I get Okay, so and finally we have two more things which is bugs like Again like internally we use this system like back to a bugzilla X to actually like So when there's a new bug report You need to get like acknowledgements from from the management that yeah This is the bug we want to fix and release and you literally need to find the right people who are able to do it And again like it can take days because maybe the persons is sleeping right now And you need to wait six hours to wait for him to get up and asking like hey Can you please tell me whether I should work on this? So again, we probably need some automation here to like make it much easier and finally manual steps Like again if you're a federal package or you know that there's a ton of manual steps to when there's a new upstream version You need to like pull the tar ball then commit to this git then build it then wait for the build then produce an update And do the same thing maybe for federal 29 and 28 So again, it takes 30 minutes when like the manual step should be like yeah I approve this change and that's all I'm doing and everything else is automated So next time yes, please It's snowing Yeah, oh, yeah, I checked before I came and it assured me in the on the interwebs that there would be no snow this week So that's nice. I Apologize to the lack of pictures on this slide. I ran out of time with with funny pictures for you but So some of this work is still going on so we've done a fair number of interviews with you know kind of one-on-one with developers Trying to keep it open-ended kind of trying to keep You know using kind of formal interviewing processes so not to try not to you know lead the the person we're talking to And so some of these are some of the highlights of things that we discovered were kind of Well, let me let me back up for a second So one of the biggest things I think we discovered it wasn't really that much of a surprise But it was interesting at how pronounced it was is that all the different kind of you know subcategories of You know Fedora do work differently, right? So, you know the desktop, you know team does stuff one way the You know the the Python team does stuff a different way And in like every regard like the end on top of that they mostly have had the same process for like many years So they're very very used to it They are absolutely certain that their way is the best way compared to all other teams However, they also all recognize that there's inefficiencies or problems with their their way of doing things So I think that was kind of the most pronounced component And so introducing change into that process for any team is very very difficult, right? It's it's hard to change your muscle memory. So that's the first problem. I think we have to get over and then Excuse me The other thing we need to deal with is that Those teams have different processes often for good reason, right? And so we need to Somebody had a great phrase with this before but it's like, you know, but now I can't think of it But basically we want to have You know the same outcomes, but we don't necessarily have to have the same process to get there But we need to recognize that different teams will have different processes And maybe need to use some of the same tools Super, I'm not sure what's up with these water bottles So but a couple other highlights that we pulled out was That a lot of developers are you know as part of that, you know different processes a team may be sharing some Like hardware components and using them together, but more often every individual developer Oh my goodness is doing things their own way with their own technology And so that can be you know, it's expensive in the sense that They have to do a lot of work to maintain that hardware or that software or that build environment or whatever It is that they happen to have either even if they're doing it as a team when you know Most of the teams are not, you know a hundred people right their teams are five So that means that you you're putting material energy into maintaining that environment Even at the team level that you know could be much more of a shared kind of infrastructure component. So That's that was one of the big drawouts that we came from And then the next one is oh, I actually mentioned this already but lack of consistency between tools and teams And then the protocols and procedures So not only do they do kind of different things technically, but then they also do different things Kind of organizationally as well, right? So like how they do code reviews say or how they do that kind of work Where you know some of them do it more so or less so and they have different processes for these So we need to make sure we recognize that teams will be different But how do we get the same output? By but still allowing them to be different Another big thing is that we're starting to see you know people starting using containers And but that's a very limited amount and I think that was one of the takeaways was like if we could help build out kind of the build-and-test infrastructure using containers that the Benefits of using a container to do that stuff is so high that we might get large adoption From the different teams which might change their processes without you know us having to introduce the change Which everyone's going to be annoyed by and not do right? So that's kind of an opportunity for us to be able to introduce a change that makes more consistency You know and more shared resources without having to fight through you know change control right? So and the reason I said so we've we've done all the interviews We're not completely finished with what we think is the like the write-up or the output So that's why I'm kind of drawn out some highlights But we should be having some proposals. We're going to talk about this in a bit But some proposals going forward Like kind of specific projects that we'd like to take on to to make some of these problems go away. Hopefully So now the good news Let's see. Oh, I'm first. Sorry. Okay, so There's a number okay, so just a little bit of background Fedora objectives is the idea of in the Fedora Council We just decided this was a few years ago now, but that we weren't getting enough things where We wanted to move Fedora the project kind of as a whole towards a particular goal and Because Fedora right in a lot of ways doesn't produce anything right it it takes Upstream components and then makes them work together, which is huge right and a lot of work But you know actually having Fedora to bra, you know Actually drive project is one of the things we wanted to kind of change and so one of the first ones was the modularity objective Right is that there's no upstream for that. It doesn't make sense as an upstream, right? It's part of the distro about how things come together and how they work together And so what we wanted to have is kind of a vehicle for that kind of work inside Fedora Which we've never really had very well before it occurred But it wasn't really formalized or discussed in per se so We introduced this concept of the objective. They're supposed to be I want to say You know nine to twelve months And then if they if they need to continue they get renewed so you write a new objective and then it gets renewed So very recently like within the last I want to say six months There was a new objective introduced by Paul Fried for called a life cycle objective, which is like how can we? improve the kind of life cycle of the Fedora releases themselves because we weren't No one's really looked at, you know, how do we at least consider like something like an LTS release? Not necessarily that it's a good idea, but what is it about that that might be interesting? And what do we what would we need to change if we wanted to go to a you know? Two-year Fedora, or what would we need to change? We want to go to a two-week Fedora And so that objective is about improving our technology and prophecies and stuff like that So we could make either one of those true and we can make either one of those true Depending on addition right so the addition could actually decide that they want to do You know much faster releases like something like the Fedora core OS or much longer releases You know which is being considered by the Fedora workstation team So that's the life cycle objective modularity objective I obviously brought up a bet a bunch which is the idea of how can we have? Applications have a separate life cycle from the operating system itself, and that's a big part of what we're trying to do there So that's the modularity objective And hopefully you guys know what that is mostly already The decade project is the one that I'm kind of leading internally that was doing the interviewing As well as trying to make some spot fixes here and there for Kind of the developer experience when building rel primarily We're looking to try to figure out how we make sure all that stuff goes upstream as well But that's where some of the things that we've been looking at so far are mostly around where the differences are in the process between Fedora and Red Hat But we're hopefully you will be doing more of that soon, and then you want to talk with the other ones Yeah, sure So maybe just for you to understand the little thing just happened. So there's no clock So I can't see what the time it is and it's really frustrating And the only thing is that that thing showed me but it blanks after 10 minutes. So it's like what time is this? Okay, so Okay, so we have more objectives So we already spoke about like packaging and how actually updating is really frustrating because it's so manual and CI and those stuff. So right now, there is new objective. So actually the name is Packager experience Objective proposal. It's not like packaging objective and it's submitted by a community member Ben Rouser and This is a very nice write-up of things we should work on in Fedora to make like packaging much easier So it's really about like improving the tools Improving the processes and like this this really could be the thing which makes it much easier to like live as a package Or in Fedora Then we have CI objective. This is led by Dominic Parpeet. So there's like so this was already done And it's like finished but right now we are working on like a next version of CI objective V2 and it will be posted I guess within a couple of weeks and Within the CI objective one of the goals is to gate Rohit. So seriously build the gate and Like put police officer in there and if there is an update coming which breaks breaks the distribution We just don't allow it and we wait for the package or to fix it or maybe or actually fix Wait for the upstream project to fix it Before it lands in Fedora and this will like greatly improve Fedora height and make it stable finally and The last point is so if you were actually yesterday our presentation with Steph We're talking about very similar topics as we are discussing today And we'll be working on an automation service for packaging. We call it packet right now So please watch the recording from yesterday The thing is that yeah We just really need to get rid of all the manual steps and bring upstream and downstream communities together Because right now we as a Fedora distribution. We don't contribute like we don't provide feedback back to upstream So when there's a new version of I don't know some Python library We don't tell the upstream developers. Hey, your newest release doesn't work with our distribution right now Or it breaks these components or here's the test failures. Please fix it or tell us how we should fix it Right now. We are doing very poor job and within the packet service. We are trying to address it So maybe there when there's a failure we send an email to upstream mailing list and tell them Hey, your component fail testing in Fedora. Maybe this is interest you or maybe not and we won't do it ever again So yes, I just wanted to add a little bit more which is just that even though I haven't written as packaging objective here It's actually it's still a proposal. It's not actually officially an objective yet And then I would just also want to mention that packet and source get are the the projects you're working on Related to this so the name of the projects they're You know, it's a mostly tied under the CI objective, but they're they're more cross-cutting than that So if you haven't seen it They've been doing some great work about kind of automatically pulling upstream code and making it land in Fedora Without as much human intervention, which is really nice. Yeah, there's the plan right All right, so next slide And this is where we we start talking to you guys You know is You know, what do you guys have for ideas? We really like feedback, you know comments what what things bother you? What things, you know, are you trying to fix what things do you have that you use that? You know, everybody else should be using to and I know it's very early on a Saturday morning So but nonetheless, I'm hoping you guys will have some ideas Anyone you were stuff I I Know some people All right, well this might get very short very fast Right, so I'm gonna repeat the comment I'm gonna repeat the comment for the for the mic right so basically the comment is You know if we want to try to get our branching down to you know buy Kind of application level, right? We might end up with challenges because there's different spec files depending on the release that that package is going out in right and If or we can you know kind of merge all that into one spec file and we have lots of conditionals in there which can be really messy and I Would come and back that you know a Okay, yes, you are correct right a hundred percent However, is there not ways that we could automate the changes between the spec files Is there not ways that the only thing different between those is literally the spec file So part of the challenge at least for me I think is I think that is part of it is that we use tar balls right rather than actually using the source trees So I think that that Introduces more complexity to how you do the branching. So yes, I agree. It's it's not an open and shut case But I think there's ways we could do it better And still reduce the number of branches and also not clutter up spec files, right? I don't do want to add to be honest. I actually disagree. So okay I understand your comment that when you have for example to okay We have two branches like Fedora and sent OS branch and we would have like two spec files And if like each of the spec files would be like very tights to the specific release You would need to like right now you are maintaining two spec files Not just one so you can't put it in upstream or anywhere else. They are just two So whenever you there's a new update you specifically need to like make the changes to one and to the other one But if you have only one spec file with all the ifs and elses and like that You are maintaining only just one so you can easily tinker it and it's just one and Yeah, so this is like really problematic So maybe we can change spec files on improve it or maybe we can template it and have like a template somewhere And you just run the templating engine and it would spit out the center spec file and just speed speed out the federal spec file Maybe that could be a solution. Yeah, so yeah, maybe I didn't explain it very well But so I guess the the the kind of spec file improvement side of it, right? Are there are there ways that we can you know either automate the generation of them to begin with? You know or whatever like this this seems like a really Like a simple problem that causes a lot of work, right is I guess my general I think Steph may have a comment So the comment is basically You know one of the one of the things that's Part of why the spec files need to change so much is the changelog Steph and others find that not particularly valuable anyway, so why don't we just dump it? Which I think is a kind of a funny recurring theme So Yes, but I don't think it really goes to the real problem, you know like as in We I think we kind of need more like a templating engine kind of model of some kind You know one of which might be simpler because you got rid of the changelog Comments so we actually already have there we already have a bunch of tools We generate spec files namely for each of the like language ecosystems and you just okay Here's the upstream project and it generates the spec file, which is like very good precise And like we probably don't even need spec files. We can generate them like up front But uh, yeah, so there and there yes, basically I think the point being is that we actually do have a lot of this automation We just don't use consistently or it's overly specific to you know a particular ecosystem And so that's something we should be really evaluating and you know encouraging into our our tool chain as a way to do things better Right so the comment is that you know There's other RPM distros like to say who have made improvements along these lines that we could probably adopt and you know, maybe get away from our fascination with Spec files being such a hand-built A piece of perfection that they really don't need to be More comments Slavic Then Like for every other Because you know Like Thank you Yep, okay, so to repeat it. Yeah, absolutely So Slavic comment is about Build routes like the place where you are doing built-ins inside the build system and that it takes a lot of time to set it up like because Cogen needs to prepare all the repositories and yeah Just takes time especially for all the architectures and Slavic comment is about That we should catch them and that should speed up the build process a lot So yeah, I agree. We actually discussed this on Monday with federal engineering when we were talking about like gating Rohit So There are actually a couple of things so I'm not really same engineer I have no idea how Koji works. I only know like only some of the things But the comment from Mikolaj is that ski was that you can actually like there's all there's already a solution in place right now That you don't need to regenerate build routes that you can actually like reference different build route And it won't be regenerated. So it would be actually loaded from cash So that would actually literally work Slavic suggesting as far as I know this is what modularity is doing like there You are not regenerating builders who are using those from cash, right? So we already have social in place. We just don't use it which is like mind-blowing Yeah, and also personally for me is like if it was more of a policy that we used You know cash build routes or whatever. I think we also get back to the my you know The other thing that I bothers me is the consistency of the build route, right? Is that you know if you can kind of control the drift of the build routes? I think that's also very useful But you know, there's a side benefit of it also being much faster, but then you also get this The fact that now you actually know what's changing which that just like I said scares me So actually there may be another comment Because we actually didn't discuss this so okay, so imagine scenario that you are updating Like N carsies in the row height, which means that you you like bring the new update And then you need to rebuild all of the packages which use the library because they would be broken otherwise because like also as presented, it's also name bump and You like you can't do it in row height because like when you land the change you break everything and everyone is really angry So you need to do it like like on the side So you create a new cog attack and this is your place of work So first you update and carsies and then you start rebuilding all the big kids until Like everything works and then you take take your work and merge it back into the like into the main line and Again, like this was one of the comments from from the survey that this is very manual like lots of typing prone to error and Like takes a long time and this is again another thing you'd like to solve By like introducing a new chain like like a new way to do it very easily so that you very easily Create new tag Put your builds you want in there? You are doing all the builds against it and then there would be a step where say okay right now We are going to like run the CI and see if all the new builds work and if everything's working we can merge it back And everything is nice and I remember this was the comment about like when we when we would actually implement this Like we would be using those cash build routes. So yeah, so this was the comment So I want to move on just a little bit, which is here are some of the questions that we had that We would like feedback on and you know and anything else obviously that we've been talking about But we're running out of time So we wanted to kind of say here's some other ideas some things that we've thought about We'd really like feedback from all of you about things that you've been thinking about or things that annoy you So that we can try to help address them going forward You know and actually add them into you know, whatever the objective work and stuff So the last thing we wanted to cover is basically like how do we how do we make this? You know, how do we get this out to people right? So how do we tell people what we've done what we're changing what we'd like to change, you know What what would work, you know, does anybody have any ideas about how you know We could kind of do this as a community more, you know one of the ideas I've had for a long time is if if Something like the FPC was less of a you know, kind of a board and more like a working group So like a place where you kind of go and talk about, you know packaging and like what problems you're having You know, maybe it's not literally the same group But you know kind of like would it help be helpful if there was a place where people would go and Talk about improvements to things like packaging You know or where where would you discover ideas about like how you could improve your own processes? And that's kind of what I was hoping to ask you and maybe do you guys have any ideas around that? Exploit Yes, absolutely. So that's why how all the generator spec files tool work So for example peep to rpm take setup dot pi finds all the dependencies and generates the spec file based on it So that all the dependencies are right. So this is how the tools work So the problem is that you have like special snowflakes like very nicely crafted, which Yeah, so he's essentially I think you're saying he's like as part almost like part of our CI infrastructure For lack of better term You know, can we actually run the general like actually walk the projects looking for things that look like the things that we have generators for Generate them build them and then propose them as changes back to the developer You know based on however, they were generated. I think it sounds like a good idea Yeah, absolutely. I agree like we can and we could even cover like I don't like 80 percent Like there still be like a lot of special packages which need like very very nice care But like for basic Python or I don't know no JS libraries. We can do it very easily This is also one of those places where you know, you've heard the you know, perfect is the enemy of the good, right? Like if we knocked out 40 percent of the packaging work that would be Ridiculously a lot of time right like we don't have to get to 95 percent for it to be useful, you know Yeah All right, we are out of time for the red sign. I just got thank you for coming and hopefully it was useful