 I want to introduce myself because you all know me and you all now have seen that the talk is changing the changes process So I have some boilerplate I start out with so let's talk about it If you want to say nice things you can there's my Twitter handle if you want to say not nice things You can send them to dev know if you have like actual constructive things to talk about Nobody ever shows up to this, but I do hold weekly office hours in IRC You do sometimes It's because you don't in my community blog post apparently no, okay, well now, you know Every word I write is golden you should read like So what we're gonna talk about in the next 24 minutes is A little bit of background about why we have a changes process What the process is just kind of as a reminder, you know Most of you are fairly familiar with it But I feel like it's always good to kind of nudge people in this way. I'm not calling out anyone specifically Now I'm gonna talk about why we're changing what we're changing and then oh, no What if it's terrible and what we do beyond that? So why have it changes process if you are at my Change it change management in open source projects talk at dev conf the next few minutes will seem very familiar But I condensed the 50 minutes down to much smaller than that So I'm not gonna get into like the more broader theory But basically start out with what a change is we're talking here about technical changes to features or processes Things that we ship to our users or ways and that we build the things that we ship So why do we have a process for this it's not just to keep me employed although I do really appreciate getting paid It's communication and these are all different variations of me saying the word communication But it looks better to have multiple bullets on the on the slide So what does it mean by communication? Well, we want to communicate with each other There's a large group of people who contribute to Fedora in some way And it's really hard to keep up with everything that goes on If you subscribe to a lot of the mailing list you mostly are really good at just marking emails as read or just dealing with The fact that you have a thousand unread messages It's good to get feedback sometimes people propose changes that are really bad I'd like to point out that Matthew submitted a change to Basically get some very raw and anonymous numbers from DNF and everyone's like actually that's kind of shady and what it happened They came up with a better idea We had recently that I I'm not here to pass judgment. I'm just reflecting the values of the community back to you So, you know and sometimes it's yeah, this is a great idea But this is gonna affect me in this way and I need time to do this so quick put it off or can we do this thing? You know, there's a lot of coordination that happens And then more communication right so like not just as as we are implementing the change and shipping it But it'd be good to tell people about what we've done one because it's nice to get you know coverage of Our work and so people recognize it and they say, oh, yeah, Fedora I should you know go install the new release because it's got all this cool stuff and Getting that into the release notes is part of the changes process Everything we ever say is immediately turned into a pharaonics article So, you know never come up with any bad ideas on a mailing list because everyone will assume that's what we're doing But process a very wise person said developers don't actually hate process. That's when they when it works they call it culture and You know part of the reason people think they hate process is sometimes the process isn't scaled to the thing So I use this you know as an example So like in for door we have a relatively heavy process because it's a really big Dynamic community with lots of moving parts people who are doing stuff because red hat pays them to people who Work at red hat and are doing stuff because they want to people who are just doing stuff because they like being in the Fedora community and they don't have any real association with red hat on the other hand there's a pearl Twitter client that I maintain or I'm the only maintainer and There's still some users, but people have mostly moved on to other tools at this point So really the change process is Ben sits down decides he wants to code something he pushes it And maybe he calls it a release at some point Not a lot of communication goes on I actually don't spend a lot of time talking to myself So what is Fedora's process well back in the old days Prior to Fedora 20 we had the features process and the main question of the features process is is this a feature or an enhancement people had to make that judgment call and It turns out it wasn't a great process when I was doing some research on this to you know sort of see what was going on I found this comment on a blog post from somebody in the community replying to somebody else in the community and Basically the gist of it. I'm not to sit here and make you try and squint at it. The gist is Why should we do this because all it does is make work? And that was really true because how we tracked the progress of changes was you went in if you were the change owner And you edited the wiki page and said I am X percent done now and you were supposed to do that You're fairly often and you know people kind of just went with 20 40 60 80 100 when they remembered to do it which was almost never and It wasn't really meaningful because 80 being 80 percent done means you've barely started, right? So it wasn't really that helpful So people started complaining people started being unhappy So the fedora engineering steering committee they identified a few issues that didn't work about the features process Feature is very ambiguous people thought it should be a feature but it's actually an enhancement some people thought it was an enhancement really is a feature and Some people didn't think it was either, but it really should have been one of the other and like nobody really knew what it meant There were a lot of different wiki pages that all asked for the same information which was a real pain It didn't account for different types or scopes of features So if you wanted to add a new desktop environment, that was the same as you know, if you said let's replace the Linux kernel with BSD They use the same process even though they are very much not the same scope and The process really didn't make the the feature proposals visible to that visible to the community Until after they were approved like after Fesco said yep, this is good then it's like oh, hey, we're going to do this thing now and Fesco is pretty great shout out to all the Fesco members in the room But there's only nine of them and they can only identify so many problems And having the community interaction early in the process is really good for identifying some problems before they get implemented So my predecessors predecessor helped design a new progress Process that we started using in Fedora 20 and it's really similar to the features process, but it's better It's probably 80 70 80 percent the same But now we don't think about feature or enhancement because those are not really helpful so we distinguish between system-wide and self-contained changes and Let's be honest. We have trouble making those distinctions sometimes too, but it's a little more clear Does somebody else have to do work because of you all right? It's probably a system-wide change is the work contained to you or the group you're working with to propose the change All right, it's probably self-contained. It's there's we still have the the rough edges, but it makes a little more sense now And changes aren't necessarily shipped code most of the changes are Some things are like oh, we're gonna change. We're gonna change what's included in the default build route The end user is never gonna actually care about that because if the package builds the package builds But community contributors that's important because all of a sudden. Oh crap. I have to update my spec file So my package is still build so the The process is fairly similar for self-contained and system-wide changes, but system-wide changes because they're a little bigger We ask a little more of the the owners for that We have they have to have contingency plans Test plans they have to say what the impact on the other developers is what the impact on upgrades is So if you go from Fedora 30 to Fedora 31 and this changes and implemented will you accidentally wipe the user's hard drive? That's if that's listed in the change proposal I get I'm guessing it will probably be rejected But it's good to identify these things right and so the way the flow kind of works Is the change owner drafts a proposal in a wiki page? They check with release engineering if they have to we have some Made fewer people that have to do that because I was tired of having mohan have to go in and be like yep It's fine. Yep. It's fine. Yep. It's fine. Like it's a waste of his time If there's a trademark approval needed because they're making a new spin or doing something They have to go to the council and the council has the authority to be like yes This is a good use of our trademark and Then if there's some policy changes like around packaging the packaging committee is the gatekeeper for that And then they go and mark the page the wiki page ready by setting it to a category and hoping They don't mistype the category because then it gets lost And so then I go in every so often and you know I refresh of the pages list of pages in that category and I say oh good. There's some new ones so I Change Wrangler post the change proposal really simplifies that I have to go through like grab the source Copy in to an email delete all the markup in the comments and make sure the link is there and make sure I type the Version number right in the subject and I copy and pasted the the title right and if you've seen the change proposals I've sent out in the last year. There's like a 1 in 20 chance that something is wrong I've put the I've said I've said system-wide when I met You know self-contained or vice versa. I've typed the wrong version number as who knows what the hell version of fedora. We're releasing I I'm not very good at this like I cannot copy paste well So anyway, if that happens we give it a week So after a week the community, you know offers their feedback you always constructive they always have these great ideas that add On to it. It's never just empty complaining And then I copy and paste more stuff into a Fesco ticket and then Fesco generally within a week sometimes really faster than that But they go through and they you know approve it or reject it Sometimes they'll go back and say can you add more to this page? We're not really sure what we're voting on or like hey, this would be great But like there here's a concern address it or you're getting can So let's assume Fesco accepts the change. I hand out a shiny new badge The the change owner does the work to implement it and people never do the implementation and then go back like oh I should probably submit a change proposal for that. No, that never happens and Then we evaluate contingency plans where they're we're applicable at the beta freeze usually Sometimes changes are actually like the contingency plan doesn't only need to be invoked until final freeze because it's something kind of small Or maybe it's sooner because it involved the mass rebuild But like for most part it's at beta freeze or some random date that sometimes people just put in And so we have a timeline basically working back from the beta freeze You know code complete a hundred percent code complete testable We start a mass rebuild you have a deadline for submitting so it actually can go through the process So that's your reminder of how the process works So why are we changing anything? I'm selfish I'm lazy and I make a lot of mistakes Yeah, I mean the selfish parts of the self-deprecating but in all honesty a lot of it is like this is just a giant pain in the butt for me and Like all this copying and pasting is not a valuable use of my time that I could be doing other things for the community and Yeah, that's kind of ties into being lazy You know, I'm a former sys admin so I'm always trying to automate myself out of a job and Yeah, I mean I wanted the one and twenty chance that something is wrong when I sent the email or I put the wrong link to The develop list thread in the Fesco ticket or I did something like it's just annoying The other thing I didn't put on there is the the dark magic of how we actually generate the bugzilla Issues once the change is accepted There's a pearl script that parses the text of the wiki page And hopes it does the right thing of pulling out the summary and the title and the change owner and shoving that into Into the bugzilla and mostly it works Right mostly it works because pearl is really good at parsing text and stuff But sometimes people just do something just unexpected enough that I have to go in like manually edit the page Before I can run the script and then I find out the email address They had for themselves on the wiki page isn't their bugzilla email address and bugzilla is like I'm not gonna create this bug I don't know who this is It's fun So as I'm doing this, you know, my idea was like I want to make these changes to make my life better I don't want to make everyone else's life worse One because I have no ability to force anyone to do anything Like if we were all red hat employees working on a red hat product I could call people's managers and be like hey, they're not being a team player Can you do something they'd be like no, but at least I'd have that option In a you know in my view everyone working on fedora is doing it in a volunteer capacity That's the way I think of it because that's that's the same default and it's true to some degree for a lot of people So it's really, you know, I have to try and make it something that people can accept Somebody else said changing people's workflow is unpopular And so this is how I'm gonna burn all the political capital I've developed in the last year doing this job by changing things for you But yeah, so really it's I want it to be something where you don't really notice if you're a change owner that much difference And I really want to automate things because I'm gonna have fewer mistakes. I Can do things faster and spend more time doing other things in fedora and Part of that is by separating out and separating out metadata Like we don't we shouldn't be parsing the wiki text to get people's email addresses We can just say here's the field where you enter your email address Here's the field where you say this is a self-contained change or this is a system-wide change You can just do things and they're separate and you can treat them separately and operate on them separately So with all of that said, what are we changing? Nothing sort of like the actual from the like the business part of the process isn't changing the flow still happens Basically the same way. It's just sort of an implementation detail about how this is gonna work is changing some tooling I have I have nothing but nice things to say from here on out Um Yeah, so it's basically the same process, but just a new tool We're using the tiger instance at teams that fedora project or org if you were in Matthew's talk this morning. He mentioned that so so what does it look like well, you just create an issue and You have Fields that are just custom fields that kind of get some of the stuff we want But then like a lot of the body of like the actual content of what you're proposing to change It's still just there except now you can use markdown, which is a lot easier It's so much easier and if you keep going if you're switching back from markdown to media wiki to back to markdown You never get the formatting right you can also do it in HTML if you really want to but I don't know why you would Maybe yeah But yeah, so like and again like if there's you know if you have some people have a list of Like they use the wiki macros like get package status and stuff You could still point out to a wiki page as like further details, but you can still have the content there Or you can use a handy new CLI tool and just do it all from the command line. So Yeah, so it better work Yeah So the idea is like a lot of the work you're doing is happening in the command line, you know at a terminal anyway You know you're making like Commits to your spec file and pushing that so like what if you just add then you went And then you had an editor come up that you entered your text in and then you could possibly create a release engineering ticket If needed and like all that magic just happened and you never even had to open your web browser that'd be kind of cool, right? So if you want to see more about the the tool come into the summer Coding showcase Saturday morning and also have about what 10 minutes to to talk about it So it'll be a pretty quick overview I'd really hoped we'd be at a point where I could like we'd have like a final version and some documentation I could show you stuff and The reason where you're not at that point is me So I'll take the heat off of you for that like that's entirely on me not having enough time in the day So yeah, so you're submitting your proposals as Tiger issues Then processing happens in an automated fashion I type a command and give it the issue number and it does the thing it sends the email and It sets the reply to so that it doesn't keep going to develop an ounce also So we don't have to keep going back and rejecting people's replies to develop an ounce because that's not useful use of that list So hey, that's an improvement right there already Then it creates, you know when I type the next command it creates the The fesco issue when it's accepted it converts it to a user story And we create the release notes issue and create the bugzilla and I don't have to copy and paste stuff Which means the thing that you will look at is actually correct And we'll still produce HTML change set summaries kind of we have on the wiki page except why deal with the wiki when we could just Write an HTML file and put it on the web Yes Because there was so there was There was enough feedback that people like to use bugzilla like the change tracker bugs to tie in Other bugs like to be related or you know blockers and stuff Yeah Yeah, so if like somebody's for example retiring Python 2 they might link a lot of bugs against the you know Get rid of Python 2 change proposal So What if it's terrible? I mean Tyga has some issues. It's not a perfect tool People already know the wiki they kind of like it some people just were even refused I Actually got that feedback like they're like no the wiki works and I'm gonna keep doing it Hey, there's hundreds of us. We all have our preferences. It's fine Here's the thing if it's terrible we can go back the wiki still exists We can find another way to do it like we can do stuff and I will do the copy pasting for you and The hopes that I can do it correctly and if it's wrong. Well, there's only so much I can do about that But again, like, you know, I really this I'm treating this as sort of an experimental thing like we're gonna try it for a while and we'll we'll improve upon it and if it's just Unacceptably bad forever It's not that's not the change owner's fault that it sucked So we'll go, you know, I will take the time to clean things up for them because I'm nice No matter what Matthew says about me All right, so what's next like I said, I really hope we'd have the CLI tool done I just kept not testing it and not giving them enough feedback and so that makes it really hard for him to You know fix things and get the polish done So then like we'll finish right up some documentation He's actually written a lot of good documentation of the usage of the command line tool itself but we need to document like the change to how you actually submit it and Gosh, there's a whole bunch of wiki pages that I keep meaning to put into Docs stuff or a project org and I just never get to it because man, that's like you got a block out I just need to like sit down like this is my afternoon where I shut everything except the wiki content and the get repo um But we already have some accepted for our 32 proposals including this one like we actually ran this through the changes process Which made me a little nervous, but then I was able to bribe fesco into accepting it. So that was great um, I Was able to use my powers of persuasion and beg fesco for lenient Much better Um, and so, you know as we do it for for for door 32, you know in the next few weeks this will be ready to do so most of the change proposals will be submitted through this and We're gonna we're gonna find things that are bad about it. So we'll fix those if we can But then mostly it'll be done and I'll find something else to improve I already have some thoughts about the release readiness meeting and I will be happy to share them later But it'll be a much that's like a real improvement that everyone can get behind I hope And so with that I am about at time. I don't want to keep people from dinner So I will stick around for questions, but don't feel obligated to to sit here and wait if you'd rather go be doing anything else Yeah So to sum it up the the concern raised is that we have a lot of things that look very similar to this and we're solving them with sort of custom solutions instead of just using Pull requests and get I would argue that anything that involves processing text is not a good fit for A poll a poll request because you can't separate the metadata from the data as easily And we've never been very good about establishing a separation Yeah, I want to go ahead and just pause on the general discussion and make sure they're you know Take any other questions about this specifically Matthew. Yes, I didn't mention that So the question was well will the user stories move through and that yes That is one of the things so right now the the source of truth is the bugzilla issue So if you set it to modify do you set it to on QA with cetera? So one of the things that will happen is there's a script that basically checks the bugzilla status and puts the card in the Right category, so you do get that dashboard view and then so each Release is an epic and tiger so you can just look at like the fedora 32 changes or you can look at all of them and see Where you know easily see what state they're in as opposed to like looking right now It's the wiki page. You have to look through each entry and that I which one's that okay? Yeah, okay So yes, you do get the sort of dashboard view This was never Yep, yeah, so that's one of and so I had originally Come up with this idea and using Pagger as the source of truth. I'd you know started talking to people about this last fall And then tiger became available. I was like, oh, that's actually for like the dashboard kind of view stuff That's actually a much better fit for this kind of workflow I'm not sure I understand the concern. Oh, so it'll be in one place Is that is that the like so PR would be you just Have you the conversation would happen in the pull request as opposed to like on mailing lists and things like that, right? Yeah, so in a lot of ways, this is very similar to how the wiki or based workflow works Yeah, it's like and that's you know, it's like there are things we could probably do better But on the other hand like it's also a way to like make it a little more incremental So like right now once I edit a wiki page, I get email notifications if someone changes it could do the same with you know tracking the the status and nice thing about Tyga versus say Pagger is You know for like issues and stuff You can actually see that the edits were made and see what they were so like it's a little bit a little better version control For the more general case you're talking about I'd love to have a good solution for that I think there are a lot of things where we would benefit from a more generalizable solution But I don't know what that looks like in a way I Mean Doing the thing where we could do all the stuff in And there would be a whole core integration that we could put issues or battle But in the different though and sink the state straight and tied up with the dashboard representation Well, still maintain the source of information Tiger and that would also allow us to do bi-directional planning across different services because everything that happens in Packers already exported out of messages, and then that allows us to change the other services that exist in the door But we didn't get that far because people stopped caring about it And then even broke a little while and then it came back And so we just hadn't gotten back to that In some ways There'll be templates for it Yeah, it's just I mean it's certainly possible it's just sort of a at some point It's not worth separating like a lot of a lot of these things when it's basically just and Honestly now that we have metadata pulled out like I care a lot less about the actual Content in there because like explain it is basically what it boils down to because all the things we need to process it or out separately I Or Changing the policy Yeah, and that's actually, and the council now is a policy for changing the policy, which is submit a pull request, and then we vote on it, so like, you know. We want to have more of this get-centric versus being diagonal, which I personally think is whatever. We want to go down that road. There are some enhancements that, as part of that, that Tiger Bridge that you and I have discussed about, making up the milestones so that they work like a compound board style, moving things and changing actions and stuff like that, and they can easily present ER issues. That would be, because technically there's nothing that stops them from supporting that. It's just that nobody expects for issues to take pull requests, so we don't do that. That's one of the advantages of everything being changed in life. So I guess, what happens to this when it is done in the archive? The one that nice things about our current would be being nice. It's like, go do a Google search for things from, you know, the door, whatever, 12 or whatever the future process started, and see what those things... Seven. It was after the core extras merged. Okay, so I think you can see when things landed there well. If these end up in just Tiger cards that are off the edge of the board in the archive, that's probably bad. Yes, that's the HTML output. That's basically going to be the same thing that we have on the Wiki now, just generated, and then I could have the scheduled directory SSHFS mounted, which I do anyway, and then I can just write a file out instead of having to edit the section and copy and paste stuff in. Yeah, it can be, like once we're producing the file, we can put it wherever. That's other questions. The old stuff when we move to the new process, because it would be nice if all of that was archived and discoverable from the same place. Wherever that is. Yeah, so it would be nice to basically go back to Fedora 7 and just copy everything over. You know, if I get bored or something, but like... I mean, you can just take the HTML-like course and save those static files. What's probably most likely is I'll just take the change set Wiki page and pipe it to a parser that produces just a regular HTML file and stick that in the same location. Yeah. And then maybe also the change pages themselves, just so that way there is some historical record that we don't lose, but not necessarily have to go through and reproduce the wheel. It shows up in the weirdest places now. Half of openSUSU's Wiki about how things are supposed to be turning out references to door changes to therapy. Incredibly detailed about how it's going to happen. Debian references are stuff half of the time for some of the things that they're thinking about maybe possibly finding to do. It's referencing not even proposed changes yet. At this point, I'm pretty sure Michael's watching the change main space and just reporting stuff. Actually, I can come in this dashboard like that and we'll make it more clear what state they're in. That's one of the things I like about this idea is that you may be less confused about what a change means. But yeah, as you come across those things, if you drop a link to me, I just kind of like to see how other people reference our stuff because that would be helpful in knowing how the information is consumed outside of the community. Going back to what I was saying about communication, that's one of the main target audiences is people who aren't coming to flock or contributing to Fedora. It's the tech press, the user community, other distros. The one that I'd like to give as the greatest example of Fedora teacher has changed that everybody just refers to as authoritative is the user group page. Because that thing is everywhere. Every single distro that has actually done it references our page on the methodology on how to do it. From Debian, just when they did their user merge, this year, Fedora, yes, this year, our page from six years ago is the primary documentation for that. And when they did it last year, same reason, same document, OpenSUSA is going through finishing user merge because I'm opening them with enough space. And again, the document shows up. It's in the page where their user merge would either say, reference a Fedora document for more details about how the book works. And it just, it's literally everywhere. So like if you want a reference about like, this is how far this spreads and we don't know it, that page is in so many places that if it disappears, we'd be in trouble. So you should also, when you come across these links, send them to Matthew so he can use them to talk to his management. Be like, hey, look at how much we're driving the rest of the ecosystem. Okay. You know, there are a lot of relief coordinates and they're equal for doing the intro, but you're doing it in an awesome way and it wants to be part of the show and actually founded by individual space. Great. Great. And I know the number of projects that have built their process based on ours. Like for example, the Mendrima family changed their whole development process just like ours after it worked out so well for us. So there was that. The early open stack stuff for their scheduling and release identification was very similar to our process before they switched to the airing model. Mainly because launch packs suck. Would you mind sending that to me if you remember? Yeah, because I'd just like to take a look. I like to see what other projects do because there aren't too many projects of our size that have like processes around stuff in this way. So it's always nice to compare notes. I don't do them, but in 2005 up until like 2010 or so, they would make these things that look kind of like ours in their wiki that they call blue friends respects and describe what they're changing, the impact, what's required and things like that. And they don't do those anymore, but it seemed like that was the genesis of the way that we do these things. Mira, I'll let you give the last one and then I'm going to get out of the AB cruiseway. Possibly unrelated pieces that are confused by something that's written on very old future pages assuming it's a current state of documentation. Is this going to help that or make it worse? Are you making sure that the documents that are generated actually from the first site are not confused with the current implementation? They will be easier to find. Hopefully they'll be a little more clear about, because it'll be like closed, implemented, but I guess that'll be like, oh, it's current state. I don't think it'll help. I hope it won't make it worse. I think doing a better job of keeping our documentation up to date and in a single, sane place. So the less documentation we have in the wiki, I think the easier it will be for people using search engines to find the correct documentation, because it'll just point to docs.fidoraproject.org instead of a wiki page that somebody wrote and never actually implemented seven years ago. This isn't a change. It has a Fidora 34 official documentation, at least the original docs. That makes sense. It's now 47 minutes after I started for a 25-minute talk. Thank you, everyone, for coming.