 Welcome to 35 Fedora releases in about 30 minutes. I have timed this talk and gotten it in 30 minutes, but I have to talk really fast. So I asked for more time here so that I could talk a little bit more slowly. I still will try and go very fast. It won't be comprehensive even with that, but we should have the ability to hit some important highlights in the history of the Fedora project and some questions after. And I think hopefully this will be fun for people who have gone through all of this, informative to people who are newer in the project and or is interested. And I think there are some lessons that we've learned along the way that hopefully we can not repeat our own mistakes and we can maybe other people can learn from them. Dan, all heckling should go later at the end of the talk. I have a slide on that later. You'll, you'll, yeah. As I said, heckling should be saved for the end. Yeah, this is me, no time to explain. If you don't know me, come talk to me later. I'll be happy to, but you know, I'm from the internet and stuff. I have 85 slides here, so I'm gonna try and get through them. There's a whole bunch of stuff about the history of Red Hat Linux and the split to rel and all of that. That's a whole nother talk. Somebody else's talk, I wasn't, I wasn't there for the behind the scenes, so this is gonna start with Fedora core one. This is the basic format of the slides, release name and number at the top, a little bits of information around. Since I'm showcasing like the desktop wallpaper over here is kind of a visual focus on things. Desktop's always been important to Fedora, but it isn't the only case. A lot of people are using Fedora for server use cases, for other things, for IoT and small devices. There's a lot of non desktop cases as well, so this shouldn't be left out, but there's the visual flare of the wallpaper, so that's kind of a big thing. I have the kernel version over here and things like that, like which init system is there, kind of put some technical things in there. The actual number, like that's not important. Over here on events, that's our big headline events. We've always had a lot of different smaller events in Fedora as well, those aren't listed. Fedora activity days, Fedora women's day, that kind of thing. And then I also have most active on the develop list, that's kind of a way to kind of get some people's names up on the screen. You may recognize some of those names as we go through. That doesn't necessarily mean that's the most productive person that release, but you know, sometimes it is. There are also obviously a lot of quieter people who've done lots of work, who don't get shown up here and a lot of people who deserve credit. I also put the Fedora project leader's name up there, which means my name will show up a lot. Doesn't mean I am the most important person, but you know, I get to be the face of things. So there it is. Here at the bottom, I have from some of my metrics the release popularity over time. It's a little bit of a spoiler here, but we'll go through that as we go. And then some pop culture things, they're just for reference as well. All right, I'll take a sip of water because here we go. So you have flowers in a pond here, nice quiet beginning for the background here. So this is Fedora Core 1, and it was really basically what would have been Red Hat Linux 10 if that had been produced. So it was basically made entirely inside Red Hat with the same mechanisms. It wasn't really a community release. It was open source, but all developed internally. It's basically just the name Fedora Core put on what would have been pretty nice or Red Hat Linux release. And then there was extras, which were a collection of add-ons that you could basically put on top of it. And that was like that. The community was welcome to maintain these extras on top of that. But actually during this release, a really important key thing happened, which I think kind of set the tone for the project overall. So AMD64, 64 bit extensions for Intel architecture were brand new. And it was kind of a server-class enterprise feature. And so Red Hat was thinking that'll be a differentiator. REL will have 64 bit, Fedora will be 32 bit and people will obviously want to pay for that distinction. There was a lot of worry about how people will understand why to pay for REL, why it's different from Fedora at the time. So they were really looking for the differentiators. But then someone in the community, actually Justin Forbes, who's still a carnal maintainer in Fedora now, went and just working under his desk in Texas, I think, went and just rebuilt everything for X8664, built a version of it. And so I wasn't there, didn't work for Red Hat at the time. But as I understand it, this caused a lot of head, people were unsure what to do. What should this be allowed? Should we say anything about it or whatever? But eventually the decision from Red Hat was, okay, well look, this Fedora community people are doing this, we should accept this. And so it actually became part of, it actually officially released at some point as the 64 bit version of Fedora. And I think that kind of set things both for multiple different architectures and just for the Fedora community as an independent, not just the community doing add-ons, but something that can actually make fundamental change in the project and that was a really key early thing. Also, I wanna point out the logo here, which is a little red hat thing, which is funny to me because we're not supposed to use hats in Fedora for, again that differentiator reason, which is funny because you know Fedora. And also it looks like Red Hat's new logo that they after much, much expense changed the shadow man thing back to this little icon here. So the next one, Tet Nang, there's a complicated process by which these names are derived and they have to relate to the other one and I've explained up at the top what the connection theoretically is there. This is basically another, like what would have been a Red Hat Linux release. It actually looks very much the same. There were some important technical changes under the hood. Had the 2.6 kernel back then that was a big jump and SE Linux was enabled. Dan Walsh is here in the corner. He can do a talk about that. I actually have my notes. That's a whole nother talk and Dan Walsh is the one for that talk. But a big question here is, what does this community stuff mean? Is it, how involved is the community here? And so actually a lot of people talk to me about this IRC, it was actually an email mimicking a chat log here that Mr. Icon, Constantine Rebetsov sort of sent to show, like this is how people are feeling. Red Haters don't seem to get it, but like it doesn't really, we don't know how the community's gonna engage here. So yeah, in the release announcement here, like it's kind of presented by Red Hat here, it isn't quite clear what the open source community as executive producer it says in the release announcement, like what does this mean? So yeah, from the outside, as a community member who was interested in this stuff, it was all pretty frustrating. People who are involved in CentOS and seeing CentOS stream and Red Hat's communication struggles with that may experience some deja vu with this. It's the same kind of thing in a lot of ways. So FC3, this one's a little bit of progress. It says in the release announcement, I don't have the release announcement up there, nope, I don't, but it says the Fedora project and Red Hat would like to announce, rather than Red Hat announcing, but the core is really still an all over Red Hat thing. There were some things happening, the Fedora extras, the add-ons, were actually now, you could get them from the same download server, it wasn't a whole different thing. There wasn't actually a central build system. The build system was you would send your spec file, your two sets that all, and he would build it for you and then put it there. But things are happening, there's like four times as many packages and extras as there were before, so people were kind of coming together and working on that. Then FC4, same wallpapers, the only time we've ever duplicated the wallpaper and it's kind of a boring wallpaper too, so oh well. This release was, I think, the worst ever. It took forever to show up and SE Linux was in a horrible state, it wasn't usable, it was very frustrating. Reviews were like, this is all about features and no one cares about stability, this is all the, it was probably all true. Also the release engineering process wasn't very good and actually there were changes, like midway through this long thing which meant that you couldn't install it anymore, you had to install the original one and then do a bunch of updates, like it was a whole horrible thing. But there was a lot of exciting community things going on. I may be biased here because I personally helped organize the first Fedora conference, the Fedora user and developer conference in Boston and I think that was actually, again, I'm biased but I really think that was an important turning point because it was a lot of red hat people came and a lot of people who weren't red hatters and it all felt collaborative and friendly and fun and there was no red hatters telling people what was going on, it was red hat engineers who were talking to other people in the community and trying to figure out and deciding what we're gonna do and people in the community presenting about how they wanted it to go and it kind of felt exciting and collaborative so that was a really, really nice moment there, I think. I'm really fond of this background, I don't know, I'm not a graphic designer but I looked at this and I was like, oh look how fancy that is, it's so, I don't know, somebody who was an artist made this one so an artist made the other ones too, that's not fair but I, so graphic designery I guess, I like it. So the big news with this release really is a story about Yum, which the precursor to DNF now the Meta package manager. So before this in the olden days, like there was just RPM and if you wanted to update your system, you basically had to download a bunch of packages and hope you got all the dependencies and it was a whole mess so there were some tools developed to do that and red hat had a huge investment in a thing called red hat network and up to date which I think was just discontinued like several years ago but it was supposed to be, like this was gonna be again a monetization thing, this was gonna be the product and they put a lot effort into it and then meanwhile people, Seth Vidal took this thing, that an open source project and put it into Fedora that did this idea of rather than just having to download themselves it would actually like figure out what you needed to do an update and retrieve all the things and get them from the mirrors and do it in kind of a nice polished way so you could actually have, you'd just run an update and it would update your system, it was amazing and so again as I understand it there was some inside red hat like consternation about this should we allow this thing, what should we do with it? You know it got put into Fedora and then eventually red hat realized okay this is actually better than the thing we have tried to develop internally and it ended up going into RHEL and being the update system for RHEL as well which was a huge kind of thing as well there was a letting go of the not invented here mentality and say okay like the innovation can come from the community it doesn't have to be something that we've designed at red hat even if it was something that was like contrary to where we thought we were going with our business plans we're gonna take a leap with this and figure out how to work with it so that was really important although still like red hat has this director control I think in the release announcement may have talked about that kind of thing again all the decisions for engineering those were red hat internal decisions made kind of I don't know I imagine some sort of shadowy back room but actually not but one funny thing happened here when this release came out that all the Fedora systems were on the same network as red hat red hat was a small company at the time and the release was like an end of quarter some sort of important business day and all the downloads for this totally broke the network and ruined everything and Mike McGrath tells a story better than I can Mike McGrath is now the VP of Linux or something maybe even fancier title than that now Big cheese says Dan yet and by the time he was the Fedora infrastructure lead but he didn't work for red hat he was a community volunteer he worked for a different company in Chicago and he was the main system man and the CEO of red hat at the time called in the FPL and was like who did this you need to get your system as men network of people right in here and so then Max Spivak explained well I could but he's in Chicago and also I can't tell him to do anything he doesn't work for us doesn't work for you so at that point very shortly I guess Mike was hired and then told to fix the problem so I don't know that's one way to get a job we also this is where the now classic actual Fedora logo was designed this is actually made by like a brand consulting thing and it took these ideas like the red hat marketing red hat marketing used to actually like do marketing stuff for Fedora and they came up with the slogan of infinity freedom voice and that's where this come together to make the classic logo there I have some fondness for it but yeah here this is a wallpaper that a lot of people love when I was talking about that people like really oh that 3D underwater rendered one wow this is one of people's favorites here people also like the name Zod it was a whole in joke thing we've got lots of in jokes but this release actually happened on exactly the day that my second daughter was born so I don't remember much about this or several months after of that personally but you know I went back and looked there was still a lot of time happening between your leases 218 days that's a lot more than six months there's a whole long story about why there was no Fedora foundation that's probably also a whole nother talk but at this point the Fedora board became an official governance body before that there was a thing called the Fedora advisory board which was really just a mailing list where anybody could give opinions and nobody made any decisions or whatever but they actually made a constitution for the project that had the Fedora board as the governing body for it and it was a mix of people who are elected by the community and appointed by the FPL and the Fedora project leader had veto power over this structure there and now finally we actually have a mirror manager system which let me get accounts so you can start seeing the statistics starting to happen at the bottom of the page there another nice wallpaper there, moonshine so this is a big thing here was core and extras got merged together so having that split not only was it starting to be kind of actually problematic practically it was showing all the frustration in the community and the stress between like who can do what and so on so Red Hat made another big decision again a lot of fighting with the internal wrangling that I'm not I don't know all the details of but eventually the decision was made we were going to merge these things together into one unified project we're not going to have this core and extras split anymore and it actually turned out to be easier to move all the stuff from inside to extras so basically core got merged into extras extras became the thing and core went away and it became a unified release this is also the first release with live images Fedora wasn't the first distro to do that but it goes way back to this cool also at the FUDCon we had again in Boston there we talked about this new file system called butterfas thought should we make that to default and at the time we decided well it's going to be about two more years before that's ready so in about two more years we'll take a look at that werewolf so seven was where these were brought together but eight was really the first community collaborative release this really felt like this was some ways this is actually the first Fedora Linux because it was not just it not just released as a unified thing but it was actually planned and put together as in the open as a community project so this was a really fundamental release here this is where the feature process which we now call the changes process or change process I've been caught and wants me to say one or the other and I cannot remember I'm sorry Ben plural or not so that was introduced and basically the idea is not that we should have a big bureaucracy but stop surprising each other with changes if you're going to do something big let's talk about it first and what impact that will have it's not meant to slow things down or stop but just make sure that we're all talking together so that we can make things work the best we can this was a very very popular release there are still at least a hundred systems out there checking into the mirrors every day to see if we will release any more updates for this just in case I enjoy that we also this is where the idea Fedora Spins came out with spin is kind of a version where instead of just you know this is like the official one desktop release we have different ways of putting together the software into a different you have different desktop different installation I think this is kind of fundamental to what's interesting about Fedora as a distribution we really try to instead of saying oh you wanted to make it different well you can fork it and make your own project that's based on it we say hey come into the project if you wanted to make you know your different version of it that works this way cool we have been trying to find a space for you to do that this wallpaper was the first one I think with a time of day feature so it changes over time this is the evening version here Sulphur the features process brings us new features I guess we had a new init system here so this is another really interesting story so init system is after the kernel boots it is the thing that brings up everything else and responsible for making sure all your services are running desktop the desktop starts all of those kind of things and so the old thing system 5 init script system V I don't know somebody can correct me on that later but that basically used a bunch of shell scripts that were kind of slow and buggy and inconsistent so at the FUDCon which was I think in Raleigh that time a guy named Casey Dolan stood up and said hi my name is Casey I'm going to replace the init system if you want to stop me come to my talk and so people came and I guess no one no one ended up stopping him so Casey was an intern at Red Hat so he was a Red Hatter but this wasn't his job it was just something he was kind of doing for fun he wasn't working on the OS at all so this is kind of a community led improvement replaced it with did I say upstart here is the system that was used there and so that was an interesting thing that also went into Raleigh releases is an improvement coming from the community that wasn't something that was started as a Red Hat decision there Cambridge Cambridge was actually going to be as I understand it the codename for the what would have been the Red Hat Linux 10 release so there was some wrangling to get it to be the Fedora 10 release behind the scenes even though nominally there's that link there that's actually the real reason this is called Cambridge this release is important I think because this is where the Fedora foundations came from there was that infinity freedom voice that had been you know basically somebody external to the project invented that and thought that would be nice which it's fine it's nice to have that support but it didn't really reflect who we are as Fedora and so there was some work on talking about who are we as a community and out of it these four foundations which we still kind of use as our guiding principles today freedom friends features and first we're there so I think that's nice that was yeah I think Paul Frieds helped finalize that and I think Max Bivac as FPL really started this down that road there this is a special wallpaper I actually never had seen this lion before before when did this you had to have two monitors on a widescreen the lion didn't show up unless you had the second monitor there so this is the those icons for the four foundations and also around this time we did talk about butterfs again and decided it's probably going to be ready in about two years and this is actually we took us exactly 196 days two releases in a row here for this one I'm going to go kind of fast through the next few ones here we had the first BudCon in South America this is a pretty nice but boring release which boring can sometimes be good here I have a note that we had a beautiful new website and I also see that I did not take a screenshot of the website but of Wikipedia this is actually not my screenshot I stole the screenshot from Wikipedia stole there's CC license information at the end of this talk it's not stolen at all I took but yeah a lot of stuff going on but this wallpaper very dramatic and I got a lot of people like is my screen broken on this one I love our community process that develops wallpaper but for some reason there's a tendency to make things that look like a broken LCD screen and this one got through here but yeah so behind the scenes in this release Jesse Keating who was the release engineer at the time did a big thing of switching everything over from using CVS as our version control to disk it so that was a lot of big work there which is actually pretty neat technology at the time because Git does not do a good job at storing large binaries so that's like at the time Git was starting to take off in popularity but there wasn't like everything is on the Git forge somewhere there were people's various things everywhere so just like today our primary way of packaging things is getting the official tar ball and so you have these official downloaded binary somewhere and those couldn't really be stuck into Git so he has a cover system for that it's all pretty neat it was innovative things coming from Fedora and I think Jesse got a job at GitHub on the strength of all this work so that's and also because in that transition we I say we but I don't know Jesse probably somebody made a tool called Fed package that basically instead of using the CVS commands it kind of abstracted all the package managing things so it didn't really matter what transition you were make what you're using underneath and we still use that Fed package tool a lot as well and kind of hide some of the implementation details I think that's actually also kind of a nice lesson because change is hard so yeah you can see you know it's not like huge growth here but we've got a nice growing growth trend around there things are going really well yeah so this really lovely wallpaper for this release this release is going to be great right I hear laughter from the audience here yeah so yeah uh oh uh goodbye to half our users here um so there's not enough time to talk about you what whether whether system D or genome are good that's we can we can have a like a whole like yelling fight about that um but uh you know this is a lot of change all at once and um I think you know if you look at these technologies today I think they're good this was this was the right choice to go in this direction but wow uh they were not ready and people were not ready and we actually I went back and looked and I was like well could we have documented things better like how could we actually put a lot of work into making this smooth there were videos about all this and it was like we tried but it was just too much change um I think you know in retrospect like that Fed package tool if we would have like worked on some things for compatibility and done some user experience things and whatever it would have been a much better experience um and it's always kind of a trade-off so one of the problems with that we switched to upstart as in a net system before but um there was so much emphasis on making it compatible that basically um not only did you not notice any of the change we ended up just keep writing the same shell scripts for it and we never took advantage of any of the possible improvements from it so there's something where uh if you're focused too much on the compatibility it can hold you back so going ahead is sometimes important but um maybe let's not do it this way again because that was not great uh this wallpaper obviously a reference to Jules Verne and 20,000 leagues under the sea um but I kind of can't help but feeling that the sort of murky depths and gloom was just kind of the feeling that my kids say the vibe of of the project at the time and it was you know kind of depressing and um so we got a depressing wallpaper here um Jared Smith who's the FPL said that was the awkward teenage years um so yeah that's made me something to it did you manage that? uh what did I what architectures? that's not a lot there's what did wait uh there was yeah sorry I'll fix my slide later um uh I don't know why we dropped uh anyways um the comment from the audience was I've accidentally left arm off the list here and I should not um but um yeah um we did consider ButterFS as the default file system um this time and um yeah we considered we decided maybe another about two years that's gonna be gonna be ready um there was actually one big important change that happened here before this Fedora had a uh contributor uh license agreement where basically you said that you were giving you know the right your license rights to Red Hat to control and since at this point and since that we've changed to a uh contributor agreement which basically says I've got the right to submit this code and it's under an open source license and um if it doesn't have an explicit license you can treat it like it has MIT or a Creative Commons license so it's it's not a thing that gives away any rights and so that was a change that um I think made the made contra being a contributor not working at Red Hat uh to Fedora much more equitable there are a lot of open source projects which have a thing where you have to assign your rights to the to the company or to some entity that works on it and often uh that's kind of a uh unilateral thing where uh everybody outside of the special entity is bound by copy left it means you know if you make a contribution that contribution can be shared with anybody and so on whereas the entity can say oh I'm licensing this as proprietary software I'm gonna charge a lot for it uh even so even your contributions that are GPL for everybody else for us they're special so we we don't do anything like that and I think Red Hat doesn't do that for anything uh I think that's pretty important that we got rid of that and I'm glad we didn't uh actually we're even looking at if that contributor agreement is even necessary for a lot of things um there's a lot of cases where um there's a concept called uh license in license out basically the assumption that when you're making a patch for something you know that that's that it's under the right license just by making the the contribution um but that's for legal to figure out um I would like to point out that on the develop list here we have Adam Williamson is both the most uh prolific thread starter and the most prolific reply to those threads there um I think that's uh um yeah so things can't all be murky depths here so we had this release um beefy miracle the mustard indicates progress um and the naming process again you can see that people were not afraid to bend the rules to get what they wanted in this one here yeah yeah all right right so oh I'm missing I'm so sorry that I've dropped the architectures well what am I missing now oh I think maybe that's supposed to say arm 64 and arm hfp okay whatever um I I told them to save the heck actually I I yeah I right I told them to save the heckling for later but it's not working anyways many people describe this as their most favorite release overall and I think you know there were a bunch of features but I think the important thing really was just kind of the camaraderie and the friends foundation of this kind of fun themed release that was very kind of a silly mascot and everything it kind of even even though things felt a little bit depressing kind of brought everybody together and the community felt like yeah we are yeah we're doing something here and so I think that was that was really a nice moment there in this release these tiny dots are actually circles because it's a idealized physics problem cow and the name that's in you know nerd jokes in here um this release was kind of a mess actually um the installer code base anaconda it had gotten kind of horrible it needed some serious refactoring which it got but that features process was very geared on what happens release to release and it turned out that rewriting the whole installer was something that needed more than one release to go through and you know we slipped all the way around from the fall release to January which was not great to say the least and also the anaconda team that doesn't sell our works on both fedora and rel and rel didn't have or an upgrade feature that's actually some being worked on for rel right now but at the time no upgrade feature and it feels pretty important to fedora but um oops somehow that hadn't made the requirements at the very last minute we realized oh no there's no way to upgrade hey we kind of need that so will woods threw together something called fed up which is a name that I love dearly I wish we still had that um and this actually is going to kind of accidentally set us on a good path really because it actually was a better upgrade process and from this we've really worked on making sure that the upgrades are a really smooth process that you don't have to stress out about it's not a oh no fedora short life cycle means I need to give it way up you know two weeks of my life every year making sure the upgrade works it's just a hey this run do the upgrade thing you'll be on the next release and smooth sailing on so I think it actually it actually turned out well through serendipity there but it wasn't wasn't the intended result another thing that happened around this time is we had a lot of pain with release engineering and I will not name names but um one person had been doing most of the work for in that area for a number of years and had ended up just from being very overworked um you know the person who knew how to do all the things and no one else did and also you know some amount of pride in that like I am the person who knows how to do all these things which is totally reasonable and a lot of things in open source are kind of driven by that like yeah I have ownership of this is my thing but it can also be pretty unhealthy for that person and for the project so there was a lot of frustration and I don't blame the person all this is a systematic like this is a fedora problem that we hadn't really addressed very well because we should have found ways to support the person in that and found ways to make sure that they could have that pride in you know being there in a way that was about the team and the sharing and success of everything and so that's uh but that was it was definitely causing problems around that time and that's kind of a thing that kind of a repeated theme a little bit here uh yeah again we've got nerd jokes here and these are boxes to put cats in the problems oh yes yeah yeah yeah um yeah here you go um so this is a good the qqa thing um it turns out that putting unicode in the name exposed a lot of exciting things everywhere um and actually um like having us I think this was the first name we had a space in it which actually like turned out to be worse like a apostrophe apostrophe yeah right I think we solved the apostrophe by making it to a unicode apostrophe um so uh yeah great um but yeah uh so all that was kind of fun but um the biggest thing that happened around this release was really an immense tragedy so Seth Vidal who I mentioned before as a really important early contributor to the project was killed while biking home um in a hit and run driver um Seth was very humble and he would have downplayed this but he was so important to uh making fedora what it is both in the community and in technology um when I was a we were both university assistants at different universities and he helped convince me that um it was worth the effort to be part of the community and give things back rather than to you know work on my own and you know do things there and um yeah he was he was kind he was funny he was brilliant and he was a dear friend and Seth we miss you very much um this wallpaper um it's a 20 xx for 20 there in the background very subtle this release had a lot going on I was actually going to put a picture of myself wearing the 10 years of fedora's shirt there but I decided if I was going to cut something out the vanity could go um the badges aren't actually in that screenshot I just decided to plaster them there because they're fun so we launched the fedora badges gamification thing which is um and something we're working on again and kind of putting more central a way to both bring people into the project find easy ways to get you know get people hooked in different areas and also you know to uh some people are very much driven by can I get a digital sticker for doing a thing so um we have we have digital stickers it's it's very fun um this is also a place where we decided you know we're at the 10-year mark it's time to do some strategic planning um we started out great and then we had this big drop um and now things are kind of you're taking uh up again we wanted to make sure that the next 10 years um would have been would be would be awesome basically uh so there's a flock to a fedora event we changed from this fudcon to this and the idea robin bergeron um my predecessor's fbl had this idea that rather than having these little distributed events with um no real focus we'd bring everybody in the world together to have a one big event where we would all talk about what we're going to do plan the next release and bring this together and so that was a focus of a lot of the strategy there um tom calloway russi lee did a lot to make that be a reality um I'm dropping names and um I probably shouldn't because I know I'm missing a lot of amazing important people's names in this so if I've left out a name I'm sorry you deserved to have your name mentioned as well but um yeah um yeah uh release engineering comment here like this was this was hard because of all the changes we were making around this um but also at this time the fedora board had kind of faded into the background and kind of become um there weren't a lot of big decisions to make but occasionally there was something like is this open sourcey enough would be raised and then the board would go off and have a debate and maybe come to a decision and then give a like a like a prophetic announcement that wasn't the question asked and it wasn't really very functional so actually fesco the engineering steering committee ended up driving a lot of the strategy work here um this was uh yeah so this thing called fedora next we came out with having different additions uh one of the big problems that had was kind of causing a lot of stress in the project was uh the earlier strategy had decided that the desktop release was the called the default offering and then we had a lot of people who were contributors in the project and they mentioned way back at the beginning that there are a lot of people using this in server use cases not the desktop and uh with with the desktop being the default a lot of the decisions were being made okay well that's what we're gonna do we can just desktop oriented decision and so it ended up that people who had sysadmin interests basically felt like all they could do was complain that's saying no no no was their role in the project and you know sysadmin's tend towards being grumpy anyways so it was easy to exacerbate that uh so part of the idea here was let's not do that let's give a positive way for um people who want to have these different use cases to work on fedora server work on iot work on what are these different cloud use cases and say okay you can make the decisions that are right for your use cases and have them even even if uh desktop can also not have to try and say i'm gonna make sure that i placate the server needs for this very different use case for here so we can have these different decisions so i think that's worked out really well um some of the other things fedora rings not quite there yet but um so did they get all the architectures right this time peter no okay awesome did not still still got the list wrong um yeah go ahead let's do that is another talk but the three additions is three additions is what were fedora server fedora workstation fedora cloud at the time and so we've expanded from that but that's basically that decision there yeah that's um yeah so uh actually because of the release of engineering problems and all that this is actually we decided to stop and actually we skipped a release for the first time and probably only time ever um to kind of give um release engineering qa time to like recover from that hard release and get the tooling up to shape to do things that turned out okay um i i think that probably staying on the cadence is something we should keep to that's a whole another big conversation um yeah a lot of people were skeptical about this additions idea it was just very fair um it was a big change um i think that idea really worked we can kind of see as these things go up here like um it was a hard decision but i think it has been see a lot of growth in the project and that that strategy and approach made things better in the community and made the release a lot more popular um and um yeah uh this is where as part of that strategy we decided to have a brochure site get fedora.org and we're going to also have something that went hand in hand with this a contributor focused site that was going to be a thing called fedora hubs the idea was um there were like 1100 IRC text chat meetings every year and a lot of stuff on mailing lists but if you looked at our website and everything that may be somebody who is new to the project might come and look it looked like we were dead they would look like there was no activity on the internet happening because you know the internet had changed from being something that was a lot of these different text based protocols to you know the web and social media and those kind of things were really uh becoming the focus of what people thought was the internet and so the project we project didn't really show up so we wanted to have kind of a social media e view of showing all the activity um that turned out to not work so well um but um that's um another talk but um we kind of come back to that idea later of making sure that the project is visible and activity is surfaced um josh boyer says that he was to blame for killing the names here but really um so we stopped having the fun code names uh the reason is we were coming up with these lists with these tenuous links we would take them to legal they would review the you know ten candidates and say these two that are the worst possible ones you can use one of those and then that was like we were using up quite a lot of legal resources on that so fine we decided okay that's it's not it's fun but it's not worth it for that we have to find our fun in other places i'm sorry um yeah um i mentioned that the board wasn't really working um not just that it wasn't very active but um it wasn't very connected to the project so we had made this you know three editions decision the board actually approved that went through this whole process of that um and then at the flock conference a little bit later we were in a room like this and a lot of people were like no one made that decision that wasn't official and like it was supposed to be official um so like that like clearly when you have a community led project a community driven project you can't have top down decisions that just declare things it's got to be something that's connected in and people need to understand okay here we have a process for decision making the system has been made and it wasn't really working very well so we came up with a new uh structure called fedora council um it maybe also can use some improvement but i think it's worked pretty well where it tries to draw people who are involved in different areas of the project into leadership and to have a more active um not just a um behind the scenes governance and dispute um resolution but actually active involvement in in leadership in the project um we also got rid of the exclusive veto for the thorough project leader by going to a consensus model where effectively everybody has an equal veto on something and we have to figure out how to all get along on things which actually has worked very well we have never gotten to a difficult situation where you haven't been able to reach a consensus um how to do consensus that's a whole another talk i would love to do but again i'm actually running way over my time here because i am going much slower than i was going to um one thing we did add um we have a thing called the f cake the fedora community action and impact coordinator that is something robin set up and it's one of the reasons i've been able to be fedora project leader for so long it is no longer just one person isolated there's a kind of another full-time person to help with community building and activities in the project which is um very very very helpful um yeah i can talk also about why that's not called community manager and a whole other talk which i will not do right now um change things up fedora kd plasmon desktop here um got a diversity inclusion advisor i think that's a thing um i would like that to be a fully paid role i can't convince redhead to do it if anybody else can come up with funding for this i would love to have that be a fully paid role um yeah a lot of things going on here i'm gonna go here xfe xfce spin here um a little bit of working in marketing fedora loves python here um one of the things that happened around here though i would have brought the the docs team kind of got into a situation where it turned out where we hadn't been paying attention and one person had become the docs team who had all the docs team knowledge and docs team work and that person got burned out and suddenly we didn't have a functioning docs team in the project and again this is not that person's fault it was something where structurally like it was going along so well um we weren't really paying attention and giving them the support and the team the support that they needed um so uh that's um yeah here it was a really short release cycle this is something we decided on purpose because previously when the release had drifted we decided we said okay well we'll go six months from now to the next release and that was causing releases to rotate all around the calendar and be unpredictable so we decided well we're gonna stick to the fall the october may release cycle and even though that this will be short um it turned out to be an okay release despite that there wasn't a lot of changes that all worked out well um and people actually really liked this one it got a lot of really good press which was very encouraging to people who kind of went through those awkward teenage years you can see in the graph here there's a big up shoot there kind of I think the things we were working on people started to take notice and be like hey this is interesting things going on in fedora these days um peter robinson I have you name dropped here uh you said you made yourself redundant for secondary architectures we got rid of this idea of having a whole another process for some of these other architectures and kind of merged that into the main thing maybe I have the list right this time who knows yeah all right I got the list right too so that works out um this is I think my favorite wallpaper I it's beautiful um this is actually not my voice print it's the designer Kyle Conway saying fedora in a voice print of that and then that becomes the trees like that's just nice and it looks pretty um we got a new mission statement I'm not gonna harp on that too much but I think it kind of goes back in a lot of ways to that thing about spins and where we try to make it so you can you can as fedora as a project we want to make something you can use as an end user but we also want to make it easy for people with ideas about how to make an os how to make things better to work on those ideas and deliver those to people um it goes yeah from fedora next rather than just like handing out a bunch of building blocks um we also we want to give out some pre-assembled things but we also want to give people those building blocks and like hey you can make your lego set and give that to people that's um something that we've tried to empower as a project um modularity is it's whole another retrospective talk I'm not even going to touch it um we finally got mp3 playback patents software patents are a huge impediment to free software um our basic tactic is to wait them out which is um not not a great way but we actually waited this one out so there we go um yeah um yeah the modularity thing was a little bit of a stress um this was um this release took took way too long um or the previous release took way too long we made this one short again um modularity was really the fire here and again that's another talk here but um yeah I was sugar on a stick which is um I user interface designed for children to be intuitive it's actually kind of amazing because you put this front in front of an adult and they're like I don't understand how to computer anymore and you put this in front of like a four-year-old and they're like whoa so it's it's it's actually it's great um yeah this is another we broke the laptop screen wallpaper I don't know why we keep doing that um so this is actually the first ever perfect execution of our planned release schedule we were um you know exactly on time um and the modularity thing actually landed so that was nice um part of the trick here was dropping alpha releases so um quality team a lot of work around that um Ben Cotton as program manager I think gets a lot of credit here as well and uh less good here we had somebody who had been you know working on the website's team who over time became the website's team and had kind of felt he had ownership of everything and once again like he was doing such an amazing job with it that we didn't really pay attention to that and when he got a different day job suddenly we didn't have a website's team and uh it turned out that when I went to redhat and said hey um can you we need to we need a website's people and they're like no that's not important for rel so that was a whole crisis about how are we going to have a a nice pretty website um so yeah actually it turns out that um this should be a whole another thing we actually have a nicely revitalized community built website's team that made a beautiful new website if you go to fedora project dot org now it's amazing and that was really done by building together a community of volunteers interested in doing this so we're getting better at this but we certainly had this problem over and over and over again I love this wallpaper it's cool um one of the big things here redhat surprised everybody by buying coro s the company they didn't know what to do with coro s the linux distribution and so I fought pretty hard to have that fedora be a home for that and I'm glad that I did if you look at the statistics in a different talk come come to flock our next conference I'll show you some statistics fedora coro s is doing really well um and from now on at least through this talk we're basically 180 days apart released release very very consistently plus or minus one I think here um fedora silver blue came out around this time and based on the coro s technology which kind of showed that you don't have to just you don't have to be just a trivial change of packages how do you're making a face what have I done oh oh we didn't um yeah yeah whatever we'll talk about that later um yeah the idea is basically you can make a big change to make in a spin it doesn't have to be just a different desktop technology and package in the same way you can change how the disc the os is put together in fundamental ways and also have a fedora spin that shows that so I think that's a big fundamental thing okay um running out of time kind of move on we dropped the 32-bit x86 that's a big thing not enough people to maintain it this is a cool release I feel like that's like a really nice 80s throwback wallpaper there yeah there's plenty of other stuff but um big highlight that Lenovo decided that they would ship fedora linux on you know you could buy it out of the box with that and one of the amazing things about this is first of all if we didn't go to Lenovo to make a deal they came to us because of customer customer demand and more than that uh they didn't want us to make a customized version they didn't want a customized version they wanted exactly what fedora as a community was producing with no changes they wanted the real fedora linux not some sort of Lenovoized version they didn't want to take control over the kernel they wanted to deliver to their users what we were making so that was a really strong boat of confidence and a really nice way to work there there's been a whole bunch of supply chain problems and covid and whatever that and there's still not as it's not available as much as it should be but um you know uh call Lenovo and ask for this and because that's really the sales people hearing it is what's causing it not to be available worldwide um yeah covid times we had nest instead of flock um virtual meeting everything i was actually kind of dreading that um thing but it actually turned out that our virtual conference turned out to be amazing and energizing and um i didn't organize it so i can say this um it was the best conference of all of covid all of the nests they were most of the things i went to you know red hat summit sorry red hat summit people were just kind of draining and awful uh it what and like you know at the best they were like oh here's a webinar and our conferences really felt like the community coming together from around the world and having a virtual party and i that was amazing so again fedora community is wonderful um this you know is the peaceful universe wallpaper i think escaping the stress side that was a nice one there's a night version of that um here's a case where agnome 40 um remember back to where we lost all the users this is a really big user interface change changing things from horizontal to vertical or the other way around and i think both fedora and you know upstream learned a lot from this and actually spent a lot of time doing actual user interviews and research before putting the change out there and actually you know the results ended up being great and i was i was worried i thought there would be drama there um and it turned out to be there's some people aren't perfectly pleased but you're never going to make everybody happy and i think the the big change landed very smoothly pipe wire uh a big audio change landed you know there's a little bit of rough there as well but this is actually a funny marketing lesson which is obvious in retrospect which is a lot of people shouldn't ever care about their audio subsystem but there's one set of people who care a whole lot and that is youtubers and podcasters and suddenly around this release we were incredibly popular with youtubers and podcasters who had an exciting thing to talk about and they really liked that and so that kind of jumped up how people were talking about us so it's like oh yeah maybe appealing to the um audiences that talk a lot about what they're doing is a way to get people to talk a lot about things and we also have our new logo here um mo duffy does a great talk about why we have a new logo and how the designing went into that um i really like it and i'm glad that it now um will not have fedoro as one of the top um typos um or searches for people who don't recognize in languages where they don't recognize this as a word assume that that's what the old logo said because it had an a it was very indistinguishable from an o at the end yeah um okay so um up at the time you know best release ever fedoro linux 35 this is the new overview the new um sideways instead of vertical thing um we didn't quite make the 182 days there so i guess more than plus or minus one but still reasonable range got things right instead of rushing um and here um some of the big changes that are not there we uh launched the chat dot fedora project dot org moving from irc to matrix and working on discussion dot fedora project dot org i i know some people are very resistant but i want to move as much mailing list stuff from there to that as well as a modern friendly interface that is transparent to people who are not necessarily uh deeply involved in the project i also want to make sure that people are deeply involved have a good experience so we'll work on that transition it's not going to be dumped on people but i think that's actually a really important thing uh we also have a renewed fedora ambassadors program around this time that's kind of getting ramped up after covid um we're working on some investments in mentoring had a mentor summit i think that was a exciting thing um and uh this website new websites team that i talked about started coming together as well and that was really with kind of lessons learned on how to make this a team success rather than something that could be it was just an individual that was really really good um yeah um right there uh kina white was somewhere on this we started having some different variants uh using os tree um that were coming around is that was that in this release here kina white was that is there um um yeah so um i i was asked at the beginning where's the net missing three releases so uh i started adding on to here but i realized i didn't really have the perspective to talk about these most recent releases so um yeah i i'm sure there's interesting things to say a lot of stuff that's happened in the last you know year or so um but i'm decided i'm going to put the seal on this talk here and end it and then maybe in you know 15 years someone else can do a sequel and talk about what's happened over the time there um but that's uh yeah i'm gonna stop this here uh so quickly um some of the hard-earned lessons what what should we not repeat um you need to find a balance when you're making big changes um you can't just document it away you've got to make sure you're listening to users and you've got to um you got to make room for mistakes but you also have to listen and deliver what people want um but also having a community that cares about the project and is you know working together um it can make the rough parts actually functional okay i'm running out of time for questions um yeah community teams you need to have momentum to serve to keep going long term gotta make sure that you're watching the people who are in in positions where there can be a bottleneck so that they don't don't use this is one of the official fedora colors so that's there there we go tune to that um yeah don't um yeah um we've got to make sure that our teams are have you know they're not just depending on one person that people should feel like they're recognized and awesome and supported but that they also if they want to go do something else they can and the thing they're working on will continue and that should be a pride of success and going kind of back to the thing they talked about um red hat you know letting letting go and letting the fedora community lead um really has brought a lot of innovation into the project and i think that's one of the things that a lot of projects really could learn from it's scary to do it that way sometimes it's hard when you have you know to figure out how to get a community to come along with the decision that you want to happen but you get better results that way and okay there i have left some time for questions here so let's all right peter rominson is it about how i've gotten the architectures wrong the first what release was that in uh uh yeah uh so fedora iot i think i mentioned it somewhere there but i may have just skipped over it fedora iot is an important addition that we added kind of one of these use cases that we thought oh we need to make sure we have something that covers this emerging area and technology so um that's a very important thing yeah and actually going through the idea of adding a new addition made us come up with a how do we add a new addition process which we had not had before so we have that as well yeah that that was a thing dan this yeah right so there's another big change we switched to something called c groups v2 which is like a kernel level thing that's used for making containers be containery um i was almost going to say containers contain but then i remembered that i got dan in this room so i shouldn't say that but uh yeah and that was that was a big change that it along with the system dn those things it's one of the situations where if you uh if no one takes that first step it's never going to actually get exposed to users and it will never get there and things will drag on forever so at some point you've got to make that change and so finding that balance of when to go to that and when to not is uh it's important and i think having in some ways like the additions split where i think what was coro s slower in that some some of the additions decided to do that yeah right so some of the some of the some of the releases uh decided okay we won't make this big change right away and kind of rolled it out slowly so that people had the option it was it was the default in a lot of things but there were also fallbacks um where you okay this was is going to just work yeah why did i did i skipped over landing my joke here i've done another version of this i landed the joke properly um but i was i had spent too much time on other things here yeah so there's actually several other times it's mentioned we actually considered butter fs throughout the years and every time it was going to be two years off um and then and at between red hat storage internally this is actually one of these uh let the community do things things red hat storage people decided we're not going to wait for this anymore we've got to focus on you know uh what what works and so they put all their effort into a thing called stratus built on lvm and xfs and so on to kind of give the same kind of features there um people in the community uh were very excited about things butterfas could bring people coming working in with ad meta where they have a bunch of big investment in butterfas finally said okay look it's ready let's let's do it and so we went through our change process and with a lot of debate and with the red hat storage team weighing in to say please don't do this um we decided you know what we've been considering this for years and years it kind of brings some neat things that we our users would like to see we're going to do it and this is again a thing where um you know uh red hat doesn't call the shots and so uh the red hat you know for four things like that um they figure out how that fedora can be a good upstream and still have different decisions on some fundamental engineering things but yeah um we we eventually did make make decision to do that um another one that's kind of along those same lines is a change in baseline architecture which is basically as new cpus come out they get new and new newer and newer features and so if you compile your code with a newer cpu it can have you know better performance but won't run on the older systems and so uh red hat for rel wants to update to the newer version of that but we have an ironic problem in fedora where because we move fast we can't move as fast on some things because we leave people behind so with rel since it's got a long life cycle if people can't use the next release on their hardware or their old hardware that's fine they'll keep running the old release for you know as long as they want to pay for it being supported like i don't i think there's like probably rel 4 out there somewhere being paid for by somebody right um you know red hat will take your money uh but fedora we're we're kind of going on to the new thing and we can't say okay well sorry um you're all your hardware from five years ago put it in the trash by new computers so we had to move more slowly and so that one of the things uh i actually asked the i we were it went the people at red hat who were working on this uh were like fedora will never accept this and i asked them to bring it to the community anyways to discuss it which we did and um and you know it was a big it was a big discussion but out of that we kind of came up with this idea of having something called eln which is a build of fedora which with uh defaults that are set for what would become the next rel so we actually have a place in fedora where people can experiment with here is a way to do things in a different way so we can do those builds even if we don't make that the mainline thing that we deliver to people um and you know in some ways you know red hat is special in order to do that because they're paying for a lot of the resources to do that but if anybody else wants to come along with a thing like that and can help provide resources so we can do it uh fedora is open to that so if you've got some other big idea you want to do uh intel if you would like to do your clear linux thing as a fedora build come to us we could we can make that happen uh yeah what is the big next big technical change for the fedora project um predicting the future is hard um i i think it's both a technical change and something that is going to be a kind of a social change i think that this rpm os tree approach that's used in silver blue kino white coro s iot i think that's the the way we want to go for all of our operating all of our defaults at least it has a lot of advantages for delivering a system that's consistent you can do cool things like do a bisect to find exactly where a problem um came from uh and it kind of goes towards a container focused separation of what you know what is the base os and what are your different applications and help solve the problem we're trying to solve the modularity which is that everybody has different opinions about the speed that they want different parts of their operating system to go and this kind of that that approach lets you separate things in a way that lets you do that so i think that change towards that i mean whatever form it'll take is probably the next really big technical change um there's also the social change of convincing everybody that that's the right way to do it rollbacks are great was that coming yeah yeah so the question is basically uh with the changes in centos project uh would fedora make sense as a cloud server operating system in actual production and yeah i can't necessarily speak to your particular use cases um there are hopefully reasons you know uh for you know redhead pays my salary um i hope that there's value that redhead's providing that for for that money but i also think that uh yeah fedora can be used in production and people are using it in production in some pretty large scale things um depending on you know your ability to work with the community on things your tolerance for risk and your your your configuration and set up your change management processes um yeah you can definitely use fedora in production for things like that okay we can talk later more about it your about your specifics as well is that is that cover what you're saying asking or is that cover what you're asking it's not crazy um yeah um or maybe maybe it's crazy in a good way i mean the right kind of crazy it can make it really work i guess but we'll talk okay i'm out of time here but thank you everybody