 Last question before we get started, which will then have more questions. Can you hear my amplified voice right now? Okay, cool. So Welcome to the what stability means and how to do better a talk. So before we get started I just want to do a quick poll. How many people in here are package maintainers in fedora? Okay, and how many people in here? work upstream on those packages Okay, so some but not all all right cool, so I think we've got the right the right Level for today Some of you are going to be very bored by this, but hopefully you can take it and make it better So just first thing on the list This is quirk and he wants to start a revolution, but he didn't print enough flyers and I too want to start a revolution, but I did not prepare enough slides and What I want to say is that stability is an open-source community problem in general This isn't just fedora, but inside fedora. We do have some Things that we have created that help us be more stable than your average upstream and we should take advantage of them so with that let's talk about what stability is because stability Fundamentally comes down to your frame of reference Like the gyroscope when it's spinning can glide along a string and not fall over But I wouldn't call it stable because I know it's going to slow down and fall down so stability In a length distribution fundamentally is changes for the better because you know any system can Can be resilient it can run fine it will crash it will corrupt data But the thing that is different the thing that is underattended are the other forms of stability So we're going to cover four of them. There is of course crashes and corruption, but What really makes things unstable is what we do as developers to the software when we push out updates when we make changes So that comes down to kind of three major categories The first one is developer ISV stability if you maintain a library like a shared library static library Anything that other people are relying on and you make an update to it if that update is incompatible then applications that are already built can can crash and If you want to rebuild them sometimes they don't rebuild there's also kind of provision and management stability where say you're making a Linux distribution and the you introduce changes into that distribution that cause it to no longer say Install or update or change and then finally there's operational stability And this is this is like the big one that we all get wrong all the time because we choose perfection of execution over like continuity of Expectation so let's just cover these a little more detail one by one So crashes and corruption. What's the problem? It's 2019. These are still a thing Why is that? There's a lot of reasons for it and the fact is the solutions are really well known and All you have to do is is do them and it'll be fine You can stop this just by like applying the best practices that are well documented throughout the industry My favorite just as a starting point though is turning on warnings and then fixing them that goes a really long way but for for like deep details on how to solve all of your Your package crashes your data corruptors and whatnot. There are a thousand talks. There are 10,000 tools there are all sorts of things and all you have to do is use them But that's not really the thrust of this because everybody knows yeah, we shouldn't crash And if you get a bug report that it crashes, it's the kind of thing you pay attention to and you're like, yeah That's definitely wrong that I did not mean for that program to crash. I didn't mean to lose your data I'm going to try fixing that we don't actually have a problem with this other than the fact that it still happens But when it does happen that we pay attention so Where do we not pay attention? It's it's in these other cases It's like developer ISV stability if I do an update on my system. I'm a developer I write an application. I want to ship that application I generally expect that that application is going to continue to run on the systems that people deploy it on Even if they do a software update if they update their operating system even if I'm not part of their operating system even if I'm not part of the infrastructure that does That does the update I expect before the update and after the update for my application to continue running and What actually happens sometimes library updates break binaries that until they're recompiled in other cases Updates to that same library. I mean that you can't recompile them because you broke the API so this is basically API API breakage and it is it's one of the things we do in rel to That that actually causes people to use rel even though it doesn't have all of the awesome features of Fedora Just because the value of this sort of stability is greater than the value of the novelty that Fedora provides So there are things that you can do there are solutions to this that anybody who is a Developer or anybody who is a maintainer can manage the first is if you're upstream and you update the library if you're going to break the ABI or API update the version number dead simple stuff right and oftentimes this doesn't happen and Also if you happen to be a maintainer don't assume that your upstream has actually done this Frequently they'll just go ahead and break ABI or API without bumping the version number But we have tools that detect even this My favorite is ABI diff which is part of live Abigail which is in Fedora and does a phenomenal job and then finally if you are a package maintainer and There is ABI breakage consider Packaging the new package Independent of the old package Use name versioning so that people don't have to go forward I mean one of the great functions of Fedora is that it causes Us to kind of take all of the packages that are inside Fedora and move them to the latest version But for anything outside of Fedora like steam or chrome or many other things that haven't like gotten into the Fedora universe That aren't part of like the Koji infrastructure those things can't Benefit if we force them to move like so if we want greater adoption outside of the Fedora Sphere then actually providing compatibility libraries for older versions is actually a really useful thing And if you do that, please set them up for parallel installation So you can have more than one available at a time best practices for this are really well documented So just any questions on this comments, I want to time box everything to four or five minutes Well the way I would do it is put it in the main repository I would not use modules for this right now unless I had to but I would also defer to Like modularity leads on that topic. This is something that there's some guidance on in Fedora But the packaging guidelines are a little bit fuzzy actually Yeah, it's in user bin user bin API diff and it's got a man page and it's very comprehensive I mean like any canoe tool it it's deeply detailed and it only scratches the surface so I Would say as if you start using it you don't find the answers post to Fedora de Vell We have the team that built it inside red hat And it's the kind of thing we want to grow the use of because the more people do a bi compatibility The easier our job is to get make the next version of rel come out and have it be just as as compatible as the previous one You know if I had my way like all future rel releases would have all old rel releases Libraries in them and you could install them side-by-side if you needed to because that would actually provide genuine value to people as long As we're like doing security updates All right, so let's let's move on Provision and management. So this is this is like a really straightforward one. How many people like if you ever use the anaconda kick starts Yeah, do you have to change them every time you go to a new Fedora release like beyond updating a path name? No, no, of course not because breaking something so fundamental would be a bad idea And and yet there is there's often a push to do this and we often have to push back And sometimes there's good reasons why you'd want to but the the basic problem is that When distributions are over 25 years old, we have really strongly established visioning and updating mechanisms We have management tools that expect them to remain the same And so anytime we actually want to make a change one of the fundamentals what we're actually saying is Everybody who's ever tried to write software that depends on the ability to install our software has to go back and update their Software to match and so this is a problem for like the virtualization team It's a problem for any of the companies that have their own management solutions. It's a problem for OpenStack It's a problem for OpenShift Kubernetes just anything that provisions and likewise if If you're in some sort of configuration management you use Ansible you use CF engine you use chef Like the ability to actually get into the system and connect and and do updates If you change one of those fundamentals then you've kind of broken provision and management and that's really bad from an ops perspective and That just makes people less likely to trust the distribution to to take the time if setting it up one time for Fedora Setting it up one time for a distro is easy if you can trust it forever more if you have to keep updating every six months it erodes trust and it it just Makes you disinclined to go there And so the more trust we build the more likely people are to join in and that's really what we want to do is just kind of grow community grow grow the ecosystem so the The basic solution to this one is don't break the old thing make a new thing like we have Fedora core OS. It's a different provisioning model But I didn't go back and like destroy server They just made a new thing and it's got its own way of deploying and it's it's good It's effective and it didn't harm anything. It's just another take on on how to do it and of course just following the Fedora packaging guidelines is good advice no matter what because all of these things have been contemplated by the greatest minds of Fedora and they know all of those evil corner cases and As as we invent new things the guidelines are updated and it just it generally works out pretty well So here is here's the big one that I think bites us the most the hardest It's just operational stability and this is this is really straightforward like Here's the problem. We have muscle memory. We have commands And they get broken by behavioral changes and packages like does anybody think that the get command line options are in any way intuitive Like it's it's confusing and you can very easily look at it and say in the clarity of hindsight If we just remapped all these things to these letters and these words then we would have Things that make a lot more sense But it would also break everything that has come before and then as time goes on those things wouldn't make sense either So at some point we have to realize that the getting to the perfect set of features or the perfect configuration of the perfect interaction is actually worse than providing continuity of what's already there and this is This is something that it's not obvious right when when we It's we've all done it We've all done it. I'm not blaming anybody it makes It is clear that we can always improve But how we value improvement if you look at it not just from what makes sense in the moment in the clarity of hindsight Is different than what makes sense if you are somebody that wasn't deeply involved if you just rely on this thing and so the the example that I have for rel 8 is that rel 7 had young and and then DNF happened and DNF came out with an explicit goal of being incompatible and There are some good reasons for this right like incompatibility was or the the force of compatibility was kind of crippling to Innovation there are a lot of dynamics in play But basically a lot of good things people wanted to do and being constrained by compatibility was a problem initially But we actually spent two and a half years After DNF was rolled out just trying to make it more compatible with young and it's not a hundred percent and Perhaps some compromises were made the wrong way but basically that compatibility mattered so much more to our Customers in rel and our product managers who are seeking new customers that that reversion was worth it So what are the actual solutions if you have powered the upstream make behavioral changes runtime configurable, so basically like If you could if you want to change the Like key bindings for instance put it in a config file and and make same defaults and and just otherwise Make it so that you can have the thing that is technically correct to you but also the thing that is Known to others to work and then make that an option It'll help you move over time and it will let them get the benefit of the changes that you've made when they're ready for them without Causing them to to try it out say it doesn't work right and then not even use it at all and then if you happen to be a maintainer and you want to maybe put in a new application that is Incompatible with an old application put it in as another module stream because then you can give that same ability You can be you can say there's this version and there's this version the new version is kind of incompatible But it's really worthwhile and then the person who's actually going to use this stuff kids gets to decide This is like one of the fundamental cool things that modules give us is just the ability to provide new features and the ability to provide Legacy support without having to compromise on either so when you get right down to its stability is the continuity of experience over time and We don't want software to stand still the all the cool stuff that happens in fedora We definitely want that to keep on happening all the stuff that happens upstream It's great But if we make just a few small changes along these lines if you make those changes opt-in You get a better customer experience to get more people that can actually trust in open source and and deploy it and Trust in fedora and deploy it and use that as their basis And I think that's what we all want is for fedora to get bigger and and address more use cases It like do the things that we wished it could do but but for somebody not being willing to and That's actually the whole thing so More questions more comments and would anybody like to maybe take this talk of Turn it into their own and like give it at Fosdame or you know kind of like spread the word of You know what you think compatibility is because I think that's the kind of thing we need is to just like grow This is a mindset inside the community You're you're in the revolution now Nice All right, so different question for you all then is there something else that you think oh this is stability This is what it's about that wasn't even close to covered in this like is there some fifth thing? Stephen That's true like the the most mature distribution Communities have like their their annual release or their quarterly release or something, but it is it does seem pretty chaotic well, yeah distributions do but like GCC you know, it's going to come out every once a year g-lib see and and and things that Components that have been around for for long enough that they can rent cars and not have to pay extra on the US those Those ones have have kind of gone through this pain But we don't want every component we ship to have to be like a quarter of a century old before we're willing to Like take the lessons of the past, right? so definitely Predictable cadence for any community is is an outstanding thing. That's a great suggestion All right, anybody else other thing you think man, I can't believe you didn't cover this All right, David one advice on how to like stop your packages from crashing and burning Like if anyone came in with that we could definitely like have a discussion, right? So a few people raised their hands for maintainer like your package maintainer in fedora Do you think you would do any of these things would you would you modularize would you version number your libraries? Yeah, if you if you see fedora as simply being the place where the latest upstreams get together and have a good time every six months It doesn't necessarily fit But when you start wouldn't when people want to take fedora a little bit further if they if you want to have Fedora core OS and fedora iot if you actually want to deploy it not just because you're developing operating system But because you want to use the result for some purpose these things start to make a little bit of sense even in the fedora context For those a similar change happened in fedora based on like what people what kinds of problems happen some of the weird things What we wind up doing as part of like our field of grace requires to make a fake package with siblings I hope that the symbols all match and then everything Versions that yeah The only case where I can think of where this might be a terrible idea to have them both at the same time We did do that and so it's like that's the only case where I think we shouldn't do it and we did it anyway This is something we could get ahead of by by having this be a standard inside the community So the question is can it be done in such a way that for for all the volunteers that are doing this packaging It's not really a great hardship to them a topic for for So many things Just the amount of times I have been comfort surprising With Unless there's a reason not to Because that might that makes it easier I So in any case we should probably start the discussion and just kind of see where we're at now So this was 25 minutes. Thank you for joining and come talk to me afterward if this is something that kind of spoke to you Thanks all