 Hi, everyone. My name's Adam Williamson, and since this talk is just going to be me telling you how to make software, I should give you my qualifications for that. I work in Fedora QA for Red Hat. I have been for the last 10 years, so my qualification is I spent the last 10 years fixing all the bugs in your software. If you don't think that qualifies me, then, you know, I have nothing to tell you and you can leave now. So I'm going to sort of work into software in a bit of a roundabout way. We're going to talk about GK Chesterton. Some of you may know who he is. For those who don't, he's one of those weird early 20th century guys who sort of wrote about anything and everything and had opinions on everything. He was a novelist, a poet. He was a sort of Catholic thinker in the vein of C.S. Lewis. They corresponded a lot. He wrote essays, and he was a pretty terrible racist. I figured I should mention that. If you look at his Wikipedia page, he said some pretty horrible things, so I'm not defending any of that. Yeah, that was bad. The thing I wanted to talk to you about is this. He wrote one of, I think, the most perceptive things. He was mainly talking about politics, but I apply it to lots of things, and I'm just going to read it through. He said, in the matter of reforming things, as distinct from deforming them, there is one plain and simple principle, which will probably be called a paradox. There exists in such a case a certain institution or law. Let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, I don't see the use of this. Let us clear it away. To which the more intelligent type of reformer would do well to answer, if you don't see the use of it, I certainly won't let you clear it away. Go away and think. Then when you can come back and tell me that you do see the use of it, I may allow you to destroy it. To put this in context a little, this is at least the way I think about it, he was talking about a political sort of distinction between what I call non-ideological, well, this distinction existed and he was working within it. Non-ideological conservatives and sort of really hard line with formers in politics, but in other things. To me, a non-ideological conservative is someone who is in favor of the system as it exists, not because they love that system, but just because it's what we've got and it works and changing it might be dangerous. Hardline reformers are those people who always believe that they can sit down and think for half an hour and come up with a better way to do everything and they're always coming up with schemes to make things better. This was a sort of ongoing political debate and much like today's in North America between liberals and conservatives, at the time there was the debate between conservatives and reformers who were the ones who wanted to change everything and make it better. So if we think about a few examples of these, I mean these are just random ones I thought up, non-ideological conservatives, we might have someone like a British monarchist, I mean some of them are monarchists because they think the monarchy is great, you know, we should absolutely always have a queen, but some of them would be like if the current system was a republic they wouldn't want to elect someone as king and change the whole system, they'd be like, let's stick with the republic. So some monarchists in the UK are like, it's what we've got, why would we go through all the trouble of changing it? Catholics, obviously there's an ideological dimension there, but Catholicism, other religions, they have this, a lot of them have this element that even in things that have nothing really to do with the theology they just don't want to change it because it's the way we've been doing it for a thousand years, let's stick to it. Examples, some examples of reformers, I mean obviously you've got the French Revolutionaries, they would be your number one example. Their theory was we're gonna do it our way and if you don't like it you can go on the highway by which I mean, have you met my friend, Mr. Ghillity? Another one that I like, you had this tradition of sort of grand architects in the early 20th century again, you know, think of Mussolini in Rome, if you go to Rome and look at all the bits of Rome that got done over by Mussolini, he got an architect and he was like just come up with this amazing plan for how this should look and the architect would get a piece of paper and draw these vast, you know, boulevards and big houses and things and then they'd go out and knock down whatever was there and just build whatever the architect had drawn on the piece of paper, didn't give a damn what was there before, just knock it down and build what you want. So to finally start sort of bringing this into a tech context, who else might be an example of a non-ideological conservative or a hard-line reformer? QA people tend to be non-ideological conservatives. It's not that we think whatever bit of software is there right now is fantastic, but we know about it, we know how it works, how it doesn't work. If you come along and try and change it, we're going to be sad because you know, the new one might not work or it might not work in different ways to how the old one didn't work. So your typical QA person just generally doesn't want you to change anything at all and we would be happy with that. Developers, some, you know, developers in general are reformers because they want to write things and make them better. Some developers are arguably, you know, hard-line reformers in that they can't look at anything without saying, hmm, I could rewrite that better and I would probably rewrite it better and go. Yeah, so that's why I think this is, this is relevant to software development, basically. So my sort of idea, my take on Chesterton's quotation, what he can say to software developers, to software people is any artificial things, which includes all software, obviously none of them just exist. You know, the code is not this, this bizarre artifact that is there and that we only know it just appeared from the sky one day. Anything that's there is probably there for a reason, like Chesterton's fence. It's not just there because a fence magically appeared one day. Someone built a fence. Someone built the fence for a reason. Any piece of code, anything, any behavior in a piece of code is probably there because someone willed it to be there for some kind of reason. So if you are a kind of hard-line reformer type or you're just in this particular case wanted to come up with a new way of doing things and you're looking at something that exists and it's getting in the way of your glorious vision, you need to know why that thing is there before you just throw it away in the service of your glorious vision about how this new software should work. You can't just be like, eh, it's probably nothing. Yeah, that's probably a bad idea. So to give an example of this in practice, I wanted to talk about, there's another really good example that I wanted to use, which I'll mention briefly. So probably a lot of people here have heard of the Debian OpenSSL bug, is this familiar to people, where they realized after it was several years at least that Debian had only been generating one of I think 65,535 possible SSL keys, which is a problem. And this is very much an example of this in that someone saw two lines in the sort of the entropy initializer for open SSL. And they were getting in the way of, I think, a code checker. I can't remember which one. And they were like, well, this doesn't look like it's doing anything terribly useful. These are just uninitialized values. So I'm just going to throw them away and then the link will look better. And it turned out they were very, very, very important. The only reason I couldn't use that example is if you try and trace back the history of those two lines, it's really difficult because they go back to like 1997. They've been there since before any kind of revision control, I think at all. So this example is easier to follow all the way through. So I'm going to talk about a bug we ran into with Anaconda, the Fedora installer. I think this was last year. And there was an update to system D came out. And some change in system D made it start choking on this line that was in the one of the Anaconda system D services. And this is kind of a mysterious line. And it's echoing dashies. It's, you know, you don't look at that and go, Oh, I know exactly what that does. Unless given the number of people in the room in the conference, we're at probably someone looks at that and knows exactly what it does. But, you know, so the developer looked at it and he, he thought about it a bit and I don't know exactly what process he went through, but the conclusion he came to, he was like, I think this is trying to ring the system bell, which, you know, interesting conclusion. But that's what he came to, but he didn't go and check exactly. He sort of looked at it and thought it through. And that was his conclusion. So he just took it out so that system D wouldn't choke on it anymore. And you know, the problem seemed to go away. Anaconda started up again. For whatever reason, I sort of got interested in this bug. Thank you. And decided to take a look at it. And I, the way I looked at it was, I dug through the actual history of how this thing came to be in the code in the first place, which is the thing I'm trying to tell you to do in this talk. And what I found out is that, does anyone have any idea what this does? Hands if anyone does? No? Okay. It's an ANSI scape code that sets the terminal to UTF-8 mode. So the interesting, I wanted to follow this up even further before this talk, but I didn't get around to it. We still don't know for sure what breaks because we're not setting the terminal to UTF-8 mode. It may be nothing at all, but it's quite likely that something breaks because we're not setting terminals to UTF-8 mode anymore. It's likely it affects text installs, and it probably only affects text installs in Japanese or something. But there's a reason someone especially set the terminal to UTF-8 mode. And if you just take that out and throw it away, we're not setting the terminal to UTF-8 mode anymore. So there's a bug link there that if you download this slide deck, you can kind of follow up on them. It's interesting. So yeah, the lessons from this example. It's great that if you want to change something, you kind of look at it and try and figure out what it does in your own head by your own knowledge. You should totally be doing that. But it's not always enough, and it's not always actually the most effective way of figuring out why something is the way it is right now. A strategy I really like to use is we recognize that someone made it that way, and they probably made it that way for a reason, and go back and find out who did that and why they did it. And we have some really good tools for doing this, which I'm going to mention in a moment, and this can actually be the easiest way to do it as well as the most effective way. Ideally, of course, you have to combine all these things together. You think about it yourself, and you also look back at the history of it. Another lesson here is that if you get something like this wrong, it's not always going to be obvious. I mean, most developers are familiar with this. When you break something, it doesn't always instantly crash. You can take something out, and it looks like it's working fine in your basic test. You took that line out, you ran the installer, it ran, it installed, so everything's fine. Not necessarily, because if this does break anything, it's going to be a corner case that you probably didn't test. There's also a corollary to this that I really want to push, which is, when you're changing things, please be very detailed about why you're doing it. I mean, everyone uses Git these days pretty much. Write a commit message that explains both what you're doing and why you're doing it. Please do that, because when someone like me is doing this process and comes back in five years and says, why is this like this? They don't want to see a commit message that just says dot, dot, dot, which I have seen. There's a project, and I can't remember which one it is, but every commit message that I get repository just says dot, dot, dot, and I really, really don't like those people, so yes. Please write good commit message. So just to go quickly, because I'm running out of time, this is a short time slot over, over how we do this. The three most important tools for doing this are Git blame, Git blame, and Git blame. Git blame. I mean, if you've never come across it, it's a fantastically useful command. What it does is it shows the file, you run it on a file, you know, Git blame, whatever file that the change you're looking at is in, and it shows you the entire file, but it also shows you for each line what commit that line was most recently touched in. So you can, you know, you look at the change that you're trying to inspect, and it tells you exactly what commit that was in, so then you go, aha, Git show that commit, and if you're lucky it'll tell you exactly why that line is that way. About half the time it doesn't tell you that, what it tells you is that someone fixed some white space in it three years ago. So then you have to dig back, and you sort of check out the commit immediately before the previous change, and you run Git blame on it again, and you go through this process, and eventually you isolate the change that really made it the way it is right now. So Git show obviously shows you what's in the commit. I, because I love embarrassing myself at these presentations, I found out about Git show when I was doing this slide deck. For the last eight years I've been doing like Git logs grep minus 15 or something to find the change. So hey, I learned something. Yes. Oh it does? That's super handy. See I learned another thing today, thank you. So Git blame dash w apparently will sort of skip white space changes for you, so that's awesome. I mean sometimes there are other changes like it moved, it moved from one file to another, and then you have to go back through the history of that. But basically this is a tool you can use to figure out why something changed and when it changed. Obviously Git log, Git diff, but you know just dig back into the archaeology of why the thing is the way it is right now. Thank you. Also if it's not Git you can learn how to do this with another repository control system or you can do something much easier which is there's like tools that convert almost any kind of RCS in the world to a Git repository or present it as a Git repository for you. So just use one of those and you can use your Git commands. You don't have to know how to do it in CVS or SVN or HGE or anything else. Mercury. The wrap-up, so my sort of takeaway from this that an easy way of thinking about all of this like a list of things to ask yourself when you want to change anything is you should be able to fill in all of these boxes before you change something. This is the behavior I want to change. It started behaving that way at exactly this point. It started behaving that way because X and the way I want to change it the effect of my change on what the original behavior was you know trying to achieve will be and the answer there should be either nothing or it will make it better. If it's going to make it worse then don't do it. That's my unless you know this is a fundamentalist position and sometimes you do have to break an existing behavior something but at least know about it and acknowledge it and say okay look I understand that I'm changing this behavior you know put it in a documentation tell people about it. So yeah I managed to get through my deck just about and we have about two three minutes for questions if anybody has any or comments or explanations as to why I'm wrong. Yes this is true yes absolutely this is how I write 250 line get commit messages that everybody loves. That's really cool. Oh sorry I should I should be repeating things. So sorry Ralph said that this slide is a great template for a get commit message which is absolutely true and sorry I can't remember your name. Kristoff says in Emacs there's a command VC-annotate version control-annotate which kind of goes through get history for you and helps you do this kind of thing. Other questions comments yes okay so apparently Japanese text install doesn't work and just falls back on English so it's probably not Japanese text install that this breaks but we really should figure out what it is this breaks. Yes we think it should be fine no I think I okay so I have the anaconda team here I really don't want to throw the anaconda team under the bus this was just an example this comes up all the time in all kinds of projects this was just the first good example I found so these guys are great and we're running out of time there so that have to do I have time for one more quick one no oh sorry I thought you just okay yes you can do it with them as well so we should get the vim guy and the evacs guy together and see what happens but yeah I'm just to more embarrassing admissions I code mainly in g-edit so they sometimes also in nano more questions or oh yes up the back have you seen any of my commit messages the question was do I have any tips on being on balancing being very complete in the commit message with being very concise and the answer is no I'm just very complete like I have written the commit message which was I think 300 lines long I try to keep it I try to keep it down I mean there are people who are better at this than me Ralph would be a good person I think okay I honestly my personal opinion is that I would much rather have a long message that explains everything in a short message that doesn't I don't know if other people agree maybe they just can't take the long explanations another thing you can do is like use references so don't you don't have to explain the entire issue in the commit message you can say this is to address bug so-and-so but make sure bug so-and-so however you format it provides enough information for a person five years later to go and find that bug report and read it right references are fine as long as I can read the message and it lets me get to the information I need to understand what you're doing that's great the bad commit messages are the ones that say oh this is to fix that bug that someone mentioned on the mailing list where blah blah happens and then you're like okay now I get to dig through mailing list archive from six years ago for the month previous to this commit and try and figure out what this guy's talking about right yes this is a good point if you're following the kernel workflow then so the point is that sometimes people explain the changes but don't do it in the individual commit so for instance if you're using the kernel workflow they put it in the 00 the cover but not in each individual commits description or if you're using a github style workflow sometimes people explain some there's a pull request with eight commits in it and they put all the explanation in the pull request so then if you just bisect something back to a commit and the commit message is something really concise which doesn't refer to the pull request at all you get a little stuck finding the explanation get it just to expand what we're saying don't necessarily want to explain everything just provide links to whatever sometimes that doesn't work at all either actually radio package you change your change is this fabricator no it was some I don't remember anyway but it was you had an explanation I'm changing this because and then links to the upstream thing yes yes this happens yeah at this something yeah sometimes this happens and it's just terrible so Kevin's point is is a good one that using references is great as long as you can be sure those references are still going to be around so something like bugzilla.redhat.com we can all be fairly sure it's either going to be there or there's going to be some kind of compatibility layer in 10 20 years hopefully random upstream projects track that link is probably not going to age terribly well and this is just another reason if you are an upstream maintainer of some kind when you throw away your old issue tracker please do something to redirect issues to your new issue tracker that really helps with this kind of process anymore okay well thank you all for coming and thanks for all the really great insights and questions and education