 Thank you very much. My name is Keith Harrison. I'm a software developer. I've been a developer for about 18 years and one of the responsibilities I've had in the various roles I've done over those years is to improve software quality in the organisations that I've worked. So that's what I'm going to talk about and how creating a culture of software craftsmanship can help do that. So, just about an overview of what I'm going to talk about. So first of all, take a bit of a step back and talk about how, before we talk about creating the culture of software craftsmanship, we're going to talk about managing complexity. So that's an issue that causes a lot of the problems that creating a culture of software craftsmanship solves. So I want to talk about that, where complexity comes from, why we have complexity, why it's so hard, what we can do about it, and how our culture of software craftsmanship can then help with that. First of all, why do projects fail? So in Steve McConnell's book Code Complete 2, he talked about the three main reasons. So they're all project management related, poor planning, poor requirements and poor management. But when they fail for technical reasons, they fail for uncontrolled complexity. So what is that? What even do we mean by complexity when we're talking about complexity in software? Fred Brooks, he's the author of Mythical Man Month, and he wrote a paper called No Silver Bullets, and he wrote this back in 1986, so there's nothing new about any of this. So he talked about two types of complexity. So he talked about, he called it accidental, I think incidental is a better phrase for it, incidental complexity. So that's the kind of complexity that engineers create, that we create, but we can also fix. So an example of that is in the languages. When we had assembly languages that were slow to work with, they were error prone, we started using high level languages, we got an order of magnitude improvement. And the same IDEs as well, if you want to start a debugger, you don't go to the command line and type in a bunch of commands or you can do, but you can go into your ID and press a button. And actually interestingly, I think with IDEs, I think we're almost at peak IDE and we're starting to see IDEs being too complicated and causing problems and we're looking for simpler IDEs as well. So what he said about these types of complexities is that initially we saw enormous improvements in that orders of magnitude, new language comes out, suddenly developers are 10 times more productive, but that's diminished over time. And as new languages, new language features, new toolings come out, we've still seen improvements, but they're much, much smaller, they're 5, 10% types of improvement. So we're not going to see enormous productivity increases. We will see some, but not these big ones from the accidental, incidental complexity. So where we are going to see them is in the other type of complexity, which is essential complexity. So in the case of software, that's the business rules that you're trying to implement interactions with the real world, like if you're using a GPS sensor in your phone, the complexities of that. And these are things that you just can't, you can't improve that away, you just have to deal with this sort of stuff. So if the business says we need these 30 features, you have to build those 30 features, so we are stuck with this complexity. So another way to think about how complex software is is to think about the scale of what we do. So in terms of data, we might be dealing with a data birth calculation, which is just dealing with a handful of bytes. Or we could be dealing with, we're working for Netflix and we're dealing with petabytes of data. So it's an enormous range, we have to think from the very, very small to the unimaginably gigantic and only getting bigger. In terms of lines of code, even a single developer working on a single system will be dealing with thousands, potentially tens of thousands, mid-size systems, hundreds of thousands, big systems, millions of lines of code. So we have enormous amounts of complexity. So what happens if we don't manage the complexity? Well, we start, everyone should know really, we start getting more defects. We get ever-increasing dev times, so our management say build these 10 features and that takes one month. They say build 10 more features of seemingly the same complexity. It takes two months, another 10 features is four months. We're getting longer and longer, the further we go on to seemingly do the same amount of work, we become afraid to change the software. Oh, we can't change that system because the person that built it left, every time we touch it, we break stuff in production. We can't reuse anything because components are so, they do so many different things in one component. You don't know, can we use this to reuse this somewhere else? Oh, we're not quite sure. If we use it, it might have this other effect. Developers get unhappy, nobody wants to be. As there's more defects, as things are taking longer, as management start to get unhappy, and developers can see these problems coming from the uncontrolled complexity. And management are unwilling to do anything about it or don't see the problem. They, at least the developers becoming unhappy. And finally rewrites, so eventually we say this system is so complicated, so difficult. The only thing we can do is throw the whole thing in the bin and rewrite it all. And that's always a costly thing from the business's point of view. They're spending a ton of money just to get back to where you started. And so it's not a productive way to be working. So the next thing to think about complexity I want to talk about is our ability to manage that complexity. So this is me here. And I'm writing a bit of code. And I'm starting to look at a function. And there's a bug in it. So management said go fix this bug. We've got a production bug. So I start looking in this first function. And that modifies a variable and then calls another function, which then modifies another variable and calls a third one and so on. Until eventually what happens is my brain fills up. And I'm juggling all these things in my head about all these functions and all these things that are changing and all the effects of that. And then eventually my brain's full. I can't, as more information comes in that I need to remember, something needs to fall out, I make a change, I forget about the side effect elsewhere and I start introducing bugs into the system, things that I didn't expect. So it was Dijkstra that pointed out back in 1972 that the known skill is big enough to keep an entire program in. And that was 45 years ago. And we've made this problem 45 years worse by continuing to write an enormously large amount of code. So we still have to work into the limitations of what we can do because we are the ones that are writing the code. You know, if you write a function that's a thousand lines long, the computer will be happy with it. But if you can't get it into your head, then it's not something that you're going to be able to work with and maintain. So now that we've talked a bit about the problem, about our limitations in dealing with this problem as well, so with all that in mind, we can start to think about the characteristics of good design. So minimise complexity. You know, no overly complicated solutions. Reduce as much as you can accidental complexity, set up CI, don't manually deploy stuff. You know, if going to production is hard, automate it all so it's not something you're thinking about or something that's holding you up so you can focus more on the essential complexity that you can't minimise. Ease of maintenance. So going back to what I said a minute ago, write code for people and not computers. Like I say, the compiler, if you have a nested if statement that's 50 lines deep, the compiler won't care, it's happy, the computer will run it. It doesn't care. But the person that comes to maintain that when you go on holiday or you, when you come to maintain it a few months later and can't actually remember why you built this code, you'll care and it will affect your ability to actually change and maintain this code. So always keep that in mind with building stuff. Loose coupling. So reduce dependencies. Have each component be as independent from every other component as much as possible. Things like test-driven development help with this because they force you down that path. It's impossible to, or very difficult it should be if you're in test-driven development to couple everything together because you won't be able to write any tests. Cohesion. So, cohesion's a degree to which elements inside a module belong together. So if you have high cohesion, that's associated with software that's robust and reliable and reusable and easy to understand. So it's something to aim for. So another way to think about this is levels of design. So going back to that picture of my head with filling up with information, we want to build a system that at every level we can fit into our head and have the complete thing in. So at a very high level, the software should be given a discrete job that everybody can clearly understand. So, for example, an order capture system. It captures orders. That's all it does. As soon as it starts getting tweaked to well, actually we want to go in and do some fulfilment in this as well and send emails and other things. All of a sudden your order capture system starts to get quite complicated and becomes more than you can manage in your head. And decomposing all your systems at a high level into all these individual parts. So, oh, there's a bug that the users are getting a bug when they put in an order. Well, it's probably an order input system. So we'll go there. And then within that individual system, break that down into subdivisions and continue that all the way down. So components for data access, components for validating the UI, components for your business logic, and so on. So you can just, at any level in the system, you can dig down through understandable units, dig down to the module and the function level to have single functions where they're only changing or using a few variables and doing a few things. So once you get into that function, the bug's in there, you do the work, job done, haven't broken anything else. Everybody's happy. So now that we've thought a bit about the problems that complexity causes us. So how can we start to go about and fix these types of problems? So a lot of this stuff is, I think people know a lot of these things and there's certainly a lot of talks today and there's a lot of, we know all this stuff, but we still don't do it. So why does that happen and how do we get our intercells into this situation? And I think by creating, it becomes a cultural piece. It's not really about, well, nobody wants to write bad code, so why does it happen? And that starts to become a cultural thing. It's not about developers being unable to do it. That's certainly not the case. So creating a culture of software craftsmanship is how you go about changing that so the culture fits with what you actually want to do. So when developers are working away, we're pulled in different directions by those competing forces pulling us in the opposite direction. So on one hand we've got the business saying we need to hit this deadline. We promised a customer that the CEO was doing a marketing event around the country on this date. We must hit it. And on the, so on one hand we've got deadlines. On the other hand we've got quality. So yes we can hit that deadline but see when we try and do the next thing. We've made such a mess hitting that deadline. We're just going to start going slower and slower. So you're getting pulled two ways. And additionally, as a developer, so you're working away on whatever particular tasks that you've been given. And you're faced with a lot of decisions and these are complicated decisions. And we're faced with them over and over again multiple times a day. What should I call a variable? Should I refactor this bit of code? Or do I just hack something in? Do what tests should I write? How much depth should I go into? We're faced with these decisions over and over again. And at such a, we have a level of expertise that means it's just impossible. You can't manage this from the top. You can't have some sort of review committee sit and go over every one of these minutiae. I mean you can do things like reviews and if you had an infinite amount of money potentially you can do reviews. But that's a different way I'm talking about like a top-down management control over all these decisions to make sure no bad decisions get made. It's just not possible. You have to trust and you have to rely on the developer while they're building the system to make the right decisions. And what decisions, all these decisions, the net effect of them is going to have an enormous effect on the long-term health of your project. So in this book, Lebrouchnyir in this book, Liars and Outliers, he's a cryptography expert. And he talked about four different social pressures that allow us to cooperate with each other. So he talks about moral. First one was moral. So I do it because it's right. So everybody when they start a new job wants to do the right thing. They want to build great code, good quality. And then they're put in a situation where they're told that here's a deadline. You've got to get this. There isn't the time to do the right thing. Just hack in a solution, cut and paste something. We don't care. And then suddenly you're faced with this choice. Do I do that? I know that's a mistake. I know that's the wrong thing to do. I know we'll pay a price down the line. But what do you do? Do you do the right thing? Or do you do what you're being pressured into? So these are the moral choices that go on. So reputational. I do it because of what others think of me. So an example of that is if you know you're going to, your code is going to have a code review. You're going to, if you're going to write some code, it's a bug fix. It's going to thrown straight into production under pressure with the best will in the world. You might think, well, you know, it might cut a few corners here. This code is good enough. It works. Let's just push it out. But if there's going to be a code review afterwards, a code review that's going to raise issues and have you do rework because, you know, this function can't be a thousand line long. You need to cut it down. Your reputation is online. Other people are going to see that and they're going to think, no, this guy didn't write very good code. As soon as that, as soon as you start thinking about that, you're going to start writing better quality up front. So institutional. So that's where your organisation rewards you or sanctions you depending on your behaviour. So for example, if you go back to the, just hit the deadline, so that's an institutional pressure to hack stuff in and to not do it correctly. Whereas, saying after you've hacked it in, right, we're going to have one month, no new features. Just tell the business they're just going to have to wait and we're going to refactor everything. That's an institutional pressure to do the right thing. And then finally, the last one is security, which I think is the least relevant to software developers. And that's simply because you're forced to. So that might be a linting tool that you can't do a check unless you don't get any errors. But ideally, we would have loads of these types of tools, but in the world we're in, we don't have the metrics yet. I've seen things like cyclomatic complexity that are useful enough for this to be a significant area. So some examples of what you can do to try and influence these types of behaviours. So I mentioned that a few times, code reviews. So set times for code reviews. And so an important thing about a code review is I've seen management say, right, guys, we need to make sure all the code gets reviewed. And then that's it. And you actually don't get any... You're barely getting the work done on time, never mind having time for all these reviews. Or the system gets put in, the review gets done, it's too late to do anything about it. So the code reviews need to be... Time needs to be set aside for them. So the business needs to say, we see the value in these. We want to invest time and money in this because we know this will save us time and money in the future. So they... And pair programming as well. So when you're with somebody else, hopefully the two of you are not going to work together and say, oh, this hack piece of rubbish this all do. Both your reputations on the line, you're both going to try and show, put your best foot forward with it. So this works at... These work at a reputational and they've done correctly an institutional level as well. So group learning. So this is something that I've organised in places I've worked. So to do a coding dojo, do an in-house training session. So what I've done that works quite well is you do in-house training and you say, right, once a month we're going to meet in a room for an hour. And what we'll take in turns in the team and each person in the team is going to run a session. So first person is going to run a TDD session and the next person is going to run a session on integration tests and the third one on a bit good naming. So by getting everyone in a room together, by getting everyone to discuss, well, why do we do a TDD? Why do we want to refactor this code? You're creating this culture where everyone's saying that these are good things and understanding the benefits of all and understanding as well that everybody is thinking this because there's no good one person saying, I know this is the right thing to do, being the only person that does it while everybody else is hacking in a mess because everyone else is digging the whole deeper, faster and you can dig it, solve the problem. So you need to get everybody to do it and these kind of sessions where, you know, if it's just one person running it, it'll be, oh, Keith's making us do another one of these sessions again, what a waste of time. If everybody's doing it, then it creates that culture where everybody cares about these sorts of things and so this works on a moral level. So I discussed this as well, project management so quality related tasks. So that's where project management would say we're going to have set tasks, okay, you've built this right now, we're going to put it up on the board and we're going to assign time to refactoring work and that's a really powerful message to say that the business cares about quality and so it works on an institutional level because you're physically setting side to time to fix things and do things right but it also works on a moral level because it's a very strong message from the business to developers that we care about this so do the right thing which then leads to the moral benefits and you get this virtuous cycle where everybody's working together to create good code and so on and that you get less defects and you hit your releases on time and you start to get a good flow going with your development. So in conclusion, create a culture of software craftsmanship to help manage your complexity and to help reduce the quality of the work you do, the developer happiness, the efficiency of your team and think about your own culture so every team is different from one person teams to large teams to teams that are physically in the same room to teams that are geographically distributed, teams of different skills, teams of different backgrounds what works for one team is not going to work for another so think about the culture of your team and think about what might work for you and adapt some of the ideas that I've suggested or even come up with your own things that would work for you. So if you found this talk interesting I, in Glasgow on just a quick ad, Glasgow on 14th of September I'm one of the organisers of CodecraftConf in Glasgow and we're a different type of conference with no speakers and all we do is talk about it's all software craftsmanship related but what's different about with no speakers and what we do is we use a format called the guided conversation so everyone sits in a circle and you pick a topic like test driven development and your time box you'll say right we'll discuss this for five minutes and we'll say the guide will have a set of questions and the first five minutes they'll say discuss what is test driven development and then go on to what is the benefits of test driven development and so on and it's about creating a conversation and having people talk about these things because these issues are largely cultural if you can't make it to Glasgow don't worry because we are an open source conference so everything that we've done we've put on, we have a guided conversations website and we've described the guided conversation process we've also put in question sets so you can come in here example question sets so we've put our question sets in here we've shared them all on github so you can pretty much copy our whole and please feel free to if you just ask for two things please tell us about it so we can come and if you create any new question sets please share them with us so we can add them to this list so other people can copy it as well and that's me any questions thank you for the nice talk and the conference looks quite exciting any questions from the audience do you have a strategy for when some members of the team get upset at what they perceive as criticism from code reviews etc or trying to guide them towards better working practice yeah, I mean that is a really good question I think often what I've found is when you put these ideas forward a lot of times it's yes well these are obvious ideas and largely it's good but people don't actually do it tends to be the problem and so I know it's kind of it's not addressing your question but I think it is a primary problem of people do know what to do they just don't do it and so then it comes back to the cultural issues but I think it's just going back to your question of resistance if you're having a conversation about it so perhaps run a lean coffee where I don't know if people know that format but you're sitting in a room and there's no agenda except you say well maybe well limit it to software craftsmanship related issues and then everybody can write what they want to talk about and you vote and whatever's got the most votes that's what you discuss people can discuss the ideas the issues that they have and share that with everybody else and hopefully bring them on that way I've actually had a case where someone responded to my code review comments by complaining to HR that I was attempting to bully him bully him and someone else gave me a copy of the previous code review comments from three years earlier that were identical and he still hadn't fixed the same issues I think eight out of the first ten were word for word identical it can be a difficult problem and that's why as a cultural piece quite often than not it's not a technical problem that's such a training issue so yeah it's hard thank you you mentioned institutional pressures and quality related tasks are there any good examples just a refactoring task oh refactoring that would be a good example of it okay thank you any more questions so I want to thank again Dave for a great talk thanks very much